mirror of
https://github.com/tree-sitter/tree-sitter.git
synced 2026-09-10 07:26:23 -04:00
Clarify behavior of advance(..., skip) in external scanner docs
Clarifies the behavior of `TSLexer::advance` when `skip=true`. The previous documentation didn't make it clear that `skip=true` should only be used before a token starts. Using it after `mark_end` can affect the token’s starting position and lead to incorrect or zero-length ranges. Fixes #2315
This commit is contained in:
parent
a8338eccef
commit
b360d655cc
|
|
@ -138,9 +138,11 @@ The second parameter to this function is the lexer, of type `TSLexer`. The `TSLe
|
|||
- **`TSSymbol result_symbol`** — The symbol that was recognized. Your scan function should _assign_ to this field one of
|
||||
the values from the `TokenType` enum, described above.
|
||||
|
||||
- **`void (*advance)(TSLexer *, bool skip)`** — A function for advancing to the next character. If you pass `true` for
|
||||
the second argument, the current character will be treated as whitespace; whitespace won't be included in the text range
|
||||
associated with tokens emitted by the external scanner.
|
||||
- **`void (*advance)(TSLexer *, bool skip)`** — A function for advancing to the next character. If `skip` is `true`, the
|
||||
current character will be treated as skipped text and will not be included in the text range associated with tokens emitted
|
||||
by the external scanner. This is typically used before a token starts, such as when skipping leading whitespace. After a
|
||||
token has started, `skip` should generally be `false`. In particular, calling `advance(lexer, true)` after `mark_end` can
|
||||
affect the token's starting position and lead to incorrect or zero-length token ranges.
|
||||
|
||||
- **`void (*mark_end)(TSLexer *)`** — A function for marking the end of the recognized token. This allows matching tokens
|
||||
that require multiple characters of lookahead. By default, (if you don't call `mark_end`), any character that you moved
|
||||
|
|
|
|||
Loading…
Reference in a new issue