`Node::kind`, `Node::grammar_name`, the field name accessors,
`TreeCursor::field_name`, and the `Language` name lookups returned `&'static
str` while pointing into storage owned by the `TSLanguage`. Releasing the last
handle to a Wasm language frees that storage.
`Node` and `TreeCursor` now return `&'tree str`, which is ok because
`ts_tree_new` takes a reference to the language via `ts_language_copy` and
holds it until `ts_tree_delete`. The `Language` lookups return strings borrowed
from `&self`.
`currentType` and `currentTypeId` return `null` before the first iteration
step, after exhaustion, and after a reset.
`currentType` also falls back to the language's own name table instead of the
literal 'ERROR' when `Language.types` has no entry (auxiliary symbols). Other
bindings report `end` where this one reported 'ERROR'.
`current_symbol` and `current_symbol_name` return an `Option` (`None` when the
iterator is not positioned on a symbol). Change `iter_names`'s item from
`&'static str` to `&str`.
Track the iterator's phase so exhaustion is sticky, and gate the symbol name on
it, so `NULL` means "not positioned on a symbol".
`ts_lookahead_iterator_new` was also the only language consuming constructor
that did not `ts_language_copy`, so an iterator outliving a wasm language read
freed memory. Retain in `_new` and `_reset`, release in `_delete`.
Previously:
- `ts_lookahead_iterator__next` left a small parse state's cursor one past its
group end, so re-advancing an exhausted iterator resumed returning `true` and
walked through the rest of `ts_small_parse_table` and off the end of it.
- `ts_lookahead_iterator_current_symbol_name` returned `NULL` only when the
exhausted symbol index happened to fall outside the names table, so a grammar
with aliases returned a real but wrong name instead.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
`QueryMatches` and `QueryCaptures` retain query cursor options while they are
being iterated. However, their types did not preserve the lifetime of the
options' progress callback, allowing the callback to be dropped while it
was still referenced by the cursor.
Add an explicit options lifetime to both iterator types and propagate it
through `matches_with_options` and `captures_with_options`. Use a static
options lifetime for iterators created without options.
The early `ts_parser__recover` dispatch for `ERROR_STATE` in
`ts_parser__advance` was dropped in `201b41cf1`. Without it, tokens
with no valid action re-enter `ts_parser__handle_error` per token
and fragment the tree instead of extending one ERROR node. Restore
it to recover the old (pre-0.25) error recovery behavior.
`Parser::set_logger` boxes the logger callback into a type-erased C
pointer, but `Parser` carries no lifetime parameter. This allows for
two kinds of UB:
- Use-after-free: a logger could capture a non-`'static` borrow, let the
borrowed value drop, and then dangle the next time the parser logged.
- Data race: a logger could capture a `!Send` value such as an `Rc`. The
parser could then be moved to another thread and logged from there
while the original thread still held a clone, racing on the reference
count.
As a fix, we just require that logger is `Send + 'static`. The
alternative here is to attach a lifetime parameter to `Parser`, but this
is highly breaking for what is mostly a debugging utility. As an escape
hatch `set_logger_unchecked` is added to the public API as an `unsafe fn`
to correctly communicate the risks and invariants that must be held.
`(P . Q* Y)` with zero `Q` dropped the leading `.`, so `Y` matched at any
position instead of being pinned to the parent's first named child. On a `?`/`*`
zero-skip, when the skipped step is the parent's first child and carries a
leading anchor, transfer that first-child requirement to the skip target.
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
Problem: A set of `array_*` macros (`array_push`, `array_extend`, etc) implicitly convert a `void*` into a different pointer type. In environments that compile these headers as C++, this implicit conversion is an error.
Solution: This commit adds an `_array_cast` macro that uses `decltype` to cast the `void*` to the proper type when compiling as C++.
A trailing `.` after an optional node, e.g. `(p (a)+ @a . (b)? @b .)`, sets
`is_last_child` on the `(b)?` step. When `(b)?` matched zero, the zero-skip
jumped past that step to completion, so the last-child requirement was never
enforced.
When a zero-skip would bypass a step carrying `is_last_child`, require that
the last matched node really is the last named child. Gated on the
`alternative_is_skip` edge marker.
A `.` anchor between a quantified pattern and a following node, e.g.
`(parent (comment)* @c . (decl))`, was treated as a leading anchor when the
quantifier matched zero. The zero-skip jumps a state directly onto the anchored
step, where `is_immediate` was enforced even though no sibling had matched before
it. Name that skip edge (`alternative_is_skip`) and, when a state follows it,
record that it skipped the quantifier (`skipped_quantifier`). The
immediate anchor is then supressoed for that state's next match.
An unanchored quantified sibling like `(program (comment)+ @doc (class))`
keeps one match state per matching sibling. A recent fix stopped over-pruning
them, but the per-node longest-match dedup is pairwise, so with n live states
matching became O(n^3).
Two states can only be capture subsets of one another if their captured
byte ranges overlap, so keep each group ordered by first-capture position
and stop the pairwise scan once the rest of the group is disjoint.
Refs neovim/neovim#40517
object rather than 4 separate optional function pointers.
Helps prevent misuse via mixing different allocators. Also update the
doc comment with relevant safety information.
This allows duplicating a query so that it can be modified by using
disable_capture or disable_pattern, without re-parsing the query. The
new `ts_query_copy` creates a deep copy of the entire query object.
A motivation for this is tags.scm code navigation queries. You might
want to have one version of the compiled query capture only definitions
and the other only capture references, since references are much much
more common than definitions in a typical codebase. Instead of
re-parsing and analyzing the query and then calling
`ts_query_disable_capture`, we can clone the built query all-at-once and
then disable captures / patterns.
Problem: In queries matching a parent node with anchored children
(siblings) where the first sibling has a quantifier and is not
captured, the check for fallible steps skips splitting the state because
the next node is a passthrough node (and not an is_immediate one). This
prevents the query from matching beyond the first occurrence.
Solution: In the check for fallible steps, skip the next steps if they
are passthrough steps.
Problem: In queries matching a parent node with anchored children
(siblings) where the first anchored sibling has a quantifier, the
deduplication logic for states is overly aggressive. The logic causes
the state split on the first capture (with the parent) to be dropped in
favor of the loopback state. This results in the query not being able to
match beyond the first occurrence.
Solution: Prevent the deduplication logic from dropping a state that is
not seeking an immediate match for one that is seeking an immediate
match, since the one not seeking for an immediate match could still
match later nodes.
Problem: In queries matching a parent node with anchored children
(siblings), the check for fallible steps only considers nodes with
children, which results in no state split being made for anchored
siblings (node1) . (node2). This prevents the query from matching beyond
the first occurrence.
Solution: Make the check for fallible steps consider also the case where
the next step is at the same depth and must be matched immediately
after.
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
The alloc.h header checked for TREE_SITTER_HIDDEN_SYMBOLS, but the
canonical macro name used everywhere else (api.h, render.rs, setup.py)
is TREE_SITTER_HIDE_SYMBOLS. This mismatch meant that defining
TREE_SITTER_HIDE_SYMBOLS (as the Python binding build does) would not
actually hide the allocator symbols in alloc.h.
Fixestree-sitter/tree-sitter#5625
Signed-off-by: Georges Savoundararadj <savoundg@amazon.com>
The wasm store gated supertype_symbols / supertype_map_slices /
supertype_map_entries copies on abi_version > LANGUAGE_VERSION_WITH_RESERVED_WORDS,
but every other consumer (language.c, query.c) treats those tables as
present when abi_version >= LANGUAGE_VERSION_WITH_RESERVED_WORDS.
A Wasm grammar built at ABI exactly 15 with supertype_count > 0 ends up
with supertype_count copied into the native TSLanguage but supertype_map_slices
left NULL. ts_query__analyze_patterns then calls ts_language_subtypes,
which dereferences self->supertype_map_slices[supertype] and crashes.
