zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
(cherry picked from commit 42f33fe2f8)
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
(cherry picked from commit 308aee0c90)
The early `ts_parser__recover` dispatch for `ERROR_STATE` in
`ts_parser__advance` was dropped in `201b41cf1`. Without it, tokens
with no valid action re-enter `ts_parser__handle_error` per token
and fragment the tree instead of extending one ERROR node. Restore
it to recover the old (pre-0.25) error recovery behavior.
(cherry picked from commit 15ea3328e1)
`(P . Q* Y)` with zero `Q` dropped the leading `.`, so `Y` matched at any
position instead of being pinned to the parent's first named child. On a `?`/`*`
zero-skip, when the skipped step is the parent's first child and carries a
leading anchor, transfer that first-child requirement to the skip target.
(cherry picked from commit 1ffd612be5)
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
(cherry picked from commit dfcf73921c)
Problem: A set of `array_*` macros (`array_push`, `array_extend`, etc) implicitly convert a `void*` into a different pointer type. In environments that compile these headers as C++, this implicit conversion is an error.
Solution: This commit adds an `_array_cast` macro that uses `decltype` to cast the `void*` to the proper type when compiling as C++.
(cherry picked from commit cc7be1fd47)
A trailing `.` after an optional node, e.g. `(p (a)+ @a . (b)? @b .)`, sets
`is_last_child` on the `(b)?` step. When `(b)?` matched zero, the zero-skip
jumped past that step to completion, so the last-child requirement was never
enforced.
When a zero-skip would bypass a step carrying `is_last_child`, require that
the last matched node really is the last named child. Gated on the
`alternative_is_skip` edge marker.
(cherry picked from commit a8486ca239)
A `.` anchor between a quantified pattern and a following node, e.g.
`(parent (comment)* @c . (decl))`, was treated as a leading anchor when the
quantifier matched zero. The zero-skip jumps a state directly onto the anchored
step, where `is_immediate` was enforced even though no sibling had matched before
it. Name that skip edge (`alternative_is_skip`) and, when a state follows it,
record that it skipped the quantifier (`skipped_quantifier`). The
immediate anchor is then supressoed for that state's next match.
(cherry picked from commit 7e7f2584d1)
A `.` at the end of a group, such as `((comment)+ @c .)`, was silently dropped
during compilation. Reject rather than dropping.
(cherry picked from commit 966dc52557)
Extract the step dump into `ts_query__dump_steps` and enrich it with the control-flow
fields and analysis annotations.
(cherry picked from commit 6bfbd9d84d)
An unanchored quantified sibling like `(program (comment)+ @doc (class))`
keeps one match state per matching sibling. A recent fix stopped over-pruning
them, but the per-node longest-match dedup is pairwise, so with n live states
matching became O(n^3).
Two states can only be capture subsets of one another if their captured
byte ranges overlap, so keep each group ordered by first-capture position
and stop the pairwise scan once the rest of the group is disjoint.
Refs neovim/neovim#40517
(cherry picked from commit 91fd04f96a)
Problem: In queries matching a parent node with anchored children
(siblings) where the first sibling has a quantifier and is not
captured, the check for fallible steps skips splitting the state because
the next node is a passthrough node (and not an is_immediate one). This
prevents the query from matching beyond the first occurrence.
Solution: In the check for fallible steps, skip the next steps if they
are passthrough steps.
(cherry picked from commit 99bceec689)
Problem: In queries matching a parent node with anchored children
(siblings) where the first anchored sibling has a quantifier, the
deduplication logic for states is overly aggressive. The logic causes
the state split on the first capture (with the parent) to be dropped in
favor of the loopback state. This results in the query not being able to
match beyond the first occurrence.
Solution: Prevent the deduplication logic from dropping a state that is
not seeking an immediate match for one that is seeking an immediate
match, since the one not seeking for an immediate match could still
match later nodes.
(cherry picked from commit 8b43f42dca)
Problem: In queries matching a parent node with anchored children
(siblings), the check for fallible steps only considers nodes with
children, which results in no state split being made for anchored
siblings (node1) . (node2). This prevents the query from matching beyond
the first occurrence.
Solution: Make the check for fallible steps consider also the case where
the next step is at the same depth and must be matched immediately
after.
(cherry picked from commit 1b110f62dd)
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
(cherry picked from commit 15ddfb21ed)
The alloc.h header checked for TREE_SITTER_HIDDEN_SYMBOLS, but the
canonical macro name used everywhere else (api.h, render.rs, setup.py)
is TREE_SITTER_HIDE_SYMBOLS. This mismatch meant that defining
TREE_SITTER_HIDE_SYMBOLS (as the Python binding build does) would not
actually hide the allocator symbols in alloc.h.
Fixestree-sitter/tree-sitter#5625
Signed-off-by: Georges Savoundararadj <savoundg@amazon.com>
(cherry picked from commit 1da46327b3)
Problem: Offsets such as parse_table, symbol_names, lex_modes, and the alias/supertype tables are used directly as indexes into the store's memory buffer. A malformed module can point one of those fields outside the current linear memory and make the host process read through an invalid pointer while loading the language.
Solution: Add a small checked-memory wrapper for Wasm language loading and routes descriptor reads, table copies, string reads, and the alias-map scan through it. Invalid descriptor addresses now fail loading with TSWasmErrorKindInstantiate.
(cherry picked from commit 21cfae7b56)
Co-authored-by: 𝙽!𝙻 <z_hakmi@estin.dz>
The wasm store gated supertype_symbols / supertype_map_slices /
supertype_map_entries copies on abi_version > LANGUAGE_VERSION_WITH_RESERVED_WORDS,
but every other consumer (language.c, query.c) treats those tables as
present when abi_version >= LANGUAGE_VERSION_WITH_RESERVED_WORDS.
A Wasm grammar built at ABI exactly 15 with supertype_count > 0 ends up
with supertype_count copied into the native TSLanguage but supertype_map_slices
left NULL. ts_query__analyze_patterns then calls ts_language_subtypes,
which dereferences self->supertype_map_slices[supertype] and crashes.
(cherry picked from commit a53c3b03a0)
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
`ts_query_cursor_next_capture` linearly scanned all finished states to
find the one with the earliest next capture byte offset. With deeply
nested code, this O(n) scan per capture caused the highlight crate to
hang for minutes on large files.
Replace the linear scan with a min-heap over the finished_states array,
keyed by (next_capture_byte_offset, pattern_index, id). The heap is
maintained lazily: ts_query_cursor__advance uses plain array_push
(preserving FIFO insertion order), and next_capture sifts new elements
into the heap on entry via a tracked heap_size boundary. This preserves
the documented "order found" guarantee for next_match while giving
next_capture O(log n) per call.
(cherry picked from commit 123fb1c13c)
Commit 1f6eac55 ("query: Use uint32_t for capture list IDs") widened
QueryState.capture_list_id to uint32_t and removed the 65536 pool cap,
but left the pool function signatures as uint16_t. This caused silent
truncation when the pool exceeded 65535 entries, leading to a segfault.
(cherry picked from commit 361f293a73)
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
(cherry picked from commit 0f6780b9a3)
Explicitly casting NULL to (Subtree *) in the ternary expression ensures
type consistency. This resolves ambiguous type inference issues encountered
by strict static analysis tools and C-to-Go transpilers (like ccgo).
Signed-off-by: lucasew <lucas59356@gmail.com>
(cherry picked from commit 4e9fededff)
Wasmtime v34 introduced breaking ABI changes to `wasmtime_func_t` and
`wasmtime_table_t` structures. The `__private` field changed from
`size_t` to `void*`, requiring code updates to store complete
`wasmtime_func_t` structures instead of raw indices.
Changes:
- `BuiltinFunctionIndices`: `uint32_t` -> `wasmtime_func_t`
- `stdlib_fn_indices`: `uint32_t*` -> `wasmtime_func_t*`
- `FunctionDefinition.storage_location`: `uint32_t*` -> `void*`
- `get_builtin_extern()`: simplified to return stored func directly
- Lexer functions: use temp array for func storage, store table index
- Zero-initialize `builtin_fn_indices` so `store_id == 0` sentinel
reliably detects missing stdlib exports
- Free `lexer_funcs` on error paths in `ts_wasm_store_new`
- Remove stale `(uint32_t *)` casts from `lexer_definitions`
Co-Authored-By: nzinfo <li.monan@gmail.com>
(cherry picked from commit a21ee02710)
When a branch inside an alternation has a + or * quantifier, the
quantifier's pass_through step loops back to the branch's first step.
The alternation linking also sets that step's alternative_index to
point to the next branch. This causes the quantifier loop-back to
incorrectly explore other branches in the case of a failed match that
follows a successful match, contaminating captures.
This is corrected by redirecting the quantifier loop-back to a
"clean" copy of the target step without the alternative index pointing
to the next alternation branch.
Co-authored-by: Riley Bruins <ribru17@hotmail.com>
(cherry picked from commit 596a4d69bb)
the region in place.
Previous changes to `malloc` caused `realloc` to sometimes pull regions
off of the free list during this optimization. Because no `memcpy` is
performed, this resulted in corrupted data returning to the caller.
Co-authored-by: trim21 <i@trim21.me>
- free memory if 0 size is passed in
- Don't `memcpy` contents if new pointer is `NULL`
- Copy old region's contents only up to size of new region
- free old region
Co-authored-by: trim21 <i@trim21.me>
Altering the `Array` type itself isn't feasible, as this causes
unacceptable breakage with existing parsers that depend on it. Instead,
pass in individual `Array` fields for to various `_array__*` functions.
Any time the `contents` of an array may be modified (`free`d, `realloc`d,
etc), return the potentially new address out by value. This prevents any
strict aliasing violations as we're no longer writing to a type-casted
pointer.
Co-authored-by: Nathaniel Wesley Filardo <nwfilardo@gmail.com>
(cherry picked from commit 5177b3dc26)
Problem: Output of `cargo xtask build-wasm-stdlib` depends on whether
`wasm-opt` is installed (since `clang` will use it by default if it
finds it).
Solution: Install it and rerun the xtask.
(cherry picked from commit 5d290a2a75)
**Problem:** A query with a `?` quantifier followed by a `+` quantifier
would hang at 100% CPU usage while iterating through a tree, regardless
of the source content.
**Solution:** Collect all quantifiers in one step, and then add the
required repeat/optional step logic *after* we have determined the
composite quantifier we need to use for the current step.
