`(P . Q* Y)` with zero `Q` dropped the leading `.`, so `Y` matched at any
position instead of being pinned to the parent's first named child. On a `?`/`*`
zero-skip, when the skipped step is the parent's first child and carries a
leading anchor, transfer that first-child requirement to the skip target.
(cherry picked from commit 1ffd612be5)
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
(cherry picked from commit dfcf73921c)
Unicode simple case folding maps two non-ASCII code points onto ASCII
letters: the long s `ſ` (U+017F) onto `s`, and the Kelvin sign `K` (U+212A)
onto `k`. So `regex_syntax` pulls them into any case-insensitive pattern,
which is virtually never intended and has two bad effects:
* such tokens can no longer be extracted as keywords, because they are not
a subset of an ASCII `word` token (#5607)
* a broad class like `[^"]` or `\p{L}` carrying `/i` loses `ſ`/`K`, even
though it legitimately contains them (#5755)
Rather than let `regex_syntax` fold, parse patterns unfolded and fold them
ourselves in `case_fold_ascii_safe`: fold via `regex_syntax`, then drop
`ſ`/`K` only when folding introduced them (they were not already in the
base set). A class that already contains them keeps them.
Regexes with the case-insensitive 'i' flag caused unicode simple case folding,
which maps two non-ASCII code points onto ASCII letters:
- `ſ` (U+017F) onto `s`
- the Kelvin sign `K` (U+212A) onto `k`
This pushed such tokens outside an ASCII `word` token, so they failed to
extract as keywords.
(cherry picked from commit 07c4ed220e)
Problem: A set of `array_*` macros (`array_push`, `array_extend`, etc) implicitly convert a `void*` into a different pointer type. In environments that compile these headers as C++, this implicit conversion is an error.
Solution: This commit adds an `_array_cast` macro that uses `decltype` to cast the `void*` to the proper type when compiling as C++.
(cherry picked from commit cc7be1fd47)
- An anchor between two patterns is vacuous when an adjacent quantifier
matches zero
- A leading or trailing anchor on a node applies to the nearest matched node
when an adjacent optional is skipped
- An anchor at the edge of a group or alternation is not allowed.
(cherry picked from commit 133d549b70)
A trailing `.` after an optional node, e.g. `(p (a)+ @a . (b)? @b .)`, sets
`is_last_child` on the `(b)?` step. When `(b)?` matched zero, the zero-skip
jumped past that step to completion, so the last-child requirement was never
enforced.
When a zero-skip would bypass a step carrying `is_last_child`, require that
the last matched node really is the last named child. Gated on the
`alternative_is_skip` edge marker.
(cherry picked from commit a8486ca239)
A `.` anchor between a quantified pattern and a following node, e.g.
`(parent (comment)* @c . (decl))`, was treated as a leading anchor when the
quantifier matched zero. The zero-skip jumps a state directly onto the anchored
step, where `is_immediate` was enforced even though no sibling had matched before
it. Name that skip edge (`alternative_is_skip`) and, when a state follows it,
record that it skipped the quantifier (`skipped_quantifier`). The
immediate anchor is then supressoed for that state's next match.
(cherry picked from commit 7e7f2584d1)
A `.` at the end of a group, such as `((comment)+ @c .)`, was silently dropped
during compilation. Reject rather than dropping.
(cherry picked from commit 966dc52557)
Extract the step dump into `ts_query__dump_steps` and enrich it with the control-flow
fields and analysis annotations.
(cherry picked from commit 6bfbd9d84d)
An unanchored quantified sibling like `(program (comment)+ @doc (class))`
keeps one match state per matching sibling. A recent fix stopped over-pruning
them, but the per-node longest-match dedup is pairwise, so with n live states
matching became O(n^3).
Two states can only be capture subsets of one another if their captured
byte ranges overlap, so keep each group ordered by first-capture position
and stop the pairwise scan once the rest of the group is disjoint.
Refs neovim/neovim#40517
(cherry picked from commit 91fd04f96a)
Problem: `zig fetch` on 0.16 can't download zip archives, which wasmtime
uses for Windows. This makes `cargo xtask upgrade-wastime` fail silently
with empty hashes for these platforms.
Solution: Re-run xtask with Zig 0.17 nightly.
(cherry picked from commit ae2081989e)
Problem: In queries matching a parent node with anchored children
(siblings) where the first sibling has a quantifier and is not
captured, the check for fallible steps skips splitting the state because
the next node is a passthrough node (and not an is_immediate one). This
prevents the query from matching beyond the first occurrence.
Solution: In the check for fallible steps, skip the next steps if they
are passthrough steps.
(cherry picked from commit 99bceec689)
Problem: In queries matching a parent node with anchored children
(siblings) where the first anchored sibling has a quantifier, the
deduplication logic for states is overly aggressive. The logic causes
the state split on the first capture (with the parent) to be dropped in
favor of the loopback state. This results in the query not being able to
match beyond the first occurrence.
Solution: Prevent the deduplication logic from dropping a state that is
not seeking an immediate match for one that is seeking an immediate
match, since the one not seeking for an immediate match could still
match later nodes.
(cherry picked from commit 8b43f42dca)
Problem: In queries matching a parent node with anchored children
(siblings), the check for fallible steps only considers nodes with
children, which results in no state split being made for anchored
siblings (node1) . (node2). This prevents the query from matching beyond
the first occurrence.
Solution: Make the check for fallible steps consider also the case where
the next step is at the same depth and must be matched immediately
after.
(cherry picked from commit 1b110f62dd)
The early exit condition to skip remaining highlights was incorrectly
checking the lexicographical order, which could lead to false
passes if a highlight was on a later row with smaller column number.
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
(cherry picked from commit 15ddfb21ed)
The alloc.h header checked for TREE_SITTER_HIDDEN_SYMBOLS, but the
canonical macro name used everywhere else (api.h, render.rs, setup.py)
is TREE_SITTER_HIDE_SYMBOLS. This mismatch meant that defining
TREE_SITTER_HIDE_SYMBOLS (as the Python binding build does) would not
actually hide the allocator symbols in alloc.h.
Fixestree-sitter/tree-sitter#5625
Signed-off-by: Georges Savoundararadj <savoundg@amazon.com>
(cherry picked from commit 1da46327b3)
Problem: Offsets such as parse_table, symbol_names, lex_modes, and the alias/supertype tables are used directly as indexes into the store's memory buffer. A malformed module can point one of those fields outside the current linear memory and make the host process read through an invalid pointer while loading the language.
Solution: Add a small checked-memory wrapper for Wasm language loading and routes descriptor reads, table copies, string reads, and the alias-map scan through it. Invalid descriptor addresses now fail loading with TSWasmErrorKindInstantiate.
(cherry picked from commit 21cfae7b56)
Co-authored-by: 𝙽!𝙻 <z_hakmi@estin.dz>
The wasm store gated supertype_symbols / supertype_map_slices /
supertype_map_entries copies on abi_version > LANGUAGE_VERSION_WITH_RESERVED_WORDS,
but every other consumer (language.c, query.c) treats those tables as
present when abi_version >= LANGUAGE_VERSION_WITH_RESERVED_WORDS.
A Wasm grammar built at ABI exactly 15 with supertype_count > 0 ends up
with supertype_count copied into the native TSLanguage but supertype_map_slices
left NULL. ts_query__analyze_patterns then calls ts_language_subtypes,
which dereferences self->supertype_map_slices[supertype] and crashes.
(cherry picked from commit a53c3b03a0)
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
The previous implementation of `InputGrammar::normalize` (inlined in
`parse_grammar` iterated over every variable, checking whether it was
reachable _backwards_ from teh root by recursing over rules that
referenced it.
This means that each and every top level call re-traversed the entire
graph. The runtime performance of this backwards walk was dependent on
the _order_ of rules as declared in `grammar.js`. All existing grammars
have an ordering that's reasonably friendly to this iteration pattern
(BFS-ish order, top down from the start rule), but this leaves us open
to a catastrophic performance cliff.
Instead, seed a `used` set with the start rule, the word token, and an
names referenced from `extras`/`externals`. Then propagate via direct rule
references. This yields anywhere from a 2-~2200x speedup for
`parse_grammar`. This greatly speeds up `--no-parser` runs, but is
relatively unimportant for `parser.c` generation for _existing_
grammars. The important piece is eliminating the potential cliff.
debug output.
The initial implementation of `--debug pretty` assumed process version
was bounded by `MAX_VERSION_COUNT`. However, we also have to account for
`MAX_VERSION_COUNT_OVERFLOW` as well as `halted_version_count`.
(cherry picked from commit 5cac4316db)
`ts_query_cursor_next_capture` linearly scanned all finished states to
find the one with the earliest next capture byte offset. With deeply
nested code, this O(n) scan per capture caused the highlight crate to
hang for minutes on large files.
Replace the linear scan with a min-heap over the finished_states array,
keyed by (next_capture_byte_offset, pattern_index, id). The heap is
maintained lazily: ts_query_cursor__advance uses plain array_push
(preserving FIFO insertion order), and next_capture sifts new elements
into the heap on entry via a tracked heap_size boundary. This preserves
the documented "order found" guarantee for next_match while giving
next_capture O(log n) per call.
(cherry picked from commit 123fb1c13c)
