The early exit condition to skip remaining highlights was incorrectly
checking the lexicographical order, which could lead to false
passes if a highlight was on a later row with smaller column number.
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
(cherry picked from commit 15ddfb21ed)
The alloc.h header checked for TREE_SITTER_HIDDEN_SYMBOLS, but the
canonical macro name used everywhere else (api.h, render.rs, setup.py)
is TREE_SITTER_HIDE_SYMBOLS. This mismatch meant that defining
TREE_SITTER_HIDE_SYMBOLS (as the Python binding build does) would not
actually hide the allocator symbols in alloc.h.
Fixestree-sitter/tree-sitter#5625
Signed-off-by: Georges Savoundararadj <savoundg@amazon.com>
(cherry picked from commit 1da46327b3)
Problem: Offsets such as parse_table, symbol_names, lex_modes, and the alias/supertype tables are used directly as indexes into the store's memory buffer. A malformed module can point one of those fields outside the current linear memory and make the host process read through an invalid pointer while loading the language.
Solution: Add a small checked-memory wrapper for Wasm language loading and routes descriptor reads, table copies, string reads, and the alias-map scan through it. Invalid descriptor addresses now fail loading with TSWasmErrorKindInstantiate.
(cherry picked from commit 21cfae7b56)
Co-authored-by: 𝙽!𝙻 <z_hakmi@estin.dz>
The wasm store gated supertype_symbols / supertype_map_slices /
supertype_map_entries copies on abi_version > LANGUAGE_VERSION_WITH_RESERVED_WORDS,
but every other consumer (language.c, query.c) treats those tables as
present when abi_version >= LANGUAGE_VERSION_WITH_RESERVED_WORDS.
A Wasm grammar built at ABI exactly 15 with supertype_count > 0 ends up
with supertype_count copied into the native TSLanguage but supertype_map_slices
left NULL. ts_query__analyze_patterns then calls ts_language_subtypes,
which dereferences self->supertype_map_slices[supertype] and crashes.
(cherry picked from commit a53c3b03a0)
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
The previous implementation of `InputGrammar::normalize` (inlined in
`parse_grammar` iterated over every variable, checking whether it was
reachable _backwards_ from teh root by recursing over rules that
referenced it.
This means that each and every top level call re-traversed the entire
graph. The runtime performance of this backwards walk was dependent on
the _order_ of rules as declared in `grammar.js`. All existing grammars
have an ordering that's reasonably friendly to this iteration pattern
(BFS-ish order, top down from the start rule), but this leaves us open
to a catastrophic performance cliff.
Instead, seed a `used` set with the start rule, the word token, and an
names referenced from `extras`/`externals`. Then propagate via direct rule
references. This yields anywhere from a 2-~2200x speedup for
`parse_grammar`. This greatly speeds up `--no-parser` runs, but is
relatively unimportant for `parser.c` generation for _existing_
grammars. The important piece is eliminating the potential cliff.
debug output.
The initial implementation of `--debug pretty` assumed process version
was bounded by `MAX_VERSION_COUNT`. However, we also have to account for
`MAX_VERSION_COUNT_OVERFLOW` as well as `halted_version_count`.
(cherry picked from commit 5cac4316db)
`ts_query_cursor_next_capture` linearly scanned all finished states to
find the one with the earliest next capture byte offset. With deeply
nested code, this O(n) scan per capture caused the highlight crate to
hang for minutes on large files.
Replace the linear scan with a min-heap over the finished_states array,
keyed by (next_capture_byte_offset, pattern_index, id). The heap is
maintained lazily: ts_query_cursor__advance uses plain array_push
(preserving FIFO insertion order), and next_capture sifts new elements
into the heap on entry via a tracked heap_size boundary. This preserves
the documented "order found" guarantee for next_match while giving
next_capture O(log n) per call.
(cherry picked from commit 123fb1c13c)
Commit 1f6eac55 ("query: Use uint32_t for capture list IDs") widened
QueryState.capture_list_id to uint32_t and removed the 65536 pool cap,
but left the pool function signatures as uint16_t. This caused silent
truncation when the pool exceeded 65535 entries, leading to a segfault.
(cherry picked from commit 361f293a73)
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
(cherry picked from commit 0f6780b9a3)
`generate_parser_for_grammar`
Passing `empty` here causes some grammars (i.e. tree-sitter-cpp) to fail
to generate.
(cherry picked from commit 457eb295b7)
This commit skips adding entries to the subtype map when the subtypes
list is empty to avoid a lookup failure in the topological sort during
node type generation.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
(cherry picked from commit 7c2e757c03)
- Cache tree byte range length to avoid redundant FFI calls per test
- Combine 5 separate XML attribute write! calls into one
- Pre-allocate format_sexp output buffer to avoid repeated growth
(cherry picked from commit 791d7cead4)
Wrap stdout in a 64KB BufWriter when writing `parse` output. The tree
walking loop makes many small write calls (parentheses, indentation,
node kinds, ranges, etc.) which are expensive without buffering.
When parsing the jquery.js corpus file, cuts the total time roughly in
half. These savings only show when piping the result to a file,
otherwise terminal rendering time usually dominates, hiding all gains.
Also do the same for the `query` command's output.
(cherry picked from commit 827bcdabd9)
The workspace dependency for `tree-sitter-generate` did not set
`default-features = false`, so Cargo always enabled its default
features (including `qjs-rt` and thus `rquickjs`) regardless of
the CLI's `--no-default-features` flag.
Additionally, `tree-sitter-generate` failed to compile without the
`load` feature due to unconditional references to `cfg`-gated items.
- Set `default-features = false` on the workspace `tree-sitter-generate`
dependency so the CLI's feature forwarding actually takes effect.
- Explicitly enable the `load` feature in the CLI's dependency on
`tree-sitter-generate`, since the CLI needs `load`-gated functions
unconditionally.
- Gate necessary imports behind `#[cfg(feature = "load")]` to fix
`tree-sitter-generate`'s build without the `load` feature.
Problem: The `clang` binary contained in the WASI-SDK releases downloaded from Github does not work on all platforms (e.g., Alpine/MUSL), but a custom (LLVM) `clang` built for the platform will default to that target, making it impossible to build wasm parsers.
Solution: Always pass `wasm32` target triple when calling clang to compile to wasm.
Notes:
* This assumes the custom `clang` is (installed or linked) to `$TREE_SITTER_WASI_SDK_PATH/bin`.
* This requires a full LLVM clang; Apple clang does not support `wasm` targets.
* Tree-sitter expects a specified version of WASI-SDK, including a specific `clang` version. Other versions may but are not guaranteed to work.
