Compiling a grammar with `-fsanitize=address` (or any other sanitizer)
emits a module constructor in the parser object file that references
runtime symbols like `__asan_init`. These runtime-resolved symbols are
incompatible with `-Wl,--no-undefined`, which breaking sanitized test
runs on clang. (gcc happens to paper over this by auto-linking libasan
into the .so)
To fix, detect `-fsanitize=` in the compile flags and omit `--no-undefined`
in that case.
The alloc.h header checked for TREE_SITTER_HIDDEN_SYMBOLS, but the
canonical macro name used everywhere else (api.h, render.rs, setup.py)
is TREE_SITTER_HIDE_SYMBOLS. This mismatch meant that defining
TREE_SITTER_HIDE_SYMBOLS (as the Python binding build does) would not
actually hide the allocator symbols in alloc.h.
Fixestree-sitter/tree-sitter#5625
Signed-off-by: Georges Savoundararadj <savoundg@amazon.com>
The wasm store gated supertype_symbols / supertype_map_slices /
supertype_map_entries copies on abi_version > LANGUAGE_VERSION_WITH_RESERVED_WORDS,
but every other consumer (language.c, query.c) treats those tables as
present when abi_version >= LANGUAGE_VERSION_WITH_RESERVED_WORDS.
A Wasm grammar built at ABI exactly 15 with supertype_count > 0 ends up
with supertype_count copied into the native TSLanguage but supertype_map_slices
left NULL. ts_query__analyze_patterns then calls ts_language_subtypes,
which dereferences self->supertype_map_slices[supertype] and crashes.
Several test functions compile the same grammar fixture.
Tests sharing a grammar name also share a src_dir. Each test also
unconditionally writes the three tree-sitter headers into
src_dir/tree_sitter/, leading to a race.
Write the three headers exactly once per src_dir, controlled via a
global `HashSet`.
The previous implementation of `InputGrammar::normalize` (inlined in
`parse_grammar` iterated over every variable, checking whether it was
reachable _backwards_ from teh root by recursing over rules that
referenced it.
This means that each and every top level call re-traversed the entire
graph. The runtime performance of this backwards walk was dependent on
the _order_ of rules as declared in `grammar.js`. All existing grammars
have an ordering that's reasonably friendly to this iteration pattern
(BFS-ish order, top down from the start rule), but this leaves us open
to a catastrophic performance cliff.
Instead, seed a `used` set with the start rule, the word token, and an
names referenced from `extras`/`externals`. Then propagate via direct rule
references. This yields anywhere from a 2-~2200x speedup for
`parse_grammar`. This greatly speeds up `--no-parser` runs, but is
relatively unimportant for `parser.c` generation for _existing_
grammars. The important piece is eliminating the potential cliff.
debug output.
The initial implementation of `--debug pretty` assumed process version
was bounded by `MAX_VERSION_COUNT`. However, we also have to account for
`MAX_VERSION_COUNT_OVERFLOW` as well as `halted_version_count`.
`ts_query_cursor_next_capture` linearly scanned all finished states to
find the one with the earliest next capture byte offset. With deeply
nested code, this O(n) scan per capture caused the highlight crate to
hang for minutes on large files.
Replace the linear scan with a min-heap over the finished_states array,
keyed by (next_capture_byte_offset, pattern_index, id). The heap is
maintained lazily: ts_query_cursor__advance uses plain array_push
(preserving FIFO insertion order), and next_capture sifts new elements
into the heap on entry via a tracked heap_size boundary. This preserves
the documented "order found" guarantee for next_match while giving
next_capture O(log n) per call.
Commit 1f6eac55 ("query: Use uint32_t for capture list IDs") widened
QueryState.capture_list_id to uint32_t and removed the 65536 pool cap,
but left the pool function signatures as uint16_t. This caused silent
truncation when the pool exceeded 65535 entries, leading to a segfault.
Solaris does not provide <endian.h> or <sys/endian.h>, but it does expose
byte-order definitions and conversion helpers via <sys/isa_defs.h> and
<sys/byteorder.h>.
Add a __sun branch so the portable header defines __BYTE_ORDER and the
htobe*/le*toh conversions on Solaris.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
