The previous implementation of `InputGrammar::normalize` (inlined in
`parse_grammar` iterated over every variable, checking whether it was
reachable _backwards_ from teh root by recursing over rules that
referenced it.
This means that each and every top level call re-traversed the entire
graph. The runtime performance of this backwards walk was dependent on
the _order_ of rules as declared in `grammar.js`. All existing grammars
have an ordering that's reasonably friendly to this iteration pattern
(BFS-ish order, top down from the start rule), but this leaves us open
to a catastrophic performance cliff.
Instead, seed a `used` set with the start rule, the word token, and an
names referenced from `extras`/`externals`. Then propagate via direct rule
references. This yields anywhere from a 2-~2200x speedup for
`parse_grammar`. This greatly speeds up `--no-parser` runs, but is
relatively unimportant for `parser.c` generation for _existing_
grammars. The important piece is eliminating the potential cliff.
debug output.
The initial implementation of `--debug pretty` assumed process version
was bounded by `MAX_VERSION_COUNT`. However, we also have to account for
`MAX_VERSION_COUNT_OVERFLOW` as well as `halted_version_count`.
`ts_query_cursor_next_capture` linearly scanned all finished states to
find the one with the earliest next capture byte offset. With deeply
nested code, this O(n) scan per capture caused the highlight crate to
hang for minutes on large files.
Replace the linear scan with a min-heap over the finished_states array,
keyed by (next_capture_byte_offset, pattern_index, id). The heap is
maintained lazily: ts_query_cursor__advance uses plain array_push
(preserving FIFO insertion order), and next_capture sifts new elements
into the heap on entry via a tracked heap_size boundary. This preserves
the documented "order found" guarantee for next_match while giving
next_capture O(log n) per call.
Commit 1f6eac55 ("query: Use uint32_t for capture list IDs") widened
QueryState.capture_list_id to uint32_t and removed the 65536 pool cap,
but left the pool function signatures as uint16_t. This caused silent
truncation when the pool exceeded 65535 entries, leading to a segfault.
Solaris does not provide <endian.h> or <sys/endian.h>, but it does expose
byte-order definitions and conversion helpers via <sys/isa_defs.h> and
<sys/byteorder.h>.
Add a __sun branch so the portable header defines __BYTE_ORDER and the
htobe*/le*toh conversions on Solaris.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
Problem: The nvim-treesitter tests use a prebuilt Neovim, which means
changes to the library are not tested and parser generation is locked at
the ABI Neovim supports.
Solution: Build Neovim against the PR version of the library on Ubuntu
and macOS. On Windows, we still download binaries (because one does not
simply build Neovim on Windows...) so we still can check backward
compatibility.
- replace `Vec<Vec<ParseStateId>>` with a flat bitset in
`CoincidentTokenIndex` for membership queries
The previous representation stored full lists of parse state IDs for
each token pair. Since only membership (yes/no) is needed for most
queries, a flat bitset (`Vec<u64>` indexed as `a * n + b`) is
better. Both `(a,b)` and `(b,a)` bits are set during construction
so `contains()` needs no min/max normalization. The original
`entries: Vec<Vec<ParseStateId>>` and `states_with()` method are
retained alongside the bitset, because `identify_keywords` needs
to iterate over the specific parse states where two tokens co-occur
rather than scanning all states. ~8% walltime reduction for
tree-sitter-bash.
- add word-aligned per-row bitsets to `CoincidentTokenIndex` for
vectorized intersection checks
Row `a` spans `[a * row_words .. (a+1) * row_words]` where
`row_words = n.div_ceil(64)`. This enables callers in
`build_lex_table` to perform word-level AND operations against
`TokenSet::terminal_bits_words()` instead of iterating individual
token indices. ~6% walltime reduction on tree-sitter-bash.
- add `TokenSet::terminal_bits_words()` accessor and `#[inline]` on
`BitVec::insert_all`
Expose the raw `&[u64]` backing the terminal bitset so callers can
perform word-level bitwise intersection. Mark `insert_all` as
`#[inline]` to allow the compiler to optimize the hot OR loop.
- precompute a flat NFA-state-to-variable-index lookup table
Build a `Vec<usize>` mapping each NFA state ID to its owning
variable index up front in `TokenConflictMap::new`, replacing
repeated binary searches in `compute_conflict_status` with
array lookups. Also simplifies `mark_fragile_tokens` to
collect terminal indices into a `Vec` rather than maintaining
a boolean mask.
- replace `FxHashSet<Vec<u32>>` visited set with `FxHashSet<u64>`
using a hash-of-states key
In `compute_conflict_status`, the visited-state-set previously
stored cloned `Vec<u32>` NFA state sets. Replace this with an
`FxHashSet<u64>` keyed by hashing the sorted state slice via
`FxHasher`. This eliminates the `Vec` clone on every BFS step.
- reduce allocations in `NfaCursor::group_transitions`
Reuse a single `CharacterSet` buffer across iterations via `assign`
and `mem::take` instead of cloning the input on every raw
transition. Also, replace `Vec::insert` with `Vec::push` when
splitting intersection transitions. The final sort makes
mid-loop ordering irrelevant, and the disjointness of the split
sets guarantees correctness.
- pre-allocate `NfaTransition` result vectors with a capacity of 8
and use `swap_remove` instead of `remove` when merging duplicate
transition entries
- lazily compute `within_separator` in `compute_conflict_status`
Most BFS states have no completions, so the separator check
(which iterates all cursor transitions) is never needed in those
iterations. Wrap it in `Option::get_or_insert_with` to defer
computation until first use.
- add `NfaCursor::transitions_and_any_sep` to fuse the transitions
and separator-check passes
Callers like `build_lex_table::add_state` previously called
`transitions()` and then `transition_chars().any(|sep|)` separately,
iterating the raw NFA transitions twice. The fused method computes
both in a single pass.
Replace `std::collections::{HashMap, HashSet}` with
`rustc_hash::{FxHashMap, FxHashSet}` throughout `generate`. `FxHash`
is _much_ faster than the default SipHash for small integer keys,
which are used extensively through `generate`. The one "risk" here is
that `FxHash` is more vulnerable to DDOS attacks, but that isn't as much
of a concern for us here.
- use bitflags for `TokenConflictStatus`
Replace the individual boolean fields on the conflict status type
with a `bitflags!` representation. This takes up significantly less
space and allows for faster combination/testing in `token_conflicts`.
- hoist loop bounds in the merge-join loops of `states_conflict`
Extract `Vec::len()` outside of `while` loops to avoid recomputing
bounds on every iteration. Remove the now-unused `row_offset` /
`does_conflict_at` indirection on `TokenConflictMap`, inlining the
conflict check directly.
- optimize `variable_index_for_nfa_state` in `grammars.rs`
Small optimization to the NFA state-to-variable index mapping.
- refactor symbol key into a proper newtype
Extract the raw `u64` symbol key bit manipulation into a `key_index`
helper function for some extra type safety.
- use a boolean `Vec` instead of `Vec::contains` in `dedup`
Replace linear containment checks (`split_state_ids.contains()`)
with a constant lookup indexed by state ID.
- narrow parse state IDs from `usize` to `u16`
Reduces memory footprint and improves packing in `ParseTable` and
`build_parse_table`. This is more cache-friendly and reduces peak rss by
~1-2%.
- precompute lookup tables for parse table minimization
Build several maps up front (symbol-to-action, nonterminal indices,
symbol keys) rather than recomputing them in the hot minimization
loop. The `SymbolKey` optimization is the largest single win: it
packs each `Symbol`'s type tag and index into a single u64, enabling
single-instruction comparisons in the merge-join that dominates
`states_conflict`. This gives a significant speedup for larger
grammars.
- store ParseTableEntry` refs in the precomputed entry maps to
avoid `IndexMap::get_index` lookups in the inner loop of
`states_conflict`
This eliminates an indirect lookup per comparison in the merge-join.
- use unchecked array accesses in hot loops
Where indices are known valid from construction (e.g. the merge-join
in `states_conflict` and dedup), skip bounds checks in the innermost
loops. The compiler can _usually_ prove this and remove the checks
itself, but not always.
