Problem: The nvim-treesitter tests use a prebuilt Neovim, which means
changes to the library are not tested and parser generation is locked at
the ABI Neovim supports.
Solution: Build Neovim against the PR version of the library on Ubuntu
and macOS. On Windows, we still download binaries (because one does not
simply build Neovim on Windows...) so we still can check backward
compatibility.
- replace `Vec<Vec<ParseStateId>>` with a flat bitset in
`CoincidentTokenIndex` for membership queries
The previous representation stored full lists of parse state IDs for
each token pair. Since only membership (yes/no) is needed for most
queries, a flat bitset (`Vec<u64>` indexed as `a * n + b`) is
better. Both `(a,b)` and `(b,a)` bits are set during construction
so `contains()` needs no min/max normalization. The original
`entries: Vec<Vec<ParseStateId>>` and `states_with()` method are
retained alongside the bitset, because `identify_keywords` needs
to iterate over the specific parse states where two tokens co-occur
rather than scanning all states. ~8% walltime reduction for
tree-sitter-bash.
- add word-aligned per-row bitsets to `CoincidentTokenIndex` for
vectorized intersection checks
Row `a` spans `[a * row_words .. (a+1) * row_words]` where
`row_words = n.div_ceil(64)`. This enables callers in
`build_lex_table` to perform word-level AND operations against
`TokenSet::terminal_bits_words()` instead of iterating individual
token indices. ~6% walltime reduction on tree-sitter-bash.
- add `TokenSet::terminal_bits_words()` accessor and `#[inline]` on
`BitVec::insert_all`
Expose the raw `&[u64]` backing the terminal bitset so callers can
perform word-level bitwise intersection. Mark `insert_all` as
`#[inline]` to allow the compiler to optimize the hot OR loop.
- precompute a flat NFA-state-to-variable-index lookup table
Build a `Vec<usize>` mapping each NFA state ID to its owning
variable index up front in `TokenConflictMap::new`, replacing
repeated binary searches in `compute_conflict_status` with
array lookups. Also simplifies `mark_fragile_tokens` to
collect terminal indices into a `Vec` rather than maintaining
a boolean mask.
- replace `FxHashSet<Vec<u32>>` visited set with `FxHashSet<u64>`
using a hash-of-states key
In `compute_conflict_status`, the visited-state-set previously
stored cloned `Vec<u32>` NFA state sets. Replace this with an
`FxHashSet<u64>` keyed by hashing the sorted state slice via
`FxHasher`. This eliminates the `Vec` clone on every BFS step.
- reduce allocations in `NfaCursor::group_transitions`
Reuse a single `CharacterSet` buffer across iterations via `assign`
and `mem::take` instead of cloning the input on every raw
transition. Also, replace `Vec::insert` with `Vec::push` when
splitting intersection transitions. The final sort makes
mid-loop ordering irrelevant, and the disjointness of the split
sets guarantees correctness.
- pre-allocate `NfaTransition` result vectors with a capacity of 8
and use `swap_remove` instead of `remove` when merging duplicate
transition entries
- lazily compute `within_separator` in `compute_conflict_status`
Most BFS states have no completions, so the separator check
(which iterates all cursor transitions) is never needed in those
iterations. Wrap it in `Option::get_or_insert_with` to defer
computation until first use.
- add `NfaCursor::transitions_and_any_sep` to fuse the transitions
and separator-check passes
Callers like `build_lex_table::add_state` previously called
`transitions()` and then `transition_chars().any(|sep|)` separately,
iterating the raw NFA transitions twice. The fused method computes
both in a single pass.
Replace `std::collections::{HashMap, HashSet}` with
`rustc_hash::{FxHashMap, FxHashSet}` throughout `generate`. `FxHash`
is _much_ faster than the default SipHash for small integer keys,
which are used extensively through `generate`. The one "risk" here is
that `FxHash` is more vulnerable to DDOS attacks, but that isn't as much
of a concern for us here.
- use bitflags for `TokenConflictStatus`
Replace the individual boolean fields on the conflict status type
with a `bitflags!` representation. This takes up significantly less
space and allows for faster combination/testing in `token_conflicts`.
- hoist loop bounds in the merge-join loops of `states_conflict`
Extract `Vec::len()` outside of `while` loops to avoid recomputing
bounds on every iteration. Remove the now-unused `row_offset` /
`does_conflict_at` indirection on `TokenConflictMap`, inlining the
conflict check directly.
- optimize `variable_index_for_nfa_state` in `grammars.rs`
Small optimization to the NFA state-to-variable index mapping.
- refactor symbol key into a proper newtype
Extract the raw `u64` symbol key bit manipulation into a `key_index`
helper function for some extra type safety.
- use a boolean `Vec` instead of `Vec::contains` in `dedup`
Replace linear containment checks (`split_state_ids.contains()`)
with a constant lookup indexed by state ID.
- narrow parse state IDs from `usize` to `u16`
Reduces memory footprint and improves packing in `ParseTable` and
`build_parse_table`. This is more cache-friendly and reduces peak rss by
~1-2%.
- precompute lookup tables for parse table minimization
Build several maps up front (symbol-to-action, nonterminal indices,
symbol keys) rather than recomputing them in the hot minimization
loop. The `SymbolKey` optimization is the largest single win: it
packs each `Symbol`'s type tag and index into a single u64, enabling
single-instruction comparisons in the merge-join that dominates
`states_conflict`. This gives a significant speedup for larger
grammars.
- store ParseTableEntry` refs in the precomputed entry maps to
avoid `IndexMap::get_index` lookups in the inner loop of
`states_conflict`
This eliminates an indirect lookup per comparison in the merge-join.
- use unchecked array accesses in hot loops
Where indices are known valid from construction (e.g. the merge-join
in `states_conflict` and dedup), skip bounds checks in the innermost
loops. The compiler can _usually_ prove this and remove the checks
itself, but not always.
Problem: Many parser projects (about 100 out of ~330 tracked by
nvim-treesitter) still do not contain a `tree-sitter.json` config file,
so `tree-sitter generate` refuses to generate ABI 15 parsers due to the
lack of version information. This prevents these parsers from profiting
from other improvements in ABI 15+ and will eventually make them
unusable when ABI 14 support is dropped.
Solution: Simply use a default version `0.0.0` if no `tree-sitter.json`
is found.
* remove obsolete msvc-dev-cmd action (runners come with build tools)
* build tree-sitter CLI with --profile optimize (longer build time due
to LTO, but generate workflow should profit)
* increase generate parallelism to 3
The previous two-phase locking scheme (probe existing lock file, then
create and compile) had TOCTOU races between phases that caused
spurious failures in CI when tests compiled grammars concurrently.
Replace with a single-phase approach using `create_new` as the sole
synchronization primitive. An RAII `LockFile` guard ensures cleanup
on drop (including panics). Only "builders" attempt to acquire the
lock. Loaders simply load the file. Builders compile a temporary path
and then rename, so loaders are guaranteed a valid shared library.
The winning builder compiles and then drops the lock. Losers poll for
lock file removal, then load. Stale locks from killed processes are
detected via a timeout and an appropriate error message is displayed
to the user.
This started as a simple one to one rewrite, just removing the regexes,
and quickly devolved into a rewrite of the test parsing logic. In
addition to the memory enhancements, the general flow should be much
clearer now. A few data points:
- JS: walltime -1.3%, peak rss -10.2%
- C: walltime -5.7%, peak rss -4.5%
- Rust: walltime -3.8%, peak rss -2.2%
This commit skips adding entries to the subtype map when the subtypes
list is empty to avoid a lookup failure in the topological sort during
node type generation.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
- Cache tree byte range length to avoid redundant FFI calls per test
- Combine 5 separate XML attribute write! calls into one
- Pre-allocate format_sexp output buffer to avoid repeated growth
Wrap stdout in a 64KB BufWriter when writing `parse` output. The tree
walking loop makes many small write calls (parentheses, indentation,
node kinds, ranges, etc.) which are expensive without buffering.
When parsing the jquery.js corpus file, cuts the total time roughly in
half. These savings only show when piping the result to a file,
otherwise terminal rendering time usually dominates, hiding all gains.
Also do the same for the `query` command's output.
We have to pay the cost of compiling the regex at runtime, and the
`LazyLock` overhead for each access. The pattern is simple enough that
we can manually parse and extract.
`grammar_json_name` is now ~116x faster on cold start, ~5.6x faster warm
(after the regex has been compiled). Both are fast enough to not matter
much in practice, but some perf gains and eliminating global state is a
win.
The `LookaheadIterator` visits symbols in group order for small parse
states, which can produce `reduce_actions` in a different order than the
original linear symbol scan (pre c1379718). Since reductions are applied
sequentially and the last reduction version survives, different orderings
lead to different error recovery outcomes (e.g. losing nodes from ERROR
trees).
Sort the reduce_actions array by symbol (descending) after collection.
The array is typically 1-5 entries, so the insertion sort cost is
negligible and the full optimization speedup is preserved.
In `ts_parser__do_all_potential_reductions`, when `lookahead_symbol` is
0 (error recovery), the code scanned every symbol from 1 to
`token_count` calling ts_language_table_entry for each with most returning
empty. Replace with `LookaheadIterator` which efficiently visits only symbols
with valid actions, yielding 7-20x fewer lookups for typical grammars.
Error recovery throughput improves ~29% (JS) / ~39% (C). Valid-code
parsing is unaffected.
Since v0.26.1, `tree-sitter build --wasm` uses wasi-sdk instead of
Emscripten. The CLI automatically downloads wasi-sdk on first use,
so Emscripten, Docker, and Podman are no longer required.
Update the web binding README to reflect this change.
See #4393 for the original switch to wasi-sdk.
The workspace dependency for `tree-sitter-generate` did not set
`default-features = false`, so Cargo always enabled its default
features (including `qjs-rt` and thus `rquickjs`) regardless of
the CLI's `--no-default-features` flag.
Additionally, `tree-sitter-generate` failed to compile without the
`load` feature due to unconditional references to `cfg`-gated items.
- Set `default-features = false` on the workspace `tree-sitter-generate`
dependency so the CLI's feature forwarding actually takes effect.
- Explicitly enable the `load` feature in the CLI's dependency on
`tree-sitter-generate`, since the CLI needs `load`-gated functions
unconditionally.
- Gate necessary imports behind `#[cfg(feature = "load")]` to fix
`tree-sitter-generate`'s build without the `load` feature.
