mirror of
https://github.com/tree-sitter/tree-sitter.git
synced 2026-09-10 07:26:23 -04:00
ec0f2b59ee
- replace `Vec<Vec<ParseStateId>>` with a flat bitset in `CoincidentTokenIndex` for membership queries The previous representation stored full lists of parse state IDs for each token pair. Since only membership (yes/no) is needed for most queries, a flat bitset (`Vec<u64>` indexed as `a * n + b`) is better. Both `(a,b)` and `(b,a)` bits are set during construction so `contains()` needs no min/max normalization. The original `entries: Vec<Vec<ParseStateId>>` and `states_with()` method are retained alongside the bitset, because `identify_keywords` needs to iterate over the specific parse states where two tokens co-occur rather than scanning all states. ~8% walltime reduction for tree-sitter-bash. - add word-aligned per-row bitsets to `CoincidentTokenIndex` for vectorized intersection checks Row `a` spans `[a * row_words .. (a+1) * row_words]` where `row_words = n.div_ceil(64)`. This enables callers in `build_lex_table` to perform word-level AND operations against `TokenSet::terminal_bits_words()` instead of iterating individual token indices. ~6% walltime reduction on tree-sitter-bash. - add `TokenSet::terminal_bits_words()` accessor and `#[inline]` on `BitVec::insert_all` Expose the raw `&[u64]` backing the terminal bitset so callers can perform word-level bitwise intersection. Mark `insert_all` as `#[inline]` to allow the compiler to optimize the hot OR loop. |
||
|---|---|---|
| .. | ||
| cli | ||
| config | ||
| generate | ||
| highlight | ||
| language | ||
| loader | ||
| tags | ||
| xtask | ||