This commit adds an `eof()` function for grammars, which is easier to
use than the NUL byte directly. It compiles down to a constraint that
the enclosing production can only reduce at end of input, not to a
shiftable token.
BREAKING CHANGE: Public error types for `tree-sitter-generate` were modified.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
* build(deps): upgrade wasmtime C API to 48.0.0
Upgrade the Rust and Zig Wasmtime dependencies and enable reference values with the null GC collector.
Wasmtime 48 requires a newer Rust toolchain, whose Clippy version identifies three item helpers that can be const. Mark them const so the workspace continues to pass Clippy with warnings denied.
* feat(benchmark): support Wasm grammars
* perf(wasm): cache language function handles
* fix(wasm): improve validation and failure cleanup
Bounds-check dylink metadata parsing, require exact import and export names, and restore memory and function-table allocation offsets when language loading fails.
* fix(wasm): copy the complete supertype map
The JSON output types now hold ids from the string pool instead of
cloning into owned `String`s, which makes `NodeTypeJSON` `Copy` and
drops most allocations when building the output.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Currently, a symbol listed in both `supertypes` and `inline` produces
a phantom supertype entry in `node-types.json` for a rule that can
never appear in a tree. `intern_symbols` now drops the supertype with
a warning. A hard error would break 29 published grammars, including
python, go, ruby, rust, and scala, so that that decision is deferred
to the next breaking release. Fixes#5218
* feat(wasm): make syntax trees sendable
* test(wasm): transfer trees across workers
* test(wasm): use JSON grammar for tree transfer
* test(wasm): edit trees across workers
* test(wasm): share dlmalloc with tree-sitter
* test(wasm): simplify worker tree exchange
* test(wasm): drive tree exchange from Rust
* test(wasm): split sendable-tree xtask
* test(wasm): generalize Rust web fixture
* test(wasm): exercise parallel Rust tree access
* fix(wasm): use Rust global allocator for C core
* test(wasm): use default Rust allocator
* feat(wasm): support external scanners in Rust web apps
* Simplify example further, add a readme
* Regenerate wasm-stdlib
* Fix wasm_stdlib check script
* Vendor the Wasm standard library subset
* Test Unicode Ruby scanner behavior in Wasm
* Make Wasm tree languages instance-aware
* Test multi-threaded use of queries in wasm32-unknown
* Refactor reference-counted language storage
* Check ABI version compat before loading rest of language
* 🎨 Remove redundant #ifdef block
* Reject unsupported Rust Wasm builds on 0.26
Problem: `Package.swift` build script is unmaintained and leads to build
errors.
Solution: Remove `Package.swift`; downstream tools should rely on the
swift-tree-sitter bindings (or, if they want a custom bare-metal build,
handle this in their own build scripts).
`test::run_wasm` gated dependency installation on `node_modules/chai` and
`node_modules/mocha`. Neither is a dependency (vitest is used instead),
so the check never passed and every `cargo xtask test-wasm`
reran `npm install`.
`Node::kind`, `Node::grammar_name`, the field name accessors,
`TreeCursor::field_name`, and the `Language` name lookups returned `&'static
str` while pointing into storage owned by the `TSLanguage`. Releasing the last
handle to a Wasm language frees that storage.
`Node` and `TreeCursor` now return `&'tree str`, which is ok because
`ts_tree_new` takes a reference to the language via `ts_language_copy` and
holds it until `ts_tree_delete`. The `Language` lookups return strings borrowed
from `&self`.
`currentType` and `currentTypeId` return `null` before the first iteration
step, after exhaustion, and after a reset.
`currentType` also falls back to the language's own name table instead of the
literal 'ERROR' when `Language.types` has no entry (auxiliary symbols). Other
bindings report `end` where this one reported 'ERROR'.
`current_symbol` and `current_symbol_name` return an `Option` (`None` when the
iterator is not positioned on a symbol). Change `iter_names`'s item from
`&'static str` to `&str`.
Track the iterator's phase so exhaustion is sticky, and gate the symbol name on
it, so `NULL` means "not positioned on a symbol".
`ts_lookahead_iterator_new` was also the only language consuming constructor
that did not `ts_language_copy`, so an iterator outliving a wasm language read
freed memory. Retain in `_new` and `_reset`, release in `_delete`.
Previously:
- `ts_lookahead_iterator__next` left a small parse state's cursor one past its
group end, so re-advancing an exhausted iterator resumed returning `true` and
walked through the rest of `ts_small_parse_table` and off the end of it.
- `ts_lookahead_iterator_current_symbol_name` returned `NULL` only when the
exhausted symbol index happened to fall outside the names table, so a grammar
with aliases returned a real but wrong name instead.
`(?-u:...)` switches `regex_syntax` to matching raw bytes, but the lexer
dispatches on decoded characters, so `expand_regex` converted the byte
class with a u8 cast. This is exact for ASCII bytes, and silently
misleading for anything aboove 0x80.
This is not reachable from grammar.js (node and QuickJS both reject
`(?-u:...)`). This change is to guard against future JS runtime changes,
as well as alternative frontends to the generate crate.
A previous fix moved case folding for `/i` patterns out of `regex_syntax` and
into `expand_regex`, so folding could drop the two non-ASCII code points
Unicode simple folding maps onto ASCII letters: the long s `ſ` (U+017F)
onto `s`, and the Kelvin sign `K` (U+212A) onto `k`. Left in, they leak
into otherwise-ASCII tokens and stop those tokens from being extracted as
keywords.
By that point, though, the HIR has already turned a negated class into a
complement, so folding it applies the fold on the wrong side of the
negation. `(?i)[^a-z]` folds a set that contains `A-Z`, which re-admits
`a-z` and leaves a class matching very nearly everything.
`regex_syntax` folds each leaf of a class expression before applying that
leaf's negation and the set algebra above it. Keep that order and change
only the fold: walk the AST, replace each leaf with its fold, and translate
with `case_insensitive(false)`.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
This is already exposed for consumers via the CLI, and is a natural way
to express some test expectations over the sexp form.
Also clean up some repeated logic in the internal test code, and narrow
the cst rendering return type to `std::io::Result` rather than
`anyhow::Result`.
Every (state, terminal) parse-table entry stored its action list inline as a
32-byte `ParseTableEntry`, but across a grammar those lists are ~98-99% _duplicates_.
The number of distinct lists is a few thousand regardless of grammar size, while
total entries scale into the hundreds of thousands.
Store each unique action list once in a shared `ActionListPool` (a flat arena of
actions plus `(offset, len)` ranges) and replace the inline entry with a 4-byte
`ActionListId` (a pool index with the `reusable` flag packed into the high bit).
- intern_table converts the freshly built `ParseTable<ParseTableEntry>` into
`ParseTable<ActionListId>`.
- minimize carries and operates on the 4-byte ids. The three global state
renumberings rewrite Shift targets once at the pool level
(`remap_terminal_references`), while the per-state unit-reduction redirects
copy the changed list into a new slot (COW). `mark_fragile_tokens` becomes a
free bit flip on the id.
- `canonicalize` dedups and compacts the pool once before `render`, dropping the
dead and duplicate slots the remaps and COW leave behind. `render` assigns the
output action-list offsets directly.
Yields ~7% wall time reduction, ~12% peak rss reduction.
`ParseStateId`, `LexStateId`, `ProductionInfoId`, and `ReservedWordSetId`
each held a `usize`, while the runtime stores state/field ids as `u16` (so
`u32` has plenty of headroom). Shrinking them saves space for every parse-table
entry, and a number of other data structures.
Reduces wall time by 2-5%, peak rss by 9-12%.
`get_auxiliary_node_info` scans every entry in the item set to collect the
non-auxiliary parents of a given auxiliary symbol. It was called once per
entry whose next symbol is auxiliary, and the same auxiliary symbol recurs
across many entries in a single state (a state's GOTO on a repeat symbol is
shared by every item advancing over it), so the same full scan was repeated
many times per state. Memoize the result per symbol within a single
`add_actions` call.
Reduces wall time 1-4%, rss flat.
`ParseTableEntry` held a `Vec<ParseAction>` per entry. `ActionList` keeps up
to one action inline and spills to a Vec only for conflict entries.
Reduces walltime by >20%, peak rss by ~30%.
The production info depends only on the production, but was recomputed and
deduplicated via a linear deep-equality scan over production_infos for every
reduce item in every state. Cache `prod_id` -> `ProductionInfoId`.
Wall time reduction 0-3%, peak rss neutral.
`ParseItemSetEntry` stores a `LookaheadSetId` into a `LookaheadSetPool` instead
of an owned `TokenSet`. Ids are canonical, so entry hash/eq/clone are integer
ops and state dedup stops walking set words. Unions and single-token inserts
are memoized by id, so the transitive closure and successor-kernel construction
stop re-materializing the same unions per state. Closure additions carry interned
ids with word-token membership precomputed.
For cpp, 3566 distinct lookahead sets back all 49,992 states. For ruby 2000 sets
for 91,991 states, and rust 646 for 16,796.
Reduces wall time by 10-15%, peak rss by 3-5%.
The state-merge pass's `token_conflicts` scanned every terminal entry of the
other state per non-shared token. Precompute flat, word-aligned bitsets,
`ConflictBits`, so each call is a handful of word ANDs.
Yields a 5-6% wall time reduction on large grammars, and flat to ~1%
loss on small grammers. Peak RSS unchanged.
Every state's kernel set was cloned into `parse_state_info` even though the
identical set already lives as the state-dedup map key at the same index
(state ids are the map's insertion order). `ParseStateInfo` now holds the
preceding-symbol sequences plus the dedup map moved out of the builder,
allowing the report path to read kernels via `get_index`. Drops the
duplicate kernel storage, allowing for a small reduction in wall time
and peak rss (1-3%).
`CoincidentTokenIndex` kept a `Vec<ParseStateId>` per token pair whose
only user was keyword identification asking one question: "do all states
where a given pair coincides also allow the word token?" Track that
directly as a second bitset built in the same pass.
Reduces walltime by ~7% for small grammars (i.e. Go), and 20-25% for
larger grammars (i.e. Rust, cpp). Reduces peak rss by ~20% for smaller
grammars and ~40-50% for larger grammars.
Replace the owned `Rule`-tree grammar with a flat, pooled representation. `Rule`
nodes live in a `RulePool` arena and reference their children and params by
index (`RuleId`). Strings are interned in a `StrPool` and referred to by `StrId`.
Productions are stored as flat `ProductionStep`/`Production` slices indexed per
variable, rather than as nested vectors hanging off each `SyntaxVariable`. This
drops the per-rule heap allocation of the owned-tree form and is the groundwork
for the later table-building optimizations.
`parse_grammar` builds the pool, the prepare passes rewrite nodes in place, and
`prepare_grammar` returns a `PreparedGrammar` bundle that owns the run's `StrPool`
alongside the grammars. `build_tables`, `node_types`, and `render` borrow that
pool and resolve `StrId`s only at the output boundary.
Shows an average ~10% walltime reduction when combined with the previous
commit, with nearly all of the savings in `build_tables`. The `prepare`
stage is also ~70% faster and much more stable, but this contributes
much less to overall generate time. Improving early stages such as
`prepare` will be important for future interactive uses.
Parse-item identity dominates parse-table construction. `ParseItem`'s
`Hash`, `Eq`, and `Ord` walk the entire production on every call,
comparing per-step precedence, aliases, and field names. These are
stored as strings for named precedences and for every alias and field.
Precompute, once per table build, two dense `u32` ids for every
`(production, dot)` slot by ranking the whole slot universe with the
item-content comparator, now extracted as `ItemContent`. Equal-content
slots are adjacent after sorting and the id increments only at content
boundaries, so comparing the `cmp` id reproduces the structural order
exactly (both an ordering and an equality). A second id, `eq_with_syms`,
refines `cmp` by the preceding symbols, which enter equality only for
items with `has_preceding_inherited_fields`.
`ParseItem` now carries its production's key slice, and `Hash`/`Eq`/`Ord`
become integer ops.
On the current string-based production storage this is performance-neutral
in isolation (the up-front ranking pays back the per-comparison savings).
It is the foundation for the following change to a pooled, integer-keyed
production representation, which makes both the ranking and the item
comparisons cheaper integer work.
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
