This commit adds an `eof()` function for grammars, which is easier to
use than the NUL byte directly. It compiles down to a constraint that
the enclosing production can only reduce at end of input, not to a
shiftable token.
BREAKING CHANGE: Public error types for `tree-sitter-generate` were modified.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
* feat(wasm): make syntax trees sendable
* test(wasm): transfer trees across workers
* test(wasm): use JSON grammar for tree transfer
* test(wasm): edit trees across workers
* test(wasm): share dlmalloc with tree-sitter
* test(wasm): simplify worker tree exchange
* test(wasm): drive tree exchange from Rust
* test(wasm): split sendable-tree xtask
* test(wasm): generalize Rust web fixture
* test(wasm): exercise parallel Rust tree access
* fix(wasm): use Rust global allocator for C core
* test(wasm): use default Rust allocator
* feat(wasm): support external scanners in Rust web apps
* Simplify example further, add a readme
* Regenerate wasm-stdlib
* Fix wasm_stdlib check script
* Vendor the Wasm standard library subset
* Test Unicode Ruby scanner behavior in Wasm
* Make Wasm tree languages instance-aware
* Test multi-threaded use of queries in wasm32-unknown
* Refactor reference-counted language storage
* Check ABI version compat before loading rest of language
* 🎨 Remove redundant #ifdef block
* Reject unsupported Rust Wasm builds on 0.26
This is already exposed for consumers via the CLI, and is a natural way
to express some test expectations over the sexp form.
Also clean up some repeated logic in the internal test code, and narrow
the cst rendering return type to `std::io::Result` rather than
`anyhow::Result`.
Replace the owned `Rule`-tree grammar with a flat, pooled representation. `Rule`
nodes live in a `RulePool` arena and reference their children and params by
index (`RuleId`). Strings are interned in a `StrPool` and referred to by `StrId`.
Productions are stored as flat `ProductionStep`/`Production` slices indexed per
variable, rather than as nested vectors hanging off each `SyntaxVariable`. This
drops the per-rule heap allocation of the owned-tree form and is the groundwork
for the later table-building optimizations.
`parse_grammar` builds the pool, the prepare passes rewrite nodes in place, and
`prepare_grammar` returns a `PreparedGrammar` bundle that owns the run's `StrPool`
alongside the grammars. `build_tables`, `node_types`, and `render` borrow that
pool and resolve `StrId`s only at the output boundary.
Shows an average ~10% walltime reduction when combined with the previous
commit, with nearly all of the savings in `build_tables`. The `prepare`
stage is also ~70% faster and much more stable, but this contributes
much less to overall generate time. Improving early stages such as
`prepare` will be important for future interactive uses.
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
A SHIFT/REDUCE conflict can bundle several shift interpretations with
different precedences against a single reduce. `handle_conflict` weighed them
with only `shift_is_less` and `shift_is_more`, so a lone lower-precedence
shift set `shift_is_less` and the REDUCE won outright, even when another
interpretation tied the REDUCE in precedence and the REDUCE was declared
right-associative. That tie's associativity should have won by shifting, so a
right-associative rule silently became left-associative as soon as an
unrelated lower-precedence rule was added to the grammar.
Track the equal-precedence case explicitly, and in the reduce-wins branch
shift instead when a tying interpretation exists and the reduce actions are
purely right-associative.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
the region in place.
Previous changes to `malloc` caused `realloc` to sometimes pull regions
off of the free list during this optimization. Because no `memcpy` is
performed, this resulted in corrupted data returning to the caller.
Co-authored-by: trim21 <i@trim21.me>
- free memory if 0 size is passed in
- Don't `memcpy` contents if new pointer is `NULL`
- Copy old region's contents only up to size of new region
- free old region
Co-authored-by: trim21 <i@trim21.me>
