Problem: Parsing UTF-16BE source containing supplementary-plane characters, such as `let emoji = "😀"`, decoded the emoji as two isolated surrogates on little-endian hosts, causing incorrect lexer lookahead and potentially shifted token boundaries.
Soluton: Fix the trailing surrogate byte-order conversion in both UTF-16LE and UTF-16BE decoders and adds a regression test for U+1F600.
`node_kind_is_named`, `node_kind_is_visible` and `node_kind_is_supertype`
forwarded their `u16` straight to `ts_language_symbol_type`, which indexes
`symbol_metadata` unchecked. Bound the id by `node_kind_count()`.
`ts_language_next_state` indexed the parse table with the caller's `state`
and `symbol` without validating either, and `ts_language_subtypes` read
`symbol_metadata` before checking `supertype` was in range.
The field handed out a slice whose lifetime outlived the streaming-iterator
loan it came from. An accessor reborrowing through `&self` bounds it correctly.
BREAKING CHANGE: In the rust bindings, replace `m.captures` with `m.captures()`.
Add documentation about version compatibility requirements between
web-tree-sitter parser ABIs for WASM file generation.
This helps users understand why Language.load() might fail when using
pre-built WASM files from third-party packages that were built with
incompatible tree-sitter-cli versions.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Browser bundlers (esbuild, webpack, vite, rollup) statically discover the
`await import("fs/promises")` and `await import("module")` calls in the
shipped ESM bundle and try to resolve them, even though both sit in
Node-only branches that never execute in a browser.
Adding a top-level `browser` field tells bundlers to substitute empty
stubs for these specifiers when targeting the browser. The runtime gates
(`globalThis.process?.versions.node` and `ENVIRONMENT_IS_NODE`) already
prevent the stubs from ever being invoked.
Fixes#5545.
* build(deps): upgrade wasmtime C API to 48.0.0
Upgrade the Rust and Zig Wasmtime dependencies and enable reference values with the null GC collector.
Wasmtime 48 requires a newer Rust toolchain, whose Clippy version identifies three item helpers that can be const. Mark them const so the workspace continues to pass Clippy with warnings denied.
* feat(benchmark): support Wasm grammars
* perf(wasm): cache language function handles
* fix(wasm): improve validation and failure cleanup
Bounds-check dylink metadata parsing, require exact import and export names, and restore memory and function-table allocation offsets when language loading fails.
* fix(wasm): copy the complete supertype map
* feat(wasm): make syntax trees sendable
* test(wasm): transfer trees across workers
* test(wasm): use JSON grammar for tree transfer
* test(wasm): edit trees across workers
* test(wasm): share dlmalloc with tree-sitter
* test(wasm): simplify worker tree exchange
* test(wasm): drive tree exchange from Rust
* test(wasm): split sendable-tree xtask
* test(wasm): generalize Rust web fixture
* test(wasm): exercise parallel Rust tree access
* fix(wasm): use Rust global allocator for C core
* test(wasm): use default Rust allocator
* feat(wasm): support external scanners in Rust web apps
* Simplify example further, add a readme
* Regenerate wasm-stdlib
* Fix wasm_stdlib check script
* Vendor the Wasm standard library subset
* Test Unicode Ruby scanner behavior in Wasm
* Make Wasm tree languages instance-aware
* Test multi-threaded use of queries in wasm32-unknown
* Refactor reference-counted language storage
* Check ABI version compat before loading rest of language
* 🎨 Remove redundant #ifdef block
* Reject unsupported Rust Wasm builds on 0.26
`Node::kind`, `Node::grammar_name`, the field name accessors,
`TreeCursor::field_name`, and the `Language` name lookups returned `&'static
str` while pointing into storage owned by the `TSLanguage`. Releasing the last
handle to a Wasm language frees that storage.
`Node` and `TreeCursor` now return `&'tree str`, which is ok because
`ts_tree_new` takes a reference to the language via `ts_language_copy` and
holds it until `ts_tree_delete`. The `Language` lookups return strings borrowed
from `&self`.
`currentType` and `currentTypeId` return `null` before the first iteration
step, after exhaustion, and after a reset.
`currentType` also falls back to the language's own name table instead of the
literal 'ERROR' when `Language.types` has no entry (auxiliary symbols). Other
bindings report `end` where this one reported 'ERROR'.
`current_symbol` and `current_symbol_name` return an `Option` (`None` when the
iterator is not positioned on a symbol). Change `iter_names`'s item from
`&'static str` to `&str`.
Track the iterator's phase so exhaustion is sticky, and gate the symbol name on
it, so `NULL` means "not positioned on a symbol".
`ts_lookahead_iterator_new` was also the only language consuming constructor
that did not `ts_language_copy`, so an iterator outliving a wasm language read
freed memory. Retain in `_new` and `_reset`, release in `_delete`.
Previously:
- `ts_lookahead_iterator__next` left a small parse state's cursor one past its
group end, so re-advancing an exhausted iterator resumed returning `true` and
walked through the rest of `ts_small_parse_table` and off the end of it.
- `ts_lookahead_iterator_current_symbol_name` returned `NULL` only when the
exhausted symbol index happened to fall outside the names table, so a grammar
with aliases returned a real but wrong name instead.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
`QueryMatches` and `QueryCaptures` retain query cursor options while they are
being iterated. However, their types did not preserve the lifetime of the
options' progress callback, allowing the callback to be dropped while it
was still referenced by the cursor.
Add an explicit options lifetime to both iterator types and propagate it
through `matches_with_options` and `captures_with_options`. Use a static
options lifetime for iterators created without options.
The early `ts_parser__recover` dispatch for `ERROR_STATE` in
`ts_parser__advance` was dropped in `201b41cf1`. Without it, tokens
with no valid action re-enter `ts_parser__handle_error` per token
and fragment the tree instead of extending one ERROR node. Restore
it to recover the old (pre-0.25) error recovery behavior.
`Parser::set_logger` boxes the logger callback into a type-erased C
pointer, but `Parser` carries no lifetime parameter. This allows for
two kinds of UB:
- Use-after-free: a logger could capture a non-`'static` borrow, let the
borrowed value drop, and then dangle the next time the parser logged.
- Data race: a logger could capture a `!Send` value such as an `Rc`. The
parser could then be moved to another thread and logged from there
while the original thread still held a clone, racing on the reference
count.
As a fix, we just require that logger is `Send + 'static`. The
alternative here is to attach a lifetime parameter to `Parser`, but this
is highly breaking for what is mostly a debugging utility. As an escape
hatch `set_logger_unchecked` is added to the public API as an `unsafe fn`
to correctly communicate the risks and invariants that must be held.
`(P . Q* Y)` with zero `Q` dropped the leading `.`, so `Y` matched at any
position instead of being pinned to the parent's first named child. On a `?`/`*`
zero-skip, when the skipped step is the parent's first child and carries a
leading anchor, transfer that first-child requirement to the skip target.
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
