Signed-off-by: cuishuang <imcusg@gmail.com>
BREAKING CHANGE: changes the signature of `assert_expected_captures`, which is exposed publicly when tree-sitter-cli is consumed as a library.
Problem: Parsing UTF-16BE source containing supplementary-plane characters, such as `let emoji = "😀"`, decoded the emoji as two isolated surrogates on little-endian hosts, causing incorrect lexer lookahead and potentially shifted token boundaries.
Soluton: Fix the trailing surrogate byte-order conversion in both UTF-16LE and UTF-16BE decoders and adds a regression test for U+1F600.
In a future feature, `RulePool` nodes can be shared between lexical and
syntax roots. No grammars generated by grammar.js/grammra.json are affected
by this issue, as they lead to a tree pool rather than a DAG.
With a DAG `RulePool` representation, however, rewriting tokens during
extraction can destroy a token body before lexical expansion consumes
it and can renumber shared syntax nodes more than once.
Record terminal rewrites while inspecting the original pool, expand tokens
and separators first, then commit the rewrites and renumber each reachable
syntax node once. Resolve grammar metadata against the pending rewrites
before mutating the pool.
`try_seq` reserves a `Seq`'s child range up front and fills slots as
recursion produces each member. The `Repeat` arm builds
`Choice(repeat(x), blank)` directly, skipping the `choice` helper.
Slightly faster and less memory used.
`node_kind_is_named`, `node_kind_is_visible` and `node_kind_is_supertype`
forwarded their `u16` straight to `ts_language_symbol_type`, which indexes
`symbol_metadata` unchecked. Bound the id by `node_kind_count()`.
`ts_language_next_state` indexed the parse table with the caller's `state`
and `symbol` without validating either, and `ts_language_subtypes` read
`symbol_metadata` before checking `supertype` was in range.
The field handed out a slice whose lifetime outlived the streaming-iterator
loan it came from. An accessor reborrowing through `&self` bounds it correctly.
BREAKING CHANGE: In the rust bindings, replace `m.captures` with `m.captures()`.
Problem: Editor-specific settings should not be part of the global
repository. (Only config files that control standard tooling in order to
enforce, e.g., formatting policies are appropriate to commit -- and only
if covered by CI.)
Solution: Drop the Zed config files from the repo.
Add documentation about version compatibility requirements between
web-tree-sitter parser ABIs for WASM file generation.
This helps users understand why Language.load() might fail when using
pre-built WASM files from third-party packages that were built with
incompatible tree-sitter-cli versions.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Browser bundlers (esbuild, webpack, vite, rollup) statically discover the
`await import("fs/promises")` and `await import("module")` calls in the
shipped ESM bundle and try to resolve them, even though both sit in
Node-only branches that never execute in a browser.
Adding a top-level `browser` field tells bundlers to substitute empty
stubs for these specifiers when targeting the browser. The runtime gates
(`globalThis.process?.versions.node` and `ENVIRONMENT_IS_NODE`) already
prevent the stubs from ever being invoked.
Fixes#5545.
Previously with the empty string base URL, the playground would not work when hosted at a subfolder of the domain. This is because playground.js has this code:
const url = `${LANGUAGE_BASE_URL}/tree-sitter-${newLanguageName}.wasm`;
With an empty string, it would look for wasm at the root of the website and 404.
Github code search shows that this is an issue which lots of people have run into, and everyone has to work around it: https://github.com/search?q=%2FLANGUAGE_BASE_URL+%3D+%22%22%2F+-language%3AHTML&type=code
Related to https://www.github.com/tree-sitter/tree-sitter/issues/5230
Require `word`, `conflicts`, `inline`, and `supertypes` callbacks
to return named grammar symbols instead of silently accepting values
that produce undefined names.
Restrict `precedences` lists to named rules and precedence names, and
preserve the existing handling of undefined and duplicate `inline`
rules.
Model grammar() as returning the evaluated grammar beneath a `grammar`
property, with callbacks replaced by their normalized values and
runtime-initialized fields represented as required properties.
Correct the evaluated rule types for named precedences and regular
expression flags. Narrow symbol-only callbacks such as `conflicts`,
`inline`, `supertypes`, and `word`, and expose the inherited values
passed to `extras` and `reserved` callbacks.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Some node-types bookkeeping still used the type name without its named
status. This caused anonymous nodes to inherit extra metadata from named
nodes with the same text, and could create false supertype cycles when a
named supertype and anonymous alias shared a name.
Key extra metadata and supertype dependencies by the complete
(type, named) identity.
Reject named aliases that collide with a canonical supertype. Such an
identity cannot be represented as both a concrete node and an abstract
supertype in node-types.json, and previously caused generation to panic.
All output entries now come from the map keyed by (type, named), so no
two entries can reach the final sort with the same identity. Remove the
unreachable root and extra tie-breakers and the now-redundant deduplication.
BREAKING CHANGE: Alters a public error enum for the generate crate.
When an anonymous token alias and an anonymous syntax-node alias have
the same name, node-types.json emits two entries with the same type and
named status. For example,
```
alias($._node, "same")
```
and
```
alias("!", "same")
```
produce separate anonymous "same" entries. Add anonymous tokens to
the node-type map instead. When a token shares an identity with a
structured node, retain its possible children and fields but mark
them as optional because the token appearance is a leaf.
A node-types entry is identified by both its type name and its `named`
value. The generator only keyed accumulated entries by name, so aliases
like
alias($._node, $.same)
alias($._node, "same")
were incorrectly merged into one entry. Whichever alias was processed
first determined the entry's `named` value, and the structures of two
distinct node types could be combined.
Key accumulated entries by `NodeTypeRef` so named and anonymous nodes
with the same type name remain separate.
Aliasing a supertype makes the aliased occurrence appear as a regular
node in the syntax tree. For example,
```
alias($.expression, $.expression_target)
```
can produce:
```
(expression_target (identifier))
```
Previously, node-types generation skipped the source supertype entirely.
It could reference `expression_target` as a field or child type without
emitting a corresponding top-level entry for it.
Continue emitting the canonical `expression` entry with its `subtypes`,
but pass aliased appearances through regular node generation so their
children and fields are recorded and merged normally.
Problem: The committed Dockerfile is not used by the standard tooling in
this repo and therefore bitrots freely, which increases maintenance
burden on the (non-Docker-using) maintainers.
Solution: Drop the Dockerfile from the repo.
This commit adds an `eof()` function for grammars, which is easier to
use than the NUL byte directly. It compiles down to a constraint that
the enclosing production can only reduce at end of input, not to a
shiftable token.
BREAKING CHANGE: Public error types for `tree-sitter-generate` were modified.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
* build(deps): upgrade wasmtime C API to 48.0.0
Upgrade the Rust and Zig Wasmtime dependencies and enable reference values with the null GC collector.
Wasmtime 48 requires a newer Rust toolchain, whose Clippy version identifies three item helpers that can be const. Mark them const so the workspace continues to pass Clippy with warnings denied.
* feat(benchmark): support Wasm grammars
* perf(wasm): cache language function handles
* fix(wasm): improve validation and failure cleanup
Bounds-check dylink metadata parsing, require exact import and export names, and restore memory and function-table allocation offsets when language loading fails.
* fix(wasm): copy the complete supertype map
