Add documentation about version compatibility requirements between
web-tree-sitter parser ABIs for WASM file generation.
This helps users understand why Language.load() might fail when using
pre-built WASM files from third-party packages that were built with
incompatible tree-sitter-cli versions.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Browser bundlers (esbuild, webpack, vite, rollup) statically discover the
`await import("fs/promises")` and `await import("module")` calls in the
shipped ESM bundle and try to resolve them, even though both sit in
Node-only branches that never execute in a browser.
Adding a top-level `browser` field tells bundlers to substitute empty
stubs for these specifiers when targeting the browser. The runtime gates
(`globalThis.process?.versions.node` and `ENVIRONMENT_IS_NODE`) already
prevent the stubs from ever being invoked.
Fixes#5545.
Previously with the empty string base URL, the playground would not work when hosted at a subfolder of the domain. This is because playground.js has this code:
const url = `${LANGUAGE_BASE_URL}/tree-sitter-${newLanguageName}.wasm`;
With an empty string, it would look for wasm at the root of the website and 404.
Github code search shows that this is an issue which lots of people have run into, and everyone has to work around it: https://github.com/search?q=%2FLANGUAGE_BASE_URL+%3D+%22%22%2F+-language%3AHTML&type=code
Related to https://www.github.com/tree-sitter/tree-sitter/issues/5230
Require `word`, `conflicts`, `inline`, and `supertypes` callbacks
to return named grammar symbols instead of silently accepting values
that produce undefined names.
Restrict `precedences` lists to named rules and precedence names, and
preserve the existing handling of undefined and duplicate `inline`
rules.
Model grammar() as returning the evaluated grammar beneath a `grammar`
property, with callbacks replaced by their normalized values and
runtime-initialized fields represented as required properties.
Correct the evaluated rule types for named precedences and regular
expression flags. Narrow symbol-only callbacks such as `conflicts`,
`inline`, `supertypes`, and `word`, and expose the inherited values
passed to `extras` and `reserved` callbacks.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Some node-types bookkeeping still used the type name without its named
status. This caused anonymous nodes to inherit extra metadata from named
nodes with the same text, and could create false supertype cycles when a
named supertype and anonymous alias shared a name.
Key extra metadata and supertype dependencies by the complete
(type, named) identity.
Reject named aliases that collide with a canonical supertype. Such an
identity cannot be represented as both a concrete node and an abstract
supertype in node-types.json, and previously caused generation to panic.
All output entries now come from the map keyed by (type, named), so no
two entries can reach the final sort with the same identity. Remove the
unreachable root and extra tie-breakers and the now-redundant deduplication.
BREAKING CHANGE: Alters a public error enum for the generate crate.
When an anonymous token alias and an anonymous syntax-node alias have
the same name, node-types.json emits two entries with the same type and
named status. For example,
```
alias($._node, "same")
```
and
```
alias("!", "same")
```
produce separate anonymous "same" entries. Add anonymous tokens to
the node-type map instead. When a token shares an identity with a
structured node, retain its possible children and fields but mark
them as optional because the token appearance is a leaf.
A node-types entry is identified by both its type name and its `named`
value. The generator only keyed accumulated entries by name, so aliases
like
alias($._node, $.same)
alias($._node, "same")
were incorrectly merged into one entry. Whichever alias was processed
first determined the entry's `named` value, and the structures of two
distinct node types could be combined.
Key accumulated entries by `NodeTypeRef` so named and anonymous nodes
with the same type name remain separate.
Aliasing a supertype makes the aliased occurrence appear as a regular
node in the syntax tree. For example,
```
alias($.expression, $.expression_target)
```
can produce:
```
(expression_target (identifier))
```
Previously, node-types generation skipped the source supertype entirely.
It could reference `expression_target` as a field or child type without
emitting a corresponding top-level entry for it.
Continue emitting the canonical `expression` entry with its `subtypes`,
but pass aliased appearances through regular node generation so their
children and fields are recorded and merged normally.
Problem: The committed Dockerfile is not used by the standard tooling in
this repo and therefore bitrots freely, which increases maintenance
burden on the (non-Docker-using) maintainers.
Solution: Drop the Dockerfile from the repo.
This commit adds an `eof()` function for grammars, which is easier to
use than the NUL byte directly. It compiles down to a constraint that
the enclosing production can only reduce at end of input, not to a
shiftable token.
BREAKING CHANGE: Public error types for `tree-sitter-generate` were modified.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
* build(deps): upgrade wasmtime C API to 48.0.0
Upgrade the Rust and Zig Wasmtime dependencies and enable reference values with the null GC collector.
Wasmtime 48 requires a newer Rust toolchain, whose Clippy version identifies three item helpers that can be const. Mark them const so the workspace continues to pass Clippy with warnings denied.
* feat(benchmark): support Wasm grammars
* perf(wasm): cache language function handles
* fix(wasm): improve validation and failure cleanup
Bounds-check dylink metadata parsing, require exact import and export names, and restore memory and function-table allocation offsets when language loading fails.
* fix(wasm): copy the complete supertype map
The JSON output types now hold ids from the string pool instead of
cloning into owned `String`s, which makes `NodeTypeJSON` `Copy` and
drops most allocations when building the output.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Currently, a symbol listed in both `supertypes` and `inline` produces
a phantom supertype entry in `node-types.json` for a rule that can
never appear in a tree. `intern_symbols` now drops the supertype with
a warning. A hard error would break 29 published grammars, including
python, go, ruby, rust, and scala, so that that decision is deferred
to the next breaking release. Fixes#5218
* feat(wasm): make syntax trees sendable
* test(wasm): transfer trees across workers
* test(wasm): use JSON grammar for tree transfer
* test(wasm): edit trees across workers
* test(wasm): share dlmalloc with tree-sitter
* test(wasm): simplify worker tree exchange
* test(wasm): drive tree exchange from Rust
* test(wasm): split sendable-tree xtask
* test(wasm): generalize Rust web fixture
* test(wasm): exercise parallel Rust tree access
* fix(wasm): use Rust global allocator for C core
* test(wasm): use default Rust allocator
* feat(wasm): support external scanners in Rust web apps
* Simplify example further, add a readme
* Regenerate wasm-stdlib
* Fix wasm_stdlib check script
* Vendor the Wasm standard library subset
* Test Unicode Ruby scanner behavior in Wasm
* Make Wasm tree languages instance-aware
* Test multi-threaded use of queries in wasm32-unknown
* Refactor reference-counted language storage
* Check ABI version compat before loading rest of language
* 🎨 Remove redundant #ifdef block
* Reject unsupported Rust Wasm builds on 0.26
Problem: `Package.swift` build script is unmaintained and leads to build
errors.
Solution: Remove `Package.swift`; downstream tools should rely on the
swift-tree-sitter bindings (or, if they want a custom bare-metal build,
handle this in their own build scripts).
`test::run_wasm` gated dependency installation on `node_modules/chai` and
`node_modules/mocha`. Neither is a dependency (vitest is used instead),
so the check never passed and every `cargo xtask test-wasm`
reran `npm install`.
`Node::kind`, `Node::grammar_name`, the field name accessors,
`TreeCursor::field_name`, and the `Language` name lookups returned `&'static
str` while pointing into storage owned by the `TSLanguage`. Releasing the last
handle to a Wasm language frees that storage.
`Node` and `TreeCursor` now return `&'tree str`, which is ok because
`ts_tree_new` takes a reference to the language via `ts_language_copy` and
holds it until `ts_tree_delete`. The `Language` lookups return strings borrowed
from `&self`.
`currentType` and `currentTypeId` return `null` before the first iteration
step, after exhaustion, and after a reset.
`currentType` also falls back to the language's own name table instead of the
literal 'ERROR' when `Language.types` has no entry (auxiliary symbols). Other
bindings report `end` where this one reported 'ERROR'.
`current_symbol` and `current_symbol_name` return an `Option` (`None` when the
iterator is not positioned on a symbol). Change `iter_names`'s item from
`&'static str` to `&str`.
Track the iterator's phase so exhaustion is sticky, and gate the symbol name on
it, so `NULL` means "not positioned on a symbol".
`ts_lookahead_iterator_new` was also the only language consuming constructor
that did not `ts_language_copy`, so an iterator outliving a wasm language read
freed memory. Retain in `_new` and `_reset`, release in `_delete`.
Previously:
- `ts_lookahead_iterator__next` left a small parse state's cursor one past its
group end, so re-advancing an exhausted iterator resumed returning `true` and
walked through the rest of `ts_small_parse_table` and off the end of it.
- `ts_lookahead_iterator_current_symbol_name` returned `NULL` only when the
exhausted symbol index happened to fall outside the names table, so a grammar
with aliases returned a real but wrong name instead.
`(?-u:...)` switches `regex_syntax` to matching raw bytes, but the lexer
dispatches on decoded characters, so `expand_regex` converted the byte
class with a u8 cast. This is exact for ASCII bytes, and silently
misleading for anything aboove 0x80.
This is not reachable from grammar.js (node and QuickJS both reject
`(?-u:...)`). This change is to guard against future JS runtime changes,
as well as alternative frontends to the generate crate.
A previous fix moved case folding for `/i` patterns out of `regex_syntax` and
into `expand_regex`, so folding could drop the two non-ASCII code points
Unicode simple folding maps onto ASCII letters: the long s `ſ` (U+017F)
onto `s`, and the Kelvin sign `K` (U+212A) onto `k`. Left in, they leak
into otherwise-ASCII tokens and stop those tokens from being extracted as
keywords.
By that point, though, the HIR has already turned a negated class into a
complement, so folding it applies the fold on the wrong side of the
negation. `(?i)[^a-z]` folds a set that contains `A-Z`, which re-admits
`a-z` and leaves a class matching very nearly everything.
`regex_syntax` folds each leaf of a class expression before applying that
leaf's negation and the set algebra above it. Keep that order and change
only the fold: walk the AST, replace each leaf with its fold, and translate
with `case_insensitive(false)`.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
This is already exposed for consumers via the CLI, and is a natural way
to express some test expectations over the sexp form.
Also clean up some repeated logic in the internal test code, and narrow
the cst rendering return type to `std::io::Result` rather than
`anyhow::Result`.
