Also use `IoError` more consistently throughout the rest of the project
where its straightforward to do so.
BREAKING CHANGE: Changes public error types for config, generate, and
loader crates.
Signed-off-by: cuishuang <imcusg@gmail.com>
BREAKING CHANGE: changes the signature of `assert_expected_captures`, which is exposed publicly when tree-sitter-cli is consumed as a library.
Problem: Parsing UTF-16BE source containing supplementary-plane characters, such as `let emoji = "😀"`, decoded the emoji as two isolated surrogates on little-endian hosts, causing incorrect lexer lookahead and potentially shifted token boundaries.
Soluton: Fix the trailing surrogate byte-order conversion in both UTF-16LE and UTF-16BE decoders and adds a regression test for U+1F600.
In a future feature, `RulePool` nodes can be shared between lexical and
syntax roots. No grammars generated by grammar.js/grammra.json are affected
by this issue, as they lead to a tree pool rather than a DAG.
With a DAG `RulePool` representation, however, rewriting tokens during
extraction can destroy a token body before lexical expansion consumes
it and can renumber shared syntax nodes more than once.
Record terminal rewrites while inspecting the original pool, expand tokens
and separators first, then commit the rewrites and renumber each reachable
syntax node once. Resolve grammar metadata against the pending rewrites
before mutating the pool.
`try_seq` reserves a `Seq`'s child range up front and fills slots as
recursion produces each member. The `Repeat` arm builds
`Choice(repeat(x), blank)` directly, skipping the `choice` helper.
Slightly faster and less memory used.
`node_kind_is_named`, `node_kind_is_visible` and `node_kind_is_supertype`
forwarded their `u16` straight to `ts_language_symbol_type`, which indexes
`symbol_metadata` unchecked. Bound the id by `node_kind_count()`.
`ts_language_next_state` indexed the parse table with the caller's `state`
and `symbol` without validating either, and `ts_language_subtypes` read
`symbol_metadata` before checking `supertype` was in range.
The field handed out a slice whose lifetime outlived the streaming-iterator
loan it came from. An accessor reborrowing through `&self` bounds it correctly.
BREAKING CHANGE: In the rust bindings, replace `m.captures` with `m.captures()`.
Problem: Editor-specific settings should not be part of the global
repository. (Only config files that control standard tooling in order to
enforce, e.g., formatting policies are appropriate to commit -- and only
if covered by CI.)
Solution: Drop the Zed config files from the repo.
Add documentation about version compatibility requirements between
web-tree-sitter parser ABIs for WASM file generation.
This helps users understand why Language.load() might fail when using
pre-built WASM files from third-party packages that were built with
incompatible tree-sitter-cli versions.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Browser bundlers (esbuild, webpack, vite, rollup) statically discover the
`await import("fs/promises")` and `await import("module")` calls in the
shipped ESM bundle and try to resolve them, even though both sit in
Node-only branches that never execute in a browser.
Adding a top-level `browser` field tells bundlers to substitute empty
stubs for these specifiers when targeting the browser. The runtime gates
(`globalThis.process?.versions.node` and `ENVIRONMENT_IS_NODE`) already
prevent the stubs from ever being invoked.
Fixes#5545.
Previously with the empty string base URL, the playground would not work when hosted at a subfolder of the domain. This is because playground.js has this code:
const url = `${LANGUAGE_BASE_URL}/tree-sitter-${newLanguageName}.wasm`;
With an empty string, it would look for wasm at the root of the website and 404.
Github code search shows that this is an issue which lots of people have run into, and everyone has to work around it: https://github.com/search?q=%2FLANGUAGE_BASE_URL+%3D+%22%22%2F+-language%3AHTML&type=code
Related to https://www.github.com/tree-sitter/tree-sitter/issues/5230
Require `word`, `conflicts`, `inline`, and `supertypes` callbacks
to return named grammar symbols instead of silently accepting values
that produce undefined names.
Restrict `precedences` lists to named rules and precedence names, and
preserve the existing handling of undefined and duplicate `inline`
rules.
Model grammar() as returning the evaluated grammar beneath a `grammar`
property, with callbacks replaced by their normalized values and
runtime-initialized fields represented as required properties.
Correct the evaluated rule types for named precedences and regular
expression flags. Narrow symbol-only callbacks such as `conflicts`,
`inline`, `supertypes`, and `word`, and expose the inherited values
passed to `extras` and `reserved` callbacks.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Some node-types bookkeeping still used the type name without its named
status. This caused anonymous nodes to inherit extra metadata from named
nodes with the same text, and could create false supertype cycles when a
named supertype and anonymous alias shared a name.
Key extra metadata and supertype dependencies by the complete
(type, named) identity.
Reject named aliases that collide with a canonical supertype. Such an
identity cannot be represented as both a concrete node and an abstract
supertype in node-types.json, and previously caused generation to panic.
All output entries now come from the map keyed by (type, named), so no
two entries can reach the final sort with the same identity. Remove the
unreachable root and extra tie-breakers and the now-redundant deduplication.
BREAKING CHANGE: Alters a public error enum for the generate crate.
When an anonymous token alias and an anonymous syntax-node alias have
the same name, node-types.json emits two entries with the same type and
named status. For example,
```
alias($._node, "same")
```
and
```
alias("!", "same")
```
produce separate anonymous "same" entries. Add anonymous tokens to
the node-type map instead. When a token shares an identity with a
structured node, retain its possible children and fields but mark
them as optional because the token appearance is a leaf.
A node-types entry is identified by both its type name and its `named`
value. The generator only keyed accumulated entries by name, so aliases
like
alias($._node, $.same)
alias($._node, "same")
were incorrectly merged into one entry. Whichever alias was processed
first determined the entry's `named` value, and the structures of two
distinct node types could be combined.
Key accumulated entries by `NodeTypeRef` so named and anonymous nodes
with the same type name remain separate.
Aliasing a supertype makes the aliased occurrence appear as a regular
node in the syntax tree. For example,
```
alias($.expression, $.expression_target)
```
can produce:
```
(expression_target (identifier))
```
Previously, node-types generation skipped the source supertype entirely.
It could reference `expression_target` as a field or child type without
emitting a corresponding top-level entry for it.
Continue emitting the canonical `expression` entry with its `subtypes`,
but pass aliased appearances through regular node generation so their
children and fields are recorded and merged normally.
Problem: The committed Dockerfile is not used by the standard tooling in
this repo and therefore bitrots freely, which increases maintenance
burden on the (non-Docker-using) maintainers.
Solution: Drop the Dockerfile from the repo.
This commit adds an `eof()` function for grammars, which is easier to
use than the NUL byte directly. It compiles down to a constraint that
the enclosing production can only reduce at end of input, not to a
shiftable token.
BREAKING CHANGE: Public error types for `tree-sitter-generate` were modified.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
* build(deps): upgrade wasmtime C API to 48.0.0
Upgrade the Rust and Zig Wasmtime dependencies and enable reference values with the null GC collector.
Wasmtime 48 requires a newer Rust toolchain, whose Clippy version identifies three item helpers that can be const. Mark them const so the workspace continues to pass Clippy with warnings denied.
* feat(benchmark): support Wasm grammars
* perf(wasm): cache language function handles
* fix(wasm): improve validation and failure cleanup
Bounds-check dylink metadata parsing, require exact import and export names, and restore memory and function-table allocation offsets when language loading fails.
* fix(wasm): copy the complete supertype map
The JSON output types now hold ids from the string pool instead of
cloning into owned `String`s, which makes `NodeTypeJSON` `Copy` and
drops most allocations when building the output.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Currently, a symbol listed in both `supertypes` and `inline` produces
a phantom supertype entry in `node-types.json` for a rule that can
never appear in a tree. `intern_symbols` now drops the supertype with
a warning. A hard error would break 29 published grammars, including
python, go, ruby, rust, and scala, so that that decision is deferred
to the next breaking release. Fixes#5218
* feat(wasm): make syntax trees sendable
* test(wasm): transfer trees across workers
* test(wasm): use JSON grammar for tree transfer
* test(wasm): edit trees across workers
* test(wasm): share dlmalloc with tree-sitter
* test(wasm): simplify worker tree exchange
* test(wasm): drive tree exchange from Rust
* test(wasm): split sendable-tree xtask
* test(wasm): generalize Rust web fixture
* test(wasm): exercise parallel Rust tree access
* fix(wasm): use Rust global allocator for C core
* test(wasm): use default Rust allocator
* feat(wasm): support external scanners in Rust web apps
* Simplify example further, add a readme
* Regenerate wasm-stdlib
* Fix wasm_stdlib check script
* Vendor the Wasm standard library subset
* Test Unicode Ruby scanner behavior in Wasm
* Make Wasm tree languages instance-aware
* Test multi-threaded use of queries in wasm32-unknown
* Refactor reference-counted language storage
* Check ABI version compat before loading rest of language
* 🎨 Remove redundant #ifdef block
* Reject unsupported Rust Wasm builds on 0.26
Problem: `Package.swift` build script is unmaintained and leads to build
errors.
Solution: Remove `Package.swift`; downstream tools should rely on the
swift-tree-sitter bindings (or, if they want a custom bare-metal build,
handle this in their own build scripts).
`test::run_wasm` gated dependency installation on `node_modules/chai` and
`node_modules/mocha`. Neither is a dependency (vitest is used instead),
so the check never passed and every `cargo xtask test-wasm`
reran `npm install`.
`Node::kind`, `Node::grammar_name`, the field name accessors,
`TreeCursor::field_name`, and the `Language` name lookups returned `&'static
str` while pointing into storage owned by the `TSLanguage`. Releasing the last
handle to a Wasm language frees that storage.
`Node` and `TreeCursor` now return `&'tree str`, which is ok because
`ts_tree_new` takes a reference to the language via `ts_language_copy` and
holds it until `ts_tree_delete`. The `Language` lookups return strings borrowed
from `&self`.
`currentType` and `currentTypeId` return `null` before the first iteration
step, after exhaustion, and after a reset.
`currentType` also falls back to the language's own name table instead of the
literal 'ERROR' when `Language.types` has no entry (auxiliary symbols). Other
bindings report `end` where this one reported 'ERROR'.
`current_symbol` and `current_symbol_name` return an `Option` (`None` when the
iterator is not positioned on a symbol). Change `iter_names`'s item from
`&'static str` to `&str`.
Track the iterator's phase so exhaustion is sticky, and gate the symbol name on
it, so `NULL` means "not positioned on a symbol".
`ts_lookahead_iterator_new` was also the only language consuming constructor
that did not `ts_language_copy`, so an iterator outliving a wasm language read
freed memory. Retain in `_new` and `_reset`, release in `_delete`.
Previously:
- `ts_lookahead_iterator__next` left a small parse state's cursor one past its
group end, so re-advancing an exhausted iterator resumed returning `true` and
walked through the rest of `ts_small_parse_table` and off the end of it.
- `ts_lookahead_iterator_current_symbol_name` returned `NULL` only when the
exhausted symbol index happened to fall outside the names table, so a grammar
with aliases returned a real but wrong name instead.
`(?-u:...)` switches `regex_syntax` to matching raw bytes, but the lexer
dispatches on decoded characters, so `expand_regex` converted the byte
class with a u8 cast. This is exact for ASCII bytes, and silently
misleading for anything aboove 0x80.
This is not reachable from grammar.js (node and QuickJS both reject
`(?-u:...)`). This change is to guard against future JS runtime changes,
as well as alternative frontends to the generate crate.
A previous fix moved case folding for `/i` patterns out of `regex_syntax` and
into `expand_regex`, so folding could drop the two non-ASCII code points
Unicode simple folding maps onto ASCII letters: the long s `ſ` (U+017F)
onto `s`, and the Kelvin sign `K` (U+212A) onto `k`. Left in, they leak
into otherwise-ASCII tokens and stop those tokens from being extracted as
keywords.
By that point, though, the HIR has already turned a negated class into a
complement, so folding it applies the fold on the wrong side of the
negation. `(?i)[^a-z]` folds a set that contains `A-Z`, which re-admits
`a-z` and leaves a class matching very nearly everything.
`regex_syntax` folds each leaf of a class expression before applying that
leaf's negation and the set algebra above it. Keep that order and change
only the fold: walk the AST, replace each leaf with its fold, and translate
with `case_insensitive(false)`.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
This is already exposed for consumers via the CLI, and is a natural way
to express some test expectations over the sexp form.
Also clean up some repeated logic in the internal test code, and narrow
the cst rendering return type to `std::io::Result` rather than
`anyhow::Result`.
Every (state, terminal) parse-table entry stored its action list inline as a
32-byte `ParseTableEntry`, but across a grammar those lists are ~98-99% _duplicates_.
The number of distinct lists is a few thousand regardless of grammar size, while
total entries scale into the hundreds of thousands.
Store each unique action list once in a shared `ActionListPool` (a flat arena of
actions plus `(offset, len)` ranges) and replace the inline entry with a 4-byte
`ActionListId` (a pool index with the `reusable` flag packed into the high bit).
- intern_table converts the freshly built `ParseTable<ParseTableEntry>` into
`ParseTable<ActionListId>`.
- minimize carries and operates on the 4-byte ids. The three global state
renumberings rewrite Shift targets once at the pool level
(`remap_terminal_references`), while the per-state unit-reduction redirects
copy the changed list into a new slot (COW). `mark_fragile_tokens` becomes a
free bit flip on the id.
- `canonicalize` dedups and compacts the pool once before `render`, dropping the
dead and duplicate slots the remaps and COW leave behind. `render` assigns the
output action-list offsets directly.
Yields ~7% wall time reduction, ~12% peak rss reduction.
`ParseStateId`, `LexStateId`, `ProductionInfoId`, and `ReservedWordSetId`
each held a `usize`, while the runtime stores state/field ids as `u16` (so
`u32` has plenty of headroom). Shrinking them saves space for every parse-table
entry, and a number of other data structures.
Reduces wall time by 2-5%, peak rss by 9-12%.
`get_auxiliary_node_info` scans every entry in the item set to collect the
non-auxiliary parents of a given auxiliary symbol. It was called once per
entry whose next symbol is auxiliary, and the same auxiliary symbol recurs
across many entries in a single state (a state's GOTO on a repeat symbol is
shared by every item advancing over it), so the same full scan was repeated
many times per state. Memoize the result per symbol within a single
`add_actions` call.
Reduces wall time 1-4%, rss flat.
`ParseTableEntry` held a `Vec<ParseAction>` per entry. `ActionList` keeps up
to one action inline and spills to a Vec only for conflict entries.
Reduces walltime by >20%, peak rss by ~30%.
The production info depends only on the production, but was recomputed and
deduplicated via a linear deep-equality scan over production_infos for every
reduce item in every state. Cache `prod_id` -> `ProductionInfoId`.
Wall time reduction 0-3%, peak rss neutral.
`ParseItemSetEntry` stores a `LookaheadSetId` into a `LookaheadSetPool` instead
of an owned `TokenSet`. Ids are canonical, so entry hash/eq/clone are integer
ops and state dedup stops walking set words. Unions and single-token inserts
are memoized by id, so the transitive closure and successor-kernel construction
stop re-materializing the same unions per state. Closure additions carry interned
ids with word-token membership precomputed.
For cpp, 3566 distinct lookahead sets back all 49,992 states. For ruby 2000 sets
for 91,991 states, and rust 646 for 16,796.
Reduces wall time by 10-15%, peak rss by 3-5%.
The state-merge pass's `token_conflicts` scanned every terminal entry of the
other state per non-shared token. Precompute flat, word-aligned bitsets,
`ConflictBits`, so each call is a handful of word ANDs.
Yields a 5-6% wall time reduction on large grammars, and flat to ~1%
loss on small grammers. Peak RSS unchanged.
Every state's kernel set was cloned into `parse_state_info` even though the
identical set already lives as the state-dedup map key at the same index
(state ids are the map's insertion order). `ParseStateInfo` now holds the
preceding-symbol sequences plus the dedup map moved out of the builder,
allowing the report path to read kernels via `get_index`. Drops the
duplicate kernel storage, allowing for a small reduction in wall time
and peak rss (1-3%).
`CoincidentTokenIndex` kept a `Vec<ParseStateId>` per token pair whose
only user was keyword identification asking one question: "do all states
where a given pair coincides also allow the word token?" Track that
directly as a second bitset built in the same pass.
Reduces walltime by ~7% for small grammars (i.e. Go), and 20-25% for
larger grammars (i.e. Rust, cpp). Reduces peak rss by ~20% for smaller
grammars and ~40-50% for larger grammars.
Replace the owned `Rule`-tree grammar with a flat, pooled representation. `Rule`
nodes live in a `RulePool` arena and reference their children and params by
index (`RuleId`). Strings are interned in a `StrPool` and referred to by `StrId`.
Productions are stored as flat `ProductionStep`/`Production` slices indexed per
variable, rather than as nested vectors hanging off each `SyntaxVariable`. This
drops the per-rule heap allocation of the owned-tree form and is the groundwork
for the later table-building optimizations.
`parse_grammar` builds the pool, the prepare passes rewrite nodes in place, and
`prepare_grammar` returns a `PreparedGrammar` bundle that owns the run's `StrPool`
alongside the grammars. `build_tables`, `node_types`, and `render` borrow that
pool and resolve `StrId`s only at the output boundary.
Shows an average ~10% walltime reduction when combined with the previous
commit, with nearly all of the savings in `build_tables`. The `prepare`
stage is also ~70% faster and much more stable, but this contributes
much less to overall generate time. Improving early stages such as
`prepare` will be important for future interactive uses.
Parse-item identity dominates parse-table construction. `ParseItem`'s
`Hash`, `Eq`, and `Ord` walk the entire production on every call,
comparing per-step precedence, aliases, and field names. These are
stored as strings for named precedences and for every alias and field.
Precompute, once per table build, two dense `u32` ids for every
`(production, dot)` slot by ranking the whole slot universe with the
item-content comparator, now extracted as `ItemContent`. Equal-content
slots are adjacent after sorting and the id increments only at content
boundaries, so comparing the `cmp` id reproduces the structural order
exactly (both an ordering and an equality). A second id, `eq_with_syms`,
refines `cmp` by the preceding symbols, which enter equality only for
items with `has_preceding_inherited_fields`.
`ParseItem` now carries its production's key slice, and `Hash`/`Eq`/`Ord`
become integer ops.
On the current string-based production storage this is performance-neutral
in isolation (the up-front ranking pays back the per-comparison savings).
It is the foundation for the following change to a pooled, integer-keyed
production representation, which makes both the ranking and the item
comparisons cheaper integer work.
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
`QueryMatches` and `QueryCaptures` retain query cursor options while they are
being iterated. However, their types did not preserve the lifetime of the
options' progress callback, allowing the callback to be dropped while it
was still referenced by the cursor.
Add an explicit options lifetime to both iterator types and propagate it
through `matches_with_options` and `captures_with_options`. Use a static
options lifetime for iterators created without options.
A SHIFT/REDUCE conflict can bundle several shift interpretations with
different precedences against a single reduce. `handle_conflict` weighed them
with only `shift_is_less` and `shift_is_more`, so a lone lower-precedence
shift set `shift_is_less` and the REDUCE won outright, even when another
interpretation tied the REDUCE in precedence and the REDUCE was declared
right-associative. That tie's associativity should have won by shifting, so a
right-associative rule silently became left-associative as soon as an
unrelated lower-precedence rule was added to the grammar.
Track the equal-precedence case explicitly, and in the reduce-wins branch
shift instead when a tying interpretation exists and the reduce actions are
purely right-associative.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
The early `ts_parser__recover` dispatch for `ERROR_STATE` in
`ts_parser__advance` was dropped in `201b41cf1`. Without it, tokens
with no valid action re-enter `ts_parser__handle_error` per token
and fragment the tree instead of extending one ERROR node. Restore
it to recover the old (pre-0.25) error recovery behavior.
- `--layout <document|line-numbers|fragment>` (default `document`): the
document structure. `document` is a self-contained page wrapping a plain
`<div class="highlight"><pre><code>` block, `line-numbers` adds a line-number
`<table>`, and `fragment` emits only the code markup with no surrounding
`<head>`/`<style>`/`<body>` so it can be embedded in an existing page.
- `--style <classes|inline|minimal>` (default `classes`): how token colors
are applied. `classes` uses `class="..."` spans plus a generated `<style>`,
`inline` bakes colors onto each span so the markup is self-contained, and
`minimal` emits bare classes with no colors (bring your own stylesheet).
Deprecates the `--css-classes` flag.
`Parser::set_logger` boxes the logger callback into a type-erased C
pointer, but `Parser` carries no lifetime parameter. This allows for
two kinds of UB:
- Use-after-free: a logger could capture a non-`'static` borrow, let the
borrowed value drop, and then dangle the next time the parser logged.
- Data race: a logger could capture a `!Send` value such as an `Rc`. The
parser could then be moved to another thread and logged from there
while the original thread still held a clone, racing on the reference
count.
As a fix, we just require that logger is `Send + 'static`. The
alternative here is to attach a lifetime parameter to `Parser`, but this
is highly breaking for what is mostly a debugging utility. As an escape
hatch `set_logger_unchecked` is added to the public API as an `unsafe fn`
to correctly communicate the risks and invariants that must be held.
`(P . Q* Y)` with zero `Q` dropped the leading `.`, so `Y` matched at any
position instead of being pinned to the parent's first named child. On a `?`/`*`
zero-skip, when the skipped step is the parent's first child and carries a
leading anchor, transfer that first-child requirement to the skip target.
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
Unicode simple case folding maps two non-ASCII code points onto ASCII
letters: the long s `ſ` (U+017F) onto `s`, and the Kelvin sign `K` (U+212A)
onto `k`. So `regex_syntax` pulls them into any case-insensitive pattern,
which is virtually never intended and has two bad effects:
* such tokens can no longer be extracted as keywords, because they are not
a subset of an ASCII `word` token (#5607)
* a broad class like `[^"]` or `\p{L}` carrying `/i` loses `ſ`/`K`, even
though it legitimately contains them (#5755)
Rather than let `regex_syntax` fold, parse patterns unfolded and fold them
ourselves in `case_fold_ascii_safe`: fold via `regex_syntax`, then drop
`ſ`/`K` only when folding introduced them (they were not already in the
base set). A class that already contains them keeps them.
Regexes with the case-insensitive 'i' flag caused unicode simple case folding,
which maps two non-ASCII code points onto ASCII letters:
- `ſ` (U+017F) onto `s`
- the Kelvin sign `K` (U+212A) onto `k`
This pushed such tokens outside an ASCII `word` token, so they failed to
extract as keywords.
Problem: A set of `array_*` macros (`array_push`, `array_extend`, etc) implicitly convert a `void*` into a different pointer type. In environments that compile these headers as C++, this implicit conversion is an error.
Solution: This commit adds an `_array_cast` macro that uses `decltype` to cast the `void*` to the proper type when compiling as C++.
- An anchor between two patterns is vacuous when an adjacent quantifier
matches zero
- A leading or trailing anchor on a node applies to the nearest matched node
when an adjacent optional is skipped
- An anchor at the edge of a group or alternation is not allowed.
A trailing `.` after an optional node, e.g. `(p (a)+ @a . (b)? @b .)`, sets
`is_last_child` on the `(b)?` step. When `(b)?` matched zero, the zero-skip
jumped past that step to completion, so the last-child requirement was never
enforced.
When a zero-skip would bypass a step carrying `is_last_child`, require that
the last matched node really is the last named child. Gated on the
`alternative_is_skip` edge marker.
A `.` anchor between a quantified pattern and a following node, e.g.
`(parent (comment)* @c . (decl))`, was treated as a leading anchor when the
quantifier matched zero. The zero-skip jumps a state directly onto the anchored
step, where `is_immediate` was enforced even though no sibling had matched before
it. Name that skip edge (`alternative_is_skip`) and, when a state follows it,
record that it skipped the quantifier (`skipped_quantifier`). The
immediate anchor is then supressoed for that state's next match.
An unanchored quantified sibling like `(program (comment)+ @doc (class))`
keeps one match state per matching sibling. A recent fix stopped over-pruning
them, but the per-node longest-match dedup is pairwise, so with n live states
matching became O(n^3).
Two states can only be capture subsets of one another if their captured
byte ranges overlap, so keep each group ordered by first-capture position
and stop the pairwise scan once the rest of the group is disjoint.
Refs neovim/neovim#40517
Problem: `zig fetch` on 0.16 can't download zip archives, which wasmtime
uses for Windows. This makes `cargo xtask upgrade-wastime` fail silently
with empty hashes for these platforms.
Solution: Re-run xtask with Zig 0.17 nightly.
object rather than 4 separate optional function pointers.
Helps prevent misuse via mixing different allocators. Also update the
doc comment with relevant safety information.
This allows duplicating a query so that it can be modified by using
disable_capture or disable_pattern, without re-parsing the query. The
new `ts_query_copy` creates a deep copy of the entire query object.
A motivation for this is tags.scm code navigation queries. You might
want to have one version of the compiled query capture only definitions
and the other only capture references, since references are much much
more common than definitions in a typical codebase. Instead of
re-parsing and analyzing the query and then calling
`ts_query_disable_capture`, we can clone the built query all-at-once and
then disable captures / patterns.
Problem: In queries matching a parent node with anchored children
(siblings) where the first sibling has a quantifier and is not
captured, the check for fallible steps skips splitting the state because
the next node is a passthrough node (and not an is_immediate one). This
prevents the query from matching beyond the first occurrence.
Solution: In the check for fallible steps, skip the next steps if they
are passthrough steps.
Problem: In queries matching a parent node with anchored children
(siblings) where the first anchored sibling has a quantifier, the
deduplication logic for states is overly aggressive. The logic causes
the state split on the first capture (with the parent) to be dropped in
favor of the loopback state. This results in the query not being able to
match beyond the first occurrence.
Solution: Prevent the deduplication logic from dropping a state that is
not seeking an immediate match for one that is seeking an immediate
match, since the one not seeking for an immediate match could still
match later nodes.
Problem: In queries matching a parent node with anchored children
(siblings), the check for fallible steps only considers nodes with
children, which results in no state split being made for anchored
siblings (node1) . (node2). This prevents the query from matching beyond
the first occurrence.
Solution: Make the check for fallible steps consider also the case where
the next step is at the same depth and must be matched immediately
after.
Problem: actions/checkout@v7 refuses to checkout fork pull request code
on `pull_request_target` trigger, breaking backport and reviewers-remove
workflows on external PRs.
Solution:
* backport: Checkout the merged-into base branch by name.
* reviewers-remove: inline script so no checkout needed.
* Update cli README.md
Added a link to CLI readme to make it clear to an outsider that
cargo-binstall is a different tool than cargo.
* Apply suggestion from @WillLillis
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
---------
Co-authored-by: Christian Clason <ch.clason+github@icloud.com>
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Clarifies the behavior of `TSLexer::advance` when `skip=true`.
The previous documentation didn't make it clear that `skip=true` should only be used before a token starts. Using it after `mark_end` can affect the token’s starting position and lead to incorrect or zero-length ranges.
Fixes#2315
The early exit condition to skip remaining highlights was incorrectly
checking the lexicographical order, which could lead to false
passes if a highlight was on a later row with smaller column number.
Additional clippy lints fire when certain modules are temporarily made
public, i.e. for testing purposes. Fixing these reduces clutter and
helps with internal development.
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
Compiling a grammar with `-fsanitize=address` (or any other sanitizer)
emits a module constructor in the parser object file that references
runtime symbols like `__asan_init`. These runtime-resolved symbols are
incompatible with `-Wl,--no-undefined`, which breaking sanitized test
runs on clang. (gcc happens to paper over this by auto-linking libasan
into the .so)
To fix, detect `-fsanitize=` in the compile flags and omit `--no-undefined`
in that case.
The alloc.h header checked for TREE_SITTER_HIDDEN_SYMBOLS, but the
canonical macro name used everywhere else (api.h, render.rs, setup.py)
is TREE_SITTER_HIDE_SYMBOLS. This mismatch meant that defining
TREE_SITTER_HIDE_SYMBOLS (as the Python binding build does) would not
actually hide the allocator symbols in alloc.h.
Fixestree-sitter/tree-sitter#5625
Signed-off-by: Georges Savoundararadj <savoundg@amazon.com>
The wasm store gated supertype_symbols / supertype_map_slices /
supertype_map_entries copies on abi_version > LANGUAGE_VERSION_WITH_RESERVED_WORDS,
but every other consumer (language.c, query.c) treats those tables as
present when abi_version >= LANGUAGE_VERSION_WITH_RESERVED_WORDS.
A Wasm grammar built at ABI exactly 15 with supertype_count > 0 ends up
with supertype_count copied into the native TSLanguage but supertype_map_slices
left NULL. ts_query__analyze_patterns then calls ts_language_subtypes,
which dereferences self->supertype_map_slices[supertype] and crashes.
Several test functions compile the same grammar fixture.
Tests sharing a grammar name also share a src_dir. Each test also
unconditionally writes the three tree-sitter headers into
src_dir/tree_sitter/, leading to a race.
Write the three headers exactly once per src_dir, controlled via a
global `HashSet`.
The previous implementation of `InputGrammar::normalize` (inlined in
`parse_grammar` iterated over every variable, checking whether it was
reachable _backwards_ from teh root by recursing over rules that
referenced it.
This means that each and every top level call re-traversed the entire
graph. The runtime performance of this backwards walk was dependent on
the _order_ of rules as declared in `grammar.js`. All existing grammars
have an ordering that's reasonably friendly to this iteration pattern
(BFS-ish order, top down from the start rule), but this leaves us open
to a catastrophic performance cliff.
Instead, seed a `used` set with the start rule, the word token, and an
names referenced from `extras`/`externals`. Then propagate via direct rule
references. This yields anywhere from a 2-~2200x speedup for
`parse_grammar`. This greatly speeds up `--no-parser` runs, but is
relatively unimportant for `parser.c` generation for _existing_
grammars. The important piece is eliminating the potential cliff.
debug output.
The initial implementation of `--debug pretty` assumed process version
was bounded by `MAX_VERSION_COUNT`. However, we also have to account for
`MAX_VERSION_COUNT_OVERFLOW` as well as `halted_version_count`.
`ts_query_cursor_next_capture` linearly scanned all finished states to
find the one with the earliest next capture byte offset. With deeply
nested code, this O(n) scan per capture caused the highlight crate to
hang for minutes on large files.
Replace the linear scan with a min-heap over the finished_states array,
keyed by (next_capture_byte_offset, pattern_index, id). The heap is
maintained lazily: ts_query_cursor__advance uses plain array_push
(preserving FIFO insertion order), and next_capture sifts new elements
into the heap on entry via a tracked heap_size boundary. This preserves
the documented "order found" guarantee for next_match while giving
next_capture O(log n) per call.
Commit 1f6eac55 ("query: Use uint32_t for capture list IDs") widened
QueryState.capture_list_id to uint32_t and removed the 65536 pool cap,
but left the pool function signatures as uint16_t. This caused silent
truncation when the pool exceeded 65535 entries, leading to a segfault.
Solaris does not provide <endian.h> or <sys/endian.h>, but it does expose
byte-order definitions and conversion helpers via <sys/isa_defs.h> and
<sys/byteorder.h>.
Add a __sun branch so the portable header defines __BYTE_ORDER and the
htobe*/le*toh conversions on Solaris.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
Problem: The nvim-treesitter tests use a prebuilt Neovim, which means
changes to the library are not tested and parser generation is locked at
the ABI Neovim supports.
Solution: Build Neovim against the PR version of the library on Ubuntu
and macOS. On Windows, we still download binaries (because one does not
simply build Neovim on Windows...) so we still can check backward
compatibility.
- replace `Vec<Vec<ParseStateId>>` with a flat bitset in
`CoincidentTokenIndex` for membership queries
The previous representation stored full lists of parse state IDs for
each token pair. Since only membership (yes/no) is needed for most
queries, a flat bitset (`Vec<u64>` indexed as `a * n + b`) is
better. Both `(a,b)` and `(b,a)` bits are set during construction
so `contains()` needs no min/max normalization. The original
`entries: Vec<Vec<ParseStateId>>` and `states_with()` method are
retained alongside the bitset, because `identify_keywords` needs
to iterate over the specific parse states where two tokens co-occur
rather than scanning all states. ~8% walltime reduction for
tree-sitter-bash.
- add word-aligned per-row bitsets to `CoincidentTokenIndex` for
vectorized intersection checks
Row `a` spans `[a * row_words .. (a+1) * row_words]` where
`row_words = n.div_ceil(64)`. This enables callers in
`build_lex_table` to perform word-level AND operations against
`TokenSet::terminal_bits_words()` instead of iterating individual
token indices. ~6% walltime reduction on tree-sitter-bash.
- add `TokenSet::terminal_bits_words()` accessor and `#[inline]` on
`BitVec::insert_all`
Expose the raw `&[u64]` backing the terminal bitset so callers can
perform word-level bitwise intersection. Mark `insert_all` as
`#[inline]` to allow the compiler to optimize the hot OR loop.
- precompute a flat NFA-state-to-variable-index lookup table
Build a `Vec<usize>` mapping each NFA state ID to its owning
variable index up front in `TokenConflictMap::new`, replacing
repeated binary searches in `compute_conflict_status` with
array lookups. Also simplifies `mark_fragile_tokens` to
collect terminal indices into a `Vec` rather than maintaining
a boolean mask.
- replace `FxHashSet<Vec<u32>>` visited set with `FxHashSet<u64>`
using a hash-of-states key
In `compute_conflict_status`, the visited-state-set previously
stored cloned `Vec<u32>` NFA state sets. Replace this with an
`FxHashSet<u64>` keyed by hashing the sorted state slice via
`FxHasher`. This eliminates the `Vec` clone on every BFS step.
- reduce allocations in `NfaCursor::group_transitions`
Reuse a single `CharacterSet` buffer across iterations via `assign`
and `mem::take` instead of cloning the input on every raw
transition. Also, replace `Vec::insert` with `Vec::push` when
splitting intersection transitions. The final sort makes
mid-loop ordering irrelevant, and the disjointness of the split
sets guarantees correctness.
- pre-allocate `NfaTransition` result vectors with a capacity of 8
and use `swap_remove` instead of `remove` when merging duplicate
transition entries
- lazily compute `within_separator` in `compute_conflict_status`
Most BFS states have no completions, so the separator check
(which iterates all cursor transitions) is never needed in those
iterations. Wrap it in `Option::get_or_insert_with` to defer
computation until first use.
- add `NfaCursor::transitions_and_any_sep` to fuse the transitions
and separator-check passes
Callers like `build_lex_table::add_state` previously called
`transitions()` and then `transition_chars().any(|sep|)` separately,
iterating the raw NFA transitions twice. The fused method computes
both in a single pass.
Replace `std::collections::{HashMap, HashSet}` with
`rustc_hash::{FxHashMap, FxHashSet}` throughout `generate`. `FxHash`
is _much_ faster than the default SipHash for small integer keys,
which are used extensively through `generate`. The one "risk" here is
that `FxHash` is more vulnerable to DDOS attacks, but that isn't as much
of a concern for us here.
- use bitflags for `TokenConflictStatus`
Replace the individual boolean fields on the conflict status type
with a `bitflags!` representation. This takes up significantly less
space and allows for faster combination/testing in `token_conflicts`.
- hoist loop bounds in the merge-join loops of `states_conflict`
Extract `Vec::len()` outside of `while` loops to avoid recomputing
bounds on every iteration. Remove the now-unused `row_offset` /
`does_conflict_at` indirection on `TokenConflictMap`, inlining the
conflict check directly.
- optimize `variable_index_for_nfa_state` in `grammars.rs`
Small optimization to the NFA state-to-variable index mapping.
- refactor symbol key into a proper newtype
Extract the raw `u64` symbol key bit manipulation into a `key_index`
helper function for some extra type safety.
- use a boolean `Vec` instead of `Vec::contains` in `dedup`
Replace linear containment checks (`split_state_ids.contains()`)
with a constant lookup indexed by state ID.
- narrow parse state IDs from `usize` to `u16`
Reduces memory footprint and improves packing in `ParseTable` and
`build_parse_table`. This is more cache-friendly and reduces peak rss by
~1-2%.
- precompute lookup tables for parse table minimization
Build several maps up front (symbol-to-action, nonterminal indices,
symbol keys) rather than recomputing them in the hot minimization
loop. The `SymbolKey` optimization is the largest single win: it
packs each `Symbol`'s type tag and index into a single u64, enabling
single-instruction comparisons in the merge-join that dominates
`states_conflict`. This gives a significant speedup for larger
grammars.
- store ParseTableEntry` refs in the precomputed entry maps to
avoid `IndexMap::get_index` lookups in the inner loop of
`states_conflict`
This eliminates an indirect lookup per comparison in the merge-join.
- use unchecked array accesses in hot loops
Where indices are known valid from construction (e.g. the merge-join
in `states_conflict` and dedup), skip bounds checks in the innermost
loops. The compiler can _usually_ prove this and remove the checks
itself, but not always.
Problem: Many parser projects (about 100 out of ~330 tracked by
nvim-treesitter) still do not contain a `tree-sitter.json` config file,
so `tree-sitter generate` refuses to generate ABI 15 parsers due to the
lack of version information. This prevents these parsers from profiting
from other improvements in ABI 15+ and will eventually make them
unusable when ABI 14 support is dropped.
Solution: Simply use a default version `0.0.0` if no `tree-sitter.json`
is found.
* remove obsolete msvc-dev-cmd action (runners come with build tools)
* build tree-sitter CLI with --profile optimize (longer build time due
to LTO, but generate workflow should profit)
* increase generate parallelism to 3
The previous two-phase locking scheme (probe existing lock file, then
create and compile) had TOCTOU races between phases that caused
spurious failures in CI when tests compiled grammars concurrently.
Replace with a single-phase approach using `create_new` as the sole
synchronization primitive. An RAII `LockFile` guard ensures cleanup
on drop (including panics). Only "builders" attempt to acquire the
lock. Loaders simply load the file. Builders compile a temporary path
and then rename, so loaders are guaranteed a valid shared library.
The winning builder compiles and then drops the lock. Losers poll for
lock file removal, then load. Stale locks from killed processes are
detected via a timeout and an appropriate error message is displayed
to the user.
This started as a simple one to one rewrite, just removing the regexes,
and quickly devolved into a rewrite of the test parsing logic. In
addition to the memory enhancements, the general flow should be much
clearer now. A few data points:
- JS: walltime -1.3%, peak rss -10.2%
- C: walltime -5.7%, peak rss -4.5%
- Rust: walltime -3.8%, peak rss -2.2%
This commit skips adding entries to the subtype map when the subtypes
list is empty to avoid a lookup failure in the topological sort during
node type generation.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
- Cache tree byte range length to avoid redundant FFI calls per test
- Combine 5 separate XML attribute write! calls into one
- Pre-allocate format_sexp output buffer to avoid repeated growth
Wrap stdout in a 64KB BufWriter when writing `parse` output. The tree
walking loop makes many small write calls (parentheses, indentation,
node kinds, ranges, etc.) which are expensive without buffering.
When parsing the jquery.js corpus file, cuts the total time roughly in
half. These savings only show when piping the result to a file,
otherwise terminal rendering time usually dominates, hiding all gains.
Also do the same for the `query` command's output.
We have to pay the cost of compiling the regex at runtime, and the
`LazyLock` overhead for each access. The pattern is simple enough that
we can manually parse and extract.
`grammar_json_name` is now ~116x faster on cold start, ~5.6x faster warm
(after the regex has been compiled). Both are fast enough to not matter
much in practice, but some perf gains and eliminating global state is a
win.
The `LookaheadIterator` visits symbols in group order for small parse
states, which can produce `reduce_actions` in a different order than the
original linear symbol scan (pre c1379718). Since reductions are applied
sequentially and the last reduction version survives, different orderings
lead to different error recovery outcomes (e.g. losing nodes from ERROR
trees).
Sort the reduce_actions array by symbol (descending) after collection.
The array is typically 1-5 entries, so the insertion sort cost is
negligible and the full optimization speedup is preserved.
In `ts_parser__do_all_potential_reductions`, when `lookahead_symbol` is
0 (error recovery), the code scanned every symbol from 1 to
`token_count` calling ts_language_table_entry for each with most returning
empty. Replace with `LookaheadIterator` which efficiently visits only symbols
with valid actions, yielding 7-20x fewer lookups for typical grammars.
Error recovery throughput improves ~29% (JS) / ~39% (C). Valid-code
parsing is unaffected.
Since v0.26.1, `tree-sitter build --wasm` uses wasi-sdk instead of
Emscripten. The CLI automatically downloads wasi-sdk on first use,
so Emscripten, Docker, and Podman are no longer required.
Update the web binding README to reflect this change.
See #4393 for the original switch to wasi-sdk.
The workspace dependency for `tree-sitter-generate` did not set
`default-features = false`, so Cargo always enabled its default
features (including `qjs-rt` and thus `rquickjs`) regardless of
the CLI's `--no-default-features` flag.
Additionally, `tree-sitter-generate` failed to compile without the
`load` feature due to unconditional references to `cfg`-gated items.
- Set `default-features = false` on the workspace `tree-sitter-generate`
dependency so the CLI's feature forwarding actually takes effect.
- Explicitly enable the `load` feature in the CLI's dependency on
`tree-sitter-generate`, since the CLI needs `load`-gated functions
unconditionally.
- Gate necessary imports behind `#[cfg(feature = "load")]` to fix
`tree-sitter-generate`'s build without the `load` feature.
Problem: Publishing the built tree-sitter-cli artifacts as gzipped
binaries prevents installing them with `cargo binstall` or packaging
them for Windows Package Manager (`winget`).
Solution: Also create a zip archive for each binary, named
`tree-sitter-cli` to match now-common downstream usage. (Most package
managers have separate `tree-sitter-cli` and `tree-sitter` (library)
packages.)
Problem: `build` tests are marked as "required" but skipped when only
touching documentation, leading to such PRs not being mergable.
Solution: Run tests on all PRs and all commits to the `master` or
`release-0.x` branches.
* reduce inner loop range in `CoincidentTokenIndex::new`
The indices computed from `a, b` and `b, a` are identical, so there's
no reason to iterate over the entire set of terminal indices in the
inner loop.
* remove redundant check for `does_match_same_string` in
`token_conflicts`
`does_match_same_string` is already covered within `does_conflict`, so
OR-ing the result of the two together is wasteful.
* mark several functions available for `inline`
Reduces walltime by 0-3% depending on the grammar.
* don't continually reserve space in render buffer
Reserving space up front for containers is great. Doing this in a loop
can lead to _more_ allocations, hurting performance.
* pre-collect terminal indices in `CoincidentTokenIndex`
In `CoincidentTokenIndex::new`, the inner loop re-iterates the
`IndexMap` keys and re-checks is_terminal() for every outer iteration.
This information can be collected once per state, avoiding redundant
calculations.
In the naive implementation, the backing arena for `BitVec` frees no
memory, resulting in an increase of ~8-15% in peak rss. To mitigate
against this, the arena manages a simple free list, keyed by block
sizes. This keeps the reduction in wall time within 3% in the worst case
(on par for most grammars) with the naive implementation, and caps the
increase in peak rss to <1% over the current master branch.
We don't pre-allocate _all_ `BitVec`s, as this causes a ~20% increase in
peak rss with little to no performance gain. Certain `TokenSet`s are
guaranteed to be filled out to a fixed size, and so reserving the
space (and thus avoiding the arena waste) is safe.
`TokenSet` operations (insert_all_terminals, insert_all, etc.) are called
_many_ times during parser generation. `SmallBitVec` performs these
bit-by-bit, which is incredibly slow compared to word-level operations.
Replace `SmallBitVec` with a custom `BitVec` backed by Vec<u64> that
operates at the word level.
Problem: The `clang` binary contained in the WASI-SDK releases downloaded from Github does not work on all platforms (e.g., Alpine/MUSL), but a custom (LLVM) `clang` built for the platform will default to that target, making it impossible to build wasm parsers.
Solution: Always pass `wasm32` target triple when calling clang to compile to wasm.
Notes:
* This assumes the custom `clang` is (installed or linked) to `$TREE_SITTER_WASI_SDK_PATH/bin`.
* This requires a full LLVM clang; Apple clang does not support `wasm` targets.
* Tree-sitter expects a specified version of WASI-SDK, including a specific `clang` version. Other versions may but are not guaranteed to work.
This commit just refactors the init command's update code, as it's quite messy. The main changes are that most bindings now go through their own generate_{lang} function.
Previously, `get_existing_tool` checked only for the existence of
binaries in the cache directory without verifying their version, meaning bumping the version files had no effect until users manually deleted the cache directories. This commit writes a `.version` marker file after downloading and checks it on subsequent runs, removing stale caches automatically when the expected version changes.
The template used `FileManager.default.fileExists(atPath: "src/scanner.c")` with a relative path, which resolves against the process cwd. When SwiftPM evaluates a dependency's manifest, the cwd is the consumer's directory, so the check returns `false` even when `scanner.c` exists, causing undefined symbol linker errors.
This commit uses `Context.packageDirectory`, which is available since `swift-tools-version:5.6`, to resolve the path against the package root. This does bump the minimum tools version from 5.3 to 5.6.
Explicitly casting NULL to (Subtree *) in the ternary expression ensures
type consistency. This resolves ambiguous type inference issues encountered
by strict static analysis tools and C-to-Go transpilers (like ccgo).
Signed-off-by: lucasew <lucas59356@gmail.com>
Wasmtime v34 introduced breaking ABI changes to `wasmtime_func_t` and
`wasmtime_table_t` structures. The `__private` field changed from
`size_t` to `void*`, requiring code updates to store complete
`wasmtime_func_t` structures instead of raw indices.
Changes:
- `BuiltinFunctionIndices`: `uint32_t` -> `wasmtime_func_t`
- `stdlib_fn_indices`: `uint32_t*` -> `wasmtime_func_t*`
- `FunctionDefinition.storage_location`: `uint32_t*` -> `void*`
- `get_builtin_extern()`: simplified to return stored func directly
- Lexer functions: use temp array for func storage, store table index
- Zero-initialize `builtin_fn_indices` so `store_id == 0` sentinel
reliably detects missing stdlib exports
- Free `lexer_funcs` on error paths in `ts_wasm_store_new`
- Remove stale `(uint32_t *)` casts from `lexer_definitions`
Co-Authored-By: Amaan Qureshi <git@amaanq.com>
Co-Authored-By: nzinfo <li.monan@gmail.com>
The current `.envrc` unconditionally runs `use flake`, which breaks multi-workspace
jj setups where workspace subdirectories each have their own `.envrc`. Entering a
workspace unloads the root's nix env even when the workspace `.envrc` is blocked,
because direnv always unloads the parent env on any `.envrc` transition.
`source_up_if_exists` re-runs the parent `.envrc` and inherits its nix env in a
multi-workspace setup (fast due to caching). The `IN_NIX_SHELL` guard then skips
`use flake path:.`, which only runs in a plain single checkout where no parent
`.envrc` exists and `IN_NIX_SHELL` is unset.
When a branch inside an alternation has a + or * quantifier, the
quantifier's pass_through step loops back to the branch's first step.
The alternation linking also sets that step's alternative_index to
point to the next branch. This causes the quantifier loop-back to
incorrectly explore other branches in the case of a failed match that
follows a successful match, contaminating captures.
This is corrected by redirecting the quantifier loop-back to a
"clean" copy of the target step without the alternative index pointing
to the next alternation branch.
Co-authored-by: Riley Bruins <ribru17@hotmail.com>
Because Rust tests run in parallel, two tests compiling a parser to wasm
can both try to download wasi-sdk and wasm-opt to a common location,
causing corrupted files and/or other failures. To prevent this, we can
guard access to these tools. `cfg(test)` isn't passed across crate
boundaries, so this lock must be present for all build configurations.
This displays the working directory of the command (if present),
compiler used, all of its arguments, any environment variables set,
and anything written to stdout/stderr by the compilation tool.
the region in place.
Previous changes to `malloc` caused `realloc` to sometimes pull regions
off of the free list during this optimization. Because no `memcpy` is
performed, this resulted in corrupted data returning to the caller.
Co-authored-by: trim21 <i@trim21.me>
Relying on a user's system's installation of `nm` has proven to be bug
prone and flaky. Instead, we can enforce that these symbols are defined
by requiring the linker to resolve them. This is already the default on
macos, but linux has looser requirements which defers the error to when
the library is opened. This check was not run on Windows (because
there's no `nm`), but the msvc linker is similiarly strict to macos's
w.r.t. resolving symbols at build time rather than runtime, so there's
no issue here.
Previously a bug in linux powerpc linkers/nm caused function symbols to
be incorrectly reported in the data "D" section. Newer toolchains now
correctly report these symbols' sections as "T". Account for both to
maintain compatibility with older toolchains
- free memory if 0 size is passed in
- Don't `memcpy` contents if new pointer is `NULL`
- Copy old region's contents only up to size of new region
- free old region
Co-authored-by: trim21 <i@trim21.me>
Altering the `Array` type itself isn't feasible, as this causes
unacceptable breakage with existing parsers that depend on it. Instead,
pass in individual `Array` fields for to various `_array__*` functions.
Any time the `contents` of an array may be modified (`free`d, `realloc`d,
etc), return the potentially new address out by value. This prevents any
strict aliasing violations as we're no longer writing to a type-casted
pointer.
Co-authored-by: Nathaniel Wesley Filardo <nwfilardo@gmail.com>
- One has to think about lifetimes if a type has one:
- `<&'a Node<'tree>>::language` now returns `LanguageRef<'tree>` instead of
`LanguageRef<'a>`, as it should;
- Remove explicit "outlives" requirements from `QueryMatches`, `QueryCaptures`,
and their impl blocks, because they're inferred
- Removed unnecessary `&mut` from `cst_render_node`'s `cursor` parameter
Problem: Output of `cargo xtask build-wasm-stdlib` depends on whether
`wasm-opt` is installed (since `clang` will use it by default if it
finds it).
Solution: Install it and rerun the xtask.
System endian conversion macros are gated behind this feature flag for
older versions of GLIBC. `_BSD_SOURCE` and `_SVID_SOURCE` were
deprecated and replaced with `_DEFAULT_SOURCE` starting with GLIBC 2.19.
Problem:
After commit f02d7e7e33
the `tree-sitter test` command no longer printed the final test summary,
leaving empty line. The `Stats` struct was embedded into `TestSummary`,
and the explicit call to print it was removed.
Solution:
Print `parse_stats` from `TestSummary.fmt()` implementation.
This fixes a potential issue with the new lock file hashing mechanism,
in which two different path literals pointing to the same location would
hash to separate lock files, allowing a race condition.
Problem:
The CST printer emits trailing whitespace after multiline text nodes.
With 1704c604bf and `:cst` corpus tests
this causes trailing spaces to appear on `test --update`.
These spaces cannot be removed afterward, as the test runner
expects an exact character-for-character match for CST tests.
Solution:
Print whitespace only if node is not multiline.
as "0.1"
If a rust project depends on both the tree-sitter lib bindings and the
language crate, cargo needs to be able to resolve a common version of
the tree-sitter-language crate. Specifying exactly "0.1.5" for the lib
bindings is overly restrictive, and could lead to future headaches. By
specifying "0.1", any "0.1.x" version should be available to resolve to.
The loader package's `ensure_wasi_sdk_exists` private method checks for
the wasi-sdk, fetching it if it can't be found. This logic was
re-implemented in xtask for `build-wasm-stdlib`, but without the
fetching functionality. We can have nice things in xtask too! Rather
than make this function a public member of `tree-sitter-loader`, we
just re-implement and leave a nice comment asking people to keep the
two in sync.
Problem: `fs::rename` fails if the parser directory and the Tree-sitter
library directory are on different file systems.
Solution: Write the library file directly to the final directory.
is disabled
This applies to the `parse` and `test` commands, but not `build` as it
doesn't require the wasm feature. Also, hide the `--wasm` options if
from the `--help` output if the feature is disabled.
**Problem:** A query with a `?` quantifier followed by a `+` quantifier
would hang at 100% CPU usage while iterating through a tree, regardless
of the source content.
**Solution:** Collect all quantifiers in one step, and then add the
required repeat/optional step logic *after* we have determined the
composite quantifier we need to use for the current step.
Since 66dab20462, bindings automatically
detect external scanner, making the instructions for manual updating
outdated. Avoids confusion about missing commented lines in Rust
bindings.
- Indicate where xtask looks for wasi-sdk
- Indicate where `build --wasm` looks for and downloads wasi-sdk binary
to
- Mark native runtime as experimental, describe limitations
- Note ABI 13 support limitations
- Mention that `test --wasm` and `parse --wasm` require
`--features=wasm` build
Closes#374.
The statement about the intended backwards compatibility is purely
speculative and provided as a "straw man" to help reviewers come up with
a better description of the intended backwards compatibility.
Pass the BUILD_TARGET variable from the build environment as 'host' for
the cc crate. Otherwise, when cross-compiled, cc will keep looking for a
cross-compiler instead of the native one on the target system.
Signed-off-by: Valeriy Kosikhin <vkosikhin@gmail.com>
Problem: "deploy docs" always pulls in the `latest` release of `mdbook`,
which now is a v0.5.0 prerelease with breaking changes -- including
removing an (apparently unused) `multilingual` config field in the TOML
that is now an error (another breaking change).
Solution: Delete the line. Add `workflow_dispatch` to the docs workflow
in case follow-up changes are needed; see
https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#05-migration-guide
The Tree-sitter CLI allows you to develop, test, and use Tree-sitter grammars from the command line. It works on `MacOS`, `Linux`, and `Windows`.
The Tree-sitter CLI allows you to develop, test, and use Tree-sitter grammars from the command line. It works on `MacOS`,
`Linux`, and `Windows`.
### Installation
You can install the `tree-sitter-cli` with `cargo`:
You can install the `tree-sitter-cli` with [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall):
```sh
cargo install --locked tree-sitter-cli
cargo binstall tree-sitter-cli
```
or with `npm`:
or you can build it from source:
```sh
npm install tree-sitter-cli
cargo install --locked tree-sitter-cli
```
You can also download a pre-built binary for your platform from [the releases page].
@ -34,9 +34,11 @@ The `tree-sitter` binary itself has no dependencies, but specific commands have
### Commands
* `generate` - The `tree-sitter generate` command will generate a Tree-sitter parser based on the grammar in the current working directory. See [the documentation] for more information.
* `generate` - The `tree-sitter generate` command will generate a Tree-sitter parser based on the grammar in the current
working directory. See [the documentation] for more information.
* `test` - The `tree-sitter test` command will run the unit tests for the Tree-sitter parser in the current working directory. See [the documentation] for more information.
* `test` - The `tree-sitter test` command will run the unit tests for the Tree-sitter parser in the current working directory.
See [the documentation] for more information.
* `parse` - The `tree-sitter parse` command will parse a file (or list of files) using Tree-sitter parsers.
