* remove obsolete msvc-dev-cmd action (runners come with build tools)
* build tree-sitter CLI with --profile optimize (longer build time due
to LTO, but generate workflow should profit)
* increase generate parallelism to 3
The previous two-phase locking scheme (probe existing lock file, then
create and compile) had TOCTOU races between phases that caused
spurious failures in CI when tests compiled grammars concurrently.
Replace with a single-phase approach using `create_new` as the sole
synchronization primitive. An RAII `LockFile` guard ensures cleanup
on drop (including panics). Only "builders" attempt to acquire the
lock. Loaders simply load the file. Builders compile a temporary path
and then rename, so loaders are guaranteed a valid shared library.
The winning builder compiles and then drops the lock. Losers poll for
lock file removal, then load. Stale locks from killed processes are
detected via a timeout and an appropriate error message is displayed
to the user.
This started as a simple one to one rewrite, just removing the regexes,
and quickly devolved into a rewrite of the test parsing logic. In
addition to the memory enhancements, the general flow should be much
clearer now. A few data points:
- JS: walltime -1.3%, peak rss -10.2%
- C: walltime -5.7%, peak rss -4.5%
- Rust: walltime -3.8%, peak rss -2.2%
This commit skips adding entries to the subtype map when the subtypes
list is empty to avoid a lookup failure in the topological sort during
node type generation.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
- Cache tree byte range length to avoid redundant FFI calls per test
- Combine 5 separate XML attribute write! calls into one
- Pre-allocate format_sexp output buffer to avoid repeated growth
Wrap stdout in a 64KB BufWriter when writing `parse` output. The tree
walking loop makes many small write calls (parentheses, indentation,
node kinds, ranges, etc.) which are expensive without buffering.
When parsing the jquery.js corpus file, cuts the total time roughly in
half. These savings only show when piping the result to a file,
otherwise terminal rendering time usually dominates, hiding all gains.
Also do the same for the `query` command's output.
We have to pay the cost of compiling the regex at runtime, and the
`LazyLock` overhead for each access. The pattern is simple enough that
we can manually parse and extract.
`grammar_json_name` is now ~116x faster on cold start, ~5.6x faster warm
(after the regex has been compiled). Both are fast enough to not matter
much in practice, but some perf gains and eliminating global state is a
win.
The `LookaheadIterator` visits symbols in group order for small parse
states, which can produce `reduce_actions` in a different order than the
original linear symbol scan (pre c1379718). Since reductions are applied
sequentially and the last reduction version survives, different orderings
lead to different error recovery outcomes (e.g. losing nodes from ERROR
trees).
Sort the reduce_actions array by symbol (descending) after collection.
The array is typically 1-5 entries, so the insertion sort cost is
negligible and the full optimization speedup is preserved.
In `ts_parser__do_all_potential_reductions`, when `lookahead_symbol` is
0 (error recovery), the code scanned every symbol from 1 to
`token_count` calling ts_language_table_entry for each with most returning
empty. Replace with `LookaheadIterator` which efficiently visits only symbols
with valid actions, yielding 7-20x fewer lookups for typical grammars.
Error recovery throughput improves ~29% (JS) / ~39% (C). Valid-code
parsing is unaffected.
Since v0.26.1, `tree-sitter build --wasm` uses wasi-sdk instead of
Emscripten. The CLI automatically downloads wasi-sdk on first use,
so Emscripten, Docker, and Podman are no longer required.
Update the web binding README to reflect this change.
See #4393 for the original switch to wasi-sdk.
The workspace dependency for `tree-sitter-generate` did not set
`default-features = false`, so Cargo always enabled its default
features (including `qjs-rt` and thus `rquickjs`) regardless of
the CLI's `--no-default-features` flag.
Additionally, `tree-sitter-generate` failed to compile without the
`load` feature due to unconditional references to `cfg`-gated items.
- Set `default-features = false` on the workspace `tree-sitter-generate`
dependency so the CLI's feature forwarding actually takes effect.
- Explicitly enable the `load` feature in the CLI's dependency on
`tree-sitter-generate`, since the CLI needs `load`-gated functions
unconditionally.
- Gate necessary imports behind `#[cfg(feature = "load")]` to fix
`tree-sitter-generate`'s build without the `load` feature.
Problem: Publishing the built tree-sitter-cli artifacts as gzipped
binaries prevents installing them with `cargo binstall` or packaging
them for Windows Package Manager (`winget`).
Solution: Also create a zip archive for each binary, named
`tree-sitter-cli` to match now-common downstream usage. (Most package
managers have separate `tree-sitter-cli` and `tree-sitter` (library)
packages.)
Problem: `build` tests are marked as "required" but skipped when only
touching documentation, leading to such PRs not being mergable.
Solution: Run tests on all PRs and all commits to the `master` or
`release-0.x` branches.
* reduce inner loop range in `CoincidentTokenIndex::new`
The indices computed from `a, b` and `b, a` are identical, so there's
no reason to iterate over the entire set of terminal indices in the
inner loop.
* remove redundant check for `does_match_same_string` in
`token_conflicts`
`does_match_same_string` is already covered within `does_conflict`, so
OR-ing the result of the two together is wasteful.
* mark several functions available for `inline`
Reduces walltime by 0-3% depending on the grammar.
* don't continually reserve space in render buffer
Reserving space up front for containers is great. Doing this in a loop
can lead to _more_ allocations, hurting performance.
* pre-collect terminal indices in `CoincidentTokenIndex`
In `CoincidentTokenIndex::new`, the inner loop re-iterates the
`IndexMap` keys and re-checks is_terminal() for every outer iteration.
This information can be collected once per state, avoiding redundant
calculations.
