Since v0.26.1, `tree-sitter build --wasm` uses wasi-sdk instead of
Emscripten. The CLI automatically downloads wasi-sdk on first use,
so Emscripten, Docker, and Podman are no longer required.
Update the web binding README to reflect this change.
See #4393 for the original switch to wasi-sdk.
The workspace dependency for `tree-sitter-generate` did not set
`default-features = false`, so Cargo always enabled its default
features (including `qjs-rt` and thus `rquickjs`) regardless of
the CLI's `--no-default-features` flag.
Additionally, `tree-sitter-generate` failed to compile without the
`load` feature due to unconditional references to `cfg`-gated items.
- Set `default-features = false` on the workspace `tree-sitter-generate`
dependency so the CLI's feature forwarding actually takes effect.
- Explicitly enable the `load` feature in the CLI's dependency on
`tree-sitter-generate`, since the CLI needs `load`-gated functions
unconditionally.
- Gate necessary imports behind `#[cfg(feature = "load")]` to fix
`tree-sitter-generate`'s build without the `load` feature.
Problem: Publishing the built tree-sitter-cli artifacts as gzipped
binaries prevents installing them with `cargo binstall` or packaging
them for Windows Package Manager (`winget`).
Solution: Also create a zip archive for each binary, named
`tree-sitter-cli` to match now-common downstream usage. (Most package
managers have separate `tree-sitter-cli` and `tree-sitter` (library)
packages.)
Problem: `build` tests are marked as "required" but skipped when only
touching documentation, leading to such PRs not being mergable.
Solution: Run tests on all PRs and all commits to the `master` or
`release-0.x` branches.
* reduce inner loop range in `CoincidentTokenIndex::new`
The indices computed from `a, b` and `b, a` are identical, so there's
no reason to iterate over the entire set of terminal indices in the
inner loop.
* remove redundant check for `does_match_same_string` in
`token_conflicts`
`does_match_same_string` is already covered within `does_conflict`, so
OR-ing the result of the two together is wasteful.
* mark several functions available for `inline`
Reduces walltime by 0-3% depending on the grammar.
* don't continually reserve space in render buffer
Reserving space up front for containers is great. Doing this in a loop
can lead to _more_ allocations, hurting performance.
* pre-collect terminal indices in `CoincidentTokenIndex`
In `CoincidentTokenIndex::new`, the inner loop re-iterates the
`IndexMap` keys and re-checks is_terminal() for every outer iteration.
This information can be collected once per state, avoiding redundant
calculations.
In the naive implementation, the backing arena for `BitVec` frees no
memory, resulting in an increase of ~8-15% in peak rss. To mitigate
against this, the arena manages a simple free list, keyed by block
sizes. This keeps the reduction in wall time within 3% in the worst case
(on par for most grammars) with the naive implementation, and caps the
increase in peak rss to <1% over the current master branch.
We don't pre-allocate _all_ `BitVec`s, as this causes a ~20% increase in
peak rss with little to no performance gain. Certain `TokenSet`s are
guaranteed to be filled out to a fixed size, and so reserving the
space (and thus avoiding the arena waste) is safe.
`TokenSet` operations (insert_all_terminals, insert_all, etc.) are called
_many_ times during parser generation. `SmallBitVec` performs these
bit-by-bit, which is incredibly slow compared to word-level operations.
Replace `SmallBitVec` with a custom `BitVec` backed by Vec<u64> that
operates at the word level.
Problem: The `clang` binary contained in the WASI-SDK releases downloaded from Github does not work on all platforms (e.g., Alpine/MUSL), but a custom (LLVM) `clang` built for the platform will default to that target, making it impossible to build wasm parsers.
Solution: Always pass `wasm32` target triple when calling clang to compile to wasm.
Notes:
* This assumes the custom `clang` is (installed or linked) to `$TREE_SITTER_WASI_SDK_PATH/bin`.
* This requires a full LLVM clang; Apple clang does not support `wasm` targets.
* Tree-sitter expects a specified version of WASI-SDK, including a specific `clang` version. Other versions may but are not guaranteed to work.
This commit just refactors the init command's update code, as it's quite messy. The main changes are that most bindings now go through their own generate_{lang} function.
Previously, `get_existing_tool` checked only for the existence of
binaries in the cache directory without verifying their version, meaning bumping the version files had no effect until users manually deleted the cache directories. This commit writes a `.version` marker file after downloading and checks it on subsequent runs, removing stale caches automatically when the expected version changes.
The template used `FileManager.default.fileExists(atPath: "src/scanner.c")` with a relative path, which resolves against the process cwd. When SwiftPM evaluates a dependency's manifest, the cwd is the consumer's directory, so the check returns `false` even when `scanner.c` exists, causing undefined symbol linker errors.
This commit uses `Context.packageDirectory`, which is available since `swift-tools-version:5.6`, to resolve the path against the package root. This does bump the minimum tools version from 5.3 to 5.6.
