* reduce inner loop range in `CoincidentTokenIndex::new`
The indices computed from `a, b` and `b, a` are identical, so there's
no reason to iterate over the entire set of terminal indices in the
inner loop.
* remove redundant check for `does_match_same_string` in
`token_conflicts`
`does_match_same_string` is already covered within `does_conflict`, so
OR-ing the result of the two together is wasteful.
* mark several functions available for `inline`
Reduces walltime by 0-3% depending on the grammar.
* don't continually reserve space in render buffer
Reserving space up front for containers is great. Doing this in a loop
can lead to _more_ allocations, hurting performance.
* pre-collect terminal indices in `CoincidentTokenIndex`
In `CoincidentTokenIndex::new`, the inner loop re-iterates the
`IndexMap` keys and re-checks is_terminal() for every outer iteration.
This information can be collected once per state, avoiding redundant
calculations.
In the naive implementation, the backing arena for `BitVec` frees no
memory, resulting in an increase of ~8-15% in peak rss. To mitigate
against this, the arena manages a simple free list, keyed by block
sizes. This keeps the reduction in wall time within 3% in the worst case
(on par for most grammars) with the naive implementation, and caps the
increase in peak rss to <1% over the current master branch.
We don't pre-allocate _all_ `BitVec`s, as this causes a ~20% increase in
peak rss with little to no performance gain. Certain `TokenSet`s are
guaranteed to be filled out to a fixed size, and so reserving the
space (and thus avoiding the arena waste) is safe.
`TokenSet` operations (insert_all_terminals, insert_all, etc.) are called
_many_ times during parser generation. `SmallBitVec` performs these
bit-by-bit, which is incredibly slow compared to word-level operations.
Replace `SmallBitVec` with a custom `BitVec` backed by Vec<u64> that
operates at the word level.
Problem: The `clang` binary contained in the WASI-SDK releases downloaded from Github does not work on all platforms (e.g., Alpine/MUSL), but a custom (LLVM) `clang` built for the platform will default to that target, making it impossible to build wasm parsers.
Solution: Always pass `wasm32` target triple when calling clang to compile to wasm.
Notes:
* This assumes the custom `clang` is (installed or linked) to `$TREE_SITTER_WASI_SDK_PATH/bin`.
* This requires a full LLVM clang; Apple clang does not support `wasm` targets.
* Tree-sitter expects a specified version of WASI-SDK, including a specific `clang` version. Other versions may but are not guaranteed to work.
This commit just refactors the init command's update code, as it's quite messy. The main changes are that most bindings now go through their own generate_{lang} function.
Previously, `get_existing_tool` checked only for the existence of
binaries in the cache directory without verifying their version, meaning bumping the version files had no effect until users manually deleted the cache directories. This commit writes a `.version` marker file after downloading and checks it on subsequent runs, removing stale caches automatically when the expected version changes.
The template used `FileManager.default.fileExists(atPath: "src/scanner.c")` with a relative path, which resolves against the process cwd. When SwiftPM evaluates a dependency's manifest, the cwd is the consumer's directory, so the check returns `false` even when `scanner.c` exists, causing undefined symbol linker errors.
This commit uses `Context.packageDirectory`, which is available since `swift-tools-version:5.6`, to resolve the path against the package root. This does bump the minimum tools version from 5.3 to 5.6.
