Also use `IoError` more consistently throughout the rest of the project
where its straightforward to do so.
BREAKING CHANGE: Changes public error types for config, generate, and
loader crates.
Compiling a grammar with `-fsanitize=address` (or any other sanitizer)
emits a module constructor in the parser object file that references
runtime symbols like `__asan_init`. These runtime-resolved symbols are
incompatible with `-Wl,--no-undefined`, which breaking sanitized test
runs on clang. (gcc happens to paper over this by auto-linking libasan
into the .so)
To fix, detect `-fsanitize=` in the compile flags and omit `--no-undefined`
in that case.
The previous two-phase locking scheme (probe existing lock file, then
create and compile) had TOCTOU races between phases that caused
spurious failures in CI when tests compiled grammars concurrently.
Replace with a single-phase approach using `create_new` as the sole
synchronization primitive. An RAII `LockFile` guard ensures cleanup
on drop (including panics). Only "builders" attempt to acquire the
lock. Loaders simply load the file. Builders compile a temporary path
and then rename, so loaders are guaranteed a valid shared library.
The winning builder compiles and then drops the lock. Losers poll for
lock file removal, then load. Stale locks from killed processes are
detected via a timeout and an appropriate error message is displayed
to the user.
We have to pay the cost of compiling the regex at runtime, and the
`LazyLock` overhead for each access. The pattern is simple enough that
we can manually parse and extract.
`grammar_json_name` is now ~116x faster on cold start, ~5.6x faster warm
(after the regex has been compiled). Both are fast enough to not matter
much in practice, but some perf gains and eliminating global state is a
win.
Problem: The `clang` binary contained in the WASI-SDK releases downloaded from Github does not work on all platforms (e.g., Alpine/MUSL), but a custom (LLVM) `clang` built for the platform will default to that target, making it impossible to build wasm parsers.
Solution: Always pass `wasm32` target triple when calling clang to compile to wasm.
Notes:
* This assumes the custom `clang` is (installed or linked) to `$TREE_SITTER_WASI_SDK_PATH/bin`.
* This requires a full LLVM clang; Apple clang does not support `wasm` targets.
* Tree-sitter expects a specified version of WASI-SDK, including a specific `clang` version. Other versions may but are not guaranteed to work.
Previously, `get_existing_tool` checked only for the existence of
binaries in the cache directory without verifying their version, meaning bumping the version files had no effect until users manually deleted the cache directories. This commit writes a `.version` marker file after downloading and checks it on subsequent runs, removing stale caches automatically when the expected version changes.
Because Rust tests run in parallel, two tests compiling a parser to wasm
can both try to download wasi-sdk and wasm-opt to a common location,
causing corrupted files and/or other failures. To prevent this, we can
guard access to these tools. `cfg(test)` isn't passed across crate
boundaries, so this lock must be present for all build configurations.
This displays the working directory of the command (if present),
compiler used, all of its arguments, any environment variables set,
and anything written to stdout/stderr by the compilation tool.
Relying on a user's system's installation of `nm` has proven to be bug
prone and flaky. Instead, we can enforce that these symbols are defined
by requiring the linker to resolve them. This is already the default on
macos, but linux has looser requirements which defers the error to when
the library is opened. This check was not run on Windows (because
there's no `nm`), but the msvc linker is similiarly strict to macos's
w.r.t. resolving symbols at build time rather than runtime, so there's
no issue here.
Previously a bug in linux powerpc linkers/nm caused function symbols to
be incorrectly reported in the data "D" section. Newer toolchains now
correctly report these symbols' sections as "T". Account for both to
maintain compatibility with older toolchains
- One has to think about lifetimes if a type has one:
- `<&'a Node<'tree>>::language` now returns `LanguageRef<'tree>` instead of
`LanguageRef<'a>`, as it should;
- Remove explicit "outlives" requirements from `QueryMatches`, `QueryCaptures`,
and their impl blocks, because they're inferred
- Removed unnecessary `&mut` from `cst_render_node`'s `cursor` parameter
This fixes a potential issue with the new lock file hashing mechanism,
in which two different path literals pointing to the same location would
hash to separate lock files, allowing a race condition.
The loader package's `ensure_wasi_sdk_exists` private method checks for
the wasi-sdk, fetching it if it can't be found. This logic was
re-implemented in xtask for `build-wasm-stdlib`, but without the
fetching functionality. We can have nice things in xtask too! Rather
than make this function a public member of `tree-sitter-loader`, we
just re-implement and leave a nice comment asking people to keep the
two in sync.
Problem: `fs::rename` fails if the parser directory and the Tree-sitter
library directory are on different file systems.
Solution: Write the library file directly to the final directory.
