This commit adds an `eof()` function for grammars, which is easier to
use than the NUL byte directly. It compiles down to a constraint that
the enclosing production can only reduce at end of input, not to a
shiftable token.
BREAKING CHANGE: Public error types for `tree-sitter-generate` were modified.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
* feat(wasm): make syntax trees sendable
* test(wasm): transfer trees across workers
* test(wasm): use JSON grammar for tree transfer
* test(wasm): edit trees across workers
* test(wasm): share dlmalloc with tree-sitter
* test(wasm): simplify worker tree exchange
* test(wasm): drive tree exchange from Rust
* test(wasm): split sendable-tree xtask
* test(wasm): generalize Rust web fixture
* test(wasm): exercise parallel Rust tree access
* fix(wasm): use Rust global allocator for C core
* test(wasm): use default Rust allocator
* feat(wasm): support external scanners in Rust web apps
* Simplify example further, add a readme
* Regenerate wasm-stdlib
* Fix wasm_stdlib check script
* Vendor the Wasm standard library subset
* Test Unicode Ruby scanner behavior in Wasm
* Make Wasm tree languages instance-aware
* Test multi-threaded use of queries in wasm32-unknown
* Refactor reference-counted language storage
* Check ABI version compat before loading rest of language
* 🎨 Remove redundant #ifdef block
* Reject unsupported Rust Wasm builds on 0.26
- `--layout <document|line-numbers|fragment>` (default `document`): the
document structure. `document` is a self-contained page wrapping a plain
`<div class="highlight"><pre><code>` block, `line-numbers` adds a line-number
`<table>`, and `fragment` emits only the code markup with no surrounding
`<head>`/`<style>`/`<body>` so it can be embedded in an existing page.
- `--style <classes|inline|minimal>` (default `classes`): how token colors
are applied. `classes` uses `class="..."` spans plus a generated `<style>`,
`inline` bakes colors onto each span so the markup is self-contained, and
`minimal` emits bare classes with no colors (bring your own stylesheet).
Deprecates the `--css-classes` flag.
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
- An anchor between two patterns is vacuous when an adjacent quantifier
matches zero
- A leading or trailing anchor on a node applies to the nearest matched node
when an adjacent optional is skipped
- An anchor at the edge of a group or alternation is not allowed.
Clarifies the behavior of `TSLexer::advance` when `skip=true`.
The previous documentation didn't make it clear that `skip=true` should only be used before a token starts. Using it after `mark_end` can affect the token’s starting position and lead to incorrect or zero-length ranges.
Fixes#2315
Compiling a grammar with `-fsanitize=address` (or any other sanitizer)
emits a module constructor in the parser object file that references
runtime symbols like `__asan_init`. These runtime-resolved symbols are
incompatible with `-Wl,--no-undefined`, which breaking sanitized test
runs on clang. (gcc happens to paper over this by auto-linking libasan
into the .so)
To fix, detect `-fsanitize=` in the compile flags and omit `--no-undefined`
in that case.
This started as a simple one to one rewrite, just removing the regexes,
and quickly devolved into a rewrite of the test parsing logic. In
addition to the memory enhancements, the general flow should be much
clearer now. A few data points:
- JS: walltime -1.3%, peak rss -10.2%
- C: walltime -5.7%, peak rss -4.5%
- Rust: walltime -3.8%, peak rss -2.2%
This displays the working directory of the command (if present),
compiler used, all of its arguments, any environment variables set,
and anything written to stdout/stderr by the compilation tool.
Since 66dab20462, bindings automatically
detect external scanner, making the instructions for manual updating
outdated. Avoids confusion about missing commented lines in Rust
bindings.
- Indicate where xtask looks for wasi-sdk
- Indicate where `build --wasm` looks for and downloads wasi-sdk binary
to
- Mark native runtime as experimental, describe limitations
- Note ABI 13 support limitations
- Mention that `test --wasm` and `parse --wasm` require
`--features=wasm` build
