The previous implementation of `InputGrammar::normalize` (inlined in
`parse_grammar` iterated over every variable, checking whether it was
reachable _backwards_ from teh root by recursing over rules that
referenced it.
This means that each and every top level call re-traversed the entire
graph. The runtime performance of this backwards walk was dependent on
the _order_ of rules as declared in `grammar.js`. All existing grammars
have an ordering that's reasonably friendly to this iteration pattern
(BFS-ish order, top down from the start rule), but this leaves us open
to a catastrophic performance cliff.
Instead, seed a `used` set with the start rule, the word token, and an
names referenced from `extras`/`externals`. Then propagate via direct rule
references. This yields anywhere from a 2-~2200x speedup for
`parse_grammar`. This greatly speeds up `--no-parser` runs, but is
relatively unimportant for `parser.c` generation for _existing_
grammars. The important piece is eliminating the potential cliff.
debug output.
The initial implementation of `--debug pretty` assumed process version
was bounded by `MAX_VERSION_COUNT`. However, we also have to account for
`MAX_VERSION_COUNT_OVERFLOW` as well as `halted_version_count`.
(cherry picked from commit 5cac4316db)
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
(cherry picked from commit 0f6780b9a3)
`generate_parser_for_grammar`
Passing `empty` here causes some grammars (i.e. tree-sitter-cpp) to fail
to generate.
(cherry picked from commit 457eb295b7)
This commit skips adding entries to the subtype map when the subtypes
list is empty to avoid a lookup failure in the topological sort during
node type generation.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
(cherry picked from commit 7c2e757c03)
- Cache tree byte range length to avoid redundant FFI calls per test
- Combine 5 separate XML attribute write! calls into one
- Pre-allocate format_sexp output buffer to avoid repeated growth
(cherry picked from commit 791d7cead4)
Wrap stdout in a 64KB BufWriter when writing `parse` output. The tree
walking loop makes many small write calls (parentheses, indentation,
node kinds, ranges, etc.) which are expensive without buffering.
When parsing the jquery.js corpus file, cuts the total time roughly in
half. These savings only show when piping the result to a file,
otherwise terminal rendering time usually dominates, hiding all gains.
Also do the same for the `query` command's output.
(cherry picked from commit 827bcdabd9)
The workspace dependency for `tree-sitter-generate` did not set
`default-features = false`, so Cargo always enabled its default
features (including `qjs-rt` and thus `rquickjs`) regardless of
the CLI's `--no-default-features` flag.
Additionally, `tree-sitter-generate` failed to compile without the
`load` feature due to unconditional references to `cfg`-gated items.
- Set `default-features = false` on the workspace `tree-sitter-generate`
dependency so the CLI's feature forwarding actually takes effect.
- Explicitly enable the `load` feature in the CLI's dependency on
`tree-sitter-generate`, since the CLI needs `load`-gated functions
unconditionally.
- Gate necessary imports behind `#[cfg(feature = "load")]` to fix
`tree-sitter-generate`'s build without the `load` feature.
Problem: The `clang` binary contained in the WASI-SDK releases downloaded from Github does not work on all platforms (e.g., Alpine/MUSL), but a custom (LLVM) `clang` built for the platform will default to that target, making it impossible to build wasm parsers.
Solution: Always pass `wasm32` target triple when calling clang to compile to wasm.
Notes:
* This assumes the custom `clang` is (installed or linked) to `$TREE_SITTER_WASI_SDK_PATH/bin`.
* This requires a full LLVM clang; Apple clang does not support `wasm` targets.
* Tree-sitter expects a specified version of WASI-SDK, including a specific `clang` version. Other versions may but are not guaranteed to work.
When a branch inside an alternation has a + or * quantifier, the
quantifier's pass_through step loops back to the branch's first step.
The alternation linking also sets that step's alternative_index to
point to the next branch. This causes the quantifier loop-back to
incorrectly explore other branches in the case of a failed match that
follows a successful match, contaminating captures.
This is corrected by redirecting the quantifier loop-back to a
"clean" copy of the target step without the alternative index pointing
to the next alternation branch.
Co-authored-by: Riley Bruins <ribru17@hotmail.com>
(cherry picked from commit 596a4d69bb)
the region in place.
Previous changes to `malloc` caused `realloc` to sometimes pull regions
off of the free list during this optimization. Because no `memcpy` is
performed, this resulted in corrupted data returning to the caller.
Co-authored-by: trim21 <i@trim21.me>
Relying on a user's system's installation of `nm` has proven to be bug
prone and flaky. Instead, we can enforce that these symbols are defined
by requiring the linker to resolve them. This is already the default on
macos, but linux has looser requirements which defers the error to when
the library is opened. This check was not run on Windows (because
there's no `nm`), but the msvc linker is similiarly strict to macos's
w.r.t. resolving symbols at build time rather than runtime, so there's
no issue here.
Previously a bug in linux powerpc linkers/nm caused function symbols to
be incorrectly reported in the data "D" section. Newer toolchains now
correctly report these symbols' sections as "T". Account for both to
maintain compatibility with older toolchains
Without this check, action ids > 65,535 can be assigned to `uint16_t`s
in parser.c. This causes parsing to either crash or silently misbehave.
(cherry picked from commit e65dc4c3b5)
- free memory if 0 size is passed in
- Don't `memcpy` contents if new pointer is `NULL`
- Copy old region's contents only up to size of new region
- free old region
Co-authored-by: trim21 <i@trim21.me>
Altering the `Array` type itself isn't feasible, as this causes
unacceptable breakage with existing parsers that depend on it. Instead,
pass in individual `Array` fields for to various `_array__*` functions.
Any time the `contents` of an array may be modified (`free`d, `realloc`d,
etc), return the potentially new address out by value. This prevents any
strict aliasing violations as we're no longer writing to a type-casted
pointer.
Co-authored-by: Nathaniel Wesley Filardo <nwfilardo@gmail.com>
(cherry picked from commit 5177b3dc26)
Set tree-sitter-php fixture to `upstream_test_fixture` branch.
Also remove the `--update` flag, as it does nothing.
(cherry picked from commit 55fdc50e81)
System endian conversion macros are gated behind this feature flag for
older versions of GLIBC. `_BSD_SOURCE` and `_SVID_SOURCE` were
deprecated and replaced with `_DEFAULT_SOURCE` starting with GLIBC 2.19.
(cherry picked from commit aefae11c0d)
Problem:
After commit f02d7e7e33
the `tree-sitter test` command no longer printed the final test summary,
leaving empty line. The `Stats` struct was embedded into `TestSummary`,
and the explicit call to print it was removed.
Solution:
Print `parse_stats` from `TestSummary.fmt()` implementation.
(cherry picked from commit 17e3c7a5c5)
