`current_symbol` and `current_symbol_name` return an `Option` (`None` when the
iterator is not positioned on a symbol). Change `iter_names`'s item from
`&'static str` to `&str`.
Track the iterator's phase so exhaustion is sticky, and gate the symbol name on
it, so `NULL` means "not positioned on a symbol".
`ts_lookahead_iterator_new` was also the only language consuming constructor
that did not `ts_language_copy`, so an iterator outliving a wasm language read
freed memory. Retain in `_new` and `_reset`, release in `_delete`.
Previously:
- `ts_lookahead_iterator__next` left a small parse state's cursor one past its
group end, so re-advancing an exhausted iterator resumed returning `true` and
walked through the rest of `ts_small_parse_table` and off the end of it.
- `ts_lookahead_iterator_current_symbol_name` returned `NULL` only when the
exhausted symbol index happened to fall outside the names table, so a grammar
with aliases returned a real but wrong name instead.
`(?-u:...)` switches `regex_syntax` to matching raw bytes, but the lexer
dispatches on decoded characters, so `expand_regex` converted the byte
class with a u8 cast. This is exact for ASCII bytes, and silently
misleading for anything aboove 0x80.
This is not reachable from grammar.js (node and QuickJS both reject
`(?-u:...)`). This change is to guard against future JS runtime changes,
as well as alternative frontends to the generate crate.
A previous fix moved case folding for `/i` patterns out of `regex_syntax` and
into `expand_regex`, so folding could drop the two non-ASCII code points
Unicode simple folding maps onto ASCII letters: the long s `ſ` (U+017F)
onto `s`, and the Kelvin sign `K` (U+212A) onto `k`. Left in, they leak
into otherwise-ASCII tokens and stop those tokens from being extracted as
keywords.
By that point, though, the HIR has already turned a negated class into a
complement, so folding it applies the fold on the wrong side of the
negation. `(?i)[^a-z]` folds a set that contains `A-Z`, which re-admits
`a-z` and leaves a class matching very nearly everything.
`regex_syntax` folds each leaf of a class expression before applying that
leaf's negation and the set algebra above it. Keep that order and change
only the fold: walk the AST, replace each leaf with its fold, and translate
with `case_insensitive(false)`.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
This is already exposed for consumers via the CLI, and is a natural way
to express some test expectations over the sexp form.
Also clean up some repeated logic in the internal test code, and narrow
the cst rendering return type to `std::io::Result` rather than
`anyhow::Result`.
Every (state, terminal) parse-table entry stored its action list inline as a
32-byte `ParseTableEntry`, but across a grammar those lists are ~98-99% _duplicates_.
The number of distinct lists is a few thousand regardless of grammar size, while
total entries scale into the hundreds of thousands.
Store each unique action list once in a shared `ActionListPool` (a flat arena of
actions plus `(offset, len)` ranges) and replace the inline entry with a 4-byte
`ActionListId` (a pool index with the `reusable` flag packed into the high bit).
- intern_table converts the freshly built `ParseTable<ParseTableEntry>` into
`ParseTable<ActionListId>`.
- minimize carries and operates on the 4-byte ids. The three global state
renumberings rewrite Shift targets once at the pool level
(`remap_terminal_references`), while the per-state unit-reduction redirects
copy the changed list into a new slot (COW). `mark_fragile_tokens` becomes a
free bit flip on the id.
- `canonicalize` dedups and compacts the pool once before `render`, dropping the
dead and duplicate slots the remaps and COW leave behind. `render` assigns the
output action-list offsets directly.
Yields ~7% wall time reduction, ~12% peak rss reduction.
`ParseStateId`, `LexStateId`, `ProductionInfoId`, and `ReservedWordSetId`
each held a `usize`, while the runtime stores state/field ids as `u16` (so
`u32` has plenty of headroom). Shrinking them saves space for every parse-table
entry, and a number of other data structures.
Reduces wall time by 2-5%, peak rss by 9-12%.
`get_auxiliary_node_info` scans every entry in the item set to collect the
non-auxiliary parents of a given auxiliary symbol. It was called once per
entry whose next symbol is auxiliary, and the same auxiliary symbol recurs
across many entries in a single state (a state's GOTO on a repeat symbol is
shared by every item advancing over it), so the same full scan was repeated
many times per state. Memoize the result per symbol within a single
`add_actions` call.
Reduces wall time 1-4%, rss flat.
`ParseTableEntry` held a `Vec<ParseAction>` per entry. `ActionList` keeps up
to one action inline and spills to a Vec only for conflict entries.
Reduces walltime by >20%, peak rss by ~30%.
The production info depends only on the production, but was recomputed and
deduplicated via a linear deep-equality scan over production_infos for every
reduce item in every state. Cache `prod_id` -> `ProductionInfoId`.
Wall time reduction 0-3%, peak rss neutral.
`ParseItemSetEntry` stores a `LookaheadSetId` into a `LookaheadSetPool` instead
of an owned `TokenSet`. Ids are canonical, so entry hash/eq/clone are integer
ops and state dedup stops walking set words. Unions and single-token inserts
are memoized by id, so the transitive closure and successor-kernel construction
stop re-materializing the same unions per state. Closure additions carry interned
ids with word-token membership precomputed.
For cpp, 3566 distinct lookahead sets back all 49,992 states. For ruby 2000 sets
for 91,991 states, and rust 646 for 16,796.
Reduces wall time by 10-15%, peak rss by 3-5%.
The state-merge pass's `token_conflicts` scanned every terminal entry of the
other state per non-shared token. Precompute flat, word-aligned bitsets,
`ConflictBits`, so each call is a handful of word ANDs.
Yields a 5-6% wall time reduction on large grammars, and flat to ~1%
loss on small grammers. Peak RSS unchanged.
Every state's kernel set was cloned into `parse_state_info` even though the
identical set already lives as the state-dedup map key at the same index
(state ids are the map's insertion order). `ParseStateInfo` now holds the
preceding-symbol sequences plus the dedup map moved out of the builder,
allowing the report path to read kernels via `get_index`. Drops the
duplicate kernel storage, allowing for a small reduction in wall time
and peak rss (1-3%).
`CoincidentTokenIndex` kept a `Vec<ParseStateId>` per token pair whose
only user was keyword identification asking one question: "do all states
where a given pair coincides also allow the word token?" Track that
directly as a second bitset built in the same pass.
Reduces walltime by ~7% for small grammars (i.e. Go), and 20-25% for
larger grammars (i.e. Rust, cpp). Reduces peak rss by ~20% for smaller
grammars and ~40-50% for larger grammars.
Replace the owned `Rule`-tree grammar with a flat, pooled representation. `Rule`
nodes live in a `RulePool` arena and reference their children and params by
index (`RuleId`). Strings are interned in a `StrPool` and referred to by `StrId`.
Productions are stored as flat `ProductionStep`/`Production` slices indexed per
variable, rather than as nested vectors hanging off each `SyntaxVariable`. This
drops the per-rule heap allocation of the owned-tree form and is the groundwork
for the later table-building optimizations.
`parse_grammar` builds the pool, the prepare passes rewrite nodes in place, and
`prepare_grammar` returns a `PreparedGrammar` bundle that owns the run's `StrPool`
alongside the grammars. `build_tables`, `node_types`, and `render` borrow that
pool and resolve `StrId`s only at the output boundary.
Shows an average ~10% walltime reduction when combined with the previous
commit, with nearly all of the savings in `build_tables`. The `prepare`
stage is also ~70% faster and much more stable, but this contributes
much less to overall generate time. Improving early stages such as
`prepare` will be important for future interactive uses.
Parse-item identity dominates parse-table construction. `ParseItem`'s
`Hash`, `Eq`, and `Ord` walk the entire production on every call,
comparing per-step precedence, aliases, and field names. These are
stored as strings for named precedences and for every alias and field.
Precompute, once per table build, two dense `u32` ids for every
`(production, dot)` slot by ranking the whole slot universe with the
item-content comparator, now extracted as `ItemContent`. Equal-content
slots are adjacent after sorting and the id increments only at content
boundaries, so comparing the `cmp` id reproduces the structural order
exactly (both an ordering and an equality). A second id, `eq_with_syms`,
refines `cmp` by the preceding symbols, which enter equality only for
items with `has_preceding_inherited_fields`.
`ParseItem` now carries its production's key slice, and `Hash`/`Eq`/`Ord`
become integer ops.
On the current string-based production storage this is performance-neutral
in isolation (the up-front ranking pays back the per-comparison savings).
It is the foundation for the following change to a pooled, integer-keyed
production representation, which makes both the ranking and the item
comparisons cheaper integer work.
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
`QueryMatches` and `QueryCaptures` retain query cursor options while they are
being iterated. However, their types did not preserve the lifetime of the
options' progress callback, allowing the callback to be dropped while it
was still referenced by the cursor.
Add an explicit options lifetime to both iterator types and propagate it
through `matches_with_options` and `captures_with_options`. Use a static
options lifetime for iterators created without options.
A SHIFT/REDUCE conflict can bundle several shift interpretations with
different precedences against a single reduce. `handle_conflict` weighed them
with only `shift_is_less` and `shift_is_more`, so a lone lower-precedence
shift set `shift_is_less` and the REDUCE won outright, even when another
interpretation tied the REDUCE in precedence and the REDUCE was declared
right-associative. That tie's associativity should have won by shifting, so a
right-associative rule silently became left-associative as soon as an
unrelated lower-precedence rule was added to the grammar.
Track the equal-precedence case explicitly, and in the reduce-wins branch
shift instead when a tying interpretation exists and the reduce actions are
purely right-associative.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
The early `ts_parser__recover` dispatch for `ERROR_STATE` in
`ts_parser__advance` was dropped in `201b41cf1`. Without it, tokens
with no valid action re-enter `ts_parser__handle_error` per token
and fragment the tree instead of extending one ERROR node. Restore
it to recover the old (pre-0.25) error recovery behavior.
- `--layout <document|line-numbers|fragment>` (default `document`): the
document structure. `document` is a self-contained page wrapping a plain
`<div class="highlight"><pre><code>` block, `line-numbers` adds a line-number
`<table>`, and `fragment` emits only the code markup with no surrounding
`<head>`/`<style>`/`<body>` so it can be embedded in an existing page.
- `--style <classes|inline|minimal>` (default `classes`): how token colors
are applied. `classes` uses `class="..."` spans plus a generated `<style>`,
`inline` bakes colors onto each span so the markup is self-contained, and
`minimal` emits bare classes with no colors (bring your own stylesheet).
Deprecates the `--css-classes` flag.
`Parser::set_logger` boxes the logger callback into a type-erased C
pointer, but `Parser` carries no lifetime parameter. This allows for
two kinds of UB:
- Use-after-free: a logger could capture a non-`'static` borrow, let the
borrowed value drop, and then dangle the next time the parser logged.
- Data race: a logger could capture a `!Send` value such as an `Rc`. The
parser could then be moved to another thread and logged from there
while the original thread still held a clone, racing on the reference
count.
As a fix, we just require that logger is `Send + 'static`. The
alternative here is to attach a lifetime parameter to `Parser`, but this
is highly breaking for what is mostly a debugging utility. As an escape
hatch `set_logger_unchecked` is added to the public API as an `unsafe fn`
to correctly communicate the risks and invariants that must be held.
