A previous fix moved case folding for `/i` patterns out of `regex_syntax` and
into `expand_regex`, so folding could drop the two non-ASCII code points
Unicode simple folding maps onto ASCII letters: the long s `ſ` (U+017F)
onto `s`, and the Kelvin sign `K` (U+212A) onto `k`. Left in, they leak
into otherwise-ASCII tokens and stop those tokens from being extracted as
keywords.
By that point, though, the HIR has already turned a negated class into a
complement, so folding it applies the fold on the wrong side of the
negation. `(?i)[^a-z]` folds a set that contains `A-Z`, which re-admits
`a-z` and leaves a class matching very nearly everything.
`regex_syntax` folds each leaf of a class expression before applying that
leaf's negation and the set algebra above it. Keep that order and change
only the fold: walk the AST, replace each leaf with its fold, and translate
with `case_insensitive(false)`.
zero quantifier skip is performed.
The zero-skip branch currently sets skipped_quantifier unconditionally.
That flag is correct only when the quantified step and its skip target
are _siblings_ at the same query depth. Otherwise, setting it allows for
a "leak" and disables unrelated, "outer" anchors.
(cherry picked from commit 42f33fe2f8)
`ts_tree_cursor_current_status`
By terminating early on `has_later_siblings`, `has_later_named_siblings`
was incorrectly reported as `false` in some cases. This led to the
execution of some queries to terminate early.
Also remove some dead branches inside `ts_tree_cursor_current_status`.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
(cherry picked from commit 308aee0c90)
A SHIFT/REDUCE conflict can bundle several shift interpretations with
different precedences against a single reduce. `handle_conflict` weighed them
with only `shift_is_less` and `shift_is_more`, so a lone lower-precedence
shift set `shift_is_less` and the REDUCE won outright, even when another
interpretation tied the REDUCE in precedence and the REDUCE was declared
right-associative. That tie's associativity should have won by shifting, so a
right-associative rule silently became left-associative as soon as an
unrelated lower-precedence rule was added to the grammar.
Track the equal-precedence case explicitly, and in the reduce-wins branch
shift instead when a tying interpretation exists and the reduce actions are
purely right-associative.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
(cherry picked from commit 0900f84eab)
The early `ts_parser__recover` dispatch for `ERROR_STATE` in
`ts_parser__advance` was dropped in `201b41cf1`. Without it, tokens
with no valid action re-enter `ts_parser__handle_error` per token
and fragment the tree instead of extending one ERROR node. Restore
it to recover the old (pre-0.25) error recovery behavior.
(cherry picked from commit 15ea3328e1)
`(P . Q* Y)` with zero `Q` dropped the leading `.`, so `Y` matched at any
position instead of being pinned to the parent's first named child. On a `?`/`*`
zero-skip, when the skipped step is the parent's first child and carries a
leading anchor, transfer that first-child requirement to the skip target.
(cherry picked from commit 1ffd612be5)
on both sides
In `A . Q* . B`, a zero-matched `Q` made both anchors vacuous, so `A` and `B`
were no longer required to be immediate siblings. Only relax the following
anchor on a `?`/`*` zero-skip when the skipped step has no leading anchor of
its own; otherwise the adjacency transfers through the empty run.
(cherry picked from commit dfcf73921c)
Unicode simple case folding maps two non-ASCII code points onto ASCII
letters: the long s `ſ` (U+017F) onto `s`, and the Kelvin sign `K` (U+212A)
onto `k`. So `regex_syntax` pulls them into any case-insensitive pattern,
which is virtually never intended and has two bad effects:
* such tokens can no longer be extracted as keywords, because they are not
a subset of an ASCII `word` token (#5607)
* a broad class like `[^"]` or `\p{L}` carrying `/i` loses `ſ`/`K`, even
though it legitimately contains them (#5755)
Rather than let `regex_syntax` fold, parse patterns unfolded and fold them
ourselves in `case_fold_ascii_safe`: fold via `regex_syntax`, then drop
`ſ`/`K` only when folding introduced them (they were not already in the
base set). A class that already contains them keeps them.
Regexes with the case-insensitive 'i' flag caused unicode simple case folding,
which maps two non-ASCII code points onto ASCII letters:
- `ſ` (U+017F) onto `s`
- the Kelvin sign `K` (U+212A) onto `k`
This pushed such tokens outside an ASCII `word` token, so they failed to
extract as keywords.
(cherry picked from commit 07c4ed220e)
Problem: A set of `array_*` macros (`array_push`, `array_extend`, etc) implicitly convert a `void*` into a different pointer type. In environments that compile these headers as C++, this implicit conversion is an error.
Solution: This commit adds an `_array_cast` macro that uses `decltype` to cast the `void*` to the proper type when compiling as C++.
(cherry picked from commit cc7be1fd47)
- An anchor between two patterns is vacuous when an adjacent quantifier
matches zero
- A leading or trailing anchor on a node applies to the nearest matched node
when an adjacent optional is skipped
- An anchor at the edge of a group or alternation is not allowed.
(cherry picked from commit 133d549b70)
A trailing `.` after an optional node, e.g. `(p (a)+ @a . (b)? @b .)`, sets
`is_last_child` on the `(b)?` step. When `(b)?` matched zero, the zero-skip
jumped past that step to completion, so the last-child requirement was never
enforced.
When a zero-skip would bypass a step carrying `is_last_child`, require that
the last matched node really is the last named child. Gated on the
`alternative_is_skip` edge marker.
(cherry picked from commit a8486ca239)
A `.` anchor between a quantified pattern and a following node, e.g.
`(parent (comment)* @c . (decl))`, was treated as a leading anchor when the
quantifier matched zero. The zero-skip jumps a state directly onto the anchored
step, where `is_immediate` was enforced even though no sibling had matched before
it. Name that skip edge (`alternative_is_skip`) and, when a state follows it,
record that it skipped the quantifier (`skipped_quantifier`). The
immediate anchor is then supressoed for that state's next match.
(cherry picked from commit 7e7f2584d1)
A `.` at the end of a group, such as `((comment)+ @c .)`, was silently dropped
during compilation. Reject rather than dropping.
(cherry picked from commit 966dc52557)
Extract the step dump into `ts_query__dump_steps` and enrich it with the control-flow
fields and analysis annotations.
(cherry picked from commit 6bfbd9d84d)
An unanchored quantified sibling like `(program (comment)+ @doc (class))`
keeps one match state per matching sibling. A recent fix stopped over-pruning
them, but the per-node longest-match dedup is pairwise, so with n live states
matching became O(n^3).
Two states can only be capture subsets of one another if their captured
byte ranges overlap, so keep each group ordered by first-capture position
and stop the pairwise scan once the rest of the group is disjoint.
Refs neovim/neovim#40517
(cherry picked from commit 91fd04f96a)
Problem: `zig fetch` on 0.16 can't download zip archives, which wasmtime
uses for Windows. This makes `cargo xtask upgrade-wastime` fail silently
with empty hashes for these platforms.
Solution: Re-run xtask with Zig 0.17 nightly.
(cherry picked from commit ae2081989e)
Problem: In queries matching a parent node with anchored children
(siblings) where the first sibling has a quantifier and is not
captured, the check for fallible steps skips splitting the state because
the next node is a passthrough node (and not an is_immediate one). This
prevents the query from matching beyond the first occurrence.
Solution: In the check for fallible steps, skip the next steps if they
are passthrough steps.
(cherry picked from commit 99bceec689)
Problem: In queries matching a parent node with anchored children
(siblings) where the first anchored sibling has a quantifier, the
deduplication logic for states is overly aggressive. The logic causes
the state split on the first capture (with the parent) to be dropped in
favor of the loopback state. This results in the query not being able to
match beyond the first occurrence.
Solution: Prevent the deduplication logic from dropping a state that is
not seeking an immediate match for one that is seeking an immediate
match, since the one not seeking for an immediate match could still
match later nodes.
(cherry picked from commit 8b43f42dca)
Problem: In queries matching a parent node with anchored children
(siblings), the check for fallible steps only considers nodes with
children, which results in no state split being made for anchored
siblings (node1) . (node2). This prevents the query from matching beyond
the first occurrence.
Solution: Make the check for fallible steps consider also the case where
the next step is at the same depth and must be matched immediately
after.
(cherry picked from commit 1b110f62dd)
The early exit condition to skip remaining highlights was incorrectly
checking the lexicographical order, which could lead to false
passes if a highlight was on a later row with smaller column number.
parser can reuse a node.
Lookahead bytes can be used to decide what a node is parsed as, so it's
resaonable to consider this as part of a node's "range" when deciding
which edits affect it.
(cherry picked from commit 15ddfb21ed)
