mirror of
https://github.com/tree-sitter/tree-sitter.git
synced 2026-09-10 07:36:22 -04:00
b6243a5234
Unicode simple case folding maps two non-ASCII code points onto ASCII
letters: the long s `ſ` (U+017F) onto `s`, and the Kelvin sign `K` (U+212A)
onto `k`. So `regex_syntax` pulls them into any case-insensitive pattern,
which is virtually never intended and has two bad effects:
* such tokens can no longer be extracted as keywords, because they are not
a subset of an ASCII `word` token (#5607)
* a broad class like `[^"]` or `\p{L}` carrying `/i` loses `ſ`/`K`, even
though it legitimately contains them (#5755)
Rather than let `regex_syntax` fold, parse patterns unfolded and fold them
ourselves in `case_fold_ascii_safe`: fold via `regex_syntax`, then drop
`ſ`/`K` only when folding introduced them (they were not already in the
base set). A class that already contains them keeps them.
|
||
|---|---|---|
| .. | ||
| cli | ||
| config | ||
| generate | ||
| highlight | ||
| language | ||
| loader | ||
| tags | ||
| xtask | ||