mirror of
https://github.com/tree-sitter/tree-sitter.git
synced 2026-09-10 07:36:22 -04:00
b6243a5234
Unicode simple case folding maps two non-ASCII code points onto ASCII
letters: the long s `ſ` (U+017F) onto `s`, and the Kelvin sign `K` (U+212A)
onto `k`. So `regex_syntax` pulls them into any case-insensitive pattern,
which is virtually never intended and has two bad effects:
* such tokens can no longer be extracted as keywords, because they are not
a subset of an ASCII `word` token (#5607)
* a broad class like `[^"]` or `\p{L}` carrying `/i` loses `ſ`/`K`, even
though it legitimately contains them (#5755)
Rather than let `regex_syntax` fold, parse patterns unfolded and fold them
ourselves in `case_fold_ascii_safe`: fold via `regex_syntax`, then drop
`ſ`/`K` only when folding introduced them (they were not already in the
base set). A class that already contains them keeps them.
|
||
|---|---|---|
| .cargo | ||
| .github | ||
| .zed | ||
| crates | ||
| docs | ||
| lib | ||
| test/fixtures | ||
| .dockerignore | ||
| .editorconfig | ||
| .envrc | ||
| .gitattributes | ||
| .gitignore | ||
| build.zig | ||
| build.zig.zon | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CMakeLists.txt | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| flake.lock | ||
| flake.nix | ||
| FUNDING.json | ||
| LICENSE | ||
| Makefile | ||
| Package.swift | ||
| README.md | ||
tree-sitter
Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. Tree-sitter aims to be:
- General enough to parse any programming language
- Fast enough to parse on every keystroke in a text editor
- Robust enough to provide useful results even in the presence of syntax errors
- Dependency-free so that the runtime library (which is written in pure C) can be embedded in any application