mirror of
https://github.com/tree-sitter/tree-sitter.git
synced 2026-09-10 07:36:22 -04:00
e4ac513afd
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
(cherry picked from commit
|
||
|---|---|---|
| .cargo | ||
| .github | ||
| .zed | ||
| crates | ||
| docs | ||
| lib | ||
| test/fixtures | ||
| .dockerignore | ||
| .editorconfig | ||
| .envrc | ||
| .gitattributes | ||
| .gitignore | ||
| build.zig | ||
| build.zig.zon | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CMakeLists.txt | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| flake.lock | ||
| flake.nix | ||
| FUNDING.json | ||
| LICENSE | ||
| Makefile | ||
| Package.swift | ||
| README.md | ||
tree-sitter
Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. Tree-sitter aims to be:
- General enough to parse any programming language
- Fast enough to parse on every keystroke in a text editor
- Robust enough to provide useful results even in the presence of syntax errors
- Dependency-free so that the runtime library (which is written in pure C) can be embedded in any application