Daniel Jalkut
0f6780b9a3
lexer: pass code-unit length to U16_NEXT in UTF-16 decoders
...
`ts_decode_utf16_le` and `ts_decode_utf16_be` passed a byte length to
`U16_NEXT_LE` and `U16_NEXT_BE`, but those macros count `uint16_t` code
units. If a lead surrogate was the last code unit in a chunk, the decoder
could peek past the chunk and combine it with adjacent memory.
This commit passes the code-unit length to the UTF-16 macros, and fails
with `TS_DECODE_ERROR` when a chunk is too short to contain even one full
code unit. This lets the lexer retry with a fresh chunk or advance through
the invalid byte as it already does.
Co-authored-by: Will Lillis <will.lillis24@gmail.com>
Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-04-26 03:38:47 -04:00
Will Lillis
d6e33d0722
test: bump c test fixture to v0.24.2
2026-04-23 03:31:00 -04:00
Will Lillis
337e20e0f2
fix(wasm): when reallocating the last allocated region, properly grow
...
the region in place.
Previous changes to `malloc` caused `realloc` to sometimes pull regions
off of the free list during this optimization. Because no `memcpy` is
performed, this resulted in corrupted data returning to the caller.
Co-authored-by: trim21 <i@trim21.me>
2026-01-31 18:24:31 -05:00
Will Lillis
8fe076695f
test: rename wasm corpus test "wasm_realloc"->"wasm_realloc_overflow_heap"
2026-01-31 18:24:31 -05:00
Will Lillis
4a2b8ed299
fix(generate): error if supertype is defined as a terminal rule
2026-01-30 22:35:41 -05:00
Will Lillis
634ba3a1f7
fix(wasm): correct several bugs in realloc
...
- free memory if 0 size is passed in
- Don't `memcpy` contents if new pointer is `NULL`
- Copy old region's contents only up to size of new region
- free old region
Co-authored-by: trim21 <i@trim21.me>
2026-01-26 21:14:14 -05:00
Will Lillis
55fdc50e81
feat(xtask): allow alternate branch for fixture grammars
...
Set tree-sitter-php fixture to `upstream_test_fixture` branch.
Also remove the `--update` flag, as it does nothing.
2026-01-24 14:18:40 -05:00
Nia
f09dc3cf46
fix(wasm): fix alias map size computation
...
This fixes a crash where parsing with certain languages can lead to a crash due to how the alias map was allocated and laid out in wasm memory
2025-09-18 18:34:27 -04:00
Amaan Qureshi
39a67eec61
feat: migrate to ESM
2025-09-16 02:24:11 -04:00
Will Lillis
5fd818babe
fix(generate): use correct state id when adding terminal states to
...
non terminal extras
2025-09-04 04:15:51 -04:00
Will Lillis
310c0b86a7
fix(generate): return error when single state transitions have
...
indirectly recursive cycles.
This can cause infinite loops in the parser near EOF.
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
2025-09-04 00:59:44 -04:00
Amaan Qureshi
6cb9486b28
test(fixtures): bump javascript grammar to 0.25.0
2025-09-04 00:59:01 -04:00
Will Lillis
62f93e221d
feat(test): pin fixture grammars to specific commits
2025-08-31 01:05:42 -04:00
Amaan Qureshi
b7f36a13ba
fix(lib): check if an ERROR node is named before assuming it's the builtin error node
2025-08-28 17:13:57 -04:00
Max Brunsfeld
f91255a201
Fix crash w/ goto_previous_sibling when parent node has leading extra child ( #4472 )
...
* Fix crash w/ goto_previous_sibling when parent node has leading extra
child Co-authored-by: Smit Barmase <heysmitbarmase@gmail.com>
Co-authored-by: Smit Barmase <heysmitbarmase@gmail.com>
* Fix lint
Co-authored-by: Smit Barmase <heysmitbarmase@gmail.com>
---------
Co-authored-by: Smit Barmase <heysmitbarmase@gmail.com>
2025-05-27 16:56:33 -07:00
Max Brunsfeld
dedcc5255a
Ignore external tokens that are zero-length and extra ( #4213 )
...
Co-authored-by: Anthony <anthony@zed.dev>
2025-02-17 15:07:44 -08:00
WillLillis
dcfc95e563
fix(cli): only remove test input's trailing '\r' if running on windows
...
CI / checks (push) Waiting to run
CI / sanitize (push) Failing after 14s
CI / build (push) Failing after 15s
platform
2025-01-11 17:52:54 -05:00
Amaan Qureshi
5a2c5ed865
docs: improve docs for contributing to docs
2025-01-10 22:57:54 -05:00
Amaan Qureshi
68e707eb4f
feat: support passing in a Rust regex in the grammar dsl
CI / checks (push) Waiting to run
CI / sanitize (push) Failing after 14s
CI / build (push) Failing after 16s
Deploy Docs / deploy-docs (push) Has been cancelled
2025-01-06 20:52:19 -05:00
Max Brunsfeld
201b41cf11
feat: add 'reserved word' construct
...
CI / sanitize (push) Failing after 13s
CI / build (push) Failing after 14s
Check Bindgen Output / check-bindgen (push) Has been cancelled
CI / checks (push) Has been cancelled
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
2024-12-23 03:06:32 -05:00
Will Lillis
5d1be545c4
fix(lib): correct next sibling of zero width node
2024-11-12 18:17:45 -05:00
Amaan Qureshi
538a197976
fix(lib): correct unexpected side effect in get_column when the lexer is at EOF
2024-10-08 23:27:42 -04:00
Amaan Qureshi
cad2d03101
chore: remove unnecessary fuzz and profile helpers
2024-10-08 21:07:41 -04:00
Ron Panduwana
a83b893016
fix: handle more cases of editing subtrees that depend on column values
2024-09-29 20:36:08 -04:00
Ron Panduwana
2bb20fe2fe
feat: allow external scanners to use the logger
...
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
2024-08-17 14:46:28 -04:00
Amaan Qureshi
6e6dcf1caf
test: modernize scanner files
2024-05-05 18:31:53 -04:00
Amaan Qureshi
627617edb4
refactor(tests): migrate remaining grammar.json tests to grammar.js
2024-04-29 00:01:35 -04:00
dundargoc
bf9154febe
docs: small fixes
...
Co-authored-by: Wang <bonede@qq.com>
Co-authored-by: Sebastiaan Speck <12570668+sebastiaanspeck@users.noreply.github.com>
2024-02-16 00:15:49 +01:00
Amaan Qureshi
9f2dc9d6b5
fix: rework parser.h includes for test grammars and multi-grammar repos
2024-02-02 10:42:39 -05:00
Amaan Qureshi
fadfa19be8
chore: update javascript error trees
2024-02-02 03:06:38 -05:00
Max Brunsfeld
360059eca1
Avoid a '../' #include in test scanner
2023-11-04 12:26:00 -07:00
Andrew Hlynskyi
5e62120050
chore: fix local fixture test for C language
2023-10-16 21:45:40 +03:00
Amaan Qureshi
ef9cabd4b5
fix: update javascript tests and use cpp/javascript master for fixtures
2023-09-20 11:31:53 -04:00
Amaan Qureshi
a4ea4737ac
fix: do not increment current_included_range_index past included_range_count in __do_advance
2023-08-27 14:16:18 +03:00
Amaan Qureshi
f9e5696bcb
ci: rework fuzzer script
2023-07-24 00:44:44 -04:00
Amaan Qureshi
fc0e3a34ba
test: update C tests from upstream
2023-07-15 20:07:46 -04:00
Andrew Helwer
89edb2ddca
fix(fuzzer): only use rule strings for fuzz dictionary
...
The `find_literals` function can also pick up tokens in `precedences`
2023-07-10 19:12:24 -04:00
Amaan Qureshi
b192200008
fix: update tests from python grammar changes
2023-07-10 14:40:18 -04:00
Max Brunsfeld
6b16a9687b
Tweak ruby error test to reflect slightly different recovery
2023-06-12 13:45:43 -07:00
Andrew Hlynskyi
85a588b5b0
Fix tests, adapt to tree-sitter-json 40a81c01a40ac48744e0c8ccabbaba1920441199
2023-04-22 12:08:50 +03:00
Max Brunsfeld
25d9c989eb
Update python error corpus to reflect grammar changes
2023-03-13 14:25:24 -07:00
Max Brunsfeld
8e3dc7cd7a
Add tests that randomly edit files with disjoint included ranges
2022-11-14 16:04:37 -08:00
Max Brunsfeld
6b87326470
Merge pull request #1787 from kianmeng/fix-typos
...
Fix typos
2022-08-25 10:25:39 -07:00
Nat Mote
4e3179fbc0
Avoid extracting default alias for extras
...
Fixes #1834
2022-08-10 07:27:34 -07:00
Nat Mote
0113bd28a4
Test for unused extra alias
...
Tests the issue described in #1834
2022-08-10 07:25:48 -07:00
Kian-Meng Ang
b8552ec6c4
Fix typos
2022-06-28 19:57:42 +08:00
Max Brunsfeld
04381dcea3
Add more python error recovery tests
2022-06-24 19:07:40 -07:00
Max Brunsfeld
d223a81b50
Allow empty external tokens during err recovery if they change the scanner's state
2022-06-24 15:58:13 -07:00
Max Brunsfeld
0fb864c1a0
Retain information about the lexer's lookahead for the token where an error was detected
2022-02-22 09:45:26 -08:00
Max Brunsfeld
584b55df8d
Delete unused code, tweak whitespace
2022-01-19 16:54:57 -08:00
