libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-10 07:26:23 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
6541 commits 19 branches 114 tags 51 MiB
Commit graph

136 commits

Author SHA1 Message Date
Nat Mote 0113bd28a4
Test for unused extra alias 
Tests the issue described in #1834
 2022-08-10 07:25:48 -07:00
Kian-Meng Ang b8552ec6c4 Fix typos 2022-06-28 19:57:42 +08:00
Max Brunsfeld 04381dcea3 Add more python error recovery tests 2022-06-24 19:07:40 -07:00
Max Brunsfeld d223a81b50 Allow empty external tokens during err recovery if they change the scanner's state 2022-06-24 15:58:13 -07:00
Max Brunsfeld 0fb864c1a0 Retain information about the lexer's lookahead for the token where an error was detected 2022-02-22 09:45:26 -08:00
Max Brunsfeld 584b55df8d Delete unused code, tweak whitespace 2022-01-19 16:54:57 -08:00
Andrew Helwer e1ee261181 Changed decimal unicode codepoint to hex 2022-01-11 19:15:36 -05:00
Andrew Helwer 69ff091a87 Added includes for macos 2022-01-11 12:31:41 -05:00
Andrew Helwer 5a6530a413 Added tests 2022-01-11 12:05:37 -05:00
Max Brunsfeld 8df0b8de7e Convert more fixture grammars from JSON to JS 2021-12-30 16:27:02 -08:00
Max Brunsfeld 52b4b65adc Convert more test grammars from JSON to JS 2021-11-19 10:35:53 -08:00
Max Brunsfeld d05c665863 Convert some of the fixture grammars from JSON to JS 
These tests are easier to write and maintain if the grammars are just JS,
like grammars normally are. It doesn't slow the tests down significantly
to shell out to `node` for each of these grammars.
 2021-10-22 18:47:23 -06:00
Andrew Helwer 0dc1bd806f Fixed some test failures due to CRLF line endings 2021-09-24 22:42:38 -04:00
FnControlOption e030434ca7 Handle aliases in unicode property escapes in regexes 2021-08-18 22:22:46 -07:00
Max Brunsfeld a40045a419 When editing, properly invalidate trees that depend on get_column 2021-03-11 14:46:13 -08:00
Max Brunsfeld e29d3714f7 Fix behavior of Lexer.get_column when at EOF 2021-03-11 12:11:33 -08:00
Max Brunsfeld dd4cba2625 Allow symbols to be used in precedence lists 2021-03-03 13:11:05 -08:00
Max Brunsfeld 344797c110 Implement named precedence comparison 2021-02-24 16:02:56 -08:00
Max Brunsfeld 9abad258c3 Expand tests for unicode property escapes 2021-02-17 13:29:30 -08:00
Max Brunsfeld e3ba701344 Start work on handling unicode property escapes in regexes 2021-01-29 16:37:45 -08:00
Max Brunsfeld 391fc8c340 Update unit tests to expect ruby grammar change 2020-12-20 19:34:45 -08:00
Max Brunsfeld 94c61de353 Update JS error recovery test to reflect grammar change 2020-12-03 10:09:58 -08:00
Max Brunsfeld 4e86b76e8c Update ruby error recovery test to reflect grammar tweaks 2020-11-03 10:28:17 -08:00
Max Brunsfeld a8d77001c2 Update c error recovery test to reflect behavior change 2020-09-21 13:20:50 -07:00
Max Brunsfeld 9a7fdd29c2 Add test for non-terminal extras that share non-extra rules 2020-07-29 09:53:07 -07:00
Max Brunsfeld 253f23c3d4 Fix error when parse error occurs after non-terminal extra 2020-07-28 13:33:13 -07:00
Max Brunsfeld 3814babfa8 Update error recovery test to reflect JS grammar changes 2020-01-28 09:16:36 -08:00
Max Brunsfeld 0cb2ef1082 Fix code paths that still conflated null characters with EOF 2019-12-06 15:29:03 -08:00
Max Brunsfeld 660efd8c0f Fix handling of token rules directly inside of alias, prec, or field. 
Fixes #483
 2019-11-15 10:53:17 -08:00
Max Brunsfeld 39ceff1908 Restore json corpus tests, which accidentally weren't running 2019-10-31 14:32:50 -07:00
Max Brunsfeld fcaabea0cf Allow non-terminal extras 2019-10-21 16:08:59 -07:00
Max Brunsfeld 0955c5b3d7 Handle named nodes aliased as anonymous nodes 
Fixes #401
 2019-08-29 14:28:44 -07:00
Max Brunsfeld 5e04daf483 Avoid non-deterministic set iteration order when handling conflicts 2019-08-13 15:57:42 -07:00
Max Brunsfeld 443ac1c1e5 Update python tests to reflect grammar changes 2019-07-22 09:55:30 -07:00
Max Brunsfeld c90a532d8f Update tests to reflect javascript grammar changes 2019-07-10 14:30:43 -05:00
Max Brunsfeld bc5e71bcc2 Fix handling of zero-width external tokens at beginning of document 
Fixes #316
 2019-04-08 15:01:52 -07:00
Max Brunsfeld 6490b1e4f2 Fix non-deterministic conflict message order 2019-03-29 17:36:19 -07:00
Max Brunsfeld 65d1ce8593 lib: Include fields in ts_node_string output 
This allows you to assert about fields in tests. But if your test 
s-expression does *not* include fields, the fields will be stripped from 
the regexp before comparison.
 2019-02-13 09:47:21 -08:00
Max Brunsfeld e579e09569 Ensure interpretations are in a predictable order in conflict messages 2019-02-08 17:11:44 -08:00
Max Brunsfeld 6105bf9909 Include error recovery examples in test suite 2019-01-20 16:58:49 -08:00
Max Brunsfeld 5c3c1dd0bd Get logging flags working properly with test script 2019-01-15 13:21:48 -08:00
Max Brunsfeld 6592fdd24c Fix parser generation error messages 2019-01-11 17:26:45 -08:00
Max Brunsfeld e64f7a64a1 Start work on running test corpus tests 2019-01-11 13:31:06 -08:00
Max Brunsfeld 00a94ac040 Fix javascript error fixture 2018-10-03 21:38:03 -07:00
Max Brunsfeld 7fc64ed25a Account for simple aliases in unit reduction elimination 2018-08-30 09:57:22 -07:00
Max Brunsfeld 41fe564a90 Update error recovery fixture 2018-08-01 15:09:45 -07:00
Max Brunsfeld cb784975a4 Add IMMEDIATE_TOKEN rule type, for enforcing no preceding extras 2018-08-01 14:00:57 -07:00
Max Brunsfeld 6ebb9195b1 Flesh out integration test for precedence within tokens 2018-07-31 10:18:49 -07:00
Max Brunsfeld 23e4596ec1 Add test for handling of precedence within tokens 2018-07-26 17:06:09 -07:00
Max Brunsfeld 379a2fd121 Incrementally build a tree of skipped tokens 
Rather than pushing them to the stack individually
 2018-04-09 12:29:22 -07:00
Max Brunsfeld 80f856cef5 Maintain a total node count on every tree 
This simplifies (and fixes bugs in) the parse stack's tracking of its
total node count since the last error, which is needed for error
recovery.
 2018-04-06 13:26:18 -07:00
Max Brunsfeld e59558c83b Allow stack versions to be temporarily paused 
This way, when detecting an error, we can defer the decision about
whether to bail or recover until all stack versions are processed.
 2018-04-06 13:26:18 -07:00
Max Brunsfeld 7183f8d3e7 Fix unit reduction elimination bugs 
* Handle 'chains' of unit reductions starting in a single state
* Avoid eliminating rules which will later receive aliases
 2018-03-12 07:54:18 -07:00
Max Brunsfeld df2430b94c Remove a C error recovery test temporarily 2018-03-08 14:18:12 -08:00
Max Brunsfeld 2daae48fe0 Handle conflicts in repeat rules after external tokens 
Signed-off-by: Rick Winfrey <rewinfrey@github.com>
 2018-02-14 11:24:51 -08:00
Max Brunsfeld 2e4f76c164 Don't allow an epsilon start rule if it is used in other rules 2018-01-23 17:05:28 -08:00
Max Brunsfeld 13adfe4927 Update error corpus 2017-12-29 16:21:04 -08:00
Max Brunsfeld e5851fd9b9 Don't use non-existent \a syntax in test grammars 2017-12-13 12:21:28 -08:00
Max Brunsfeld f426b61e7c Fix expectation around preproc directive in C error test 2017-12-13 12:21:13 -08:00
Max Brunsfeld fbcefe25f7 Avoid creating external tokens that start after they end 2017-12-07 11:50:27 -08:00
Max Brunsfeld c0073c5b72 Update error corpus to reflect C grammar changes 2017-10-04 15:06:12 -07:00
Max Brunsfeld d291af9a31 Refactor error comparisons 
* Deal with mergeability outside of error comparison function
* Make `better_version_exists` function pure (don't halt other versions
as a side effect).
* Tweak error comparison logic

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
 2017-09-13 16:38:15 -07:00
Max Brunsfeld 99d048e016 Simplify error recovery; eliminate recovery states 
The previous approach to error recovery relied on special error-recovery
states in the parse table. For each token T, there was an error recovery
state in which the parser looked for *any* token that could follow T.
Unfortunately, sometimes the set of tokens that could follow T contained
conflicts. For example, in JS, the token '}' can be followed by the
open-ended 'template_chars' token, but also by ordinary tokens like
'identifier'. So with the old algorithm, when recovering from an
unexpected '}' token, the lexer had no way to distinguish identifiers
from template_chars.

This commit drops the error recovery states. Instead, when we encounter
an unexpected token T, we recover from the error by finding a previous
state S in the stack in which T would be valid, popping all of the nodes
after S, and wrapping them in an error.

This way, the lexer is always invoked in a normal parse state, in which
it is looking for a non-conflicting set of tokens. Eliminating the error
recovery states also shrinks the lex state machine significantly.

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
 2017-09-11 15:22:52 -07:00
Max Brunsfeld 94dc703bfc Require that grammars' start rules be visible 2017-08-04 17:07:37 -07:00
Max Brunsfeld 1dca3a0b58 Simplify parse version reordering 2017-08-04 14:51:14 -07:00
Max Brunsfeld e5c3bf742d Update fixture grammars 2017-08-03 16:32:39 -07:00
Max Brunsfeld 84e4114f79 Allow conflicts involving repeat rules to be whitelisted, via their parent rule 2017-08-03 15:18:29 -07:00
Max Brunsfeld 119c67dd78 Fix conflict reporting for shift/reduce conflicts w/ multiple reductions 
We were failing to rule out shift actions with lower precedence.

Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-08-02 15:13:30 -07:00
Max Brunsfeld 09f4796f6b Get tests passing w/ new alias API 2017-08-01 14:35:34 -07:00
Max Brunsfeld cb5fe80348 Rename RENAME rule to ALIAS, allow it to create anonymous nodes 2017-07-31 16:41:11 -07:00
Max Brunsfeld cbdfd89675 Mark reductions as fragile based on their final properties 
We previously maintained a set of individual productions that were
involved in conflicts, but that was subtly incorrect because
we don't compare productions themselves when comparing parse items;
we only compare the parse items properties that could affect the
final reduce actions.
 2017-07-21 09:54:24 -07:00
Max Brunsfeld 7d9d8bce79 Handle inlined rules that contain other inlined rules 2017-07-20 15:29:06 -07:00
Max Brunsfeld 45c40c8742 Update test grammars to use new serialization API 2017-07-17 17:46:46 -07:00
Max Brunsfeld 4b40a1ed6c Support anonymous tokens inside of RENAME rules 2017-07-14 10:19:58 -07:00
Max Brunsfeld b3a72954ff Introduce RENAME rule type 2017-07-13 17:17:22 -07:00
Max Brunsfeld e4f57d6fee Test more cases in fixture grammar with inline rules 2017-07-12 10:12:42 -07:00
Max Brunsfeld 65bf1389e1 Add a way to automatically inline rules 2017-07-11 23:13:44 -07:00
Max Brunsfeld a98abde529 Provide all preceding symbols as context when reporting conflicts 2017-07-07 14:52:56 -07:00
Max Brunsfeld d8e9d04fe7 Add PREC_DYNAMIC rule for resolving runtime ambiguities 2017-07-06 15:24:45 -07:00
Max Brunsfeld 8f028ebf68 Avoid deep tree comparison when both trees have errors 2017-07-05 17:33:35 -07:00
Max Brunsfeld 009d6d1534 Improve heuristics for pruning parse versions based on errors 
* Rewrite the error cost comparison in terms of explicit, discrete
conditions.
* Allow merging versions have different error costs.
* Store the depth of each stack version since the last error. Use this
state to prevent incorrect merging.
* Sort the stack versions in order of preference and put a hard limit on
the version count.
 2017-06-29 15:00:20 -07:00
Max Brunsfeld 6db12ab44e Adjust C error recovery tests to reflect grammar changes 2017-06-21 16:58:40 -07:00
Max Brunsfeld af553420bf Update JS error corpus 2017-03-19 22:21:12 -07:00
Max Brunsfeld ed8fbff175 Allow anonymous tokens to be used in grammars' external token lists 2017-03-17 16:31:29 -07:00
Max Brunsfeld f04d7c5860 Handle unused tokens 2017-03-09 21:16:37 -08:00
Max Brunsfeld 6dc0ff359d Rename spec -> test 
'Test' is a lot more straightforward of a name.
 2017-03-09 20:40:01 -08:00