libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-13 08:56:23 -04:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity
5714 commits 19 branches 114 tags 51 MiB
Commit graph

547 commits

Author SHA1 Message Date
Max Brunsfeld 0e02ead0de Update tags test to reflect new handling of escapes in queries 2020-03-10 15:53:42 -07:00
Max Brunsfeld 4531130b44 Merge branch 'master' into tags 2020-03-10 15:50:27 -07:00
Max Brunsfeld 741eed01b7 query: Handle escape sequences and escaped quotes in string literals 2020-03-10 15:50:06 -07:00
Max Brunsfeld fc4d5c3a33 🔥 Dead test code 2020-03-10 12:23:13 -07:00
Max Brunsfeld 17cc38678c Get generate_tags with the new iterator API 2020-03-10 12:05:09 -07:00
Max Brunsfeld 0eb162c685 wip: converting generate_tags to return an iterator 2020-03-10 11:45:31 -07:00
Max Brunsfeld 157258d881 tags: Implement strip regex for docs processing 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-10 10:43:23 -07:00
Max Brunsfeld 7f4828254f Fix criteria for detecting when an aborted parse is resuming 2020-03-09 11:30:08 -07:00
Patrick Thomson 00dcc1eaa6 Need to use expression_statement here. 2020-03-06 17:48:55 -05:00
Max Brunsfeld 680a9e0531 wip 2020-03-06 13:24:03 -08:00
Max Brunsfeld a3f0087b11 Start work on tagging unit test 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-05 13:04:49 -08:00
Max Brunsfeld ee46218a73 Fix incremental parsing problem with non-terminal extras 
Also add PHP grammar as a fixture to test against.
 2020-03-02 14:17:12 -08:00
Max Brunsfeld 570b83e2b2 query: Add immediate child operator 2020-02-19 11:47:52 -08:00
Max Brunsfeld 950a89a525 query: Differentiate between wildcard '*' and named wildcard '(*)' 2020-02-19 09:42:29 -08:00
Max Brunsfeld de8e3ee188 query: Allow multiple captures on a single node 2020-02-11 16:02:32 -08:00
Max Brunsfeld e23f518915 highlight: add built-in support for carriage-return highlight 2020-01-28 14:47:21 -08:00
Max Brunsfeld d06407aca2 Update highlight test for JSDoc grammar changes 2020-01-27 13:08:30 -08:00
Max Brunsfeld 3f109a3cb5 highlight: Fix logic for handling empty injections with no highlights 2020-01-27 12:32:37 -08:00
Max Brunsfeld 9f63139a10 Fix error when set_included_ranges is called with an invalid range list 2020-01-17 10:31:28 -08:00
Max Brunsfeld 9dfd03e79a highlight: Sipmlify injection API w/ new injection.combined property 2020-01-16 12:43:31 -08:00
Max Brunsfeld f3747863df Add ts_query_disable_pattern API 2020-01-15 17:08:55 -08:00
Max Brunsfeld c2e253a4c8 Update unit test to reflect javascript query change 2019-12-10 13:24:36 -08:00
Max Brunsfeld 0cb2ef1082 Fix code paths that still conflated null characters with EOF 2019-12-06 15:29:03 -08:00
Max Brunsfeld d426f46087 Add unit test for symbols respecting simple aliases 2019-12-06 12:47:27 -08:00
Max Brunsfeld d6c7b243a7
Merge pull request #499 from tree-sitter/highlight-test 
Add a system for testing syntax highlighting queries
 2019-12-05 15:55:24 -08:00
Max Brunsfeld 4b9feccd13 Run syntax highlighting tests as part of the main test command 
Also, allow `corpus` to be in the `test/corpus` directory, and expect 
highlighting tests to be in the `test/highlight` directory.
 2019-12-05 15:28:16 -08:00
Max Brunsfeld 6dbfbaed07 Start work on a system for testing syntax highlighting 2019-12-05 09:45:54 -08:00
Max Brunsfeld c957a5b8e7 Add test for handling of too many states in query captures 2019-11-22 11:54:32 -08:00
Max Brunsfeld 1909ae2cee binding_rust: Improve node children APIs 
Closes #480
 2019-11-08 14:20:10 -08:00
Max Brunsfeld 39ceff1908 Restore json corpus tests, which accidentally weren't running 2019-10-31 14:32:50 -07:00
Max Brunsfeld a62b7a70f3 Lexer: track EOF state without relying on null character as lookahead 2019-10-31 14:11:52 -07:00
Max Brunsfeld e14e285a10 cli: Check queries when running tree-sitter test 2019-10-18 14:44:16 -07:00
Max Brunsfeld 64c6cf4473 Implicitly reset parser's state if language is changed after a timeout 2019-10-18 11:28:59 -07:00
Max Brunsfeld 084406148b Tweak highlight name matching semantics, add doc comments 2019-10-17 12:03:34 -07:00
Max Brunsfeld b79295e1a0 🔥 Everything related to property sheets 2019-10-17 11:00:31 -07:00
Max Brunsfeld fa43ce01a6 Allow queries to capture ERROR nodes 2019-10-16 11:54:32 -07:00
Max Brunsfeld 060e00463d Implement include-children directive in injection queries 2019-10-14 17:38:42 -07:00
Max Brunsfeld b3809274f0 Load highlight queries correctly in highlight unit tests 2019-10-14 17:24:16 -07:00
Max Brunsfeld 324c259cbb rust: Create readable messages for query syntax errors 2019-10-14 12:30:22 -07:00
Max Brunsfeld 4c17af3ecd Allow queries with no patterns 2019-10-14 12:30:22 -07:00
Max Brunsfeld f4903578f8 Start reimplementing highlight crate with tree queries 2019-10-14 12:30:22 -07:00
Max Brunsfeld 02b6397aa3 Add test for advancing query cursor after has already finished 2019-10-03 12:55:10 -07:00
Max Brunsfeld 9872a083b7 rust: Change QueryCursor::captures to expose the full match 2019-10-03 12:45:58 -07:00
Max Brunsfeld 27149902f8 Handle is?, is-not?, and set! predicate functions in queries 2019-09-25 10:21:20 -07:00
Max Brunsfeld b15e90bd26 Handle set! predicate function in queries 2019-09-24 11:54:24 -07:00
Max Brunsfeld ff9a2c1f53 Make queries work in languages with simple aliases 2019-09-24 11:54:24 -07:00
Max Brunsfeld 070f11b8bf Use ptr::NonNull in Rust bindings 2019-09-23 17:00:52 -07:00
Max Brunsfeld 598e4e4cf4 Add a unit test to verify captures are sorted by start and end position 2019-09-20 10:12:45 -07:00
Max Brunsfeld a6b6a681ec Fix a bug that prevented early termination of query matches 2019-09-18 16:13:10 -07:00
Max Brunsfeld d9b8bae629 rust: Include pattern indices when iterating query captures 2019-09-18 14:22:55 -07:00
Max Brunsfeld 186b08381c Terminate failed query matches before descending whenever possible 
When iterating over captures, this prevents reasonable queries from 
forcing the tree cursor to buffer matches unnecessarily.
 2019-09-18 11:37:49 -07:00
Max Brunsfeld 374a7ac81e Ensure that duplicate captures are ordered by pattern index 2019-09-17 16:27:16 -07:00
Max Brunsfeld 82955759c0 Add an API for getting a pattern's start offset in the source code 2019-09-17 16:19:58 -07:00
Max Brunsfeld 2d1ca8bc9f Fix match return order fom ts_query_cursor_next_match 2019-09-17 14:52:27 -07:00
Max Brunsfeld 7793bf2a5a Clean up query code 2019-09-16 11:33:22 -07:00
Max Brunsfeld d4d554b2ae Add wasm bindings for predicates 2019-09-16 10:25:44 -07:00
Max Brunsfeld 096126d039 Allow predicates in queries, to match on nodes' text 2019-09-15 22:06:51 -07:00
Max Brunsfeld 86205b9e6d Fix infinite loop on unterminated string in query 2019-09-13 15:19:21 -07:00
Max Brunsfeld a1fec71b19 Tweak QueryCursor to allow iterating either matches or captures 
For syntax highlighting, we want to iterate over all of the captures in 
order, and don't care about grouping the captures by pattern.
 2019-09-13 15:19:04 -07:00
Max Brunsfeld 49ce2fddb9 Add wasm binding for running tree queries in a limited range 2019-09-12 12:14:04 -07:00
Max Brunsfeld 67dcbc3e73 Detect error when a prefix of a valid node name is used in a query 2019-09-11 21:23:46 -07:00
Max Brunsfeld c71de5bd81 Tweak query interface 
* Rename TSQueryContext -> TSQueryCursor
* Remove the permanent association between the cursor and its query. The 
cursor can now be used again for a different query.
 2019-09-11 17:33:48 -07:00
Max Brunsfeld c8c75782e3 Allow tree queries to execute within a limited range 2019-09-11 16:49:29 -07:00
Max Brunsfeld beb5eec7d9 Fix handling of single-node patterns in tree queries 2019-09-11 15:22:32 -07:00
Max Brunsfeld d674bc139a Fix more bugs in binary search used in tree queries 
This binary search implementation differs from Rust's
`slice::binary_search_by` method in how they deal with ties.

In Rust's implementation:

> If there are multiple matches, then any one of the matches
> could be returned.

This implementation needs to return the index of the *first* match.
 2019-09-11 14:45:14 -07:00
Max Brunsfeld 33f89522f6 Allow lisp-style comments in tree queries 2019-09-11 12:16:05 -07:00
Max Brunsfeld 4fa0b02d67 Fix management of capture lists in query execution 2019-09-11 12:06:38 -07:00
Max Brunsfeld 60467ae701 Fix bugs in binary search used in tree queries 2019-09-10 22:30:27 -07:00
Max Brunsfeld fe7c74e7aa Start work on an API for querying trees 2019-09-10 20:53:57 -07:00
Ika d88dae7a3e feat(cli): support snapshot testing with --update flag 
This PR adds an `--update` flag to the `tree-sitter test` command, which adds the ability to replace the _expected_ output in the corpus.txt with the _actual_ output produced by the parser, that is, we can now simply use this `--update` flag to write all the corresponding parser output back to the corpus.txt, and we just need to check the output without typing its actual sexp.

- use the same output format as `tree-sitter parse`, except there won't be any position information printed.
- the corpus.txt won't be touched if there's no difference between the _expected_ output and the _actual_ output in that file.
- if there're differences between _expected_ and _actual_, _expected_ will be replaced by _actual_ and the whole file will be reformatted, i.e., all the output sexp will be formatted just like the output from `tree-sitter parse` and all the delimiters `===`/`---` will be normalized as 80-column long.
- this flag also works with `--filter` flag.
 2019-09-06 10:57:59 +08:00
Max Brunsfeld 62538ed410 highlight iterator: Return byte offset ranges instead of string slices 
Refs #443
 2019-09-04 17:29:31 -07:00
Max Brunsfeld 30e73505e1 Fix handling of extra tokens in ts_tree_cursor_current_field_id 2019-09-04 09:04:31 -07:00
Max Brunsfeld 0955c5b3d7 Handle named nodes aliased as anonymous nodes 
Fixes #401
 2019-08-29 14:28:44 -07:00
Max Brunsfeld b3ab2e07a2 binding_rust: Generalize the interface to callback-based parse methods 
Fixes #386
 2019-08-29 10:39:51 -07:00
Tuấn-Anh Nguyễn d96ba09391 Make Tree::changed_ranges return an Iterator instead of a Vec (#437) 
* Make Tree::changed_ranges return an Iterator instead of a Vec

* Remove CBufferIter.free parameter
 2019-08-26 19:36:16 -07:00
Max Brunsfeld 8e3ff6376f Upgrade rand, dirs 2019-08-19 17:31:35 -07:00
Max Brunsfeld 0afbc31789 Automatically skip BOM characters at beginnings of files 
Refs tree-sitter/tree-sitter-python#48
 2019-08-02 12:03:04 -07:00
Max Brunsfeld a21d355ade Fix handling of potentially overlapping selectors in property sheets 2019-07-22 16:17:10 -07:00
Max Brunsfeld f4740a1beb Make properties generation aware of which nodes are leaves 2019-07-22 09:30:05 -07:00
Max Brunsfeld ea515b6967 Fix ts_node_string behavior for anonymous leaf nodes 
Refs tree-sitter/py-tree-sitter#8
 2019-07-18 12:39:15 -07:00
Max Brunsfeld f505613af3 Fix bug introduced when optimizing property sheet JSON files 2019-07-17 17:00:25 -07:00
Max Brunsfeld f64ee1eb75 Fix incorrect handling of field names in property sheets 2019-07-17 14:28:44 -07:00
Max Brunsfeld 7cd1d77c08 Loosen assertion in timeout unit test to avoid spurious CI failures 2019-07-17 14:28:44 -07:00
Max Brunsfeld bd466febb4 highlight: Fix panic when cancelled before parsing a nested document 2019-07-16 14:34:24 -07:00
Max Brunsfeld c90a532d8f Update tests to reflect javascript grammar changes 2019-07-10 14:30:43 -05:00
Max Brunsfeld ead37c29c6 Tighten timeouts in parse timeout tests 2019-06-20 09:57:38 -07:00
Max Brunsfeld 28011b1e60 Add ts_node_is_extra API 2019-06-19 15:58:29 -07:00
Max Brunsfeld 80b785daee Improve behavior of node descendant queries with single positions 2019-06-13 10:53:59 -07:00
Max Brunsfeld a1682eb81c Handle injection-includes-children in highlighting property sheets 2019-06-11 16:59:33 -07:00
Max Brunsfeld d274e81d0d Overhaul CLI error handling to allow multiple levels of context 2019-05-30 16:52:55 -07:00
Max Brunsfeld 7bc7306afb Add a unit test for local variable tracking in syntax highlighting 2019-05-10 14:38:43 -07:00
Max Brunsfeld 958ab27efb Update JS highlighting unit test 2019-05-10 10:32:31 -07:00
Max Brunsfeld d78ac581f3 Rename scope -> highlight in highlighting property sheets 
We need to use the word `scope` for a different purpose: tracking local 
scopes.
 2019-05-09 09:42:40 -07:00
Max Brunsfeld 572f290ec0 Fix highlighting in the presence of node fields 2019-05-07 11:17:34 -07:00
Max Brunsfeld 58e1a0fee7 In node range queries, treat the end coordinate as exclusive 2019-04-26 13:36:28 -07:00
Max Brunsfeld 701d63dab7 Add --edit flag to parse command 2019-04-08 09:21:03 -07:00
Max Brunsfeld 5035e194ff Merge branch 'master' into node-fields 2019-03-26 11:58:21 -07:00
Max Brunsfeld 0ccb910922 Use a size_t instead of a uint32_t for cancellation flag 2019-03-21 11:26:05 -07:00
Max Brunsfeld 3340168097 Fix backwards logic for cancellation flag 2019-03-20 17:02:07 -07:00
Max Brunsfeld 8941dc1dda Add cancellation flag parameter to highlight API 2019-03-18 09:52:02 -07:00
Max Brunsfeld 0ae304f582 Lib: Rework the API for cancelling a parse 
Also, use beta on CI until atomic::AtomicU32 lands in stable.
 2019-03-18 09:51:21 -07:00
Jacob Mitchell 15b096d695 Extract graph log validation into a specialized test 2019-03-14 22:21:19 -07:00
Jacob Mitchell c8d040ca26 Use 1-indexed rows in CLI and log output (resolves #287) 2019-03-14 22:21:19 -07:00
Max Brunsfeld 1aaad66a03
Merge pull request #301 from tree-sitter/clock-based-timeouts 
Replace operation limit API with a clock-based timeout API
 2019-03-14 16:50:44 -07:00
Max Brunsfeld 9ae594a507 Be more loose with timeout unit test assertions 2019-03-14 15:53:45 -07:00
Max Brunsfeld 006a931ab8 Tests: Prevent array reallocations during ts_stack_print_dot_graph 
When debugging a test with 'script/test -D', the DOT-graph generation
code was sometimes causing reallocations that were not captured by the
allocation tracker, because we explicitly disable allocation-tracking
for that method in order to reduce noise when debugging memory leaks.

By growing the relevant array *prior* to turning off allocation
tracking, we can ensure that it is not reallocated within that function,
avoiding false positive memory leak errors.

Fixes #302
 2019-03-14 13:59:12 -07:00
Max Brunsfeld cddb3e416d Replace operation limit API with a clock-based timeout API 2019-03-14 11:13:38 -07:00
Max Brunsfeld 083e813218 Highlight: fix handling of root node injections like in ERB, EJS 2019-03-13 15:51:50 -07:00
Max Brunsfeld 98e4fd22ef Start work on a C API for syntax highlighting 2019-03-08 13:13:02 -08:00
Max Brunsfeld f52271352b Merge branch 'master' into node-fields 2019-03-05 08:08:05 -08:00
Max Brunsfeld 14b7af3436 highlight: Fix HTML rendering of empty lines 2019-02-22 11:48:29 -08:00
Max Brunsfeld e239aa8229 highlight: don't include scope in ScopeEnd events 
When there are embedded documents, multiple scopes can start or
end at the same position. Previously, there was no guarantee that
the ScopeEnd events would always occur in the reverse order of the
ScopeStart events. The easiest way to avoid exposing inconsistency
is to not surface the scopes being ended.
 2019-02-20 16:45:51 -08:00
Max Brunsfeld a46515b80f Replace LanguageRegistry trait with a simple callback 2019-02-19 17:07:12 -08:00
Max Brunsfeld e89b6b2402 Add a highlight subcommand 2019-02-19 12:32:03 -08:00
Max Brunsfeld 4f069fbe3b Merge branch 'master' into node-fields 2019-02-14 09:35:47 -08:00
Max Brunsfeld 57c528b6c5 CLI: Determine language symbol from grammar, not package.json 
Fixes #272
Fixes #277
 2019-02-13 19:31:26 -08:00
Max Brunsfeld 65d1ce8593 lib: Include fields in ts_node_string output 
This allows you to assert about fields in tests. But if your test 
s-expression does *not* include fields, the fields will be stripped from 
the regexp before comparison.
 2019-02-13 09:47:21 -08:00
Max Brunsfeld 9f608435ee Fix errors in when languages have no fields 2019-02-12 17:20:12 -08:00
Max Brunsfeld a7206b1b8b Add some assertions to node field test 2019-02-08 16:25:27 -08:00
Max Brunsfeld 1d1674811c Fully implement ts_node_child_by_field_id 2019-02-08 15:16:56 -08:00
Max Brunsfeld 18a13b457d Get basic field API working 2019-02-08 15:16:56 -08:00
Max Brunsfeld bf4e1304f8 Start work on new ref API, for giving names to nodes' children 
Co-Authored-By: Ayman Nadeem <aymannadeem@gmail.com>
 2019-02-08 15:16:56 -08:00
Vicent Marti a8cc082ce2 binding: Make walk_with_properties take an &[u8] 2019-02-06 10:10:55 +01:00
Max Brunsfeld 50281637d7 binding: Make parse methods more convenient 
* Rename parse_str to parse and make it polymorphic.
* Rename parse_utf8 to parse_with, since it is now the callback-based
  version of parse
* Add a parse_utf16 method analogous to parse
* Rename existing parse_utf16 method to parse_utf16_with

This brings in the changes from tree-sitter/rust-tree-sitter#5
 2019-02-05 10:59:33 -08:00
Max Brunsfeld efe79889be Port node tests 2019-02-04 20:42:56 -08:00
Max Brunsfeld 9a8cf39277 Add incremental parsing unit tests 2019-02-04 16:43:21 -08:00
Max Brunsfeld 4a98f0b87e Port unit test for missing tokens and included ranges 2019-02-04 14:44:06 -08:00
Max Brunsfeld e62a8a2302 Port more parser unit tests 2019-02-04 10:38:44 -08:00
Max Brunsfeld 6b8483c53c Start work on porting included range unit tests 2019-02-02 22:57:04 -08:00
Max Brunsfeld d465850aba Add unit tests for ts_tree_get_changed_ranges 2019-02-02 14:00:11 -08:00
Max Brunsfeld f263a4fbe3 Separate walk_with_properties tests from parser tests 2019-02-01 21:20:27 -08:00
Max Brunsfeld 4cac85fec4 Add benchmark script 
* Structure `cli` crate as both a library and an executable, so that
benchmarks can import code from the crate.
* Import macros in the Rust 2018 style.
 2019-02-01 15:17:35 -08:00
Max Brunsfeld e26cbb62a5 Add Tree::edit unit tests 2019-01-31 08:15:30 -08:00
Max Brunsfeld 5927e104c2 Check tree consistency in randomized tests 2019-01-26 22:22:29 -08:00
Max Brunsfeld af83e8034e Move test helpers into their own folder 2019-01-25 16:40:26 -08:00
Max Brunsfeld 5a12fbd927 Verify changed ranges in randomized tests 2019-01-25 15:20:34 -08:00
Max Brunsfeld 233d616ebf Add random mutation tests 2019-01-25 12:05:21 -08:00
Max Brunsfeld 196339aaa9 Assert no memory leaks by stubbing malloc/free in the test suite 2019-01-21 14:22:35 -08:00
Max Brunsfeld 6105bf9909 Include error recovery examples in test suite 2019-01-20 16:58:49 -08:00
Max Brunsfeld ed195de8b6 rustfmt 2019-01-17 17:16:04 -08:00
Max Brunsfeld a0a3903f76 Generate binding.gyp, binding.cc, and index.js 2019-01-16 13:53:01 -08:00
Max Brunsfeld d8ab36b2a5 Fix bugs in handling tokens that overlap with separators 2019-01-15 13:21:48 -08:00
Max Brunsfeld a8292f4fe9 Load all fixture grammars dynamically 
This way the build doesn't take forever any time a single grammar has 
been regenerated.
 2019-01-15 13:21:48 -08:00
Max Brunsfeld 5c3c1dd0bd Get logging flags working properly with test script 2019-01-15 13:21:48 -08:00
Max Brunsfeld 6592fdd24c Fix parser generation error messages 2019-01-11 17:26:45 -08:00
Max Brunsfeld e64f7a64a1 Start work on running test corpus tests 2019-01-11 13:31:06 -08:00
Max Brunsfeld 272046a250 Reorganize tests - move them all into the CLI crate 2019-01-10 17:11:57 -08:00