libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-12 08:26:20 -04:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
5714 commits 19 branches 114 tags 51 MiB
Commit graph

1363 commits

Author SHA1 Message Date
Max Brunsfeld e49a56ea4c
Merge pull request #939 from tree-sitter/partial-order-precedence 
Allow precedences to be specified using strings and a partial ordering relation
 2021-02-25 13:16:01 -08:00
Max Brunsfeld d8a235faa1 Add further static validation of named precedences 2021-02-25 11:54:21 -08:00
Patrick Thomson 44010d69ea Walk query files recursively in tree-sitter test. 
We were only walking one level of depth into the `queries/` folder
during invocations of `test`, which made us attempt to open folders
rather than recurse into them.

We have to pull in the `walkdir` crate, which is required for
cross-platform walking of directories.

Fixes #938.
 2021-02-25 10:24:54 -05:00
Max Brunsfeld 344797c110 Implement named precedence comparison 2021-02-24 16:02:56 -08:00
Max Brunsfeld d40f118370 Generalize precedence datatype to include strings 
Right now, the strings are not used in comparisons, but they
are passed through the grammar processing pipeline, and are
available to the parse table construction algorithm.

This also cleans up a confusing aspect of the parse table
construction, in which precedences and associativities were
temporarily stored in the parse table data structure itself.
 2021-02-23 20:48:39 -08:00
Douglas Creager a7e2b6a8b2 Remove allocations_stubs 
These were used to provide the `ts_record_*` functions regardless of
whether the `allocation-tracking` feature is enabled.  The allocation
tracking code is now implemented entirely in the `lib` crate, and only
when the feature is enabled, and so these stubs are no longer needed.
 2021-02-23 09:39:39 -05:00
Douglas Creager a29c8d9264 Move allocation tracking into lib crate 
We have several test cases defined in the `cli` crate that depend on the
`lib` crate's `allocation-tracking` feature.  The implementation of the
actual allocation tracker used to live in the `cli` crate, close to the
test cases that use it.  The `allocation-tracking` feature in the `lib`
crate was just used to tell the tree-sitter implementation to expect
that the allocation tracker exists, and to use it.

That pattern meant that we had a circular dependency: `cli` depends on
`lib`, but `lib` required some code that was implemented in `cli`.
That, in turn, caused linker errors — but only when compiling in certain
configurations! [1]

This patch moves all of the allocation tracking implementation into the
`lib` crate, gated on the existing `allocation-tracking` feature, which
fixes the circular dependency.

Note that this patch does **not** fix the fact that feature unification
causes the `lib` crate to be built with the `allocation-tracking`
feature enabled, even though it's not a default.  Fixing that depends on
the forthcoming version 2 feature resolver [2], or using the `dev_dep`
workaround [3] in the meantime.

[1] https://github.com/tree-sitter/tree-sitter/issues/919
[2] https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-resolver-version-2
[3] https://github.com/tree-sitter/tree-sitter/issues/919#issuecomment-777107086
 2021-02-23 09:16:37 -05:00
Max Brunsfeld 2f28a35e1b Handle unicode property escapes inside bracketed char classes 
Refs #906
 2021-02-18 22:27:44 -08:00
Max Brunsfeld 29bc26ecd5 Fix test failure after non-terminal extras change 2021-02-18 15:43:01 -08:00
Max Brunsfeld 86a891fa63 Fix bugs in parser generation for non-terminal extras 
Previously, we attempted to completely separate the parse states
for item sets with non-terminal extras from the parse states
for other rules. But there was not a complete separation.

It actually isn't necessary to separate the parse states in this way.
The only special behavior for parse states with non-terminal extra rules
is what happens at the *end* of the rule: these parse states need to
perform an unconditional reduction.

Luckily, it's possible to distinguish these *non-terminal extra ending*
states from other states just based on their normal structure, with
no additional state.
 2021-02-18 14:14:22 -08:00
Max Brunsfeld b46d51f224 Add a unit test for all unicode character escape forms 2021-02-17 17:49:01 -08:00
Max Brunsfeld 5b630054c6 Handle negated unicode property escapes in regexes 
Refs #380
 2021-02-17 17:22:33 -08:00
Max Brunsfeld 6ae04051e7 Tweak whitespace in generated character set functions 2021-02-17 16:32:49 -08:00
Max Brunsfeld 9d9eb2234f
Merge pull request #906 from tree-sitter/unicode-property-escapes 
Handle simple unicode property escapes in regexes
 2021-02-17 16:14:42 -08:00
Max Brunsfeld dad8546776 Generate more compact code for character set binary search 2021-02-17 13:52:23 -08:00
Max Brunsfeld 6132a10b1c Use binary search in generated character set functions 2021-02-17 13:08:56 -08:00
Max Brunsfeld f5a4c14dbe Add some doc comments to CharacterSet 2021-02-16 21:37:52 -08:00
Max Brunsfeld 73e252e39b Add unit test for child_by_field_name w/ hidden nodes 2021-02-10 16:14:24 -08:00
Max Brunsfeld 6dbe6a3a90 Merge branch 'master' into actions-ci 2021-02-05 10:19:05 -08:00
Max Brunsfeld 2b0de9dfec Fix small bugs in conflict reporting 
* Negative precedence values were not displayed
* Rule names were repeated in resolution suggestions
 2021-02-01 13:30:06 -08:00
Max Brunsfeld e3ba701344 Start work on handling unicode property escapes in regexes 2021-01-29 16:37:45 -08:00
Max Brunsfeld 38444ea7f9
Merge pull request #904 from tree-sitter/character-set-ranges 
Represent CharacterSet internally as a vector of ranges
 2021-01-29 13:35:48 -08:00
Andrew Hlynskyi 2b9e5f6c4b Fix hiding problems in ./build/Debug/tree_sitter_*_binding 
In debug building modules also may happen errors and a current implementation
completely hides them, so errors like 'undefined symbol' can't be
easily identified due to wrong traceback and error message.
 2021-01-29 15:54:10 +02:00
Max Brunsfeld ab78ab3f9b Represent CharacterSet internally as a vector of ranges 2021-01-28 16:10:39 -08:00
marceloll 057d299297 Allow comments only at line start 2021-01-08 13:28:40 -03:00
marceloll 0f5563c536 Add test for ; comments 2021-01-08 13:28:40 -03:00
marceloll 9500aff052 Use ; as comment in tests segxp section 2021-01-08 13:28:40 -03:00
Yijun Yu 9e08712773 Add --xml option to save the parsed code into XML, using node.start_byte() and node.end_byte() 2021-01-04 22:07:38 +00:00
Max Brunsfeld 391fc8c340 Update unit tests to expect ruby grammar change 2020-12-20 19:34:45 -08:00
Max Brunsfeld 96f259d8c5 Run rustfmt 2020-12-03 09:48:20 -08:00
Max Brunsfeld 026231e93d Merge branch 'master' into HEAD 2020-12-03 09:44:33 -08:00
Max Brunsfeld b661050a61 Simplify setup for enabling/disabling allocation recording in the C lib 2020-12-02 15:35:13 -08:00
Max Brunsfeld 751ffd2ee1 Use new emscripten when building with docker 2020-12-01 11:04:06 -08:00
Patrick Thomson cc8f978b3b inline this lambda 2020-11-23 12:05:32 -05:00
Patrick Thomson e1da6e554b Remove fanciful nomenclature. 2020-11-23 12:01:08 -05:00
Patrick Thomson 6764b803a0 Allow overlap in specs. 2020-11-23 11:58:07 -05:00
Patrick Thomson 0b4661e401 Really fix the tests. 2020-11-23 11:41:16 -05:00
Patrick Thomson f3d16f4770 Fix tests. 2020-11-23 11:34:56 -05:00
Patrick Thomson c1a8985712 Merge remote-tracking branch 'origin/master' into query-testy 2020-11-23 11:10:53 -05:00
Max Brunsfeld b267f90e64 Update unit tests to reflect python and ruby grammar changes 2020-11-16 10:51:08 -08:00
Patrick Thomson 4604b40b72 better name for capture regex 2020-11-10 16:23:39 -05:00
Patrick Thomson 50bccdf5da rename Assertion.expected to expected_capture_name 2020-11-10 16:20:51 -05:00
Patrick Thomson c9c886d971 Merge remote-tracking branch 'origin/master' into query-testy 2020-11-10 16:16:53 -05:00
Max Brunsfeld 99cd283e39 query: Fix detection of repeated field names 
Fixes #790
 2020-11-02 14:07:39 -08:00
Max Brunsfeld 3497f34dd7 Fix parser-generation bugs introduced in #782 2020-11-02 13:43:28 -08:00
Arthur Baars d62e7f7d75 Add test case with extra_symbols 2020-10-30 10:58:41 +01:00
Arthur Baars f07dda692e Ensure "extras" symbols are included in the node-types.json file 
The symbols marked as "extras" are the start symbols of secondary
languages. These should be included in the aliases map just as done
for start symbol of the main language to ensure their node type and
field information is included in the node-types.json file.
 2020-10-29 18:05:24 +01:00
Max Brunsfeld c2c63baf5b query: Fix escape sequence parsing in anonymous node patterns 
Fixes #776
Fixes #760
 2020-10-28 13:55:13 -07:00
Max Brunsfeld 071f4e40f1 Fix generate error when there are aliases in unused rules 2020-10-28 12:34:16 -07:00
Max Brunsfeld a2d760e426 Ensure nodes are aliased consistently within syntax error nodes 
Co-Authored-By: Rick Winfrey <rewinfrey@github.com>
 2020-10-27 15:46:09 -07:00
Patrick Thomson 0bd223f032 Better naming for this regex. 2020-10-27 13:11:57 -04:00
Patrick Thomson 938eae8536 Merge remote-tracking branch 'origin/master' into query-testy 2020-10-27 09:16:58 -04:00
Patrick Thomson 1aee60a7c0 propitiate the tests 2020-10-26 14:35:18 -04:00
Patrick Thomson 6adeb7b40d move shared code to query_testing 2020-10-26 14:27:33 -04:00
Patrick Thomson 1012bea3f4 let's start sharing this code 2020-10-26 13:35:10 -04:00
Patrick Thomson f364ce2304 Remove old assertion stuff 2020-10-26 13:22:12 -04:00
Patrick Thomson 9af9d66e19 it works 2020-10-26 13:13:25 -04:00
Patrick Thomson 3e18e97f7c start pulling in the stuff from test_highlight 2020-10-26 12:58:32 -04:00
Max Brunsfeld b972a7158d Tweak cancellation logic for CLI commands 
In 'parse' and 'highlight' cancel on stdin if stdin is a tty.
 2020-10-23 12:15:44 -07:00
Patrick Thomson e370c5053e this is nicer, though 2020-10-23 14:11:46 -04:00
Patrick Thomson 363a0ce4fc things are working: time to piggyback off the highlighter's parser 2020-10-21 14:54:47 -04:00
Patrick Thomson 0dfe89f353 parse assertions from regex capture 2020-10-21 13:32:04 -04:00
Patrick Thomson c691df5ae2 reading in the source correctly 2020-10-21 12:56:11 -04:00
Patrick Thomson 947528f019 use our Result type here 2020-10-21 12:49:41 -04:00
Patrick Thomson 91d5d59d85 Introduce query/assert and call it in query.rs. 2020-10-21 12:37:24 -04:00
Max Brunsfeld 8bb8e9b8b3 Initialize TSLanguage fields in order of their declaration 
This makes parser.c valid under the C++20 standard
 2020-10-15 07:20:12 -07:00
Max Brunsfeld 0a46033391 Remove duplication of LossyUtf8 helper 2020-10-14 11:35:50 -07:00
Max Brunsfeld 857a9ed07b query: Handle captured wildcard nodes at the root of patterns 2020-10-08 12:34:08 -07:00
Patrick Thomson 1f3248a3e0
Merge pull request #749 from tree-sitter/ensure-extras-is-array 
Fix crash when extras function doesn't return an array.
 2020-10-05 16:57:27 -04:00
Patrick Thomson b9b478873a
Merge pull request #748 from tree-sitter/fix-nonexistent-file-crash 
Fix crash when nonexistent files were passed to `parse`.
 2020-10-05 16:14:09 -04:00
Patrick Thomson adce3cb8e2 Merge remote-tracking branch 'origin/master' into ensure-extras-is-array 2020-10-05 16:12:37 -04:00
Max Brunsfeld d1c95193c1 query: Fix invalid use of slice::from_raw_parts 2020-10-05 12:08:53 -07:00
Patrick Thomson 683a2da055 Fix crash when extras function doesn't return an array. 
Fixes #745, which failed due to attempting to call `map` on a
non-array. This bails out at the same spot, but with a more
illuminating error message.
 2020-09-30 16:21:20 -04:00
Patrick Thomson 470733b323 Fix crash when nonexistent files were passed to parse. 
We were unwrapping the result of counting the characters in the vector
returned by collect_files(), which, if that vector is empty, returns
None. The most correct behavior is to halt if a nonexistent filename
was provided or a glob failed.
 2020-09-30 15:52:21 -04:00
Patrick Thomson 33435f43c0 Take Max's suggestions. 2020-09-30 09:28:58 -04:00
Patrick Thomson 16bd061b33 Have the caller track stats here. 2020-09-29 15:43:30 -04:00
Patrick Thomson 939cdf12b9 Add --stats flag for reporting parse information. 2020-09-29 12:34:25 -04:00
Max Brunsfeld ba239ce4ab Make query error line numbers consistently display 1-indexed 2020-09-24 15:03:51 -07:00
Max Brunsfeld 518916f221 Return correct path and line in query errors from the CLI 2020-09-24 13:47:27 -07:00
Max Brunsfeld 297e2bcb28 static query analysis: Fix handling of fields in hidden nodes 2020-09-23 16:55:48 -07:00
Max Brunsfeld ffd3bdc4c1 Escape ? in C string literals 
Fixes #714
 2020-09-23 13:06:06 -07:00
Max Brunsfeld cb343cad5e Avoid reusing the root node during incremental parsing 
Fixes #712
 2020-09-23 12:59:27 -07:00
Max Brunsfeld 21c3bbc4b4 Account for supertypes during query analysis 2020-09-23 10:55:31 -07:00
Max Brunsfeld 5003064da7 Make supertypes automatically hidden, without underscore prefix 2020-09-23 09:35:14 -07:00
Max Brunsfeld 8835dfda99 Fix test for supertypes in queries 2020-09-21 13:11:54 -07:00
Max Brunsfeld b5a9adb555 Allow queries to match on supertypes 
Co-authored-by: Ayman Nadeem <aymannadeem@github.com>
 2020-09-21 12:34:48 -07:00
Max Brunsfeld f10a8448ed
Merge pull request #723 from tree-sitter/comprehensive-wasm-help 
Conditionally handle the presence/absence of build-wasm files.
 2020-09-15 11:54:57 -07:00
Max Brunsfeld ff488f89c9 Make the --prev-abi flag work w/ the newest abi change 2020-09-08 10:58:20 -07:00
Patrick Thomson 2a1bd3dbc2 Better naming. 2020-09-08 09:25:38 -04:00
Patrick Thomson 865f59ad74 No need for platformish logic in build.rs. 2020-09-08 09:23:54 -04:00
Patrick Thomson 92a17e782f Conditionally compile with a cfg variable instead. 2020-09-02 15:32:12 -04:00
Max Brunsfeld 36a8821f3e Fix behavior of the last child operator in tree queries 2020-09-02 12:10:04 -07:00
Max Brunsfeld 4aba684d66 Control recursion depth explicitly during query analysis 2020-08-24 16:55:56 -07:00
Max Brunsfeld 2eb04094f8 Handle aliased parent nodes in query analysis 2020-08-21 14:12:04 -07:00
Max Brunsfeld 456b1f6771 Fix handling of alternations and optional nodes in query analysis 2020-08-20 16:28:54 -07:00
Max Brunsfeld 9daec9cb22 Tweak impossible pattern error messages 2020-08-20 13:35:11 -07:00
Max Brunsfeld 4301110c12 query: Indicate specific step that's impossible 2020-08-20 13:06:38 -07:00
Max Brunsfeld bd42729a41 query: Avoid early-returning captures due to predicates 2020-08-18 13:01:45 -07:00
Max Brunsfeld 604f9e8148 query: Assign is_definite correctly for steps within nested sub-patterns 2020-08-18 10:55:03 -07:00
Max Brunsfeld 91fc9f5399 Use is_definite flag in ts_query_cursor_next_capture 2020-08-17 16:50:59 -07:00
Max Brunsfeld 228a9e28e1 Add tests for impossible queries 2020-08-17 13:27:17 -07:00
Max Brunsfeld c3f9b2b377 Fix query analysis bugs found in ruby tags query 2020-08-17 09:57:06 -07:00
Max Brunsfeld 1ea29053e1 Merge branch 'master' into query-pattern-is-definite 2020-08-14 09:31:55 -07:00
Patrick Thomson 94ab884ee4 Add a test. 2020-08-05 12:16:09 -04:00
Patrick Thomson 5c86a9c654 Fix the tests 2020-08-05 11:52:07 -04:00
Patrick Thomson f4108056b0 Remove otiose pattern match. 2020-08-05 11:33:04 -04:00
Patrick Thomson 5a52dc2cd7 Return an iterator-bool tuple instead of just an iterator. 2020-08-05 11:18:59 -04:00
Max Brunsfeld af655547e5 Fix handling of queries with many patterns with leading repetitions 2020-07-31 12:47:58 -07:00
Max Brunsfeld e89a19a158 tags: Add @ignore capture 2020-07-29 15:33:48 -07:00
Max Brunsfeld 81bbdf19f4 Fix handling of non-terminal extras that share non-extra rules 
Fixes #701
 2020-07-29 09:50:13 -07:00
Max Brunsfeld 1ae5cbc851 query: Handle #not-match? in rust, wasm bindings 2020-07-24 12:15:23 -07:00
Max Brunsfeld 32099050d6 node_types: Fix panic when field is associated with a hidden token 
Fixes #695
 2020-07-24 09:26:56 -07:00
Max Brunsfeld de2b71d465 Fix query bug when max permutations are exceeded 2020-07-23 16:05:50 -07:00
Tuấn-Anh Nguyễn 740d864e67 Add '.' as a valid start of a predicate, in addition to '#' 
See https://github.com/ubolonton/emacs-tree-sitter/issues/38
 2020-07-19 15:46:39 +07:00
Max Brunsfeld 82aa1462fd Clean up get_variable_info function 2020-07-17 15:12:13 -07:00
Max Brunsfeld c4fca5f73e node types: Fix handling of repetitions inside of fields 
Fixes #676
 2020-07-17 14:19:59 -07:00
Max Brunsfeld f4adf0269a Propagate dynamic precedence correctly for inlined rules 
Fixes #683
 2020-07-17 09:53:01 -07:00
Max Brunsfeld 4535efce69 query: Prevent dropping of matches when exceeding range maximum 
Fixes #685
 2020-07-17 09:39:06 -07:00
Max Brunsfeld 91a715799e Accept a paths file to most CLI subcommands 2020-07-14 15:04:39 -07:00
Tuấn-Anh Nguyễn c2fb0f5229 cli: Add --byte-range flag to query command 2020-07-12 20:47:23 +07:00
Max Brunsfeld 0c2dc4c1e9
Merge pull request #648 from tree-sitter/tagging-improvements 
Tagging improvements
 2020-07-10 13:48:23 -07:00
Max Brunsfeld e4e785b567 Remove unused flags from tags CLI command 2020-07-10 13:47:56 -07:00
Max Brunsfeld 0bfd47e2e5 Improve error message when failing to run graphviz stuff 
Fixes #682
 2020-07-10 10:13:19 -07:00
Max Brunsfeld b52f28d6d5 Allow measuring time for tags subcommand 2020-07-09 11:28:07 -07:00
Max Brunsfeld fef72fb434 Merge branch 'tags-utf16-columns' into tagging-improvements 2020-07-09 10:40:46 -07:00
Max Brunsfeld 255cf0a9cf tags: Add utf16 column ranges to tags 
Also, ensure that line ranges contain only valid UTF8.

Co-authored-by: Tim Clem <tclem@github.com>
Co-authored-by: Beka Valentine <bekavalentine@github.com>
 2020-07-08 16:13:58 -07:00
Max Brunsfeld d614c14c2c tags: Make spans refer to name, not entire def/ref 
Co-authored-by: Tim Clem <tclem@github.com>
Co-authored-by: Beka Valentine <bekavalentine@github.com>
 2020-07-08 12:37:01 -07:00
Max Brunsfeld 0bf2450b4a Always enforce stack version limit during reductions 
Fixes #669
 2020-07-06 15:58:33 -07:00
Max Brunsfeld cc37da7457 Query analysis: fix propagation of uncertainty from later siblings 2020-06-26 16:46:12 -07:00
Max Brunsfeld 19baa5fd5e Clean up and document query analysis code 2020-06-25 17:56:43 -07:00
Max Brunsfeld 891de051e2 Fix population of subgraph nodes when analyzing queries 2020-06-25 15:06:27 -07:00
Max Brunsfeld 7f955419a8 Start work on recognizing impossible patterns 2020-06-25 15:06:27 -07:00
Max Brunsfeld 4c2f36a07b Mark steps as definite on query construction 
* Add a ts_query_pattern_is_definite API, just for debugging this
* Store state_count on TSLanguage structs, to allow for scanning parse tables
 2020-06-25 15:06:27 -07:00
Max Brunsfeld deeeb67a3b
query: Fix handling of alternations under field names (#661) 2020-06-24 14:20:56 -07:00
Timothy Clem f166947abb Test updates, definition/reference prefix is now required 2020-06-18 15:05:27 -07:00
Timothy Clem 17d26c0d5a Improved errors 2020-06-18 14:43:27 -07:00
Timothy Clem b6ae67a610 Fix up CLI, use new syntax_type_name 2020-06-18 14:43:10 -07:00
Timothy Clem 75724698f0 Fix up tests 2020-06-18 14:42:41 -07:00
Timothy Clem 3bcb1f8c94 Assert line trimming 2020-06-18 10:48:33 -07:00
Timothy Clem f24a952cb4 Minor output changes 2020-06-17 15:54:36 -07:00
Timothy Clem 929bb40adc Shorten to def/ref 2020-06-17 10:34:55 -07:00
Timothy Clem 80f5c52259 Tests compile 2020-06-16 17:19:35 -07:00
Timothy Clem d802b37791 Bring back a SyntaxType enum 2020-06-16 17:09:34 -07:00
Timothy Clem 9bf4939b9a Show if tag is a def/ref in the cli 2020-06-16 16:04:22 -07:00
Max Brunsfeld a6f71328fe Avoid whitelist/blacklist terminology in test comments 2020-06-16 09:22:34 -07:00
Max Brunsfeld 519a1369ce In highlight test, reset included ranges before finding assertions 2020-06-15 10:51:34 -07:00
Max Brunsfeld 0e5ff14976
Requery the parse table when breaking down the parse stack on invalid lookahead (#636) 
* Requery parse table after breaking down parse stack due to invalid lookahead

* Include Ruby parser in randomized test suite

Ruby and PHP are our only two languages that use non-terminal extras.
Adding Ruby uncovered some bugs.

* Print edited source code when running parse --edit w/ debug flag

* Recompute lookahead when breaking down stack on invalid lookahead

* Fix stack summary leak when there are two discontinuities on a stack version
 2020-06-04 13:40:04 -07:00
Max Brunsfeld 47d607da8d
Add alternative syntax in queries (#630) 
* Add alternative syntax in queries

* Add tests and tweak error handling for alternatives in queries
 2020-06-01 13:23:07 -07:00
Max Brunsfeld ec870e9e66 Avoid extracting helpers for char sets that are only used once 2020-05-26 16:37:45 -07:00
Max Brunsfeld 911fb7f1b2
Extract helper functions to reduce the code size of the lexer function (#626) 
* Extract helper functions to reduce code size of ts_lex

* Name char set helper functions based on token name
 2020-05-26 13:39:11 -07:00
Max Brunsfeld 462c86903f
Improve tree queries' ability to handle large numbers of nested matches (#624) 
* query: Acquire capture lists lazily, allow more concurrent states

* Fix some static analysis warnings
 2020-05-18 13:40:24 -07:00
lerencao 37ee7acc9e
[cli]: add an option to no open browser in web-ui command (#620) 2020-05-15 23:56:40 -07:00
Алексей Пастухов 38d32c018b
add Rust into languages list for corpus tests (#619) 2020-05-15 16:02:39 -07:00
Max Brunsfeld 9d182bb078 node-types: Fix bug w/ required property when multiple rules aliased as same 2020-05-14 10:51:18 -07:00
Max Brunsfeld b66d149b74 Fix inconsistent whitespace after '{' in generated parser 2020-05-13 15:56:49 -07:00
Max Brunsfeld 40993195b8
Fix wasm tests on CI (#616) 
* wasm: Improve error message on missing language symbol

* Fix source file existence checks in build-wasm command
 2020-05-13 15:14:43 -07:00
Max Brunsfeld 2934e219cf Fix build-wasm command 2020-05-12 16:56:21 -07:00
Max Brunsfeld cdc973866f Fix build-wasm command on latest emscripten 2020-05-12 15:42:11 -07:00
Max Brunsfeld 85c998d572 Change the wildcard syntax in tree queries 
1. Use '_' instead of '*'.
2. Add '*' as a postfix operator for zero-or-more repetitions

Signed-off-by: Patrick Thomson <patrickt@github.com>
 2020-05-11 13:04:04 -07:00
Max Brunsfeld 40262483a9 Change query syntax for predicates 
Signed-off-by: Patrick Thomson <patrickt@github.com>
 2020-05-11 12:35:51 -07:00
Max Brunsfeld b14f564550 Add a test for nested optionals in queries 2020-05-11 10:42:48 -07:00
Max Brunsfeld 9c0535cea6 Fix logic for aborting failed matches 2020-05-08 14:15:25 -07:00
Max Brunsfeld b47c170c75 Query: fix bugs and add tests for top-level and nested repetitions 2020-05-08 12:10:01 -07:00
Max Brunsfeld 3ad71625dd Fix query bugs, expand and clean up query tests 2020-05-07 14:22:15 -07:00
Max Brunsfeld 3456a21f0d Start work on restructuring query implementation to deal w/ optionals and repeats better 2020-05-07 12:41:25 -07:00
Riccardo Schirone 780e9cecc9 Do not use multiple unnamed structs inside of unions 2020-04-29 20:42:45 +02:00
Max Brunsfeld 35f82ce301 Fix incorrect parent values after call to child_by_field_name 
Refs tree-sitter/node-tree-sitter#61
Refs tree-sitter/tree-sitter-javascript#127
 2020-04-03 11:21:51 -07:00
Max Brunsfeld 8eac81b8df Merge branch 'master' into tags 2020-03-25 12:49:15 -07:00
Max Brunsfeld 783c087aec tags: Handle cancellation 2020-03-25 11:26:52 -07:00
Max Brunsfeld 59c457c5cf tags: Fix typo in tag ranges 2020-03-20 11:35:27 -07:00
Max Brunsfeld a003e5f6bd generate: Avoid duplicate string tokens in unique symbol map 2020-03-20 11:35:11 -07:00
Max Brunsfeld ae1c51051a Fix tag order in JS tags test 2020-03-20 11:34:33 -07:00
Max Brunsfeld aedab72afa tags: Start work on handling local variables for ruby support 2020-03-20 10:13:03 -07:00
Max Brunsfeld 651fa38c93 Add unit test for tagging via C API. Fix docs handling 2020-03-18 10:40:15 -07:00
Max Brunsfeld 591e066226 tags: Make cli output more human readable 2020-03-17 12:05:09 -07:00
Max Brunsfeld 94a60b8e13 tags: Start adapting Tag struct for use in C API 2020-03-17 11:19:30 -07:00
Max Brunsfeld d0325579ad Use Arc to avoid use-after-free in threaded cancellation unit test 
Fixes #579
 2020-03-16 14:46:05 -07:00
Max Brunsfeld 94bbf14d0e tags: Add test where no comments are adjacent to definition 2020-03-16 14:28:58 -07:00
Max Brunsfeld 65f2874b9e query: Optimize handling of patterns with a wildcard at the root 
Avoid adding and removing states for these patterns on every node in the tree
by just skipping the wildcard step of the matching process
 2020-03-16 14:02:31 -07:00
Max Brunsfeld 1b3a67834b cli: Fix loading of tags query 2020-03-13 16:13:31 -07:00
Max Brunsfeld b5483c67ab query: allow repetition operator to be used on non-terminal nodes 2020-03-13 16:12:39 -07:00
Max Brunsfeld b5f2ed83fe tags: Implement select-adjacent! predicate 2020-03-13 13:02:56 -07:00
Max Brunsfeld 0457736766 rust: add handling of arbitrary predicate operators 2020-03-13 13:02:34 -07:00
Max Brunsfeld 6e2df06dc2 Start proving out tags support for JavaScript 2020-03-12 16:33:52 -07:00
Max Brunsfeld 6f636a0357 query: Add postfix '+' operator for token repetition 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-12 15:10:58 -07:00
Max Brunsfeld 05c1d44e80 Merge branch 'master' into tags 2020-03-11 13:15:26 -07:00
Max Brunsfeld e3aad995f6 query: Fix handling of patterns with wildcards at the root 2020-03-11 13:14:16 -07:00
Patrick Thomson d798bd6bd9 Slice out the line associated with a tag. 2020-03-10 20:39:04 -04:00
Max Brunsfeld 4996cbe830 cli: Move more of the tags code from main into the tags module 2020-03-10 16:52:10 -07:00
Max Brunsfeld 0e02ead0de Update tags test to reflect new handling of escapes in queries 2020-03-10 15:53:42 -07:00
Max Brunsfeld 4531130b44 Merge branch 'master' into tags 2020-03-10 15:50:27 -07:00
Max Brunsfeld 741eed01b7 query: Handle escape sequences and escaped quotes in string literals 2020-03-10 15:50:06 -07:00
Max Brunsfeld fc4d5c3a33 🔥 Dead test code 2020-03-10 12:23:13 -07:00
Max Brunsfeld 17cc38678c Get generate_tags with the new iterator API 2020-03-10 12:05:09 -07:00
Max Brunsfeld 0eb162c685 wip: converting generate_tags to return an iterator 2020-03-10 11:45:31 -07:00
Max Brunsfeld 157258d881 tags: Implement strip regex for docs processing 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-10 10:43:23 -07:00
Max Brunsfeld 7f4828254f Fix criteria for detecting when an aborted parse is resuming 2020-03-09 11:30:08 -07:00
Patrick Thomson 00dcc1eaa6 Need to use expression_statement here. 2020-03-06 17:48:55 -05:00
Max Brunsfeld 680a9e0531 wip 2020-03-06 13:24:03 -08:00
Max Brunsfeld a3f0087b11 Start work on tagging unit test 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-05 13:04:49 -08:00
Max Brunsfeld feac368a30 Start work on new tree-sitter-tags crate 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-04 14:27:31 -08:00
Mark Schmitz b1c7768cc2
Output also HTML_FOOTER with highlight --html (#550) 
* Output also HTML_FOOTER with highlight --html

* move html footer output after end of for loop, as only one closing tag
is needed
 2020-03-02 15:04:15 -08:00
Max Brunsfeld ee46218a73 Fix incremental parsing problem with non-terminal extras 
Also add PHP grammar as a fixture to test against.
 2020-03-02 14:17:12 -08:00
Max Brunsfeld 6cb8d24de2
Merge pull request #542 from SKalt/issue-524-document-supertypes-in-grammar-schema 
feat(cli): documented optional supertypes string[] in grammar schema
 2020-02-24 16:14:49 -08:00
Steven Kalt d82ee739e9
Update cli/src/generate/grammar-schema.json 
Co-Authored-By: Max Brunsfeld <maxbrunsfeld@github.com>
 2020-02-24 18:13:38 -05:00
Max Brunsfeld 709ddfebe9 docs: Add explanation of syntax highlighting configuration for CLI 2020-02-21 11:39:27 -08:00
Max Brunsfeld 360b188644 cli: Handle 'underline' styling when highlighting w/ HTML output 2020-02-20 09:35:46 -08:00
Max Brunsfeld 570b83e2b2 query: Add immediate child operator 2020-02-19 11:47:52 -08:00
Max Brunsfeld 950a89a525 query: Differentiate between wildcard '*' and named wildcard '(*)' 2020-02-19 09:42:29 -08:00
Alyssa Verkade 0e689657b7 Add a language linkage declaration to parsers 
Previously, in order to compile a `tree-sitter` grammar that contained
c++ source in the parser (ie the `scanner.cc` file), you would have to
compile the `parser.c` file separately from the c++ files. For example,
in rust this would result in a `build.rs` close to the following:
```
extern crate cc;

fn main() {
  let dir: PathBuf = ["tree-sitter-ruby", "src"].iter().collect();

  cc::Build::new()
    .include(&dir)
    .cpp(true)
    .file(dir.join("scanner.cc"))
    // NOTE: must have a name that differs from the c static lib
    .compile("tree-sitter-ruby-scanner");

  cc::Build::new()
    .include(&dir)
    .file(dir.join("parser.c"))
    // NOTE: must have a name that differs from the c++ static lib
    .compile("tree-sitter-ruby-parser");
}
```

This was necessary at the time for the following grammars: `ruby`,
`php`, `python`, `embedded-template`, `html`, `cpp`, `ocaml`,
`bash`, `agda`, and `haskell`.

To solve this, we specify an `extern "C"` language linkage declaration
to the functions that must be linked against to compile a parser with the
scanner, making parsers linkable against c++ source.
On all major compilers (gcc, clang, and msvc) this should be the only
change needed due to the combination of clang and gcc both supporting
designated initialization for years and msvc 2019 adopting designated
initializers as a part of the C++20 conformance push.

Subsequently, for rust projects, the necessary `build.rs` would become
(which also brings these parsers into sync with the current docs):
```
extern crate cc;

fn main() {
  let dir: PathBuf = ["tree-sitter-ruby", "src"].iter().collect();

  cc::Build::new()
    .include(&dir)
    .cpp(true)
    .file(dir.join("scanner.cc"))
    .file(dir.join("parser.c"))
    .compile("tree-sitter-ruby");
}
```
 2020-02-18 19:46:59 -08:00
Steven Kalt c5ca259d09
feat(cli): documented optional supertypes Array<string> in grammar-schema.json 2020-02-15 11:23:14 -05:00
Max Brunsfeld de8e3ee188 query: Allow multiple captures on a single node 2020-02-11 16:02:32 -08:00
Max Brunsfeld 5922064b75 node-types: Fix ambiguity warning on rustc 1.41 2020-02-10 10:26:12 -08:00
Max Brunsfeld e23f518915 highlight: add built-in support for carriage-return highlight 2020-01-28 14:47:21 -08:00
Max Brunsfeld d06407aca2 Update highlight test for JSDoc grammar changes 2020-01-27 13:08:30 -08:00
Max Brunsfeld 3f109a3cb5 highlight: Fix logic for handling empty injections with no highlights 2020-01-27 12:32:37 -08:00
Max Brunsfeld 8dd68c360a Fix logic for generating unique symbol map 
Previously, this didn't correctly handle the case where *multiple* 
symbols were all simply-aliased to the same *other* symbol.

Refs #500
 2020-01-27 12:06:48 -08:00
Max Brunsfeld 9f63139a10 Fix error when set_included_ranges is called with an invalid range list 2020-01-17 10:31:28 -08:00
Max Brunsfeld 7421836ee4 Improve cli error message on invalid glob 2020-01-16 16:17:33 -08:00
Max Brunsfeld 9d460e6d01
Merge pull request #525 from SKalt/add-field-rule-to-grammar-json-schema 
added field-rule to grammar-schema.json
 2020-01-16 16:06:48 -08:00
Max Brunsfeld 9dfd03e79a highlight: Sipmlify injection API w/ new injection.combined property 2020-01-16 12:43:31 -08:00
Max Brunsfeld f3747863df Add ts_query_disable_pattern API 2020-01-15 17:08:55 -08:00
Max Brunsfeld 58617cfa0c Improve output format for query subcommand 2020-01-15 17:08:31 -08:00
Steven Kalt 619d7cd65a
added field-rule to grammar-schema.json 2020-01-12 19:40:21 -05:00
Steven Kalt e69430ae7d removed duplicate key (#521) 
The key "required" was duplicated on "symbol-rule". I removed the more permissive copy.
 2020-01-09 17:34:07 -08:00
Max Brunsfeld 9a73277389 web-ui: Load static files from disk if TREE_SITTER_BASE_DIR var is set 2019-12-19 11:23:14 -08:00
Max Brunsfeld f53e7377dc Allow highlight to command to take glob patterns 2019-12-17 15:49:05 -08:00
Max Brunsfeld f0e4d630d5 Reliably avoid duplicate subtypes in node-types.json 2019-12-17 14:55:51 -08:00
Max Brunsfeld 3ac047679c Fix children.required when multiple nodes are aliased as the same 2019-12-16 12:39:57 -08:00
Max Brunsfeld 7c711c5537 Move the private functions in node_types.rs to the end of the file 2019-12-16 12:38:40 -08:00
Max Brunsfeld 5edf97bca9 Remove accidentally-commited garbage line in arg parsing config 2019-12-16 11:59:26 -08:00
Max Brunsfeld 9a0cfa2376 Fix 'required' field when rules with diff fields are aliased to look the same 2019-12-12 11:26:58 -08:00
Max Brunsfeld fc19312913 Fix node-types bugs involving aliases and external tokens 2019-12-12 10:06:18 -08:00
Max Brunsfeld 1b5ae380ce
Don't attempt to extract keywords that don't entirely match word token (#505) 2019-12-11 17:18:15 -08:00
Max Brunsfeld c2e253a4c8 Update unit test to reflect javascript query change 2019-12-10 13:24:36 -08:00
Max Brunsfeld a5a9000e29 generate: Ensure that field_map_slices array is long enough 2019-12-09 11:46:32 -08:00
Steven Kalt 09fb24c03d add encoding to web_ui.html (#504) 
addressing #503 .
 2019-12-06 19:28:08 -08:00
Max Brunsfeld 0cb2ef1082 Fix code paths that still conflated null characters with EOF 2019-12-06 15:29:03 -08:00
Max Brunsfeld d426f46087 Add unit test for symbols respecting simple aliases 2019-12-06 12:47:27 -08:00
Max Brunsfeld 7032dae4f6 Include alias symbols in unique symbol map 2019-12-06 12:11:09 -08:00
Max Brunsfeld bea6e0e28b cli: Generate parseres with the new ABI by default 2019-12-06 11:51:55 -08:00
Max Brunsfeld bd49fbab12 cli: Allow globs and exclusions (via '!' prefix) as args to parse 2019-12-06 11:41:21 -08:00
Max Brunsfeld 56c620c005 Store a mapping to ensure no two symbols map to the same metadata 2019-12-05 17:21:46 -08:00
Max Brunsfeld d6c7b243a7
Merge pull request #499 from tree-sitter/highlight-test 
Add a system for testing syntax highlighting queries
 2019-12-05 15:55:24 -08:00
Max Brunsfeld 4b9feccd13 Run syntax highlighting tests as part of the main test command 
Also, allow `corpus` to be in the `test/corpus` directory, and expect 
highlighting tests to be in the `test/highlight` directory.
 2019-12-05 15:28:16 -08:00
Max Brunsfeld 6dbfbaed07 Start work on a system for testing syntax highlighting 2019-12-05 09:45:54 -08:00
Max Brunsfeld c957a5b8e7 Add test for handling of too many states in query captures 2019-11-22 11:54:32 -08:00
Max Brunsfeld 5a979d1457 node-types: Add test for field merging with aliases 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-11-19 14:31:56 -08:00
Max Brunsfeld 1d63434664 Refactor node-type merging 2019-11-19 11:57:21 -08:00
Max Brunsfeld 5489bc4dc5 Fix small issues with merging node types 
* Merge the `required` field with an 'and', not an 'or'
* Merge field info in addition to children info
 2019-11-19 11:55:21 -08:00
Timothy Clem e2325102d1 No printing 2019-11-19 09:03:35 -08:00
Timothy Clem c346ce4a5e Try not to loose existing children 2019-11-18 14:48:24 -08:00
Timothy Clem 5c72642634 A test demonstrating the issue with named aliases 2019-11-18 14:47:48 -08:00
Max Brunsfeld 660efd8c0f Fix handling of token rules directly inside of alias, prec, or field. 
Fixes #483
 2019-11-15 10:53:17 -08:00
Max Brunsfeld d45b98c7d9 node-types - Fix duplicate when a terminal is aliased with the same name as a non-terminal 2019-11-14 09:31:25 -08:00
Max Brunsfeld 5767bbc806 Avoid generating C char literals with control characters 
Fixes #487
 2019-11-13 10:54:34 -08:00
Max Brunsfeld a2bbc7391b node-types: Fix handling of simple aliased inlined rules 2019-11-13 10:38:47 -08:00
Max Brunsfeld 1909ae2cee binding_rust: Improve node children APIs 
Closes #480
 2019-11-08 14:20:10 -08:00
Max Brunsfeld 39ceff1908 Restore json corpus tests, which accidentally weren't running 2019-10-31 14:32:50 -07:00
Max Brunsfeld d765332c61 Don't rely on new eof ABI in parsers unless --next-abi is passed 2019-10-31 14:32:50 -07:00
Max Brunsfeld d3b7caa565 Add a TSLexer.eof() API, use it in generated parsers 2019-10-31 14:11:52 -07:00
Max Brunsfeld a62b7a70f3 Lexer: track EOF state without relying on null character as lookahead 2019-10-31 14:11:52 -07:00
Max Brunsfeld 0cceca7b4e Rename extra_tokens -> extra_symbols 2019-10-21 17:26:01 -07:00
Max Brunsfeld fcaabea0cf Allow non-terminal extras 2019-10-21 16:08:59 -07:00
Max Brunsfeld 49c632ae90 Fix error in test command when queries folder does not exist 
Fixes #468
 2019-10-21 09:22:24 -07:00
Max Brunsfeld e14e285a10 cli: Check queries when running tree-sitter test 2019-10-18 14:44:16 -07:00
Max Brunsfeld 64c6cf4473 Implicitly reset parser's state if language is changed after a timeout 2019-10-18 11:28:59 -07:00
Max Brunsfeld 084406148b Tweak highlight name matching semantics, add doc comments 2019-10-17 12:03:34 -07:00
Max Brunsfeld b79295e1a0 🔥 Everything related to property sheets 2019-10-17 11:00:31 -07:00
Max Brunsfeld 075a552430 Tweak how highlight query paths are specified in package.json 2019-10-17 10:14:05 -07:00
Max Brunsfeld fa43ce01a6 Allow queries to capture ERROR nodes 2019-10-16 11:54:32 -07:00
Max Brunsfeld 060e00463d Implement include-children directive in injection queries 2019-10-14 17:38:42 -07:00
Max Brunsfeld b3809274f0 Load highlight queries correctly in highlight unit tests 2019-10-14 17:24:16 -07:00
Max Brunsfeld 7c8216c692 highlight: Handle injections correctly, using a separate query cursor 2019-10-14 12:30:22 -07:00
Max Brunsfeld 324c259cbb rust: Create readable messages for query syntax errors 2019-10-14 12:30:22 -07:00
Max Brunsfeld 4c17af3ecd Allow queries with no patterns 2019-10-14 12:30:22 -07:00
Max Brunsfeld f4903578f8 Start reimplementing highlight crate with tree queries 2019-10-14 12:30:22 -07:00
Max Brunsfeld dc7997fdbb node-types: Rework the approach to computing multiple and required 2019-10-11 13:59:02 -07:00
Max Brunsfeld 02b6397aa3 Add test for advancing query cursor after has already finished 2019-10-03 12:55:10 -07:00
Max Brunsfeld 9872a083b7 rust: Change QueryCursor::captures to expose the full match 2019-10-03 12:45:58 -07:00
Ika f191858bae fix: handle UNEXPECTED node 2019-10-03 21:24:17 +08:00
Max Brunsfeld fb7a5369fe Fix regression in node-types for rules with simple aliases 2019-10-02 14:11:21 -07:00
Max Brunsfeld 754cfd42bf Respect nodes' aliases when generating node types 
Fixes #455
 2019-10-02 13:59:00 -07:00
Max Brunsfeld 27149902f8 Handle is?, is-not?, and set! predicate functions in queries 2019-09-25 10:21:20 -07:00
Max Brunsfeld b15e90bd26 Handle set! predicate function in queries 2019-09-24 11:54:24 -07:00
Max Brunsfeld ff9a2c1f53 Make queries work in languages with simple aliases 2019-09-24 11:54:24 -07:00
Max Brunsfeld 070f11b8bf Use ptr::NonNull in Rust bindings 2019-09-23 17:00:52 -07:00
Max Brunsfeld 598e4e4cf4 Add a unit test to verify captures are sorted by start and end position 2019-09-20 10:12:45 -07:00
Max Brunsfeld c9f46b8242 Fix false negative in token conflict detection 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-09-19 11:50:38 -07:00
Max Brunsfeld a6b6a681ec Fix a bug that prevented early termination of query matches 2019-09-18 16:13:10 -07:00
Max Brunsfeld d9b8bae629 rust: Include pattern indices when iterating query captures 2019-09-18 14:22:55 -07:00
Max Brunsfeld 186b08381c Terminate failed query matches before descending whenever possible 
When iterating over captures, this prevents reasonable queries from 
forcing the tree cursor to buffer matches unnecessarily.
 2019-09-18 11:37:49 -07:00
Max Brunsfeld 374a7ac81e Ensure that duplicate captures are ordered by pattern index 2019-09-17 16:27:16 -07:00
Max Brunsfeld 82955759c0 Add an API for getting a pattern's start offset in the source code 2019-09-17 16:19:58 -07:00
Max Brunsfeld 2d1ca8bc9f Fix match return order fom ts_query_cursor_next_match 2019-09-17 14:52:27 -07:00
Max Brunsfeld 7793bf2a5a Clean up query code 2019-09-16 11:33:22 -07:00
Max Brunsfeld d4d554b2ae Add wasm bindings for predicates 2019-09-16 10:25:44 -07:00
Max Brunsfeld 096126d039 Allow predicates in queries, to match on nodes' text 2019-09-15 22:06:51 -07:00
Max Brunsfeld 0d913dec65 Fix layout issues in web-ui 2019-09-13 15:19:31 -07:00
Max Brunsfeld 86205b9e6d Fix infinite loop on unterminated string in query 2019-09-13 15:19:21 -07:00
Max Brunsfeld a1fec71b19 Tweak QueryCursor to allow iterating either matches or captures 
For syntax highlighting, we want to iterate over all of the captures in 
order, and don't care about grouping the captures by pattern.
 2019-09-13 15:19:04 -07:00
Max Brunsfeld f08767c482 Add tree query editor to web UI and playground 2019-09-12 14:07:17 -07:00
Max Brunsfeld 49ce2fddb9 Add wasm binding for running tree queries in a limited range 2019-09-12 12:14:04 -07:00
Max Brunsfeld 67dcbc3e73 Detect error when a prefix of a valid node name is used in a query 2019-09-11 21:23:46 -07:00
Max Brunsfeld c71de5bd81 Tweak query interface 
* Rename TSQueryContext -> TSQueryCursor
* Remove the permanent association between the cursor and its query. The 
cursor can now be used again for a different query.
 2019-09-11 17:33:48 -07:00
Max Brunsfeld c8c75782e3 Allow tree queries to execute within a limited range 2019-09-11 16:49:29 -07:00
Max Brunsfeld beb5eec7d9 Fix handling of single-node patterns in tree queries 2019-09-11 15:22:32 -07:00
Max Brunsfeld ad3f21b0e5 Add simple CLI command for running tree queries 2019-09-11 15:12:45 -07:00
Max Brunsfeld d674bc139a Fix more bugs in binary search used in tree queries 
This binary search implementation differs from Rust's
`slice::binary_search_by` method in how they deal with ties.

In Rust's implementation:

> If there are multiple matches, then any one of the matches
> could be returned.

This implementation needs to return the index of the *first* match.
 2019-09-11 14:45:14 -07:00
Max Brunsfeld 33f89522f6 Allow lisp-style comments in tree queries 2019-09-11 12:16:05 -07:00
Max Brunsfeld 4fa0b02d67 Fix management of capture lists in query execution 2019-09-11 12:06:38 -07:00
Max Brunsfeld 60467ae701 Fix bugs in binary search used in tree queries 2019-09-10 22:30:27 -07:00
Max Brunsfeld fe7c74e7aa Start work on an API for querying trees 2019-09-10 20:53:57 -07:00
Ika d88dae7a3e feat(cli): support snapshot testing with --update flag 
This PR adds an `--update` flag to the `tree-sitter test` command, which adds the ability to replace the _expected_ output in the corpus.txt with the _actual_ output produced by the parser, that is, we can now simply use this `--update` flag to write all the corresponding parser output back to the corpus.txt, and we just need to check the output without typing its actual sexp.

- use the same output format as `tree-sitter parse`, except there won't be any position information printed.
- the corpus.txt won't be touched if there's no difference between the _expected_ output and the _actual_ output in that file.
- if there're differences between _expected_ and _actual_, _expected_ will be replaced by _actual_ and the whole file will be reformatted, i.e., all the output sexp will be formatted just like the output from `tree-sitter parse` and all the delimiters `===`/`---` will be normalized as 80-column long.
- this flag also works with `--filter` flag.
 2019-09-06 10:57:59 +08:00
Ika 9770a0c9f6 Merge branch 'master' of https://github.com/tree-sitter/tree-sitter into feat/snapshot-testing 2019-09-06 10:44:32 +08:00
Ika 807fdf3ef0 Revert "feat(cli): support snapshot testing with --update flag" 
This reverts commit 1b033fdfa4.
 2019-09-06 10:42:37 +08:00
Max Brunsfeld a8dedbee0d In test command, use the longest line of dashes as the test divider 2019-09-05 09:59:59 -07:00
Max Brunsfeld 62538ed410 highlight iterator: Return byte offset ranges instead of string slices 
Refs #443
 2019-09-04 17:29:31 -07:00
Max Brunsfeld 30e73505e1 Fix handling of extra tokens in ts_tree_cursor_current_field_id 2019-09-04 09:04:31 -07:00
Ika 1b033fdfa4 feat(cli): support snapshot testing with --update flag 
This PR adds an `--update` flag to the `tree-sitter test` command, which adds the ability to replace the _expected_ output in the corpus.txt with the _actual_ output produced by the parser, that is, we can now simply use this `--update` flag to write all the corresponding parser output back to the corpus.txt, and we just need to check the output without typing its actual sexp.

- use the same output format as `tree-sitter parse`, except there won't be any position information printed.
- the corpus.txt won't be touched if there's no difference between the _expected_ output and the _actual_ output in that file.
- if there're differences between _expected_ and _actual_, only the test case that is different will be replaced, the rest test cases will stay as-is. (All the delimiters `===`/`---` will be normalized as 80-column long, though.)
- this flag also works with `--filter` flag.
 2019-09-01 23:52:39 +08:00
Ika 4b0489e2f3 fix: allow lowercase unicode escape (#440) 2019-08-31 23:30:33 -07:00
Max Brunsfeld 69ab405325 In next ABI, group symbols by action in small parse state table 
This is a more compact representation because in most states, many 
symbols share the same actions.
 2019-08-30 20:29:55 -07:00
Max Brunsfeld 8037607583 Only generate the new parse table format if --next-abi flag is used 2019-08-29 17:37:33 -07:00
Max Brunsfeld aeb2f895b4 Add --report-states flag for reporting state counts for each rule 2019-08-29 17:36:39 -07:00
Max Brunsfeld 82ff542d3b Appease MSVC by avoiding empty arrays 2019-08-29 17:31:44 -07:00
Max Brunsfeld 09a2755399 Store parse states with few lookahead symbols in a more compact way 2019-08-29 15:52:23 -07:00
Max Brunsfeld 48a883c1d4 Move external token state id computation out of render module 2019-08-29 15:48:22 -07:00
Max Brunsfeld 759c1d6e65 Reorder parse states by descending symbol count 2019-08-29 15:28:29 -07:00
Max Brunsfeld 2430733ee8 Avoid iterating hashmaps in places where order matters 2019-08-29 15:26:05 -07:00
Max Brunsfeld f371507d39 Move TokenSet to rules module 2019-08-29 15:25:45 -07:00
Max Brunsfeld 0955c5b3d7 Handle named nodes aliased as anonymous nodes 
Fixes #401
 2019-08-29 14:28:44 -07:00
Max Brunsfeld b3ab2e07a2 binding_rust: Generalize the interface to callback-based parse methods 
Fixes #386
 2019-08-29 10:39:51 -07:00
Max Brunsfeld 09b46b87dd Remove unnecessary blank lines from parse command output on error 2019-08-28 15:29:44 -07:00
Tuấn-Anh Nguyễn d96ba09391 Make Tree::changed_ranges return an Iterator instead of a Vec (#437) 
* Make Tree::changed_ranges return an Iterator instead of a Vec

* Remove CBufferIter.free parameter
 2019-08-26 19:36:16 -07:00
Max Brunsfeld d5b5d473ab Add missing comparisons to property item hash & equality impls 
Fixes the inconsistent failurs of property sheet unit tests
 2019-08-26 09:31:48 -07:00
Max Brunsfeld 8e3ff6376f Upgrade rand, dirs 2019-08-19 17:31:35 -07:00
Rob Donnelly 9ba5f25594 Ignore hidden files in grammar test directories (#430) 
This fixes "stream did not contain valid UTF-8" error due to
`tree-sitter test` attempting to Vim's parse hidden binary swap files.
 2019-08-19 09:03:12 -07:00
Max Brunsfeld 5e04daf483 Avoid non-deterministic set iteration order when handling conflicts 2019-08-13 15:57:42 -07:00
Max Brunsfeld 56ce4e5d50 Upgrade rsass, remove hashbrown 2019-08-13 10:08:58 -07:00
Max Brunsfeld 5f369a5870 Fix another empty array literal for MSVC compatibility 2019-08-12 15:13:41 -07:00
Max Brunsfeld 13c0aa7dbb Avoid empty initializer list for ts_alias_sequences 
Fixes a bug introduced in 68b089b41e
 2019-08-12 14:11:40 -07:00
Max Brunsfeld 68b089b41e cli: Fix generation of parsers with fields but no aliases 
Fixes #419
 2019-08-11 09:22:30 -07:00
Max Brunsfeld 4d63697036 cli: Fix loading of parsers with no tree-sitter section in package.json 2019-08-11 09:21:49 -07:00
Max Brunsfeld 0dff46865c web-ui: Fix content-type of library wasm 2019-08-10 13:00:55 -07:00
Max Brunsfeld d78204b713 cli: Don't regenerate index.js if it already exists 2019-08-08 15:02:36 -07:00
Max Brunsfeld 93f7de03e2 cli: Handle multi-parser repos, content-regex property 
Prompted by tree-sitter/tree-sitter-typescript#68
 2019-08-07 17:41:45 -07:00
Max Brunsfeld 0afbc31789 Automatically skip BOM characters at beginnings of files 
Refs tree-sitter/tree-sitter-python#48
 2019-08-02 12:03:04 -07:00
Max Brunsfeld 9e1649d9f8 node-types: Fix children descriptors' multiple and required properties 2019-07-30 12:07:11 -07:00
Max Brunsfeld a21d355ade Fix handling of potentially overlapping selectors in property sheets 2019-07-22 16:17:10 -07:00
Max Brunsfeld f4740a1beb Make properties generation aware of which nodes are leaves 2019-07-22 09:30:05 -07:00
Max Brunsfeld 0a3f2131c8 Move state splitting algorithm into its own file 2019-07-19 12:39:52 -07:00
Max Brunsfeld f85ce2fbfa Remove state-ids-to-log flag 2019-07-18 16:06:13 -07:00
Max Brunsfeld ea515b6967 Fix ts_node_string behavior for anonymous leaf nodes 
Refs tree-sitter/py-tree-sitter#8
 2019-07-18 12:39:15 -07:00
Max Brunsfeld f505613af3 Fix bug introduced when optimizing property sheet JSON files 2019-07-17 17:00:25 -07:00
Max Brunsfeld f64ee1eb75 Fix incorrect handling of field names in property sheets 2019-07-17 14:28:44 -07:00
Max Brunsfeld 7cd1d77c08 Loosen assertion in timeout unit test to avoid spurious CI failures 2019-07-17 14:28:44 -07:00
Max Brunsfeld def1e1e91a Reduce bloat in generated property sheet JSON files 2019-07-17 14:28:44 -07:00
Max Brunsfeld d84e6fafdd
Merge pull request #375 from mkrupcale/wasm-docker-selinux 
Use Z option for docker-run volume mount
 2019-07-16 15:10:42 -07:00
Max Brunsfeld bd466febb4 highlight: Fix panic when cancelled before parsing a nested document 2019-07-16 14:34:24 -07:00
Max Brunsfeld c90a532d8f Update tests to reflect javascript grammar changes 2019-07-10 14:30:43 -05:00
Paul Young 98d98ca171
Set NODEJS_CATCH_EXIT=0 when invoking emcc 2019-07-10 13:41:16 +01:00
Max Brunsfeld a9ca6ff73c Fix non-deterministic ordering of lex states 2019-07-03 16:34:06 -07:00
Matthew Krupcale 71d96e813f Use Z option for docker-run volume mount 
This is necessary on systems with SELinux to prevent file permission/access errors when building WASM output

 * cli/src/wasm.rs: Use Z volume mount option for docker-run
 * script/build-wasm: Likewise.
 2019-06-29 15:49:25 -04:00
Max Brunsfeld 223a656fc8 Fix another bug in lex state merging 
Reuse more logic for lex and parse state merging algorithms
 2019-06-21 13:12:09 -07:00
Max Brunsfeld dfd012d13d
Merge pull request #367 from tree-sitter/node-type-children 
Include info about children w/o fields in node-types.json
 2019-06-20 14:06:27 -07:00
Max Brunsfeld 70dc79b412 Merge lex states more liberally 2019-06-20 14:05:15 -07:00
Max Brunsfeld fe6a69a626 Sort lex states 2019-06-20 13:01:28 -07:00
Max Brunsfeld ead37c29c6 Tighten timeouts in parse timeout tests 2019-06-20 09:57:38 -07:00
Max Brunsfeld 5b38ff5f78 Loosen lex state equality check to catch some spurious duplicates 2019-06-20 09:57:38 -07:00
Max Brunsfeld e4873191d6 Refactor generated lex function to use fewer instructions per state 2019-06-20 09:57:38 -07:00
Max Brunsfeld 28011b1e60 Add ts_node_is_extra API 2019-06-19 15:58:29 -07:00
Max Brunsfeld 0ad4e147e2 Dedup types in children property of node-types 2019-06-19 15:35:53 -07:00
Max Brunsfeld ee1d4d6b2c Fix handling of inlined rules in node-types generation 2019-06-19 15:30:48 -07:00
Max Brunsfeld 0a2b183bd0 Add a unit test for the children property in node-types 2019-06-17 13:24:13 -07:00
Max Brunsfeld 381cb09c64 Include info about children w/o fields in node-types.json 2019-06-17 11:35:06 -07:00
Max Brunsfeld 80b785daee Improve behavior of node descendant queries with single positions 2019-06-13 10:53:59 -07:00
Max Brunsfeld a1682eb81c Handle injection-includes-children in highlighting property sheets 2019-06-11 16:59:33 -07:00
Max Brunsfeld 4d4e4f3909 Fix problems with build-wasm command 
* Check that docker is on the PATH before trying to run it
* Don't try to use id(1) if using docker on windows
* Add error message if neither emcc nor docker are available

Fixes #358
 2019-06-10 11:40:36 -07:00
Max Brunsfeld f7d25a5934 Fix missed opportunities to merge parse states 2019-06-06 15:29:22 -07:00
Max Brunsfeld fa68ef4cea Make git sha optional at build time 
Fixes #352
 2019-06-06 13:12:53 -07:00
Max Brunsfeld d274e81d0d Overhaul CLI error handling to allow multiple levels of context 2019-05-30 16:52:55 -07:00
Max Brunsfeld f5163126c8 generate-fixtures-wasm: handle language repos w/ multiple grammars 
Refs tree-sitter/tree-sitter-typescript#68
 2019-05-30 12:34:03 -07:00
Max Brunsfeld 5bb06682d5 Allow dynamically loading grammars w/ no package.json file 2019-05-30 12:05:53 -07:00
Max Brunsfeld 23c6d78674 Fix presentation of missing nodes in parse command 2019-05-15 16:25:53 -07:00
Max Brunsfeld 9674df0c54 Avoid introducing certain auxiliary repeat rules in hidden rules 2019-05-15 12:36:54 -07:00
Max Brunsfeld a008ce63e6 Don't include wasm library in the CLI binary on windows 2019-05-14 15:51:12 -07:00
Max Brunsfeld ad43b211f4 Allow building the wasm libs with native emscripten instead of docker 
And build them on the mac CI as well as the linux CI
 2019-05-14 14:02:22 -07:00
Max Brunsfeld a1ed12f4f4 Start work on web UI command 2019-05-13 21:51:17 -07:00
Max Brunsfeld 7bc7306afb Add a unit test for local variable tracking in syntax highlighting 2019-05-10 14:38:43 -07:00
Max Brunsfeld 958ab27efb Update JS highlighting unit test 2019-05-10 10:32:31 -07:00
Max Brunsfeld b25af0f05f Fix property state merging bug 2019-05-09 20:34:34 -07:00
Max Brunsfeld a7d02e7276 Add support for highlight properties that track local variables 2019-05-09 20:34:34 -07:00
Max Brunsfeld d78ac581f3 Rename scope -> highlight in highlighting property sheets 
We need to use the word `scope` for a different purpose: tracking local 
scopes.
 2019-05-09 09:42:40 -07:00
Max Brunsfeld 572f290ec0 Fix highlighting in the presence of node fields 2019-05-07 11:17:34 -07:00
Max Brunsfeld a6a50a6320 Avoid some bloat in wasm build 2019-05-01 12:44:58 -07:00
Max Brunsfeld 7d1aa10f73 Update generated binding.cc to use newer NAN APIs 2019-04-30 17:23:36 -07:00
Max Brunsfeld 8315a6277b Increase memory limit for language wasm files 2019-04-27 17:54:00 -07:00
Max Brunsfeld 66e006105c Build and test wasm on CI 2019-04-26 14:38:13 -07:00
Max Brunsfeld 58e1a0fee7 In node range queries, treat the end coordinate as exclusive 2019-04-26 13:36:28 -07:00
Max Brunsfeld 1fc0525940 Start work on a WASM binding 2019-04-25 17:27:39 -07:00
Ervin Oro e5584f82d3 Add test to verify regex unicode codepoints work 2019-04-09 21:55:49 +03:00
Ervin Oro 8c845f29e0 Allow hex characters in unicode code points 2019-04-09 20:37:36 +03:00
Max Brunsfeld 701d63dab7 Add --edit flag to parse command 2019-04-08 09:21:03 -07:00
Max Brunsfeld 2bd14a8463 Build parsers with debug symbols 2019-04-05 13:04:37 -07:00
Max Brunsfeld a1eee4a193 Catch missing precedence values in JS 
Fixes #305
 2019-04-05 13:04:12 -07:00
Max Brunsfeld df76aef067 CLI: In lex function, merge branches with the same body 2019-04-04 16:02:50 -07:00
Max Brunsfeld 6490b1e4f2 Fix non-deterministic conflict message order 2019-03-29 17:36:19 -07:00
Max Brunsfeld a133afe9ac Clean up ParseItem::cmp 2019-03-29 12:49:51 -07:00
Max Brunsfeld 9bdac0e693 CLI: Account for field names when comparing parse items 2019-03-29 12:13:24 -07:00
Max Brunsfeld 09b902f6a0 cli: require node-types.json in generated index.js files 2019-03-29 09:07:32 -07:00
Max Brunsfeld eb96dd6ddb node types: Preserve all supertypes in field type lists 2019-03-27 16:32:02 -07:00
Max Brunsfeld 451478c620 Include tokens in node-types.json 2019-03-26 16:57:55 -07:00
Max Brunsfeld e9afdd72b4 node-types: Fix incorrect named value for subclasses 2019-03-26 15:27:07 -07:00
Max Brunsfeld af09e65efc Move node types code to its own module 2019-03-26 14:42:32 -07:00
Max Brunsfeld 6c65d74810 Restructure node-types.json output 2019-03-26 13:43:10 -07:00
Max Brunsfeld 5035e194ff Merge branch 'master' into node-fields 2019-03-26 11:58:21 -07:00
Max Brunsfeld b79bd8693b Start work on handling node supertypes 2019-03-26 11:51:02 -07:00
Max Brunsfeld 82358d3f2f
Merge pull request #307 from tree-sitter/pointer-sized-cancellation-flag 
Make the cancellation flag a size_t, not a uint32_t
 2019-03-22 12:30:39 -07:00
Max Brunsfeld 5a59f19b69 Use explicit syntax for functions with no parameters 2019-03-21 16:06:06 -07:00
Max Brunsfeld 0ccb910922 Use a size_t instead of a uint32_t for cancellation flag 2019-03-21 11:26:05 -07:00
Max Brunsfeld 2fd9ffa8e5 Add --cancel flag to parse command to allow command-line testing 2019-03-21 10:56:32 -07:00
Max Brunsfeld 3340168097 Fix backwards logic for cancellation flag 2019-03-20 17:02:07 -07:00
Max Brunsfeld 8941dc1dda Add cancellation flag parameter to highlight API 2019-03-18 09:52:02 -07:00
Max Brunsfeld 0ae304f582 Lib: Rework the API for cancelling a parse 
Also, use beta on CI until atomic::AtomicU32 lands in stable.
 2019-03-18 09:51:21 -07:00
Jacob Mitchell 15b096d695 Extract graph log validation into a specialized test 2019-03-14 22:21:19 -07:00
Jacob Mitchell c8d040ca26 Use 1-indexed rows in CLI and log output (resolves #287) 2019-03-14 22:21:19 -07:00
Max Brunsfeld 1aaad66a03
Merge pull request #301 from tree-sitter/clock-based-timeouts 
Replace operation limit API with a clock-based timeout API
 2019-03-14 16:50:44 -07:00
Max Brunsfeld 9ae594a507 Be more loose with timeout unit test assertions 2019-03-14 15:53:45 -07:00
Max Brunsfeld 006a931ab8 Tests: Prevent array reallocations during ts_stack_print_dot_graph 
When debugging a test with 'script/test -D', the DOT-graph generation
code was sometimes causing reallocations that were not captured by the
allocation tracker, because we explicitly disable allocation-tracking
for that method in order to reduce noise when debugging memory leaks.

By growing the relevant array *prior* to turning off allocation
tracking, we can ensure that it is not reallocated within that function,
avoiding false positive memory leak errors.

Fixes #302
 2019-03-14 13:59:12 -07:00
Max Brunsfeld 88e3907cc0 Use QueryPerformanceFrequency as clock on windows 2019-03-14 13:42:31 -07:00
Max Brunsfeld e30e827c5f CLI: Add timeout flag to parse command 2019-03-14 11:52:50 -07:00
Max Brunsfeld cddb3e416d Replace operation limit API with a clock-based timeout API 2019-03-14 11:13:38 -07:00
Max Brunsfeld abcac40f2d cli: Add a --time flag to highlight subcommand 2019-03-13 15:52:29 -07:00
Max Brunsfeld 083e813218 Highlight: fix handling of root node injections like in ERB, EJS 2019-03-13 15:51:50 -07:00
Max Brunsfeld b89348e7fe Merge branch 'master' into c-highlight 2019-03-12 15:27:12 -07:00
Max Brunsfeld 2de54c101e cli: put a newline character at the end of grammar.json files 2019-03-12 11:54:31 -07:00
Max Brunsfeld d19c875134 cli: ⬆️ rsass 2019-03-12 11:50:16 -07:00
Max Brunsfeld 98e4fd22ef Start work on a C API for syntax highlighting 2019-03-08 13:13:02 -08:00
Max Brunsfeld f52271352b Merge branch 'master' into node-fields 2019-03-05 08:08:05 -08:00
Max Brunsfeld a20fc3c111 Remove unnecessary character escape processing for regexes 
Fixes #289
 2019-03-04 10:10:04 -08:00
Max Brunsfeld 08ac66a656 Don't generate prop sheets when a specific grammar path is passed 2019-03-03 18:53:24 -08:00
Max Brunsfeld 363079bf3e Re-enable HTML language in benchmarks 
Made possible by this error recovery performance fix:
4c7d5fe26c
 2019-03-03 18:23:01 -08:00
Max Brunsfeld 1bad6dc41e Add ~/.tree-sitter/config.json file, init-config command 
Right now this is just used for two things:
* Specifying folders for locarting parsers to use with `tree-sitter 
parse` and `tree-sitter highlight`
* Specifying colors to use for `tree-sitter-highlight`
 2019-02-25 12:33:24 -08:00
Max Brunsfeld 858b4ba8ac cli: Fix handling of import chains in property sheets 2019-02-22 16:15:36 -08:00
Max Brunsfeld 14b7af3436 highlight: Fix HTML rendering of empty lines 2019-02-22 11:48:29 -08:00
Max Brunsfeld e239aa8229 highlight: don't include scope in ScopeEnd events 
When there are embedded documents, multiple scopes can start or
end at the same position. Previously, there was no guarantee that
the ScopeEnd events would always occur in the reverse order of the
ScopeStart events. The easiest way to avoid exposing inconsistency
is to not surface the scopes being ended.
 2019-02-20 16:45:51 -08:00
Max Brunsfeld d2264d597f cli: Add --scope flag to highlight command 2019-02-20 14:38:19 -08:00
Max Brunsfeld 2ee5cbbc1d highlight: take callback parameters by value 2019-02-20 10:27:08 -08:00
Max Brunsfeld a46515b80f Replace LanguageRegistry trait with a simple callback 2019-02-19 17:07:12 -08:00
Max Brunsfeld e89b6b2402 Add a highlight subcommand 2019-02-19 12:32:03 -08:00
Max Brunsfeld 0dd15e2b02 Ensure deterministic order of values in property sheet JSON files 2019-02-19 10:40:34 -08:00
Max Brunsfeld 9185f6c168 Clarify parse command error message when failing to read source file 2019-02-19 10:40:27 -08:00
Max Brunsfeld 4db132ff94 In property sheets, avoid converting numeric values to strings 2019-02-15 20:48:36 -08:00
Max Brunsfeld 405c200786 Merge branch 'master' into node-fields 2019-02-14 16:27:22 -08:00
Max Brunsfeld 5debf3cbcc cli: Create src directory before writing grammar.json 
Fixes #278
 2019-02-14 10:57:33 -08:00
Max Brunsfeld 4f069fbe3b Merge branch 'master' into node-fields 2019-02-14 09:35:47 -08:00
Max Brunsfeld 57c528b6c5 CLI: Determine language symbol from grammar, not package.json 
Fixes #272
Fixes #277
 2019-02-13 19:31:26 -08:00
Max Brunsfeld 9f3134dace Allow fields to be used in property sheets 2019-02-13 19:14:33 -08:00
Max Brunsfeld 65d1ce8593 lib: Include fields in ts_node_string output 
This allows you to assert about fields in tests. But if your test 
s-expression does *not* include fields, the fields will be stripped from 
the regexp before comparison.
 2019-02-13 09:47:21 -08:00
Max Brunsfeld 9f608435ee Fix errors in when languages have no fields 2019-02-12 17:20:12 -08:00
Max Brunsfeld 56309a1c28 Generate node-fields.json file 2019-02-12 11:06:18 -08:00
Max Brunsfeld b7e38ccc96 Allow using fields in inlined rules 2019-02-08 17:12:08 -08:00
Max Brunsfeld eb1e7af5ec Normalize rule argument to field function 2019-02-08 17:11:58 -08:00
Max Brunsfeld e579e09569 Ensure interpretations are in a predictable order in conflict messages 2019-02-08 17:11:44 -08:00
Max Brunsfeld a7206b1b8b Add some assertions to node field test 2019-02-08 16:25:27 -08:00
Max Brunsfeld 79d90f0d3e Restore naming of alias sequence lengths 
Fields aren't stored in sequences now, so the max length
is back to being just for aliases.
 2019-02-08 16:14:18 -08:00
Max Brunsfeld d8a2c0dda2 Use a separate type for storing field map headers 2019-02-08 16:06:29 -08:00
Max Brunsfeld 51a9f14f7d Ensure symbols are written in a predictable order in conflict messages 2019-02-08 15:16:56 -08:00
Max Brunsfeld 1d1674811c Fully implement ts_node_child_by_field_id 2019-02-08 15:16:56 -08:00
Max Brunsfeld 18a13b457d Get basic field API working 2019-02-08 15:16:56 -08:00
Max Brunsfeld 108ca989ea Start work on including child refs in generated parsers 2019-02-08 15:16:56 -08:00
Max Brunsfeld bf4e1304f8 Start work on new ref API, for giving names to nodes' children 
Co-Authored-By: Ayman Nadeem <aymannadeem@gmail.com>
 2019-02-08 15:16:56 -08:00
Max Brunsfeld dac13af206 cli: Always write parser.h in generate command 2019-02-08 15:16:38 -08:00
Max Brunsfeld acd30d4693 Write grammar.json file when generating based on grammar.js 2019-02-08 15:15:47 -08:00
Max Brunsfeld 4dc475b18c cli: Compile C++ external scanners with exceptions disabled 2019-02-06 16:19:08 -08:00
Max Brunsfeld 9b8bf8dfe2 cli: Compute fragile tokens *after* merging compatibile parse states 
Previously, we failed to mark as fragile some tokens that *should* be 
fragile because of tokens that were introduced during parse state 
merging.
 2019-02-06 16:18:49 -08:00
Max Brunsfeld af694b4c13 cli: Fix exit code of parse subcommand 2019-02-06 16:17:35 -08:00
Max Brunsfeld ccbb8c1cce Add context to error message when opening dynamic libraries 2019-02-06 16:03:50 -08:00
Max Brunsfeld 18085b9eb9 cli: Compile C++ files first when loading parsers 2019-02-06 13:07:03 -08:00
Max Brunsfeld 92a6a69625 cli: Don't error if parser load path contains a non-existent dir 2019-02-06 12:59:19 -08:00
Max Brunsfeld 4badd7cc40 Disable compiler optimizations for lex functions in more cases 
* Reduce the lexer state count threshold from 500 to 300
* Disable optimizations on clang and gcc in addition to MSVC

Optimizations in these source files don't seem to make any impact on
parsing performance, but they slow down compile time substantially.
 2019-02-06 11:50:37 -08:00
Max Brunsfeld db8de605e6
Merge pull request #270 from vmg/vmg/fixes 
Bytes/C Fixes
 2019-02-06 10:41:41 -08:00
Max Brunsfeld 5c925b3868 cli: Check that extra rules aren't passed to functions like 'optional' 
Fixes #247
 2019-02-06 09:31:32 -08:00
Vicent Marti a8cc082ce2 binding: Make walk_with_properties take an &[u8] 2019-02-06 10:10:55 +01:00
Max Brunsfeld d7bc61cada Include parser header content in library crate 2019-02-05 11:59:34 -08:00
Max Brunsfeld 50281637d7 binding: Make parse methods more convenient 
* Rename parse_str to parse and make it polymorphic.
* Rename parse_utf8 to parse_with, since it is now the callback-based
  version of parse
* Add a parse_utf16 method analogous to parse
* Rename existing parse_utf16 method to parse_utf16_with

This brings in the changes from tree-sitter/rust-tree-sitter#5
 2019-02-05 10:59:33 -08:00
Max Brunsfeld efe79889be Port node tests 2019-02-04 20:42:56 -08:00
Max Brunsfeld 9a8cf39277 Add incremental parsing unit tests 2019-02-04 16:43:21 -08:00
Max Brunsfeld 4a98f0b87e Port unit test for missing tokens and included ranges 2019-02-04 14:44:06 -08:00
Max Brunsfeld e62a8a2302 Port more parser unit tests 2019-02-04 10:38:44 -08:00
Max Brunsfeld 59f7511b1c Fix test command's exit code 2019-02-04 09:12:25 -08:00
Max Brunsfeld 6b8483c53c Start work on porting included range unit tests 2019-02-02 22:57:04 -08:00
Max Brunsfeld d465850aba Add unit tests for ts_tree_get_changed_ranges 2019-02-02 14:00:11 -08:00
Max Brunsfeld f263a4fbe3 Separate walk_with_properties tests from parser tests 2019-02-01 21:20:27 -08:00
Max Brunsfeld e143710f4a Move properties module out of generate 2019-02-01 19:57:00 -08:00
Max Brunsfeld 4cac85fec4 Add benchmark script 
* Structure `cli` crate as both a library and an executable, so that
benchmarks can import code from the crate.
* Import macros in the Rust 2018 style.
 2019-02-01 15:17:35 -08:00
Max Brunsfeld e26cbb62a5 Add Tree::edit unit tests 2019-01-31 08:15:30 -08:00
Max Brunsfeld f6d014f3f4 Write tree_sitter/parser.h file in generate command 2019-01-28 14:23:41 -08:00
Max Brunsfeld 6d8ef48dad Make test subcommand exit 1 if tests fail 2019-01-27 09:53:49 -08:00
Max Brunsfeld 5927e104c2 Check tree consistency in randomized tests 2019-01-26 22:22:29 -08:00
Max Brunsfeld af83e8034e Move test helpers into their own folder 2019-01-25 16:40:26 -08:00
Max Brunsfeld 5a12fbd927 Verify changed ranges in randomized tests 2019-01-25 15:20:34 -08:00
Max Brunsfeld 233d616ebf Add random mutation tests 2019-01-25 12:05:21 -08:00
Max Brunsfeld e305012b31 Loosen keyword identification criteria slightly 2019-01-21 15:33:43 -08:00
Max Brunsfeld 196339aaa9 Assert no memory leaks by stubbing malloc/free in the test suite 2019-01-21 14:22:35 -08:00
Max Brunsfeld 6105bf9909 Include error recovery examples in test suite 2019-01-20 16:58:49 -08:00
Max Brunsfeld f6cdd5e3d4 Loosen criteria for identifying conflict-free tokens for error recovery 2019-01-20 16:58:31 -08:00
Max Brunsfeld ff41f05a20 Fix computation of following tokens 2019-01-18 15:16:20 -08:00
Max Brunsfeld 31bdf5eb97 Fix handling of JavaScript errors 
Refs #258
 2019-01-18 09:40:09 -08:00
Max Brunsfeld ed195de8b6 rustfmt 2019-01-17 17:16:04 -08:00
Max Brunsfeld cbcc61a8cf Get parse command handling multiple files, add --time, --quiet flags 2019-01-17 17:15:10 -08:00
Max Brunsfeld c204b5e728 Print help/version info when run w/ no subcommand 2019-01-17 15:17:36 -08:00
Max Brunsfeld c27f776d41 Fix word token index issue in a different way 
Refs https://github.com/tree-sitter/tree-sitter/issues/258
 2019-01-17 13:18:59 -08:00
Max Brunsfeld 8f4096e5cb Give more informative error messages when failing to write files 2019-01-17 12:52:05 -08:00
Max Brunsfeld bb5dedfb1e Fix another token conflict detection bug 2019-01-17 12:44:35 -08:00
Max Brunsfeld 9f7079c9c5 Ensure that the word token has a low numerical index 
Fixes https://github.com/tree-sitter/tree-sitter/issues/258
 2019-01-17 12:44:14 -08:00
Max Brunsfeld 3d11388cd1 Fix test subcommand bugs 
* Log session was dropped before the parser
* Whitespace between close parens was not stripped
 2019-01-17 12:40:21 -08:00
Max Brunsfeld d52a11fd03 Avoid using a string literal to pass grammar path to JS 
Backslashes in windows path were getting interpeted as escape characters.
 2019-01-17 10:09:03 -08:00
Max Brunsfeld d903371709 Remove noisy logging 2019-01-17 10:07:58 -08:00
Max Brunsfeld e4b9d9dfa9 Fix token conflict detection bugs 2019-01-16 20:57:31 -08:00
Max Brunsfeld ae07d2d6e4 Build 32-bit executables on 32-bit appveyor builds 2019-01-16 19:22:33 -08:00
Max Brunsfeld a0a3903f76 Generate binding.gyp, binding.cc, and index.js 2019-01-16 13:53:01 -08:00
Max Brunsfeld ef735eb942 Upload binary artifacts from CI builds 2019-01-16 10:12:43 -08:00
Max Brunsfeld d23a03bdf1 Represent ParseItemSet as a sorted Vec, not a BTreeMap 2019-01-15 16:37:54 -08:00
Max Brunsfeld 0ee11584a7 Add -xc compiler flag for pure-C external scanners 2019-01-15 16:19:47 -08:00
Max Brunsfeld ceff3936ef Unify logic for handling tokens that match separators into one place 2019-01-15 16:10:52 -08:00
Max Brunsfeld 522021b107 Fix NFA generation w/ nested groups 2019-01-15 15:57:29 -08:00
Max Brunsfeld d8ab36b2a5 Fix bugs in handling tokens that overlap with separators 2019-01-15 13:21:48 -08:00
Max Brunsfeld b799b46f79 Handle repetition range operators with commas in regexes 2019-01-15 13:21:48 -08:00
Max Brunsfeld 0a2d72d956 Determine language name from package.json, not directory 2019-01-15 13:21:48 -08:00
Max Brunsfeld a8292f4fe9 Load all fixture grammars dynamically 
This way the build doesn't take forever any time a single grammar has 
been regenerated.
 2019-01-15 13:21:48 -08:00
Max Brunsfeld 5c3c1dd0bd Get logging flags working properly with test script 2019-01-15 13:21:48 -08:00
Max Brunsfeld 19b2addcc4 Fix bug in symbol enum code generation 2019-01-14 14:08:07 -08:00
Max Brunsfeld def5884b59 Allow passing grammar JS or JSON path to generate command 2019-01-14 14:07:42 -08:00
Max Brunsfeld e2717a6ad1 Preprocess regexes to allow non-standard escape sequences 
Also allow unescaped curly braces to match literal curly braces when 
they don't form a valid repetition operator.
 2019-01-14 14:05:19 -08:00
Max Brunsfeld 2e009f7177 Avoid writing empty initializer list for alias sequences 2019-01-12 21:57:34 -08:00
Max Brunsfeld 545e840a08 Remove stray single quotes in symbol name strings 2019-01-12 21:42:31 -08:00
Max Brunsfeld 6f242fda0c Fix edge case in flatten rule 2019-01-11 17:43:42 -08:00
Max Brunsfeld c76a155174 Fix escaping of characters in C strings 2019-01-11 17:43:27 -08:00
Max Brunsfeld 1468b349b5 Ensure 'src' directory exists before writing src/parser.c 2019-01-11 17:39:16 -08:00
Max Brunsfeld 6592fdd24c Fix parser generation error messages 2019-01-11 17:26:45 -08:00
Max Brunsfeld 88f1c4af8e Ensure the .tree-sitter directory exists 2019-01-11 14:56:33 -08:00
Max Brunsfeld fa283dcf27 Use the compiler environment vars computed by the cc config 2019-01-11 14:44:32 -08:00
Max Brunsfeld e64f7a64a1 Start work on running test corpus tests 2019-01-11 13:31:06 -08:00
Max Brunsfeld 0d85a1ef53 Exclude final newlines from inputs when parsing corpus files 2019-01-11 09:48:45 -08:00
Max Brunsfeld cffe80bfad Fix tie-breaking via cascade ordering in property sheets 2019-01-11 09:48:19 -08:00
Max Brunsfeld 272046a250 Reorganize tests - move them all into the CLI crate 2019-01-10 17:11:57 -08:00
Max Brunsfeld ae6dbb945b Avoid using unix-specific methods on windows 2019-01-10 16:11:19 -08:00
Max Brunsfeld 38417fc8a1 Port over remaining tests about property sheet generation 2019-01-10 15:17:38 -08:00
Max Brunsfeld 1dfbe495ed Get property sheet compilation working 2019-01-10 13:12:16 -08:00
Max Brunsfeld 6bd550ca87 Start work on property sheet compilation 2019-01-09 18:10:32 -08:00
Max Brunsfeld c0fad8b3c4 Write parser.c in generate command 2019-01-09 14:43:49 -08:00
Max Brunsfeld 2e8b2ab8fb Give strings more implicit precedence than immediate tokens 2019-01-09 09:59:46 -08:00
Max Brunsfeld 6972a8e3e8 Add logging when deciding not to merge parse states 2019-01-09 09:58:45 -08:00
Max Brunsfeld 98807d2053 Add debug and debug-graph flags to parse and test commands 2019-01-08 21:03:51 -08:00
Max Brunsfeld 6c4d00aad5 Print diffs when tests fail 2019-01-07 22:01:40 -08:00
Max Brunsfeld 20fcffb393 Add parse subcommand 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-01-07 17:57:36 -08:00
Max Brunsfeld 8291d294fb Add test subcommand 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-01-07 17:57:27 -08:00
Max Brunsfeld f059557a9d Move parser generation code in to 'generate' module within CLI crate 2019-01-07 10:23:01 -08:00
Max Brunsfeld 4e29fe69df Reduce lex table size by merging compatible entry point states 2019-01-07 09:59:04 -08:00
Max Brunsfeld 001f8c8f55 Rename LookaheadSet -> TokenSet 
Also, replace non-standard `with` method with a `FromIterator` 
implementation.
 2019-01-07 08:39:47 -08:00
Max Brunsfeld 47607cecf4 Reorganize repo, add rust CLI and binding code, 2019-01-04 17:31:49 -08:00
Max Brunsfeld 5b0e12ea33 Move code into cli directory 2019-01-04 16:50:52 -08:00