libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-12 00:16:30 -04:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
5714 commits 19 branches 114 tags 51 MiB
Commit graph

1363 commits

Author SHA1 Message Date
Ervin Oro e5584f82d3 Add test to verify regex unicode codepoints work 2019-04-09 21:55:49 +03:00
Ervin Oro 8c845f29e0 Allow hex characters in unicode code points 2019-04-09 20:37:36 +03:00
Max Brunsfeld 701d63dab7 Add --edit flag to parse command 2019-04-08 09:21:03 -07:00
Max Brunsfeld 2bd14a8463 Build parsers with debug symbols 2019-04-05 13:04:37 -07:00
Max Brunsfeld a1eee4a193 Catch missing precedence values in JS 
Fixes #305
 2019-04-05 13:04:12 -07:00
Max Brunsfeld df76aef067 CLI: In lex function, merge branches with the same body 2019-04-04 16:02:50 -07:00
Max Brunsfeld 6490b1e4f2 Fix non-deterministic conflict message order 2019-03-29 17:36:19 -07:00
Max Brunsfeld a133afe9ac Clean up ParseItem::cmp 2019-03-29 12:49:51 -07:00
Max Brunsfeld 9bdac0e693 CLI: Account for field names when comparing parse items 2019-03-29 12:13:24 -07:00
Max Brunsfeld 09b902f6a0 cli: require node-types.json in generated index.js files 2019-03-29 09:07:32 -07:00
Max Brunsfeld eb96dd6ddb node types: Preserve all supertypes in field type lists 2019-03-27 16:32:02 -07:00
Max Brunsfeld 451478c620 Include tokens in node-types.json 2019-03-26 16:57:55 -07:00
Max Brunsfeld e9afdd72b4 node-types: Fix incorrect named value for subclasses 2019-03-26 15:27:07 -07:00
Max Brunsfeld af09e65efc Move node types code to its own module 2019-03-26 14:42:32 -07:00
Max Brunsfeld 6c65d74810 Restructure node-types.json output 2019-03-26 13:43:10 -07:00
Max Brunsfeld 5035e194ff Merge branch 'master' into node-fields 2019-03-26 11:58:21 -07:00
Max Brunsfeld b79bd8693b Start work on handling node supertypes 2019-03-26 11:51:02 -07:00
Max Brunsfeld 82358d3f2f
Merge pull request #307 from tree-sitter/pointer-sized-cancellation-flag 
Make the cancellation flag a size_t, not a uint32_t
 2019-03-22 12:30:39 -07:00
Max Brunsfeld 5a59f19b69 Use explicit syntax for functions with no parameters 2019-03-21 16:06:06 -07:00
Max Brunsfeld 0ccb910922 Use a size_t instead of a uint32_t for cancellation flag 2019-03-21 11:26:05 -07:00
Max Brunsfeld 2fd9ffa8e5 Add --cancel flag to parse command to allow command-line testing 2019-03-21 10:56:32 -07:00
Max Brunsfeld 3340168097 Fix backwards logic for cancellation flag 2019-03-20 17:02:07 -07:00
Max Brunsfeld 8941dc1dda Add cancellation flag parameter to highlight API 2019-03-18 09:52:02 -07:00
Max Brunsfeld 0ae304f582 Lib: Rework the API for cancelling a parse 
Also, use beta on CI until atomic::AtomicU32 lands in stable.
 2019-03-18 09:51:21 -07:00
Jacob Mitchell 15b096d695 Extract graph log validation into a specialized test 2019-03-14 22:21:19 -07:00
Jacob Mitchell c8d040ca26 Use 1-indexed rows in CLI and log output (resolves #287) 2019-03-14 22:21:19 -07:00
Max Brunsfeld 1aaad66a03
Merge pull request #301 from tree-sitter/clock-based-timeouts 
Replace operation limit API with a clock-based timeout API
 2019-03-14 16:50:44 -07:00
Max Brunsfeld 9ae594a507 Be more loose with timeout unit test assertions 2019-03-14 15:53:45 -07:00
Max Brunsfeld 006a931ab8 Tests: Prevent array reallocations during ts_stack_print_dot_graph 
When debugging a test with 'script/test -D', the DOT-graph generation
code was sometimes causing reallocations that were not captured by the
allocation tracker, because we explicitly disable allocation-tracking
for that method in order to reduce noise when debugging memory leaks.

By growing the relevant array *prior* to turning off allocation
tracking, we can ensure that it is not reallocated within that function,
avoiding false positive memory leak errors.

Fixes #302
 2019-03-14 13:59:12 -07:00
Max Brunsfeld 88e3907cc0 Use QueryPerformanceFrequency as clock on windows 2019-03-14 13:42:31 -07:00
Max Brunsfeld e30e827c5f CLI: Add timeout flag to parse command 2019-03-14 11:52:50 -07:00
Max Brunsfeld cddb3e416d Replace operation limit API with a clock-based timeout API 2019-03-14 11:13:38 -07:00
Max Brunsfeld abcac40f2d cli: Add a --time flag to highlight subcommand 2019-03-13 15:52:29 -07:00
Max Brunsfeld 083e813218 Highlight: fix handling of root node injections like in ERB, EJS 2019-03-13 15:51:50 -07:00
Max Brunsfeld b89348e7fe Merge branch 'master' into c-highlight 2019-03-12 15:27:12 -07:00
Max Brunsfeld 2de54c101e cli: put a newline character at the end of grammar.json files 2019-03-12 11:54:31 -07:00
Max Brunsfeld d19c875134 cli: ⬆️ rsass 2019-03-12 11:50:16 -07:00
Max Brunsfeld 98e4fd22ef Start work on a C API for syntax highlighting 2019-03-08 13:13:02 -08:00
Max Brunsfeld f52271352b Merge branch 'master' into node-fields 2019-03-05 08:08:05 -08:00
Max Brunsfeld a20fc3c111 Remove unnecessary character escape processing for regexes 
Fixes #289
 2019-03-04 10:10:04 -08:00
Max Brunsfeld 08ac66a656 Don't generate prop sheets when a specific grammar path is passed 2019-03-03 18:53:24 -08:00
Max Brunsfeld 363079bf3e Re-enable HTML language in benchmarks 
Made possible by this error recovery performance fix:
4c7d5fe26c
 2019-03-03 18:23:01 -08:00
Max Brunsfeld 1bad6dc41e Add ~/.tree-sitter/config.json file, init-config command 
Right now this is just used for two things:
* Specifying folders for locarting parsers to use with `tree-sitter 
parse` and `tree-sitter highlight`
* Specifying colors to use for `tree-sitter-highlight`
 2019-02-25 12:33:24 -08:00
Max Brunsfeld 858b4ba8ac cli: Fix handling of import chains in property sheets 2019-02-22 16:15:36 -08:00
Max Brunsfeld 14b7af3436 highlight: Fix HTML rendering of empty lines 2019-02-22 11:48:29 -08:00
Max Brunsfeld e239aa8229 highlight: don't include scope in ScopeEnd events 
When there are embedded documents, multiple scopes can start or
end at the same position. Previously, there was no guarantee that
the ScopeEnd events would always occur in the reverse order of the
ScopeStart events. The easiest way to avoid exposing inconsistency
is to not surface the scopes being ended.
 2019-02-20 16:45:51 -08:00
Max Brunsfeld d2264d597f cli: Add --scope flag to highlight command 2019-02-20 14:38:19 -08:00
Max Brunsfeld 2ee5cbbc1d highlight: take callback parameters by value 2019-02-20 10:27:08 -08:00
Max Brunsfeld a46515b80f Replace LanguageRegistry trait with a simple callback 2019-02-19 17:07:12 -08:00
Max Brunsfeld e89b6b2402 Add a highlight subcommand 2019-02-19 12:32:03 -08:00
Max Brunsfeld 0dd15e2b02 Ensure deterministic order of values in property sheet JSON files 2019-02-19 10:40:34 -08:00
Max Brunsfeld 9185f6c168 Clarify parse command error message when failing to read source file 2019-02-19 10:40:27 -08:00
Max Brunsfeld 4db132ff94 In property sheets, avoid converting numeric values to strings 2019-02-15 20:48:36 -08:00
Max Brunsfeld 405c200786 Merge branch 'master' into node-fields 2019-02-14 16:27:22 -08:00
Max Brunsfeld 5debf3cbcc cli: Create src directory before writing grammar.json 
Fixes #278
 2019-02-14 10:57:33 -08:00
Max Brunsfeld 4f069fbe3b Merge branch 'master' into node-fields 2019-02-14 09:35:47 -08:00
Max Brunsfeld 57c528b6c5 CLI: Determine language symbol from grammar, not package.json 
Fixes #272
Fixes #277
 2019-02-13 19:31:26 -08:00
Max Brunsfeld 9f3134dace Allow fields to be used in property sheets 2019-02-13 19:14:33 -08:00
Max Brunsfeld 65d1ce8593 lib: Include fields in ts_node_string output 
This allows you to assert about fields in tests. But if your test 
s-expression does *not* include fields, the fields will be stripped from 
the regexp before comparison.
 2019-02-13 09:47:21 -08:00
Max Brunsfeld 9f608435ee Fix errors in when languages have no fields 2019-02-12 17:20:12 -08:00
Max Brunsfeld 56309a1c28 Generate node-fields.json file 2019-02-12 11:06:18 -08:00
Max Brunsfeld b7e38ccc96 Allow using fields in inlined rules 2019-02-08 17:12:08 -08:00
Max Brunsfeld eb1e7af5ec Normalize rule argument to field function 2019-02-08 17:11:58 -08:00
Max Brunsfeld e579e09569 Ensure interpretations are in a predictable order in conflict messages 2019-02-08 17:11:44 -08:00
Max Brunsfeld a7206b1b8b Add some assertions to node field test 2019-02-08 16:25:27 -08:00
Max Brunsfeld 79d90f0d3e Restore naming of alias sequence lengths 
Fields aren't stored in sequences now, so the max length
is back to being just for aliases.
 2019-02-08 16:14:18 -08:00
Max Brunsfeld d8a2c0dda2 Use a separate type for storing field map headers 2019-02-08 16:06:29 -08:00
Max Brunsfeld 51a9f14f7d Ensure symbols are written in a predictable order in conflict messages 2019-02-08 15:16:56 -08:00
Max Brunsfeld 1d1674811c Fully implement ts_node_child_by_field_id 2019-02-08 15:16:56 -08:00
Max Brunsfeld 18a13b457d Get basic field API working 2019-02-08 15:16:56 -08:00
Max Brunsfeld 108ca989ea Start work on including child refs in generated parsers 2019-02-08 15:16:56 -08:00
Max Brunsfeld bf4e1304f8 Start work on new ref API, for giving names to nodes' children 
Co-Authored-By: Ayman Nadeem <aymannadeem@gmail.com>
 2019-02-08 15:16:56 -08:00
Max Brunsfeld dac13af206 cli: Always write parser.h in generate command 2019-02-08 15:16:38 -08:00
Max Brunsfeld acd30d4693 Write grammar.json file when generating based on grammar.js 2019-02-08 15:15:47 -08:00
Max Brunsfeld 4dc475b18c cli: Compile C++ external scanners with exceptions disabled 2019-02-06 16:19:08 -08:00
Max Brunsfeld 9b8bf8dfe2 cli: Compute fragile tokens *after* merging compatibile parse states 
Previously, we failed to mark as fragile some tokens that *should* be 
fragile because of tokens that were introduced during parse state 
merging.
 2019-02-06 16:18:49 -08:00
Max Brunsfeld af694b4c13 cli: Fix exit code of parse subcommand 2019-02-06 16:17:35 -08:00
Max Brunsfeld ccbb8c1cce Add context to error message when opening dynamic libraries 2019-02-06 16:03:50 -08:00
Max Brunsfeld 18085b9eb9 cli: Compile C++ files first when loading parsers 2019-02-06 13:07:03 -08:00
Max Brunsfeld 92a6a69625 cli: Don't error if parser load path contains a non-existent dir 2019-02-06 12:59:19 -08:00
Max Brunsfeld 4badd7cc40 Disable compiler optimizations for lex functions in more cases 
* Reduce the lexer state count threshold from 500 to 300
* Disable optimizations on clang and gcc in addition to MSVC

Optimizations in these source files don't seem to make any impact on
parsing performance, but they slow down compile time substantially.
 2019-02-06 11:50:37 -08:00
Max Brunsfeld db8de605e6
Merge pull request #270 from vmg/vmg/fixes 
Bytes/C Fixes
 2019-02-06 10:41:41 -08:00
Max Brunsfeld 5c925b3868 cli: Check that extra rules aren't passed to functions like 'optional' 
Fixes #247
 2019-02-06 09:31:32 -08:00
Vicent Marti a8cc082ce2 binding: Make walk_with_properties take an &[u8] 2019-02-06 10:10:55 +01:00
Max Brunsfeld d7bc61cada Include parser header content in library crate 2019-02-05 11:59:34 -08:00
Max Brunsfeld 50281637d7 binding: Make parse methods more convenient 
* Rename parse_str to parse and make it polymorphic.
* Rename parse_utf8 to parse_with, since it is now the callback-based
  version of parse
* Add a parse_utf16 method analogous to parse
* Rename existing parse_utf16 method to parse_utf16_with

This brings in the changes from tree-sitter/rust-tree-sitter#5
 2019-02-05 10:59:33 -08:00
Max Brunsfeld efe79889be Port node tests 2019-02-04 20:42:56 -08:00
Max Brunsfeld 9a8cf39277 Add incremental parsing unit tests 2019-02-04 16:43:21 -08:00
Max Brunsfeld 4a98f0b87e Port unit test for missing tokens and included ranges 2019-02-04 14:44:06 -08:00
Max Brunsfeld e62a8a2302 Port more parser unit tests 2019-02-04 10:38:44 -08:00
Max Brunsfeld 59f7511b1c Fix test command's exit code 2019-02-04 09:12:25 -08:00
Max Brunsfeld 6b8483c53c Start work on porting included range unit tests 2019-02-02 22:57:04 -08:00
Max Brunsfeld d465850aba Add unit tests for ts_tree_get_changed_ranges 2019-02-02 14:00:11 -08:00
Max Brunsfeld f263a4fbe3 Separate walk_with_properties tests from parser tests 2019-02-01 21:20:27 -08:00
Max Brunsfeld e143710f4a Move properties module out of generate 2019-02-01 19:57:00 -08:00
Max Brunsfeld 4cac85fec4 Add benchmark script 
* Structure `cli` crate as both a library and an executable, so that
benchmarks can import code from the crate.
* Import macros in the Rust 2018 style.
 2019-02-01 15:17:35 -08:00
Max Brunsfeld e26cbb62a5 Add Tree::edit unit tests 2019-01-31 08:15:30 -08:00
Max Brunsfeld f6d014f3f4 Write tree_sitter/parser.h file in generate command 2019-01-28 14:23:41 -08:00
Max Brunsfeld 6d8ef48dad Make test subcommand exit 1 if tests fail 2019-01-27 09:53:49 -08:00
Max Brunsfeld 5927e104c2 Check tree consistency in randomized tests 2019-01-26 22:22:29 -08:00
Max Brunsfeld af83e8034e Move test helpers into their own folder 2019-01-25 16:40:26 -08:00
Max Brunsfeld 5a12fbd927 Verify changed ranges in randomized tests 2019-01-25 15:20:34 -08:00
Max Brunsfeld 233d616ebf Add random mutation tests 2019-01-25 12:05:21 -08:00
Max Brunsfeld e305012b31 Loosen keyword identification criteria slightly 2019-01-21 15:33:43 -08:00
Max Brunsfeld 196339aaa9 Assert no memory leaks by stubbing malloc/free in the test suite 2019-01-21 14:22:35 -08:00
Max Brunsfeld 6105bf9909 Include error recovery examples in test suite 2019-01-20 16:58:49 -08:00
Max Brunsfeld f6cdd5e3d4 Loosen criteria for identifying conflict-free tokens for error recovery 2019-01-20 16:58:31 -08:00
Max Brunsfeld ff41f05a20 Fix computation of following tokens 2019-01-18 15:16:20 -08:00
Max Brunsfeld 31bdf5eb97 Fix handling of JavaScript errors 
Refs #258
 2019-01-18 09:40:09 -08:00
Max Brunsfeld ed195de8b6 rustfmt 2019-01-17 17:16:04 -08:00
Max Brunsfeld cbcc61a8cf Get parse command handling multiple files, add --time, --quiet flags 2019-01-17 17:15:10 -08:00
Max Brunsfeld c204b5e728 Print help/version info when run w/ no subcommand 2019-01-17 15:17:36 -08:00
Max Brunsfeld c27f776d41 Fix word token index issue in a different way 
Refs https://github.com/tree-sitter/tree-sitter/issues/258
 2019-01-17 13:18:59 -08:00
Max Brunsfeld 8f4096e5cb Give more informative error messages when failing to write files 2019-01-17 12:52:05 -08:00
Max Brunsfeld bb5dedfb1e Fix another token conflict detection bug 2019-01-17 12:44:35 -08:00
Max Brunsfeld 9f7079c9c5 Ensure that the word token has a low numerical index 
Fixes https://github.com/tree-sitter/tree-sitter/issues/258
 2019-01-17 12:44:14 -08:00
Max Brunsfeld 3d11388cd1 Fix test subcommand bugs 
* Log session was dropped before the parser
* Whitespace between close parens was not stripped
 2019-01-17 12:40:21 -08:00
Max Brunsfeld d52a11fd03 Avoid using a string literal to pass grammar path to JS 
Backslashes in windows path were getting interpeted as escape characters.
 2019-01-17 10:09:03 -08:00
Max Brunsfeld d903371709 Remove noisy logging 2019-01-17 10:07:58 -08:00
Max Brunsfeld e4b9d9dfa9 Fix token conflict detection bugs 2019-01-16 20:57:31 -08:00
Max Brunsfeld ae07d2d6e4 Build 32-bit executables on 32-bit appveyor builds 2019-01-16 19:22:33 -08:00
Max Brunsfeld a0a3903f76 Generate binding.gyp, binding.cc, and index.js 2019-01-16 13:53:01 -08:00
Max Brunsfeld ef735eb942 Upload binary artifacts from CI builds 2019-01-16 10:12:43 -08:00
Max Brunsfeld d23a03bdf1 Represent ParseItemSet as a sorted Vec, not a BTreeMap 2019-01-15 16:37:54 -08:00
Max Brunsfeld 0ee11584a7 Add -xc compiler flag for pure-C external scanners 2019-01-15 16:19:47 -08:00
Max Brunsfeld ceff3936ef Unify logic for handling tokens that match separators into one place 2019-01-15 16:10:52 -08:00
Max Brunsfeld 522021b107 Fix NFA generation w/ nested groups 2019-01-15 15:57:29 -08:00
Max Brunsfeld d8ab36b2a5 Fix bugs in handling tokens that overlap with separators 2019-01-15 13:21:48 -08:00
Max Brunsfeld b799b46f79 Handle repetition range operators with commas in regexes 2019-01-15 13:21:48 -08:00
Max Brunsfeld 0a2d72d956 Determine language name from package.json, not directory 2019-01-15 13:21:48 -08:00
Max Brunsfeld a8292f4fe9 Load all fixture grammars dynamically 
This way the build doesn't take forever any time a single grammar has 
been regenerated.
 2019-01-15 13:21:48 -08:00
Max Brunsfeld 5c3c1dd0bd Get logging flags working properly with test script 2019-01-15 13:21:48 -08:00
Max Brunsfeld 19b2addcc4 Fix bug in symbol enum code generation 2019-01-14 14:08:07 -08:00
Max Brunsfeld def5884b59 Allow passing grammar JS or JSON path to generate command 2019-01-14 14:07:42 -08:00
Max Brunsfeld e2717a6ad1 Preprocess regexes to allow non-standard escape sequences 
Also allow unescaped curly braces to match literal curly braces when 
they don't form a valid repetition operator.
 2019-01-14 14:05:19 -08:00
Max Brunsfeld 2e009f7177 Avoid writing empty initializer list for alias sequences 2019-01-12 21:57:34 -08:00
Max Brunsfeld 545e840a08 Remove stray single quotes in symbol name strings 2019-01-12 21:42:31 -08:00
Max Brunsfeld 6f242fda0c Fix edge case in flatten rule 2019-01-11 17:43:42 -08:00
Max Brunsfeld c76a155174 Fix escaping of characters in C strings 2019-01-11 17:43:27 -08:00
Max Brunsfeld 1468b349b5 Ensure 'src' directory exists before writing src/parser.c 2019-01-11 17:39:16 -08:00
Max Brunsfeld 6592fdd24c Fix parser generation error messages 2019-01-11 17:26:45 -08:00
Max Brunsfeld 88f1c4af8e Ensure the .tree-sitter directory exists 2019-01-11 14:56:33 -08:00
Max Brunsfeld fa283dcf27 Use the compiler environment vars computed by the cc config 2019-01-11 14:44:32 -08:00
Max Brunsfeld e64f7a64a1 Start work on running test corpus tests 2019-01-11 13:31:06 -08:00
Max Brunsfeld 0d85a1ef53 Exclude final newlines from inputs when parsing corpus files 2019-01-11 09:48:45 -08:00
Max Brunsfeld cffe80bfad Fix tie-breaking via cascade ordering in property sheets 2019-01-11 09:48:19 -08:00
Max Brunsfeld 272046a250 Reorganize tests - move them all into the CLI crate 2019-01-10 17:11:57 -08:00
Max Brunsfeld ae6dbb945b Avoid using unix-specific methods on windows 2019-01-10 16:11:19 -08:00
Max Brunsfeld 38417fc8a1 Port over remaining tests about property sheet generation 2019-01-10 15:17:38 -08:00
Max Brunsfeld 1dfbe495ed Get property sheet compilation working 2019-01-10 13:12:16 -08:00
Max Brunsfeld 6bd550ca87 Start work on property sheet compilation 2019-01-09 18:10:32 -08:00
Max Brunsfeld c0fad8b3c4 Write parser.c in generate command 2019-01-09 14:43:49 -08:00
Max Brunsfeld 2e8b2ab8fb Give strings more implicit precedence than immediate tokens 2019-01-09 09:59:46 -08:00
Max Brunsfeld 6972a8e3e8 Add logging when deciding not to merge parse states 2019-01-09 09:58:45 -08:00
Max Brunsfeld 98807d2053 Add debug and debug-graph flags to parse and test commands 2019-01-08 21:03:51 -08:00
Max Brunsfeld 6c4d00aad5 Print diffs when tests fail 2019-01-07 22:01:40 -08:00
Max Brunsfeld 20fcffb393 Add parse subcommand 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-01-07 17:57:36 -08:00
Max Brunsfeld 8291d294fb Add test subcommand 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-01-07 17:57:27 -08:00
Max Brunsfeld f059557a9d Move parser generation code in to 'generate' module within CLI crate 2019-01-07 10:23:01 -08:00
Max Brunsfeld 4e29fe69df Reduce lex table size by merging compatible entry point states 2019-01-07 09:59:04 -08:00
Max Brunsfeld 001f8c8f55 Rename LookaheadSet -> TokenSet 
Also, replace non-standard `with` method with a `FromIterator` 
implementation.
 2019-01-07 08:39:47 -08:00
Max Brunsfeld 47607cecf4 Reorganize repo, add rust CLI and binding code, 2019-01-04 17:31:49 -08:00
Max Brunsfeld 5b0e12ea33 Move code into cli directory 2019-01-04 16:50:52 -08:00