libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-11 16:06:21 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
5714 commits 20 branches 114 tags 51 MiB
Commit graph

1363 commits

Author SHA1 Message Date
Andrew Hlynskyi 4a007259fc Fix warning from #2454 in more clear way 2023-08-10 03:59:34 +03:00
Brian Strauch d0afe1ad0a fix: pad hex numbers with zeros when writing colors 
This solves a bug where a hex color with a zero can get truncated
 2023-08-08 09:04:15 -04:00
Amaan Qureshi f35752e3ac
feat!: support the case-insensitive regex flag 2023-08-06 23:40:29 -04:00
Amaan Qureshi 96919cae5d
fix(generate): add Vertical Tab and Form Feed to the whitespace class 2023-08-06 20:41:09 -04:00
Andrew Hlynskyi e024653cb0
Merge pull request #2403 from jakesarjeant/master 
feat(cli): add option to select JS runtime other than node
 2023-08-03 22:29:22 +03:00
Jake Sarjeant 61b70943b1 feat(cli): add option to select JS runtime other than node 2023-08-03 21:34:47 +03:00
Eddie Hatfield 14aa23b1e1 feat: run highlight tests in subdirectories 2023-08-03 19:11:28 +01:00
Andrew Hlynskyi 485d192880 binding_rust: set_max_start_depth accepts optional to reset limit 2023-08-03 11:45:36 +03:00
Samuel Moelius 7b01d8ee05 Add test_query_max_start_depth_more 2023-08-03 11:45:34 +03:00
Samuel Moelius 41ec8b4166 Partially revert d4d5e29 2023-08-03 11:41:50 +03:00
Andrew Hlynskyi 7816397026 Add compile fail tests for LookaheadIterator 2023-08-03 03:49:57 +03:00
Andrew Hlynskyi 4eb22ed3ae Fix UB for LookaheadIterator 2023-08-03 03:49:57 +03:00
Andrew Hlynskyi 4f28ce4e3b Add async context tests for Node and TreeCursor 2023-08-03 03:03:03 +03:00
Michael Davis 3e499d675a
CLI: Re-use highlight cancellation flag for HTML highlights 
The `html` highlight function created its own cancellation flag which
conflicts with the Ctrl-c handler set up in the CLI's `main` block
for `tree-sitter highlight`. We can re-use the cancellation flag from
that block to avoid a panic that happens when using `tree-sitter
highlight -H <file>`

    thread 'main' panicked at 'Error setting Ctrl-C handler: MultipleHandlers', cli/src/util.rs:31:6

This change also aligns the parameters that `highlight::ansi` and
`highlight::html` take.
 2023-08-02 08:56:26 -05:00
Amaan Qureshi acef7a3cdf chore: add tests 2023-08-02 10:47:28 +01:00
Amaan Qureshi b8fe5fe21b fix: do not allow eof to advance states if the new state is the same state 2023-08-02 10:47:27 +01:00
Andrew Hlynskyi a08c61235d Fix, reference by value for Language as everywhere 2023-08-02 00:04:24 +03:00
Daumantas Kavolis ec90c215ae Add tests for bidirectional cursor 2023-08-02 00:02:33 +03:00
Daumantas Kavolis 1cb378b445 Add rust tests for lookahead iterator 2023-08-02 00:02:33 +03:00
Andrew Hlynskyi 19e665ef79 Add tests for TextProvider 2023-08-01 09:12:45 +03:00
Andrew Hlynskyi a2f834d846 More error contexts + conv panics to errors with context 2023-07-30 21:16:45 +03:00
Amaan Qureshi f4e788b28e
feat: warn when unused conflicts are present in a grammar 2023-07-28 00:23:28 -04:00
Amaan Qureshi b8f7645ae2
Merge pull request #2434 from amaanq/test-update 
fix: retain header and divider delimiter lengths when updating tests
 2023-07-27 13:40:53 -04:00
Amaan Qureshi 36c6c8aadf
Merge pull request #2423 from amaanq/nonzero-field-id 
feat!: use `Option<NonZeroU16>` for TSFieldIds
 2023-07-27 13:04:34 -04:00
Amaan Qureshi 0787c24a14
Merge pull request #2430 from amaanq/negative-assertions 
Negative assertions
 2023-07-27 12:44:12 -04:00
Amaan Qureshi 99366f3b5d
ci(sparc64): increase timeout time 2023-07-27 12:26:25 -04:00
Amaan Qureshi 192793a6ab
fix: avoid panic if input is empty 2023-07-27 03:37:02 -04:00
Amaan Qureshi ee399e67d4
fix: retain header and divider delimiter lengths when updating tests 
This helps to prevent visual bloat in diffs when updating tests where the delimiter is not 80 chars long
 2023-07-27 03:31:46 -04:00
Amaan Qureshi 68b1006a3b
feat: add negative assertions, remove duplicate code 2023-07-25 00:27:34 -04:00
Amaan Qureshi c9fd357c06
fix: avoid infinite loop bug 
If the current tag is the last tag and the test fails, the j loop neverends. This prevents that by checking that in the j loop
 2023-07-24 23:43:25 -04:00
Amaan Qureshi a809274009
Merge pull request #2426 from amaanq/sort-tests 
fix(tests): sort categories alphabetically
 2023-07-24 23:21:57 -04:00
Amaan Qureshi 36deb567c8
fix(tests): sort categories alphabetically 2023-07-24 03:05:09 -04:00
Amaan Qureshi 75e1bcf70a
refactor!: rename proc_macro to avoid conflicts when building std with -Z build-std 2023-07-24 00:44:44 -04:00
Amaan Qureshi c521e9c18e
chore: improve error message in some spots loading grammar.json 2023-07-24 00:44:44 -04:00
Amaan Qureshi 9e99789e4c
Merge pull request #2407 from amaanq/bump-deps 
Bump deps
 2023-07-19 17:05:10 -04:00
Amaan Qureshi fc1514d0cd
Merge pull request #2401 from amaanq/fix-inline-bug 
fix: disallow inlining the first rule
 2023-07-19 17:04:17 -04:00
Amaan Qureshi 5fba369c4a
fix: disallow inlining the first rule 
This prevents a panic when indexing symbol_ids during the generation process
 2023-07-19 16:14:58 -04:00
Samuel Moelius d9d4116105
Add test_consecutive_zero_or_modifiers 2023-07-19 03:27:43 -04:00
Amaan Qureshi 0e53b66931
chore: remove useless use 2023-07-19 03:12:13 -04:00
Max Brunsfeld 8310fd423e Update node unit test to reflect C grammar change 2023-07-18 17:58:19 -07:00
Max Brunsfeld 40f7b2ec97 Fix parsing of queries that start with repetitions followed by alternatives 2023-07-18 17:57:52 -07:00
Amaan Qureshi e77405841e
refactor: make a subdirectory for the scratch dir depending on the machine's features 2023-07-18 20:14:21 -04:00
Amaan Qureshi 76f5f81aee
chore: update test 2023-07-18 19:29:40 -04:00
Amaan Qureshi c436897662
test: bump start_time limit for sparc64 2023-07-18 19:07:51 -04:00
Andrew Hlynskyi e1ea3328f4
Merge pull request #2394 from amaanq/hmmcipt2 
Add more architectures to CICD
 2023-07-16 17:23:26 +03:00
Amaan Qureshi 975f3f4686 refactor!: use AtomicUsize for AllocationRecorder for portability 2023-07-16 05:14:11 -04:00
Amaan Qureshi 2a277879ab
Merge pull request #2340 from tree-sitter/highlights-conformance 
feat: add --check flag to `tree-sitter highlight`
 2023-07-16 04:57:23 -04:00
Andrew Hlynskyi 0558f1e0bf test: add retries for all timeout tests 2023-07-16 05:05:36 +03:00
Patrick Thomson cb58bc593f
Add --check flag to tree-sitter highlight. 
Recently I've been pulling a lot of grammars into GitHub's highlighting backend,
replacing legacy language support with tree-sitter highlighting queries.
Our backend systems have a standard set of highlight captures we expect, very
similar to the standard tagging captures we expect. Though end-user applications
are free to choose whatever tagging nomenclature they want, I think it's nice to
include a checking stage that will help us ensure that we know whether a capture
might be recognized or not. It will also help us figure out where we need to
expand our standard set of captures (see #1539).
 2023-07-15 20:49:03 -04:00
Andrew Hlynskyi 0b0cc6c429 Fix rustc 1.71.0 warnings 2023-07-13 17:50:04 +03:00
Amaan Qureshi 9d669abac4 feat: add encoding flag and automatically check if a file might be utf-16 2023-07-10 16:43:35 -04:00
Max Brunsfeld 356f68293a Fix false positive query match bug, introduced in #2085 2023-07-10 16:12:59 -04:00
Amaan Qureshi 3504aa3260 fix(deps): bump proc-macro2 as an unstable feature became stabilized 2023-07-10 16:12:19 -04:00
Amaan Qureshi b192200008 fix: update tests from python grammar changes 2023-07-10 14:40:18 -04:00
Max Brunsfeld 4922602369
Merge pull request #2316 from tree-sitter/node-descendant-index 
Add APIs for retrieving descendant counts, seeking to nodes by their descendant index
 2023-06-12 15:13:43 -07:00
Max Brunsfeld 3375527a89 Fix bug in ts_tree_cursor_goto_descendant 2023-06-12 14:45:30 -07:00
Max Brunsfeld 9dd725b4e3 Start work on exposing APIs for node descendant counts and indices 2023-06-09 08:51:47 -07:00
Samuel Moelius 67f4f2328c Add test_query_error_does_not_oob 2023-05-19 21:31:01 -04:00
Andrew Hlynskyi da269ac350 test: add a test case to demonstrate the #2107 issue 2023-05-13 19:40:49 +03:00
Andrew Hlynskyi ceac2380ed chore(test): reorganize test helpers 2023-05-12 19:29:51 +03:00
Andrew Hlynskyi e966c5ad5b fix: use SIGINT instead of stdin for interruption to don't mess up piped commands 2023-05-02 10:43:46 +03:00
Andrew Hlynskyi 85a588b5b0 Fix tests, adapt to tree-sitter-json 40a81c01a40ac48744e0c8ccabbaba1920441199 2023-04-22 12:08:50 +03:00
Andrew Hlynskyi d4d5e29c91 feat(lib): ts_query_cursor_set_max_start_depth - use 0 to reset 2023-04-17 11:16:04 +03:00
Lewis Russell 1e81a1b67f feat(lib): add ts_query_cursor_set_max_start_depth query API 
This allows configuring cursors from traversing too deep into a tree.
 2023-04-17 11:15:13 +03:00
Joel Spadin 10178ade35 fix: Use / paths when building WASM 
Changed the build-wasm command to always use forward slashes in paths,
since using Windows style paths breaks if the build is run with Docker.

Fixes #532
 2023-04-07 13:07:45 -05:00
Andrew Hlynskyi 0d326824d2 test: add a reproducing test for #2162 2023-04-06 04:49:39 +03:00
Andreas Deininger 0751736d17 docs: convert various links to https protocol 2023-04-04 18:05:46 +03:00
Andrew Hlynskyi c1f784498f chore: nit picking in internal proc_macro crate 2023-04-04 13:51:10 +03:00
Andrew Hlynskyi da894afef5 cargo: specify minimum supported Rust version 2023-04-04 03:42:55 +03:00
Andrew Hlynskyi ddb0af9509 test: use random SEED numbers 
This is needed to omit occurrences of the same seed in a sequence of
following seeds due to the reason of that two initial seed are very
close if based on unix epoch seconds.
 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi 3aeef44eb6 test: run all corpus tests with 10 retries 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi 588549c093 test: run test_parsing_with_a_timeout with 10 retries 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi 62f8c431ae test: add retry and test_with_seed proc macros 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi ca152a93dd ci: fix tests 2023-03-14 20:06:31 +02:00
Max Brunsfeld 8dcf851739 Add unit test for querying within a range of a long top-level repetition 2023-02-16 12:03:51 -08:00
Max Brunsfeld 837899e456 Add API for checking if a pattern in a query is non-local 2023-02-16 11:59:34 -08:00
Max Brunsfeld 40703f110c Fix bug in maintenance of query cursor's tree depth 2023-02-16 11:59:34 -08:00
Max Brunsfeld bd63fb2a0d Tweak query tests 2023-02-16 11:59:34 -08:00
Max Brunsfeld ff2436a6f8 Add --row-range, --quiet, and --time flags to query subcommand 2023-02-14 14:41:25 -08:00
Max Brunsfeld 125503ff3b Fix CLI build on windows 2023-02-13 23:40:06 -08:00
Max Brunsfeld 97fd990822 Add --dot flag to parse subcommand, for printing tree as DOT graph 2023-02-13 12:33:34 -08:00
Andrew Hlynskyi 6f42c78f2d
Merge pull request #2019 from verhovsky/error-message 
cli: make error message more specific for building in `docker`
 2023-01-27 11:45:04 +02:00
Max Brunsfeld 555277a102 Allow testing highlight and tag queries when testing wasm build 
Replace non-mutating `ts_parser_wasm_store` function with
`ts_parser_take_wasm_store`, which removes and returns the wasm
store, in order to facilitate single ownership.
 2023-01-23 11:46:59 -08:00
Andrew Hlynskyi c7d431b53e Add setting TREE_SITTER_DEBUG in the test subcommand 
Were asked in #1218
 2023-01-10 10:47:24 +02:00
Boris Verkhovskiy 61b85b2664 Make error message more specific 2023-01-08 08:10:14 -07:00
Andrew Hlynskyi 108d0ecede loader: add TREE_SITTER_LIBDIR; cli: add --libdir to tree-sitter generate 
Closes #1336
 2023-01-06 10:28:47 +02:00
Andrew Hlynskyi 5088781ef9 cli: add -b, --build flags for tree-sitter generate 2023-01-06 10:28:47 +02:00
Andrew Hlynskyi ad45f5cd2c Remove unused no-minimize arg for the generate command 2023-01-06 10:28:47 +02:00
Andrew Hlynskyi 870fb8772f
Merge pull request #1953 from tree-sitter/fix/playground-port-bind-racing 
fix(cli): Racing on playground webserver port binding
 2022-11-16 06:28:43 +02:00
Max Brunsfeld 51720beeb0 Start work on a --wasm flag for the test subcommand 2022-11-15 17:14:33 -08:00
Max Brunsfeld d47713ee4a Integrate WASM compilation into the CLI's Loader 2022-11-15 17:14:33 -08:00
Max Brunsfeld 3f1a7f9cd4 Start work on ability to load wasm languages from native lib, via wasmtime 2022-11-15 17:14:33 -08:00
Max Brunsfeld 8883d43bee
Merge pull request #1941 from mliszcz/master 
Fix test output formatting for rules starting with M/U
 2022-11-15 17:13:39 -08:00
Jonathan Arnett 813216be9a Configure compiled WASM grammars to not catch rejections 2022-11-15 16:39:17 -08:00
mliszcz 7bf2484d81 Fix test output formatting for rules starting with M/U 
Previously the rule names could not begin with an uppercase M or U
because the test output formatter assumed that they represent special
tokens: MISSING or UEXPECTED.

Fixes #1940.
 2022-11-15 20:20:44 +01:00
Max Brunsfeld 8e3dc7cd7a Add tests that randomly edit files with disjoint included ranges 2022-11-14 16:04:37 -08:00
Max Brunsfeld d07f864815 Fix parse error when reusing a node at the end of an included range 2022-11-11 16:34:57 -08:00
Andrew Hlynskyi a33e9141a8 fix(cli): Racing on playground webserver port binding 2022-11-09 04:56:52 +02:00
Max Brunsfeld da6e24de17 Fix adjustment of trees' included ranges on edits 
Previously, when an included range started or ended *inside* of
an edit, that range did not get updated correctly.

Co-authored-by: Mikayla Maki <mikayla@zed.dev>
 2022-11-07 16:52:19 -08:00
Max Brunsfeld f0177f216e Use library-configured allocator to free intermediate included ranges 2022-11-07 10:48:32 -08:00
Max Brunsfeld e2fe380a08 Generate parsers with ABI version 14 by default 2022-09-02 12:31:29 -07:00
Max Brunsfeld 1b86ccecc8
Merge pull request #1845 from tree-sitter/root-node-with-offset 
Add API for applying a positional offset when accessing a tree's nodes
 2022-08-25 10:51:20 -07:00
Max Brunsfeld 6b87326470
Merge pull request #1787 from kianmeng/fix-typos 
Fix typos
 2022-08-25 10:25:39 -07:00
Max Brunsfeld be442a8f3a
Merge pull request #1831 from shaded-enmity/addr-port-env-vars 
Add env vars for port and address
 2022-08-25 10:23:27 -07:00
Max Brunsfeld 53ed4cf037 Tolerate tree edits whose old range extends beyond the end of the tree 2022-08-25 10:20:40 -07:00
Max Brunsfeld 477b667753 Add ts_tree_root_node_with_offset API 2022-08-18 13:48:47 -07:00
Nat Mote 4e3179fbc0
Avoid extracting default alias for extras 
Fixes #1834
 2022-08-10 07:27:34 -07:00
Pavel Odvody d6c451dc60
Add env vars for port and address 
Make it possible to pass port and address via
pair of environmental variables TREE_SITTER_PLAYGROUND_PORT
and TREE_SITTER_PLAYGROUND_ADDR, respectively.

Signed-off-by: Pavel Odvody <pavel@redhat.com>
 2022-08-02 14:54:45 +02:00
Max Brunsfeld 548c12fb88 Fix bug where patterns with top-level alternatives were not considered 'rooted' 2022-07-07 17:53:54 -07:00
Max Brunsfeld 1401767689 query: Don't attempt to match top-level sibling patterns directly in ERROR nodes 
Co-authored-by: Keith Simmons <keith@zed.dev>
 2022-07-07 15:27:00 -07:00
Kian-Meng Ang b8552ec6c4 Fix typos 2022-06-28 19:57:42 +08:00
Max Brunsfeld 8b5ff2ec8f Loosen randomized changed range assertions on newline characters 
For now, don't error if the scope changes for a newline character.
 2022-06-25 17:13:46 -07:00
Max Brunsfeld 58b719541b Fix failure to match queries with wildcard at root with range restrictions 2022-06-22 15:54:06 -07:00
Aleksei Bavshin b27b4665ac
test: fix incorrect uses of i8 instead of c_char 
Fixes build on aarch64, ppc64le and other platforms that have c_char
defined as u8.
 2022-03-23 00:00:26 -07:00
Max Brunsfeld 7170ec7c96 Improve randomized testing setup 
* Allow iterations to be specified via an env var
* Randomly decide the edit count, with a maximum
  specified via an env var.
* Instead of separate env vars for starting seed + trial, just accept a seed
* Remove some noisy output
 2022-03-02 17:12:25 -08:00
Max Brunsfeld 4b93326898 Don't generate primary states array if it will be unused due to abi version setting 2022-03-02 14:57:59 -08:00
Max Brunsfeld a494d6aa28
Merge pull request #1668 from tree-sitter/remember-lookahead-bytes-on-error-detection 
Remember lookahead bytes on error detection
 2022-02-26 14:59:40 -08:00
Max Brunsfeld c697ebfb27 Add explicit unit test for error detection lookahead bug 2022-02-26 14:33:09 -08:00
Max Brunsfeld 9866674cf8
Merge pull request #1660 from alex-pinkus/expanded-regex-support 
Expand regex support to include emojis and binary ops
 2022-02-24 17:14:23 -08:00
Max Brunsfeld 5eb0a3090f
Merge pull request #1547 from the-mikedavis/md-test-tags 
test tags queries in 'tree-sitter test'
 2022-02-24 15:22:15 -08:00
Alex Pinkus 8fadf18655 Expand regex support to include emojis and binary ops 
The `Emoji` property alias is already present, but the actual property
is not available since it lives in a new file. This adds that file to
the `generate-unicode-categories-json`.

The `emoji-data` file follows the same format as the ones we already
consume in `generate-unicode-categories-json`, so adding emoji support
is fairly easy. his, grammars would need to hard-code a set of
unicode ranges in their own regex. The Javascript library `emoji-regex`
cannot be used because of #451.

For unclear reasons, the characters #, *, and 0-9 are marked as
`Emoji=Yes` by `emoji-data.txt`. Because of this, a grammar that wishes
to use emojis is likely to want to exclude those characters. For that
reason, this change also adds support for binary operations in regexes,
e.g. `[\p{Emoji}&&[^#*0-9]]`.

Lastly (and perhaps controversially), this change introduces new
variables available at grammar compile time, for the major, minor, and
patch versions of the tree-sitter CLI used to compile the grammar. This
will allow grammars to conditionally adopt these new regex features
while remaining backward compatible with older versions of the CLI.
Without this part of the change, grammar authors who do not precompile
and check-in their `grammar.json` would need to wait for downstream
systems to adopt a newer tree-sitter CLI version before they could begin
to use these features.
 2022-02-19 11:41:36 -08:00
Max Brunsfeld cb4317ba8e Change goto_first_child_for_{byte,point} to compare nodes' ranges inclusively 
Co-Authored-By: Antonio Scandurra <me@as-cii.com>
 2022-02-04 12:38:33 -08:00
Max Brunsfeld 994cb61f2c Always generate parser.h, regardless of chosen ABI version 
For some ABI changes, we may need to make changes to the parser.h in order
to restore a previous binary format, but for the current range of supported
ABI versions (13 + 14), the current parser.h is fine.

Refs #1599
 2022-01-23 10:29:52 -08:00
Max Brunsfeld 82ceebc10d 🎨 Use base struct syntax to clean up grammar expectations 2022-01-20 17:17:46 -08:00
Max Brunsfeld fce23d63b3
Merge pull request #1602 from the-mikedavis/md-ignore-future-matches-for-non-local-patterns 
prevent future captures for `#is-not? local` matches
 2022-01-19 16:40:30 -08:00
Michael Davis 02abc2a063
add test for removals in eager query matches 2022-01-18 20:54:55 -06:00
Alex Pinkus 858ea5782b Fix back compat by moving primary_field_ids to the end 
Due to an oversight in #1589, I added `primary_field_ids` into the
`TSLanguage` struct in a place that wasn't the end. This is not actually
backwards compatible and causes downstream failures :(
 2022-01-17 17:23:02 -08:00
Max Brunsfeld 516fd6f6de Add --abi flag to generate command, generate version 13 by default 2022-01-17 14:50:47 -08:00
Max Brunsfeld aaf4572727
Merge pull request #1589 from alex-pinkus/deduplicate-core-ids 
Ignore duplicate states when initializing subgraphs in `ts_query__analyze_patterns`
 2022-01-17 13:54:31 -08:00
Alex Pinkus eaf9b170f1 Don't start with duplicate states in ts_query__analyze_patterns 
This change exposes a new `primary_state_ids` field on the `TSLanguage`
struct, and populates it by tracking the first encountered state with a
given `core_id`. (For posterity: the initial change just exposed
`core_id` and deduplicated within `ts_analyze_query`).

With this `primary_state_ids` field in place, the
`ts_query__analyze_patterns` function only needs to populate its
subgraphs with starting states that are _primary_, since non-primary
states behave identically to primary ones. This leads to large savings
across the board, since most states are not primary.
 2022-01-16 11:17:47 -08:00
Hendrik van Antwerpen ec9b00e5c6 Handle multiple top-level alternations correctly 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen a1a241b013 Expose quantifiers per pattern, instead of merging for all patterns in a query 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen 1f1a449c76 Improve capture quantifier computation 
Compute quantifiers in a bottom-up manner, which allows more precise
results for alternations, where the quantifiers are now precisly joined.
 2022-01-11 18:33:36 +01:00
Max Brunsfeld 622359b400 Simplify allocation-recording in test suite using new ts_set_allocator API 2021-12-30 16:09:07 -08:00
Max Brunsfeld 1d77b9ea47 cli: Fix parsing of test files with newlines in test names 
Fixes #1527
 2021-12-23 14:14:27 -08:00
Michael Davis e7f0cc4e24
add python test case for tag testing 2021-12-18 11:07:46 -06:00
Michael Davis dba7a13808
refactor out get_tag_positions helper function 2021-12-18 11:07:42 -06:00
Michael Davis 47dc88b647
add tags config fixture test helper 2021-12-18 11:07:18 -06:00
Michael Davis 067f742ad3
run test_tag in 'tree-sitter test' when tags test dir exists 2021-12-18 11:07:18 -06:00
Michael Davis 7df82c825f
add module for testing tags 2021-12-18 11:07:11 -06:00
Patrick Thomson f5d1c0b860 Support @module as a highlight key. 
Some languages have the notion of modules, and to represent those
we've started to use a `@module` tag, as discussed in
https://github.com/elixir-lang/tree-sitter-elixir/issues/15.
Because historically we've used the constructor highlight color for
modules in JS/Ruby, it's defined to map to the same color.
 2021-12-15 17:33:57 -05:00
Romanos Skiadas 4784ecaf0a fix crash on empty sexp in test, () 
Now this no longer crashes:

=====
a test
=====
-----
()

...

tree-sitter test:

1 failure:

expected / actual

  1. a test:

    (source_file)

    ()

fixes #1537
 2021-12-15 19:10:26 +02:00
Max Brunsfeld 836d753c73 Fix include_bytes error when building cli outside of source tree 
Refs #1523
 2021-12-13 14:22:33 -08:00
Max Brunsfeld 9f535f95bd Always include playground.html in the CLI binary 
Fixes #1523
 2021-12-06 16:52:14 -08:00
Max Brunsfeld 65746afeff playground: Make query error underlines render in safari 2021-11-22 16:04:45 -08:00
Max Brunsfeld 86b408412c Use serde's derive feature everywhere 2021-11-21 13:39:30 -08:00
Max Brunsfeld 26dac9b2dd Fix query bugs revealed by randomized tests 
* Fix bugs related to named wildcard patterns vs regular wildcard patterns.
* Fix handling of extra nodes during query analysis. Previously, the
expected child_index was updated incorrectly after an extra node,
leading to false "impossible pattern" errors.
* Refine logic for avoiding unnecessary state-splitting due to fallible steps.
Compute *two* different analysis results related to step fallibility:
  * `root_pattern_guaranteed` which, like before, summarizes whether the
    entire pattern is guaranteed to match once this step is reached.
  * `parent_pattern_guaranteed` - which just indicates whether the
    immediate parent pattern is guaranteed. This is now used when
    deciding whether it's necessary to split a match state.
 2021-11-21 12:02:58 -08:00
Max Brunsfeld 142f4b6438 Rename Query::step_is_definite -> is_pattern_guaranteed_at_step 2021-11-21 11:37:52 -08:00
Max Brunsfeld f69c4861c3 Add a randomized test for query matching 2021-11-21 11:29:36 -08:00
Max Brunsfeld a0c085bbec Return an error when trying to inline a token 
Fixes #1420
 2021-11-19 13:02:04 -08:00
Max Brunsfeld 2f9b7ac465 Fix logging of seed in randomized test 2021-11-19 12:16:54 -08:00
Jostein Kjønigsen d0559ca43f Fix whitespace issues when using paths-file. 2021-10-25 23:51:01 +02:00
Max Brunsfeld d05c665863 Convert some of the fixture grammars from JSON to JS 
These tests are easier to write and maintain if the grammars are just JS,
like grammars normally are. It doesn't slow the tests down significantly
to shell out to `node` for each of these grammars.
 2021-10-22 18:47:23 -06:00
Max Brunsfeld ddb12dc0c6 query: Return error on unclosed tree pattern in alternation 
Fixes #1436
 2021-10-12 09:20:43 -07:00
Max Brunsfeld e78413832b Restructure test suite's allocation recording so that tests can run in parallel 2021-10-11 17:24:37 -07:00
Razze 956705a23d Update to unicode standard 14 2021-10-10 16:40:31 +02:00
Max Brunsfeld 21fd9ee36c
Merge pull request #1412 from ahelwer/windows-test-failures 
Fixed some test failures due to CRLF line endings
 2021-10-05 14:13:26 -07:00
Amin Yahyaabadi 8a5620ce15 Use serde's derive feature 2021-10-05 02:38:18 -05:00
Andrew Helwer 993e93eeee Fix linux build 2021-09-28 19:06:05 -04:00
Andrew Helwer 3b5f9cb642 Fixed cargo fmt 2021-09-28 18:53:24 -04:00
Andrew Helwer 8eef790a61 Build rust with warnings=errors 2021-09-28 18:51:09 -04:00
Andrew Helwer e8e4dafa92 Fixed cargo fmt suggestions 2021-09-28 18:31:04 -04:00
Andrew Helwer 3d1f760e31 Simplified corpus test output comparisons 2021-09-28 18:27:25 -04:00
Andrew Helwer 0c3fb6e6a2 Fixed cargo fmt issues 2021-09-24 22:49:25 -04:00
Andrew Helwer 0dc1bd806f Fixed some test failures due to CRLF line endings 2021-09-24 22:42:38 -04:00
Max Brunsfeld 0e3a2780cd Avoid dynamic regex construction when parsing test files 2021-09-21 21:35:14 -07:00
Andrew Hlynskyi 16470bc0b1 chore(cli): Rename all internal web_ui stuff to playground 2021-09-22 04:28:08 +03:00
Andrew Hlynskyi 2abd6bc318 fix(cli): Fix playground opening in a browser, regression in #1304 2021-09-22 01:38:52 +03:00
Andrew Hlynskyi e2de738026 cli(query): Improve and unify query subcommand output 2021-09-21 23:10:00 +03:00
Andrew Hlynskyi 62d79b80e1 feat(cli): add a flag to compile a parser in debug mode with -O0 C/C++ compiler flag 2021-09-13 13:48:02 +03:00
Max Brunsfeld 4fa5ab8a52
Merge pull request #1348 from tlaplus-community/testfile-separator 
Add support for arbitrary suffixes in test file headers and separators
 2021-09-07 22:58:05 -07:00
Max Brunsfeld 23b28f6f36 Fix 'include!' error when building the CLI outside of the repo 2021-09-03 13:45:13 -07:00
Max Brunsfeld 8fa875b1a4 Fix possible infinite loop when running syntax highlighting tests 
Fixes #1347
 2021-08-29 15:04:23 -07:00
Andrew Helwer 03b6a00bb8 Support for suffixes in test file separators 
Some languages use the non-suffixed separators in their syntax
Fixes #982
 2021-08-19 09:42:46 -04:00
FnControlOption e030434ca7 Handle aliases in unicode property escapes in regexes 2021-08-18 22:22:46 -07:00
Paul Gey a533e4d7bb Remove unnecessary borrows 
This produces an `unused_must_use` warning on nightly:
https://github.com/rust-lang/rust/pull/86426
 2021-08-14 15:44:24 +02:00
Max Brunsfeld c6dd5da5e6
Merge pull request #1329 from narpfel/improve-performance 
Improve performance of `tree-sitter generate`
 2021-08-11 16:08:23 -07:00
Paul Gey 965e3c9e5e Generator::add_parse_table: Store entries in hash map 
This avoids a quadratic behaviour due to repeatedly using `find` on a
growing `Vec`.
 2021-08-08 21:45:43 +02:00
Paul Gey cf69a2c94c Use IndexMap and FxHash for some hot hash maps 2021-08-08 21:45:43 +02:00
Andrew Hlynskyi 533073cdb5 fix(cli): Remove tree-sitter grammar ./... call limitation 2021-08-06 02:11:35 +03:00
Kolja Lampe c21bec3716 Always print where the playground is running 2021-08-06 01:58:24 +03:00
Kolja Lampe 99243ddaf2 Correct quiet description for playground 2021-08-06 01:58:24 +03:00
Andrew Hlynskyi bc13d6e504
Merge pull request #1319 from ahlinc/fix/dead-code-in-logger 
fix(cli): allow dead code in Logger; fixes CI failure
 2021-08-06 01:08:58 +03:00
Andrew Hlynskyi d8ac8f5d16 fix(cli): allow dead code in Logger 2021-08-04 03:38:58 +03:00
Max Brunsfeld c7998d93d0
Merge pull request #1278 from CyberShadow/pull-20210719-104342 
feat(cli): Make "test" output more readable
 2021-07-29 15:06:18 -07:00
Max Brunsfeld 59cd1c3962
Merge pull request #1235 from ahlinc/feat/cli-help-messages 
feat(cli): Add a lot of help messages for CLI options
 2021-07-19 11:32:32 -07:00
Vladimir Panteleev 505bc5e1af
feat(cli): Make "test" output more readable 2021-07-19 10:56:21 +00:00
Andrew Hlynskyi 7f538170bf fix(parser): count rows in the debug log from 0 2021-07-15 11:47:14 +03:00
Max Brunsfeld 9cb732859f Update unit test to reflect HTML grammar change 2021-07-12 12:21:10 -07:00
Andrew Hlynskyi b4208ee307 feat(cli): Add a lot of help messages for CLI options 2021-07-03 16:14:57 +03:00
Andrew Hlynskyi bc94c0cc2f fix(cli): fix theme key loading from config.json, closes #1232 2021-07-03 03:38:24 +03:00
Andrew Hlynskyi 60f0953b65 feat(cli): Set TREE_SITTER_DEBUG env var on 'tree-sitter parse -d' 2021-06-30 18:51:43 +03:00
Max Brunsfeld 190fee3625
Merge pull request #1177 from forbisc/master 
update set_included_ranges to modify extent if the current position is at the very beginning of the range.
 2021-06-30 08:38:59 -07:00
Max Brunsfeld c512a0eed7
Merge pull request #1194 from ahlinc/fix/1032 
Close #1032 - fix all weirdness in the generated Cargo.toml
 2021-06-29 16:48:23 -07:00
Max Brunsfeld bd1f7f6881
Merge pull request #1196 from ahlinc/feat/independent-binding-files-gen 
feat(cli): Independent binding files generation
 2021-06-29 16:28:46 -07:00
Max Brunsfeld af0ed73f76
Merge branch 'master' into fix/1032 2021-06-29 15:59:15 -07:00
Max Brunsfeld 23433f4826
Merge pull request #1210 from ahlinc/fix/init-config-experience 
fix(cli): init-config experience, it shouldn't silently overwrite existing config file
 2021-06-29 15:53:36 -07:00
Max Brunsfeld e7d3b29470
Merge pull request #1180 from CyberShadow/pull-20210618-110411 
Pin emscripten/emsdk Docker version
 2021-06-29 15:45:24 -07:00
Vladimir Panteleev b14ea51e3d
Refactor emscripten/emsdk version to a single file 2021-06-29 21:39:12 +00:00
Andrew Hlynskyi f22d62393b fix(cli): actual Rust binding version in generated Cargo.toml 2021-06-30 00:36:11 +03:00
Cameron Forbis 1ea797fbd2 run fmt 2021-06-29 13:59:21 -07:00
Cameron Forbis ae5340901e address code review comments 2021-06-29 13:49:43 -07:00
Andrew Hlynskyi 202a37f1a1 chore(cli): Use DeriveDisplayOrder Clap's setting 
To show sub commands in the defined order that looks more accurate
and have more meaning for sub commands use order.
 2021-06-27 03:37:56 +03:00
Andrew Hlynskyi 2e2b239c42 fix(cli): Don't silently overwrite existing config file 2021-06-27 03:15:35 +03:00
Vladimir Panteleev 5adf7bdcde
cli/src/main.rs: Fix panic with empty file list to parse 
E.g.: tree-sitter parse --paths /dev/null
 2021-06-26 18:11:30 +00:00
Vladimir Panteleev 725f3f7f2b
Pin emscripten/emsdk Docker version 
Fixes issues caused by incompatible changes in Emscripten since the
time that tree-sitter was built.
 2021-06-26 18:07:12 +00:00
Max Brunsfeld d7746bec99
Merge pull request #1198 from ahlinc/feat/color-and-aliases 
feat(cli): colored help and frequent command aliases
 2021-06-24 09:12:48 -07:00
Max Brunsfeld cf8e7bd413
Merge pull request #1199 from dcreager/edit-file 
cli: Allow `$` in `tree-sitter parse --edit`
 2021-06-24 09:07:38 -07:00
Andrew Hlynskyi bd0e60240e fix(cli): Change -h -> -H for the highlight subcommand to don't clash with the help flag 2021-06-24 12:09:00 +03:00
Andrew Hlynskyi 650ee94800 fix(cli): disable help subcommand, -h, --help are enough 2021-06-24 12:04:12 +03:00
Andrew Hlynskyi 1220ec3852 feat(cli): Rename web-ui -> playground with fallback alias 2021-06-24 12:04:12 +03:00
Andrew Hlynskyi 80ee33b073 feat(cli): Add short aliases for frequent subcommands 2021-06-24 11:55:50 +03:00
Andrew Hlynskyi 04badf8a8e feat(cli): Enable clap colored help 2021-06-24 11:55:50 +03:00
Douglas Creager 71c43a869b Allow $ in tree-sitter parse --edit 
You can now specify `$` as the position to apply an edit, signifying the
end of the file.  (That prevents you from having to calculate the size
of the file yourself.)
 2021-06-23 10:04:29 -04:00
Andrew Hlynskyi 421815a68d feat(cli): Make more clearer sub command selection 2021-06-23 10:51:55 +03:00
Andrew Hlynskyi d3527109a8 Updating of binding.gyp should depend on its content instead of bindings/node folder 2021-06-23 02:42:48 +03:00
Andrew Hlynskyi 22d63338a2 Use double quoted patterns for more precise pattern matching in the binding.gyp files 2021-06-23 02:41:30 +03:00
Andrew Hlynskyi 86b8137457 Add create_path_else fn to handle creation or modification 2021-06-23 02:40:32 +03:00
Andrew Hlynskyi 797c7668c1 feat(cli): Independant language binding files generation 2021-06-23 02:39:38 +03:00
Andrew Hlynskyi 4578e58794 fix(cli): close #1032 - fix repository template url generation in cargo.toml 2021-06-23 01:02:29 +03:00
Cameron Forbis 9182ebef86 update set_included_ranges to modify extent if the current position is at the very beginning of the included range 2021-06-17 16:42:25 -07:00
Edgar 9feca80b08
Make HighlightConfiguration::configure use into iterator instead of a Vec<String> 2021-06-16 19:14:45 +02:00
Andrew Hlynskyi 318b00e437 fix(cli): Don't print 'Error:' prefix 2021-06-14 10:46:39 +03:00
Douglas Creager b44270efab cli: Missing config file shouldn't be an error 
Just fall back on the default values for each configuration option.
 2021-06-10 10:22:20 -04:00
Douglas Creager e841fcfa1b cli: Extract CLI configuration into separate crate 
This patch adds the `tree-sitter-config` crate, which manages
tree-sitter's configuration file.  This new setup allows different
components to define their own serializable configuration types, instead
of having to create a single monolithic configuration type.  But the
configuration itself is still stored in a single JSON file.

Before, the default location for the configuration file was
`~/.tree-sitter/config.json`.  This patch updates the default location
to follow the XDG Base Directory spec (or other relevant platform-
specific spec).  So on Linux, for instance, the new default location is
`~/.config/tree-sitter/config.json`.  We will look in the new location
_first_, and fall back on reading from the legacy location if we can't
find anything.
 2021-06-10 10:21:22 -04:00
Douglas Creager 66c30648c2 rust: Extract runtime language detection into separate crate 
This patch adds a new `tree-sitter-loader` crate, which holds the CLI's
logic for finding and building local grammar definitions at runtime.
This allows other command-line tools to use this logic too!
 2021-06-09 17:42:30 -04:00
Douglas Creager 75da247317 cli: Ignore BrokenPipe errors again 
With the change to anyhow in the previous commit, we stopped ignoring
BrokenPipe errors.  Now we do again, not as a core part of our error
type, but as part of the `main` functions reaction to any error that
occurs.
 2021-06-09 16:17:23 -04:00
Douglas Creager d2d01e77e3 cli: Use anyhow and thiserror for errors 
This patch updates the CLI to use anyhow and thiserror for error
management.  The main feature that our custom `Error` type was providing
was a _list_ of messages, which would allow us to annotate "lower-level"
errors with more contextual information.  This is exactly what's
provided by anyhow's `Context` trait.

(This is setup work for a future PR that will pull the `config` and
`loader` modules out into separate crates; by using `anyhow` we wouldn't
have to deal with a circular dependency between with the new crates.)
 2021-06-09 16:17:23 -04:00
Max Brunsfeld 9d77561c43
Merge pull request #1150 from ahlinc/fix/panic-on-broken-pipe 
Stylistic fix for #1135 - Panic on broken pipe
 2021-06-08 11:21:04 -07:00
Andrew Hlynskyi 8bbf27b82b chore(cli): Panic on broken pipe; stylistic fix 2021-06-08 21:04:26 +03:00
Max Brunsfeld 22620dee73
Merge pull request #1135 from ahlinc/fix/panic-on-broken-pipe 
fix(cli): Panic on broken pipe
 2021-06-07 11:15:37 -07:00
Andrew Hlynskyi ce2bb561b4 chore(cli): Panic on broken pipe; review fixes 2021-06-07 21:11:49 +03:00
Andrew Hlynskyi dbdda25224 fix(cli): Panic on broken pipe; use conditional match arm with ErrorKind 2021-06-07 20:22:12 +03:00
Max Brunsfeld f3ea60e23f Merge branch 'master' into query-cursor-api 2021-06-02 11:51:26 -07:00
Douglas Creager cd96552448 query: Allow configurable match limit 
The default is now a whopping 64K matches, which "should be enough for
everyone".  You can use the new `ts_query_cursor_set_match_limit`
function to set this to a lower limit, such as the previous default of
32.
 2021-06-02 11:30:55 -04:00
Andrew Hlynskyi 65dc922944 fix(cli): Panic on broken pipe for: 'tree-sitter parse ... | head' 
Introduce a concept of ignored errors when the Error wrapper struct
contains None instead of Vec of String messages.
 2021-05-31 18:34:09 +03:00
Max Brunsfeld d72771a19f Make ::set_{byte,point}_range methods take a Range 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 14:21:11 -07:00
Max Brunsfeld 851f55afce Report non-rooted matches that intersect cursor's range restriction 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 11:58:38 -07:00
Max Brunsfeld 919e9745a6 Add ts_tree_cursor_goto_first_child_for_point function 
This function (and the similar `ts_tree_cursor_goto_first_child_for_byte`)
allows you to efficiently seek the tree cursor to a given position,
exploiting the tree's internal balancing, without having to visit
all of the preceding siblings of each node.
 2021-05-27 12:30:19 -07:00
Max Brunsfeld fda35894d4 Stop matching new patterns past the end of QueryCursor's range 
This restores the original signatures of the `set_byte_range` and
`set_point_range` functions. Now, the QueryCursor will properly report
matches that intersect, but are not fully contained by its range.

Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-25 18:02:35 -07:00
Max Brunsfeld f597cc6a75 Preserve matches that contain the QueryCursor's start byte 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
Co-Authored-By: Antonio Scandurra <me@as-cii.com>
 2021-05-25 13:06:24 -07:00
Max Brunsfeld a61f25bc58 Add APIs for advancing a QueryCursor to an arbitrary position 2021-05-24 21:07:59 -07:00
Douglas Creager 78010722a4 query: Allow unlimited pending matches 
Well, not completely unlimited — we're still using a 16-bit counter to
keep track of them.  But we longer have a static maximum of 32 pending
matches when executing a query.
 2021-05-24 11:02:58 -04:00
Max Brunsfeld 8c3d1466ec Allow QueryCursor's text callback to return an iterator 2021-05-23 21:05:26 -07:00
Andrew Hlynskyi 0ae70289ae fix(cli): Correct fallback on docker compilation for all platforms, fixes #1124 2021-05-23 17:04:34 +03:00
Niranjan Hasabnis c0e2b43d34 Fixing Rust formatting issue 2021-05-21 15:43:10 +00:00
Niranjan Hasabnis c31acb8fec Changing API name; Adding unit test and Rust bindings 2021-05-21 01:50:10 +00:00
Andrew Hlynskyi 3c0152a331 chore(fmt): Apply 'cargo fmt' to the whole code base 2021-05-19 23:21:43 +03:00
Markus F.X.J. Oberhumer cc519b3121 cli: Improve const-correctness of the generated parsers (part 2 of 2). 
This is a follow-up to my previous commit 1badd131f9 .

I've made this an extra patch as it requires a minor
API change in <tree_sitter/parser.h>.

This commit moves the remaining generated tables into
the read-only segment.

Before:
  $ for f in bash c cpp go html java javascript jsdoc json php python ruby rust; do \
       gcc -o $f.o -O2 -Ilib/include -c test/fixtures/grammars/$f/src/parser.c; \
    done
  $ size --totals *.o
      text    data     bss     dec     hex filename
   5353477   24472       0 5377949  520f9d (TOTALS)

After:
  $ for f in bash c cpp go html java javascript jsdoc json php python ruby rust; do \
       gcc -o $f.o -O2 -Ilib/include -c test/fixtures/grammars/$f/src/parser.c; \
    done
  $ size --totals *.o
   5378147       0       0 5378147  521063 (TOTALS)
 2021-05-19 12:49:57 +02:00
Max Brunsfeld fe5a29a1c6
Merge pull request #1081 from ahlinc/feat/fail-ci-on-compiler-warnings 
feat: Fail CI on compiler warnings
 2021-05-01 09:26:44 -07:00
Andrew Hlynskyi b856f7e1bd Remove unneeded dead_code annotations 2021-04-30 06:55:00 +03:00
Douglas Creager 108679ffdc
Merge pull request #1076 from tree-sitter/cli-match-limit-warning 
cli: Warn about queries that exceed match limit
 2021-04-28 15:37:43 -04:00
Douglas Creager 850007718a cli: Warn about queries that exceed match limit 2021-04-27 09:38:43 -04:00
Markus F.X.J. Oberhumer 1badd131f9 cli: Improve const-correctness of the generated parsers. 
This moves most of the generated tables from the data segment into
the text segment (read-only memory) so that it can be shared between
different processes.

As a bonus side effect we can also remove all casts in the generated parsers.

Before:
  size --totals target/scratch/*.so
      text    data     bss     dec     hex filename
    853623 4684560    2160 5540343  5489f7 (TOTALS)

After:
  size --totals target/scratch/*.so
      text    data     bss     dec     hex filename
   5472086   68616     480 5541182  548d3e (TOTALS)
 2021-04-27 09:22:18 +02:00
Max Brunsfeld 347b3eb243
Merge pull request #1069 from ahlinc/fix/package-json-generation 
package.json generation improvements
 2021-04-22 14:27:36 -07:00
Andrew Hlynskyi b57266cbb8 test(binding_rust): Use some redundancy to trigger more clear comliler error on the master 2021-04-22 21:00:58 +03:00
Andrew Hlynskyi 2d9a2a1855 test(binding_rust): test query captures/matches lifetime is separate from nodes lifetime 2021-04-22 21:00:58 +03:00
Andrew Hlynskyi 7aa538dd97 fix(cli): use dashed language name in generated package.json and Cargo.toml files 2021-04-22 16:29:48 +03:00
Andrew Hlynskyi 9416f975d3 fix(cli): set actual cli version in generated package.json 2021-04-22 16:29:48 +03:00
an-kumar aabe6100d0
Update generated Cargo.toml's tree-sitter dependency 
tree-sitter 0.19.0 bumped the language version from 12 to 13. `npm install tree-sitter-cli` gets a recent version of tree-sitter, which generates languages with language version 13. However, the Cargo.toml generated from `tree-sitter generate` still has a an old tree-sitter as a dependency. This causes the rust bindings to not work out of the box, as the tree-sitter library expects language version 12.

It would be nice to add a test for this in CI.  `tree-sitter generate` already creates a test for the rust binding, and that test fails out of the box due to the language mismatch.
 2021-04-09 10:59:51 -07:00
Amin Yahyaabadi 90c9a3a2f8 Refactor get_emcc_path and use PathBuf 2021-04-04 21:16:53 -05:00
Amin Yahyaabadi 919eab023f Fix build-wasm on Windows 2021-04-04 19:07:16 -05:00
Max Brunsfeld c3eb5daa31 Include has_preceding_inherited_fields in Item's hash impl 2021-03-27 10:08:24 -07:00
Max Brunsfeld 72143b17ff Fix handling of queries with field names at the roots of patterns 
Fixes #1018
 2021-03-27 09:36:13 -07:00
Max Brunsfeld 89e1157a29 Fix handling of repetitions in query analysis 
Fixes #1007
 2021-03-19 11:00:31 -07:00
Max Brunsfeld a440e11197
Merge pull request #996 from ahlinc/fix/test-precondition 
fix: assertion in highlight::tests::test_parse_style test
 2021-03-18 10:25:02 -07:00
Andrew Hlynskyi 192f51324e Added more test for byte and point ranges 2021-03-17 19:11:20 +02:00
Andrew Hlynskyi 148680b3b9 fix: assertion in highlight::tests::test_parse_style test 2021-03-17 15:52:21 +02:00
pbug90 9d04efda94 update wording 2021-03-13 19:55:04 +01:00
pbug90 9a833d165e update test command filter help 2021-03-13 19:53:10 +01:00
pbug90 858160172f update test command help outputs 2021-03-13 19:45:55 +01:00
Max Brunsfeld bc0ef5f373 Add negated field patterns to queries 2021-03-12 14:38:02 -08:00
Max Brunsfeld 08d17e33bd Fix unused import warning on windows 2021-03-11 16:43:20 -08:00
Max Brunsfeld a40045a419 When editing, properly invalidate trees that depend on get_column 2021-03-11 14:46:13 -08:00
Max Brunsfeld 57036b4f8a Extract lexer helper functions for all large char sets 
No need to restrict it to char sets used in multiple places.
This is important because the helper functions are now implemented
more efficiently than the inline comparisons (using a binary search).
 2021-03-11 11:48:48 -08:00
Max Brunsfeld 7ebdebeafd
Merge pull request #871 from marcel0ll/master 
Add ";" as comment in s-expressions
 2021-03-11 09:24:50 -08:00
Andrew Hlynskyi a331607f4e dsl.js: Reuse sym() in RuleBuilder 2021-03-10 23:06:53 +02:00
Max Brunsfeld 9dc88061d5 query: Fix last child operator with multiple children 
Fixes #806
 2021-03-09 13:40:15 -08:00
Max Brunsfeld 9e50befcf8 For node-types.json, process supertypes in a stable order 2021-03-08 12:02:01 -08:00
Max Brunsfeld 8e894ff3f1 Add --no-bindings flag to generate subcommand 2021-03-08 12:01:45 -08:00
Max Brunsfeld 6ba11d7778 Fix query analysis with top-level non-terminal extras 
Fixes https://github.com/tree-sitter/tree-sitter/issues/968#issuecomment-792314281
 2021-03-08 09:33:35 -08:00
Max Brunsfeld 5a2534ac4a Fix crash on unterminated alternation in query 
Fixes #968
 2021-03-07 08:48:03 -08:00
Max Brunsfeld f18c36ca62 Fix handling of extra nodes in query analysis 
Fixes #967
 2021-03-06 14:35:41 -08:00
Max Brunsfeld db6e1d9bdc Add ts_query_cursor_did_exceed_match_limit API 2021-03-05 15:02:49 -08:00
Max Brunsfeld 047d33eecf Remove bad logic for detecting later siblings with the same field 
Co-authored-by: Douglas Creager <dcreager@dcreager.net>
 2021-03-05 14:50:16 -08:00
Max Brunsfeld 7300249d20 Fix incorrect merging of states with different inherited fields 
Co-authored-by: Douglas Creager <dcreager@dcreager.net>
 2021-03-05 14:49:28 -08:00
Max Brunsfeld e20aff9a9c Fix templates for rust binding files 2021-03-04 14:22:31 -08:00
Max Brunsfeld e12093e8df Fix regression introduced in CharacterSet optimization 2021-03-04 13:50:27 -08:00
Max Brunsfeld dd4cba2625 Allow symbols to be used in precedence lists 2021-03-03 13:11:05 -08:00
lazytype 0c4ea79e2e Slightly improve highlight test hygiene 2021-03-01 14:13:02 -05:00
Max Brunsfeld 592fd8678d Organize TSLanguage fields 
Due to the breaking ABI change in #943, this is our chance
to reorder the fields in a more logical way.
 2021-03-01 10:27:22 -08:00
Max Brunsfeld a6b267201f
Merge pull request #952 from lazytype/master 
Support highlighting in truecolor, falling back to the closest xterm …
 2021-03-01 09:19:31 -08:00
Max Brunsfeld 3f315f167b
Merge pull request #945 from hvithrafn/update-dependencies 
Update dependencies; fix breakage
 2021-03-01 09:15:38 -08:00
lazytype 5de649b7aa Support highlighting in truecolor, falling back to the closest xterm color if the terminal does not support it 
Fixes #758
 2021-02-28 16:22:45 -05:00
Max Brunsfeld d56f9ebe4e Re-enable --prev-abi flag to generate command 2021-02-26 14:51:01 -08:00
Max Brunsfeld 075bf2bd5c In generate, create rust bindings 
Also, migrate node binding files into the same 'bindings' folder.
 2021-02-26 13:24:21 -08:00
hvithrafn 05f79f0f90 Update dependencies; fix breakage 2021-02-26 10:55:20 -07:00
Max Brunsfeld c25fa9910e Fix query match failure when indefinite steps have no captures 
Fixes #937
 2021-02-25 21:41:49 -08:00
Max Brunsfeld c1639cc456 Add production_id_count field to Language objects 
I think this is the last additional field that's needed so
that every array member of TSLanguage has a length that
can be calculated at runtime.
 2021-02-25 16:32:05 -08:00
Patrick Thomson 63616289dd
Merge pull request #941 from tree-sitter/walk-query-files-recursively 
Walk query files recursively in `tree-sitter test`.
 2021-02-25 16:33:05 -05:00
Max Brunsfeld e49a56ea4c
Merge pull request #939 from tree-sitter/partial-order-precedence 
Allow precedences to be specified using strings and a partial ordering relation
 2021-02-25 13:16:01 -08:00
Max Brunsfeld d8a235faa1 Add further static validation of named precedences 2021-02-25 11:54:21 -08:00
Patrick Thomson 44010d69ea Walk query files recursively in tree-sitter test. 
We were only walking one level of depth into the `queries/` folder
during invocations of `test`, which made us attempt to open folders
rather than recurse into them.

We have to pull in the `walkdir` crate, which is required for
cross-platform walking of directories.

Fixes #938.
 2021-02-25 10:24:54 -05:00
Max Brunsfeld 344797c110 Implement named precedence comparison 2021-02-24 16:02:56 -08:00
Max Brunsfeld d40f118370 Generalize precedence datatype to include strings 
Right now, the strings are not used in comparisons, but they
are passed through the grammar processing pipeline, and are
available to the parse table construction algorithm.

This also cleans up a confusing aspect of the parse table
construction, in which precedences and associativities were
temporarily stored in the parse table data structure itself.
 2021-02-23 20:48:39 -08:00
Douglas Creager a7e2b6a8b2 Remove allocations_stubs 
These were used to provide the `ts_record_*` functions regardless of
whether the `allocation-tracking` feature is enabled.  The allocation
tracking code is now implemented entirely in the `lib` crate, and only
when the feature is enabled, and so these stubs are no longer needed.
 2021-02-23 09:39:39 -05:00
Douglas Creager a29c8d9264 Move allocation tracking into lib crate 
We have several test cases defined in the `cli` crate that depend on the
`lib` crate's `allocation-tracking` feature.  The implementation of the
actual allocation tracker used to live in the `cli` crate, close to the
test cases that use it.  The `allocation-tracking` feature in the `lib`
crate was just used to tell the tree-sitter implementation to expect
that the allocation tracker exists, and to use it.

That pattern meant that we had a circular dependency: `cli` depends on
`lib`, but `lib` required some code that was implemented in `cli`.
That, in turn, caused linker errors — but only when compiling in certain
configurations! [1]

This patch moves all of the allocation tracking implementation into the
`lib` crate, gated on the existing `allocation-tracking` feature, which
fixes the circular dependency.

Note that this patch does **not** fix the fact that feature unification
causes the `lib` crate to be built with the `allocation-tracking`
feature enabled, even though it's not a default.  Fixing that depends on
the forthcoming version 2 feature resolver [2], or using the `dev_dep`
workaround [3] in the meantime.

[1] https://github.com/tree-sitter/tree-sitter/issues/919
[2] https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-resolver-version-2
[3] https://github.com/tree-sitter/tree-sitter/issues/919#issuecomment-777107086
 2021-02-23 09:16:37 -05:00
Max Brunsfeld 2f28a35e1b Handle unicode property escapes inside bracketed char classes 
Refs #906
 2021-02-18 22:27:44 -08:00
Max Brunsfeld 29bc26ecd5 Fix test failure after non-terminal extras change 2021-02-18 15:43:01 -08:00
Max Brunsfeld 86a891fa63 Fix bugs in parser generation for non-terminal extras 
Previously, we attempted to completely separate the parse states
for item sets with non-terminal extras from the parse states
for other rules. But there was not a complete separation.

It actually isn't necessary to separate the parse states in this way.
The only special behavior for parse states with non-terminal extra rules
is what happens at the *end* of the rule: these parse states need to
perform an unconditional reduction.

Luckily, it's possible to distinguish these *non-terminal extra ending*
states from other states just based on their normal structure, with
no additional state.
 2021-02-18 14:14:22 -08:00
Max Brunsfeld b46d51f224 Add a unit test for all unicode character escape forms 2021-02-17 17:49:01 -08:00
Max Brunsfeld 5b630054c6 Handle negated unicode property escapes in regexes 
Refs #380
 2021-02-17 17:22:33 -08:00
Max Brunsfeld 6ae04051e7 Tweak whitespace in generated character set functions 2021-02-17 16:32:49 -08:00
Max Brunsfeld 9d9eb2234f
Merge pull request #906 from tree-sitter/unicode-property-escapes 
Handle simple unicode property escapes in regexes
 2021-02-17 16:14:42 -08:00
Max Brunsfeld dad8546776 Generate more compact code for character set binary search 2021-02-17 13:52:23 -08:00
Max Brunsfeld 6132a10b1c Use binary search in generated character set functions 2021-02-17 13:08:56 -08:00
Max Brunsfeld f5a4c14dbe Add some doc comments to CharacterSet 2021-02-16 21:37:52 -08:00
Max Brunsfeld 73e252e39b Add unit test for child_by_field_name w/ hidden nodes 2021-02-10 16:14:24 -08:00
Max Brunsfeld 6dbe6a3a90 Merge branch 'master' into actions-ci 2021-02-05 10:19:05 -08:00
Max Brunsfeld 2b0de9dfec Fix small bugs in conflict reporting 
* Negative precedence values were not displayed
* Rule names were repeated in resolution suggestions
 2021-02-01 13:30:06 -08:00
Max Brunsfeld e3ba701344 Start work on handling unicode property escapes in regexes 2021-01-29 16:37:45 -08:00
Max Brunsfeld 38444ea7f9
Merge pull request #904 from tree-sitter/character-set-ranges 
Represent CharacterSet internally as a vector of ranges
 2021-01-29 13:35:48 -08:00
Andrew Hlynskyi 2b9e5f6c4b Fix hiding problems in ./build/Debug/tree_sitter_*_binding 
In debug building modules also may happen errors and a current implementation
completely hides them, so errors like 'undefined symbol' can't be
easily identified due to wrong traceback and error message.
 2021-01-29 15:54:10 +02:00
Max Brunsfeld ab78ab3f9b Represent CharacterSet internally as a vector of ranges 2021-01-28 16:10:39 -08:00
marceloll 057d299297 Allow comments only at line start 2021-01-08 13:28:40 -03:00
marceloll 0f5563c536 Add test for ; comments 2021-01-08 13:28:40 -03:00
marceloll 9500aff052 Use ; as comment in tests segxp section 2021-01-08 13:28:40 -03:00
Yijun Yu 9e08712773 Add --xml option to save the parsed code into XML, using node.start_byte() and node.end_byte() 2021-01-04 22:07:38 +00:00
Max Brunsfeld 391fc8c340 Update unit tests to expect ruby grammar change 2020-12-20 19:34:45 -08:00
Max Brunsfeld 96f259d8c5 Run rustfmt 2020-12-03 09:48:20 -08:00
Max Brunsfeld 026231e93d Merge branch 'master' into HEAD 2020-12-03 09:44:33 -08:00
Max Brunsfeld b661050a61 Simplify setup for enabling/disabling allocation recording in the C lib 2020-12-02 15:35:13 -08:00
Max Brunsfeld 751ffd2ee1 Use new emscripten when building with docker 2020-12-01 11:04:06 -08:00
Patrick Thomson cc8f978b3b inline this lambda 2020-11-23 12:05:32 -05:00
Patrick Thomson e1da6e554b Remove fanciful nomenclature. 2020-11-23 12:01:08 -05:00
Patrick Thomson 6764b803a0 Allow overlap in specs. 2020-11-23 11:58:07 -05:00
Patrick Thomson 0b4661e401 Really fix the tests. 2020-11-23 11:41:16 -05:00
Patrick Thomson f3d16f4770 Fix tests. 2020-11-23 11:34:56 -05:00
Patrick Thomson c1a8985712 Merge remote-tracking branch 'origin/master' into query-testy 2020-11-23 11:10:53 -05:00
Max Brunsfeld b267f90e64 Update unit tests to reflect python and ruby grammar changes 2020-11-16 10:51:08 -08:00
Patrick Thomson 4604b40b72 better name for capture regex 2020-11-10 16:23:39 -05:00
Patrick Thomson 50bccdf5da rename Assertion.expected to expected_capture_name 2020-11-10 16:20:51 -05:00
Patrick Thomson c9c886d971 Merge remote-tracking branch 'origin/master' into query-testy 2020-11-10 16:16:53 -05:00
Max Brunsfeld 99cd283e39 query: Fix detection of repeated field names 
Fixes #790
 2020-11-02 14:07:39 -08:00
Max Brunsfeld 3497f34dd7 Fix parser-generation bugs introduced in #782 2020-11-02 13:43:28 -08:00
Arthur Baars d62e7f7d75 Add test case with extra_symbols 2020-10-30 10:58:41 +01:00
Arthur Baars f07dda692e Ensure "extras" symbols are included in the node-types.json file 
The symbols marked as "extras" are the start symbols of secondary
languages. These should be included in the aliases map just as done
for start symbol of the main language to ensure their node type and
field information is included in the node-types.json file.
 2020-10-29 18:05:24 +01:00
Max Brunsfeld c2c63baf5b query: Fix escape sequence parsing in anonymous node patterns 
Fixes #776
Fixes #760
 2020-10-28 13:55:13 -07:00
Max Brunsfeld 071f4e40f1 Fix generate error when there are aliases in unused rules 2020-10-28 12:34:16 -07:00
Max Brunsfeld a2d760e426 Ensure nodes are aliased consistently within syntax error nodes 
Co-Authored-By: Rick Winfrey <rewinfrey@github.com>
 2020-10-27 15:46:09 -07:00
Patrick Thomson 0bd223f032 Better naming for this regex. 2020-10-27 13:11:57 -04:00
Patrick Thomson 938eae8536 Merge remote-tracking branch 'origin/master' into query-testy 2020-10-27 09:16:58 -04:00
Patrick Thomson 1aee60a7c0 propitiate the tests 2020-10-26 14:35:18 -04:00
Patrick Thomson 6adeb7b40d move shared code to query_testing 2020-10-26 14:27:33 -04:00
Patrick Thomson 1012bea3f4 let's start sharing this code 2020-10-26 13:35:10 -04:00
Patrick Thomson f364ce2304 Remove old assertion stuff 2020-10-26 13:22:12 -04:00
Patrick Thomson 9af9d66e19 it works 2020-10-26 13:13:25 -04:00
Patrick Thomson 3e18e97f7c start pulling in the stuff from test_highlight 2020-10-26 12:58:32 -04:00
Max Brunsfeld b972a7158d Tweak cancellation logic for CLI commands 
In 'parse' and 'highlight' cancel on stdin if stdin is a tty.
 2020-10-23 12:15:44 -07:00
Patrick Thomson e370c5053e this is nicer, though 2020-10-23 14:11:46 -04:00
Patrick Thomson 363a0ce4fc things are working: time to piggyback off the highlighter's parser 2020-10-21 14:54:47 -04:00
Patrick Thomson 0dfe89f353 parse assertions from regex capture 2020-10-21 13:32:04 -04:00
Patrick Thomson c691df5ae2 reading in the source correctly 2020-10-21 12:56:11 -04:00
Patrick Thomson 947528f019 use our Result type here 2020-10-21 12:49:41 -04:00
Patrick Thomson 91d5d59d85 Introduce query/assert and call it in query.rs. 2020-10-21 12:37:24 -04:00
Max Brunsfeld 8bb8e9b8b3 Initialize TSLanguage fields in order of their declaration 
This makes parser.c valid under the C++20 standard
 2020-10-15 07:20:12 -07:00
Max Brunsfeld 0a46033391 Remove duplication of LossyUtf8 helper 2020-10-14 11:35:50 -07:00
Max Brunsfeld 857a9ed07b query: Handle captured wildcard nodes at the root of patterns 2020-10-08 12:34:08 -07:00
Patrick Thomson 1f3248a3e0
Merge pull request #749 from tree-sitter/ensure-extras-is-array 
Fix crash when extras function doesn't return an array.
 2020-10-05 16:57:27 -04:00
Patrick Thomson b9b478873a
Merge pull request #748 from tree-sitter/fix-nonexistent-file-crash 
Fix crash when nonexistent files were passed to `parse`.
 2020-10-05 16:14:09 -04:00
Patrick Thomson adce3cb8e2 Merge remote-tracking branch 'origin/master' into ensure-extras-is-array 2020-10-05 16:12:37 -04:00
Max Brunsfeld d1c95193c1 query: Fix invalid use of slice::from_raw_parts 2020-10-05 12:08:53 -07:00
Patrick Thomson 683a2da055 Fix crash when extras function doesn't return an array. 
Fixes #745, which failed due to attempting to call `map` on a
non-array. This bails out at the same spot, but with a more
illuminating error message.
 2020-09-30 16:21:20 -04:00
Patrick Thomson 470733b323 Fix crash when nonexistent files were passed to parse. 
We were unwrapping the result of counting the characters in the vector
returned by collect_files(), which, if that vector is empty, returns
None. The most correct behavior is to halt if a nonexistent filename
was provided or a glob failed.
 2020-09-30 15:52:21 -04:00
Patrick Thomson 33435f43c0 Take Max's suggestions. 2020-09-30 09:28:58 -04:00
Patrick Thomson 16bd061b33 Have the caller track stats here. 2020-09-29 15:43:30 -04:00
Patrick Thomson 939cdf12b9 Add --stats flag for reporting parse information. 2020-09-29 12:34:25 -04:00
Max Brunsfeld ba239ce4ab Make query error line numbers consistently display 1-indexed 2020-09-24 15:03:51 -07:00
Max Brunsfeld 518916f221 Return correct path and line in query errors from the CLI 2020-09-24 13:47:27 -07:00
Max Brunsfeld 297e2bcb28 static query analysis: Fix handling of fields in hidden nodes 2020-09-23 16:55:48 -07:00
Max Brunsfeld ffd3bdc4c1 Escape ? in C string literals 
Fixes #714
 2020-09-23 13:06:06 -07:00
Max Brunsfeld cb343cad5e Avoid reusing the root node during incremental parsing 
Fixes #712
 2020-09-23 12:59:27 -07:00
Max Brunsfeld 21c3bbc4b4 Account for supertypes during query analysis 2020-09-23 10:55:31 -07:00
Max Brunsfeld 5003064da7 Make supertypes automatically hidden, without underscore prefix 2020-09-23 09:35:14 -07:00
Max Brunsfeld 8835dfda99 Fix test for supertypes in queries 2020-09-21 13:11:54 -07:00
Max Brunsfeld b5a9adb555 Allow queries to match on supertypes 
Co-authored-by: Ayman Nadeem <aymannadeem@github.com>
 2020-09-21 12:34:48 -07:00
Max Brunsfeld f10a8448ed
Merge pull request #723 from tree-sitter/comprehensive-wasm-help 
Conditionally handle the presence/absence of build-wasm files.
 2020-09-15 11:54:57 -07:00
Max Brunsfeld ff488f89c9 Make the --prev-abi flag work w/ the newest abi change 2020-09-08 10:58:20 -07:00
Patrick Thomson 2a1bd3dbc2 Better naming. 2020-09-08 09:25:38 -04:00
Patrick Thomson 865f59ad74 No need for platformish logic in build.rs. 2020-09-08 09:23:54 -04:00
Patrick Thomson 92a17e782f Conditionally compile with a cfg variable instead. 2020-09-02 15:32:12 -04:00
Max Brunsfeld 36a8821f3e Fix behavior of the last child operator in tree queries 2020-09-02 12:10:04 -07:00
Max Brunsfeld 4aba684d66 Control recursion depth explicitly during query analysis 2020-08-24 16:55:56 -07:00
Max Brunsfeld 2eb04094f8 Handle aliased parent nodes in query analysis 2020-08-21 14:12:04 -07:00
Max Brunsfeld 456b1f6771 Fix handling of alternations and optional nodes in query analysis 2020-08-20 16:28:54 -07:00
Max Brunsfeld 9daec9cb22 Tweak impossible pattern error messages 2020-08-20 13:35:11 -07:00
Max Brunsfeld 4301110c12 query: Indicate specific step that's impossible 2020-08-20 13:06:38 -07:00
Max Brunsfeld bd42729a41 query: Avoid early-returning captures due to predicates 2020-08-18 13:01:45 -07:00
Max Brunsfeld 604f9e8148 query: Assign is_definite correctly for steps within nested sub-patterns 2020-08-18 10:55:03 -07:00
Max Brunsfeld 91fc9f5399 Use is_definite flag in ts_query_cursor_next_capture 2020-08-17 16:50:59 -07:00
Max Brunsfeld 228a9e28e1 Add tests for impossible queries 2020-08-17 13:27:17 -07:00
Max Brunsfeld c3f9b2b377 Fix query analysis bugs found in ruby tags query 2020-08-17 09:57:06 -07:00
Max Brunsfeld 1ea29053e1 Merge branch 'master' into query-pattern-is-definite 2020-08-14 09:31:55 -07:00
Patrick Thomson 94ab884ee4 Add a test. 2020-08-05 12:16:09 -04:00
Patrick Thomson 5c86a9c654 Fix the tests 2020-08-05 11:52:07 -04:00
Patrick Thomson f4108056b0 Remove otiose pattern match. 2020-08-05 11:33:04 -04:00
Patrick Thomson 5a52dc2cd7 Return an iterator-bool tuple instead of just an iterator. 2020-08-05 11:18:59 -04:00
Max Brunsfeld af655547e5 Fix handling of queries with many patterns with leading repetitions 2020-07-31 12:47:58 -07:00
Max Brunsfeld e89a19a158 tags: Add @ignore capture 2020-07-29 15:33:48 -07:00
Max Brunsfeld 81bbdf19f4 Fix handling of non-terminal extras that share non-extra rules 
Fixes #701
 2020-07-29 09:50:13 -07:00
Max Brunsfeld 1ae5cbc851 query: Handle #not-match? in rust, wasm bindings 2020-07-24 12:15:23 -07:00
Max Brunsfeld 32099050d6 node_types: Fix panic when field is associated with a hidden token 
Fixes #695
 2020-07-24 09:26:56 -07:00
Max Brunsfeld de2b71d465 Fix query bug when max permutations are exceeded 2020-07-23 16:05:50 -07:00
Tuấn-Anh Nguyễn 740d864e67 Add '.' as a valid start of a predicate, in addition to '#' 
See https://github.com/ubolonton/emacs-tree-sitter/issues/38
 2020-07-19 15:46:39 +07:00
Max Brunsfeld 82aa1462fd Clean up get_variable_info function 2020-07-17 15:12:13 -07:00
Max Brunsfeld c4fca5f73e node types: Fix handling of repetitions inside of fields 
Fixes #676
 2020-07-17 14:19:59 -07:00
Max Brunsfeld f4adf0269a Propagate dynamic precedence correctly for inlined rules 
Fixes #683
 2020-07-17 09:53:01 -07:00
Max Brunsfeld 4535efce69 query: Prevent dropping of matches when exceeding range maximum 
Fixes #685
 2020-07-17 09:39:06 -07:00
Max Brunsfeld 91a715799e Accept a paths file to most CLI subcommands 2020-07-14 15:04:39 -07:00
Tuấn-Anh Nguyễn c2fb0f5229 cli: Add --byte-range flag to query command 2020-07-12 20:47:23 +07:00
Max Brunsfeld 0c2dc4c1e9
Merge pull request #648 from tree-sitter/tagging-improvements 
Tagging improvements
 2020-07-10 13:48:23 -07:00
Max Brunsfeld e4e785b567 Remove unused flags from tags CLI command 2020-07-10 13:47:56 -07:00
Max Brunsfeld 0bfd47e2e5 Improve error message when failing to run graphviz stuff 
Fixes #682
 2020-07-10 10:13:19 -07:00
Max Brunsfeld b52f28d6d5 Allow measuring time for tags subcommand 2020-07-09 11:28:07 -07:00
Max Brunsfeld fef72fb434 Merge branch 'tags-utf16-columns' into tagging-improvements 2020-07-09 10:40:46 -07:00
Max Brunsfeld 255cf0a9cf tags: Add utf16 column ranges to tags 
Also, ensure that line ranges contain only valid UTF8.

Co-authored-by: Tim Clem <tclem@github.com>
Co-authored-by: Beka Valentine <bekavalentine@github.com>
 2020-07-08 16:13:58 -07:00
Max Brunsfeld d614c14c2c tags: Make spans refer to name, not entire def/ref 
Co-authored-by: Tim Clem <tclem@github.com>
Co-authored-by: Beka Valentine <bekavalentine@github.com>
 2020-07-08 12:37:01 -07:00
Max Brunsfeld 0bf2450b4a Always enforce stack version limit during reductions 
Fixes #669
 2020-07-06 15:58:33 -07:00
Max Brunsfeld cc37da7457 Query analysis: fix propagation of uncertainty from later siblings 2020-06-26 16:46:12 -07:00
Max Brunsfeld 19baa5fd5e Clean up and document query analysis code 2020-06-25 17:56:43 -07:00
Max Brunsfeld 891de051e2 Fix population of subgraph nodes when analyzing queries 2020-06-25 15:06:27 -07:00
Max Brunsfeld 7f955419a8 Start work on recognizing impossible patterns 2020-06-25 15:06:27 -07:00
Max Brunsfeld 4c2f36a07b Mark steps as definite on query construction 
* Add a ts_query_pattern_is_definite API, just for debugging this
* Store state_count on TSLanguage structs, to allow for scanning parse tables
 2020-06-25 15:06:27 -07:00
Max Brunsfeld deeeb67a3b
query: Fix handling of alternations under field names (#661) 2020-06-24 14:20:56 -07:00
Timothy Clem f166947abb Test updates, definition/reference prefix is now required 2020-06-18 15:05:27 -07:00
Timothy Clem 17d26c0d5a Improved errors 2020-06-18 14:43:27 -07:00
Timothy Clem b6ae67a610 Fix up CLI, use new syntax_type_name 2020-06-18 14:43:10 -07:00
Timothy Clem 75724698f0 Fix up tests 2020-06-18 14:42:41 -07:00
Timothy Clem 3bcb1f8c94 Assert line trimming 2020-06-18 10:48:33 -07:00
Timothy Clem f24a952cb4 Minor output changes 2020-06-17 15:54:36 -07:00
Timothy Clem 929bb40adc Shorten to def/ref 2020-06-17 10:34:55 -07:00
Timothy Clem 80f5c52259 Tests compile 2020-06-16 17:19:35 -07:00
Timothy Clem d802b37791 Bring back a SyntaxType enum 2020-06-16 17:09:34 -07:00
Timothy Clem 9bf4939b9a Show if tag is a def/ref in the cli 2020-06-16 16:04:22 -07:00
Max Brunsfeld a6f71328fe Avoid whitelist/blacklist terminology in test comments 2020-06-16 09:22:34 -07:00
Max Brunsfeld 519a1369ce In highlight test, reset included ranges before finding assertions 2020-06-15 10:51:34 -07:00
Max Brunsfeld 0e5ff14976
Requery the parse table when breaking down the parse stack on invalid lookahead (#636) 
* Requery parse table after breaking down parse stack due to invalid lookahead

* Include Ruby parser in randomized test suite

Ruby and PHP are our only two languages that use non-terminal extras.
Adding Ruby uncovered some bugs.

* Print edited source code when running parse --edit w/ debug flag

* Recompute lookahead when breaking down stack on invalid lookahead

* Fix stack summary leak when there are two discontinuities on a stack version
 2020-06-04 13:40:04 -07:00
Max Brunsfeld 47d607da8d
Add alternative syntax in queries (#630) 
* Add alternative syntax in queries

* Add tests and tweak error handling for alternatives in queries
 2020-06-01 13:23:07 -07:00
Max Brunsfeld ec870e9e66 Avoid extracting helpers for char sets that are only used once 2020-05-26 16:37:45 -07:00
Max Brunsfeld 911fb7f1b2
Extract helper functions to reduce the code size of the lexer function (#626) 
* Extract helper functions to reduce code size of ts_lex

* Name char set helper functions based on token name
 2020-05-26 13:39:11 -07:00
Max Brunsfeld 462c86903f
Improve tree queries' ability to handle large numbers of nested matches (#624) 
* query: Acquire capture lists lazily, allow more concurrent states

* Fix some static analysis warnings
 2020-05-18 13:40:24 -07:00
lerencao 37ee7acc9e
[cli]: add an option to no open browser in web-ui command (#620) 2020-05-15 23:56:40 -07:00
Алексей Пастухов 38d32c018b
add Rust into languages list for corpus tests (#619) 2020-05-15 16:02:39 -07:00
Max Brunsfeld 9d182bb078 node-types: Fix bug w/ required property when multiple rules aliased as same 2020-05-14 10:51:18 -07:00
Max Brunsfeld b66d149b74 Fix inconsistent whitespace after '{' in generated parser 2020-05-13 15:56:49 -07:00
Max Brunsfeld 40993195b8
Fix wasm tests on CI (#616) 
* wasm: Improve error message on missing language symbol

* Fix source file existence checks in build-wasm command
 2020-05-13 15:14:43 -07:00
Max Brunsfeld 2934e219cf Fix build-wasm command 2020-05-12 16:56:21 -07:00
Max Brunsfeld cdc973866f Fix build-wasm command on latest emscripten 2020-05-12 15:42:11 -07:00
Max Brunsfeld 85c998d572 Change the wildcard syntax in tree queries 
1. Use '_' instead of '*'.
2. Add '*' as a postfix operator for zero-or-more repetitions

Signed-off-by: Patrick Thomson <patrickt@github.com>
 2020-05-11 13:04:04 -07:00
Max Brunsfeld 40262483a9 Change query syntax for predicates 
Signed-off-by: Patrick Thomson <patrickt@github.com>
 2020-05-11 12:35:51 -07:00
Max Brunsfeld b14f564550 Add a test for nested optionals in queries 2020-05-11 10:42:48 -07:00
Max Brunsfeld 9c0535cea6 Fix logic for aborting failed matches 2020-05-08 14:15:25 -07:00
Max Brunsfeld b47c170c75 Query: fix bugs and add tests for top-level and nested repetitions 2020-05-08 12:10:01 -07:00
Max Brunsfeld 3ad71625dd Fix query bugs, expand and clean up query tests 2020-05-07 14:22:15 -07:00
Max Brunsfeld 3456a21f0d Start work on restructuring query implementation to deal w/ optionals and repeats better 2020-05-07 12:41:25 -07:00
Riccardo Schirone 780e9cecc9 Do not use multiple unnamed structs inside of unions 2020-04-29 20:42:45 +02:00
Max Brunsfeld 35f82ce301 Fix incorrect parent values after call to child_by_field_name 
Refs tree-sitter/node-tree-sitter#61
Refs tree-sitter/tree-sitter-javascript#127
 2020-04-03 11:21:51 -07:00
Max Brunsfeld 8eac81b8df Merge branch 'master' into tags 2020-03-25 12:49:15 -07:00
Max Brunsfeld 783c087aec tags: Handle cancellation 2020-03-25 11:26:52 -07:00
Max Brunsfeld 59c457c5cf tags: Fix typo in tag ranges 2020-03-20 11:35:27 -07:00
Max Brunsfeld a003e5f6bd generate: Avoid duplicate string tokens in unique symbol map 2020-03-20 11:35:11 -07:00
Max Brunsfeld ae1c51051a Fix tag order in JS tags test 2020-03-20 11:34:33 -07:00
Max Brunsfeld aedab72afa tags: Start work on handling local variables for ruby support 2020-03-20 10:13:03 -07:00
Max Brunsfeld 651fa38c93 Add unit test for tagging via C API. Fix docs handling 2020-03-18 10:40:15 -07:00
Max Brunsfeld 591e066226 tags: Make cli output more human readable 2020-03-17 12:05:09 -07:00
Max Brunsfeld 94a60b8e13 tags: Start adapting Tag struct for use in C API 2020-03-17 11:19:30 -07:00
Max Brunsfeld d0325579ad Use Arc to avoid use-after-free in threaded cancellation unit test 
Fixes #579
 2020-03-16 14:46:05 -07:00
Max Brunsfeld 94bbf14d0e tags: Add test where no comments are adjacent to definition 2020-03-16 14:28:58 -07:00
Max Brunsfeld 65f2874b9e query: Optimize handling of patterns with a wildcard at the root 
Avoid adding and removing states for these patterns on every node in the tree
by just skipping the wildcard step of the matching process
 2020-03-16 14:02:31 -07:00
Max Brunsfeld 1b3a67834b cli: Fix loading of tags query 2020-03-13 16:13:31 -07:00
Max Brunsfeld b5483c67ab query: allow repetition operator to be used on non-terminal nodes 2020-03-13 16:12:39 -07:00
Max Brunsfeld b5f2ed83fe tags: Implement select-adjacent! predicate 2020-03-13 13:02:56 -07:00
Max Brunsfeld 0457736766 rust: add handling of arbitrary predicate operators 2020-03-13 13:02:34 -07:00
Max Brunsfeld 6e2df06dc2 Start proving out tags support for JavaScript 2020-03-12 16:33:52 -07:00
Max Brunsfeld 6f636a0357 query: Add postfix '+' operator for token repetition 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-12 15:10:58 -07:00
Max Brunsfeld 05c1d44e80 Merge branch 'master' into tags 2020-03-11 13:15:26 -07:00
Max Brunsfeld e3aad995f6 query: Fix handling of patterns with wildcards at the root 2020-03-11 13:14:16 -07:00
Patrick Thomson d798bd6bd9 Slice out the line associated with a tag. 2020-03-10 20:39:04 -04:00
Max Brunsfeld 4996cbe830 cli: Move more of the tags code from main into the tags module 2020-03-10 16:52:10 -07:00
Max Brunsfeld 0e02ead0de Update tags test to reflect new handling of escapes in queries 2020-03-10 15:53:42 -07:00
Max Brunsfeld 4531130b44 Merge branch 'master' into tags 2020-03-10 15:50:27 -07:00
Max Brunsfeld 741eed01b7 query: Handle escape sequences and escaped quotes in string literals 2020-03-10 15:50:06 -07:00
Max Brunsfeld fc4d5c3a33 🔥 Dead test code 2020-03-10 12:23:13 -07:00
Max Brunsfeld 17cc38678c Get generate_tags with the new iterator API 2020-03-10 12:05:09 -07:00
Max Brunsfeld 0eb162c685 wip: converting generate_tags to return an iterator 2020-03-10 11:45:31 -07:00
Max Brunsfeld 157258d881 tags: Implement strip regex for docs processing 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-10 10:43:23 -07:00
Max Brunsfeld 7f4828254f Fix criteria for detecting when an aborted parse is resuming 2020-03-09 11:30:08 -07:00
Patrick Thomson 00dcc1eaa6 Need to use expression_statement here. 2020-03-06 17:48:55 -05:00
Max Brunsfeld 680a9e0531 wip 2020-03-06 13:24:03 -08:00
Max Brunsfeld a3f0087b11 Start work on tagging unit test 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-05 13:04:49 -08:00
Max Brunsfeld feac368a30 Start work on new tree-sitter-tags crate 
Co-Authored-By: Patrick Thomson <patrickt@users.noreply.github.com>
 2020-03-04 14:27:31 -08:00
Mark Schmitz b1c7768cc2
Output also HTML_FOOTER with highlight --html (#550) 
* Output also HTML_FOOTER with highlight --html

* move html footer output after end of for loop, as only one closing tag
is needed
 2020-03-02 15:04:15 -08:00
Max Brunsfeld ee46218a73 Fix incremental parsing problem with non-terminal extras 
Also add PHP grammar as a fixture to test against.
 2020-03-02 14:17:12 -08:00
Max Brunsfeld 6cb8d24de2
Merge pull request #542 from SKalt/issue-524-document-supertypes-in-grammar-schema 
feat(cli): documented optional supertypes string[] in grammar schema
 2020-02-24 16:14:49 -08:00
Steven Kalt d82ee739e9
Update cli/src/generate/grammar-schema.json 
Co-Authored-By: Max Brunsfeld <maxbrunsfeld@github.com>
 2020-02-24 18:13:38 -05:00
Max Brunsfeld 709ddfebe9 docs: Add explanation of syntax highlighting configuration for CLI 2020-02-21 11:39:27 -08:00
Max Brunsfeld 360b188644 cli: Handle 'underline' styling when highlighting w/ HTML output 2020-02-20 09:35:46 -08:00
Max Brunsfeld 570b83e2b2 query: Add immediate child operator 2020-02-19 11:47:52 -08:00
Max Brunsfeld 950a89a525 query: Differentiate between wildcard '*' and named wildcard '(*)' 2020-02-19 09:42:29 -08:00
Alyssa Verkade 0e689657b7 Add a language linkage declaration to parsers 
Previously, in order to compile a `tree-sitter` grammar that contained
c++ source in the parser (ie the `scanner.cc` file), you would have to
compile the `parser.c` file separately from the c++ files. For example,
in rust this would result in a `build.rs` close to the following:
```
extern crate cc;

fn main() {
  let dir: PathBuf = ["tree-sitter-ruby", "src"].iter().collect();

  cc::Build::new()
    .include(&dir)
    .cpp(true)
    .file(dir.join("scanner.cc"))
    // NOTE: must have a name that differs from the c static lib
    .compile("tree-sitter-ruby-scanner");

  cc::Build::new()
    .include(&dir)
    .file(dir.join("parser.c"))
    // NOTE: must have a name that differs from the c++ static lib
    .compile("tree-sitter-ruby-parser");
}
```

This was necessary at the time for the following grammars: `ruby`,
`php`, `python`, `embedded-template`, `html`, `cpp`, `ocaml`,
`bash`, `agda`, and `haskell`.

To solve this, we specify an `extern "C"` language linkage declaration
to the functions that must be linked against to compile a parser with the
scanner, making parsers linkable against c++ source.
On all major compilers (gcc, clang, and msvc) this should be the only
change needed due to the combination of clang and gcc both supporting
designated initialization for years and msvc 2019 adopting designated
initializers as a part of the C++20 conformance push.

Subsequently, for rust projects, the necessary `build.rs` would become
(which also brings these parsers into sync with the current docs):
```
extern crate cc;

fn main() {
  let dir: PathBuf = ["tree-sitter-ruby", "src"].iter().collect();

  cc::Build::new()
    .include(&dir)
    .cpp(true)
    .file(dir.join("scanner.cc"))
    .file(dir.join("parser.c"))
    .compile("tree-sitter-ruby");
}
```
 2020-02-18 19:46:59 -08:00
Steven Kalt c5ca259d09
feat(cli): documented optional supertypes Array<string> in grammar-schema.json 2020-02-15 11:23:14 -05:00
Max Brunsfeld de8e3ee188 query: Allow multiple captures on a single node 2020-02-11 16:02:32 -08:00
Max Brunsfeld 5922064b75 node-types: Fix ambiguity warning on rustc 1.41 2020-02-10 10:26:12 -08:00
Max Brunsfeld e23f518915 highlight: add built-in support for carriage-return highlight 2020-01-28 14:47:21 -08:00
Max Brunsfeld d06407aca2 Update highlight test for JSDoc grammar changes 2020-01-27 13:08:30 -08:00
Max Brunsfeld 3f109a3cb5 highlight: Fix logic for handling empty injections with no highlights 2020-01-27 12:32:37 -08:00
Max Brunsfeld 8dd68c360a Fix logic for generating unique symbol map 
Previously, this didn't correctly handle the case where *multiple* 
symbols were all simply-aliased to the same *other* symbol.

Refs #500
 2020-01-27 12:06:48 -08:00
Max Brunsfeld 9f63139a10 Fix error when set_included_ranges is called with an invalid range list 2020-01-17 10:31:28 -08:00
Max Brunsfeld 7421836ee4 Improve cli error message on invalid glob 2020-01-16 16:17:33 -08:00
Max Brunsfeld 9d460e6d01
Merge pull request #525 from SKalt/add-field-rule-to-grammar-json-schema 
added field-rule to grammar-schema.json
 2020-01-16 16:06:48 -08:00
Max Brunsfeld 9dfd03e79a highlight: Sipmlify injection API w/ new injection.combined property 2020-01-16 12:43:31 -08:00
Max Brunsfeld f3747863df Add ts_query_disable_pattern API 2020-01-15 17:08:55 -08:00
Max Brunsfeld 58617cfa0c Improve output format for query subcommand 2020-01-15 17:08:31 -08:00
Steven Kalt 619d7cd65a
added field-rule to grammar-schema.json 2020-01-12 19:40:21 -05:00
Steven Kalt e69430ae7d removed duplicate key (#521) 
The key "required" was duplicated on "symbol-rule". I removed the more permissive copy.
 2020-01-09 17:34:07 -08:00
Max Brunsfeld 9a73277389 web-ui: Load static files from disk if TREE_SITTER_BASE_DIR var is set 2019-12-19 11:23:14 -08:00
Max Brunsfeld f53e7377dc Allow highlight to command to take glob patterns 2019-12-17 15:49:05 -08:00
Max Brunsfeld f0e4d630d5 Reliably avoid duplicate subtypes in node-types.json 2019-12-17 14:55:51 -08:00
Max Brunsfeld 3ac047679c Fix children.required when multiple nodes are aliased as the same 2019-12-16 12:39:57 -08:00
Max Brunsfeld 7c711c5537 Move the private functions in node_types.rs to the end of the file 2019-12-16 12:38:40 -08:00
Max Brunsfeld 5edf97bca9 Remove accidentally-commited garbage line in arg parsing config 2019-12-16 11:59:26 -08:00
Max Brunsfeld 9a0cfa2376 Fix 'required' field when rules with diff fields are aliased to look the same 2019-12-12 11:26:58 -08:00
Max Brunsfeld fc19312913 Fix node-types bugs involving aliases and external tokens 2019-12-12 10:06:18 -08:00
Max Brunsfeld 1b5ae380ce
Don't attempt to extract keywords that don't entirely match word token (#505) 2019-12-11 17:18:15 -08:00
Max Brunsfeld c2e253a4c8 Update unit test to reflect javascript query change 2019-12-10 13:24:36 -08:00
Max Brunsfeld a5a9000e29 generate: Ensure that field_map_slices array is long enough 2019-12-09 11:46:32 -08:00
Steven Kalt 09fb24c03d add encoding to web_ui.html (#504) 
addressing #503 .
 2019-12-06 19:28:08 -08:00
Max Brunsfeld 0cb2ef1082 Fix code paths that still conflated null characters with EOF 2019-12-06 15:29:03 -08:00
Max Brunsfeld d426f46087 Add unit test for symbols respecting simple aliases 2019-12-06 12:47:27 -08:00
Max Brunsfeld 7032dae4f6 Include alias symbols in unique symbol map 2019-12-06 12:11:09 -08:00
Max Brunsfeld bea6e0e28b cli: Generate parseres with the new ABI by default 2019-12-06 11:51:55 -08:00
Max Brunsfeld bd49fbab12 cli: Allow globs and exclusions (via '!' prefix) as args to parse 2019-12-06 11:41:21 -08:00
Max Brunsfeld 56c620c005 Store a mapping to ensure no two symbols map to the same metadata 2019-12-05 17:21:46 -08:00
Max Brunsfeld d6c7b243a7
Merge pull request #499 from tree-sitter/highlight-test 
Add a system for testing syntax highlighting queries
 2019-12-05 15:55:24 -08:00
Max Brunsfeld 4b9feccd13 Run syntax highlighting tests as part of the main test command 
Also, allow `corpus` to be in the `test/corpus` directory, and expect 
highlighting tests to be in the `test/highlight` directory.
 2019-12-05 15:28:16 -08:00
Max Brunsfeld 6dbfbaed07 Start work on a system for testing syntax highlighting 2019-12-05 09:45:54 -08:00
Max Brunsfeld c957a5b8e7 Add test for handling of too many states in query captures 2019-11-22 11:54:32 -08:00
Max Brunsfeld 5a979d1457 node-types: Add test for field merging with aliases 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-11-19 14:31:56 -08:00
Max Brunsfeld 1d63434664 Refactor node-type merging 2019-11-19 11:57:21 -08:00
Max Brunsfeld 5489bc4dc5 Fix small issues with merging node types 
* Merge the `required` field with an 'and', not an 'or'
* Merge field info in addition to children info
 2019-11-19 11:55:21 -08:00