libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-11 07:56:28 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
5714 commits 20 branches 114 tags 51 MiB
Commit graph

1363 commits

Author SHA1 Message Date
Amaan Qureshi 274d0cda0a fix: binding.cc overwrite should replace PARSER_NAME 2024-03-01 10:59:25 -05:00
Amaan Qureshi 6b2879144f build(deps): bump deps & lockfile 2024-02-29 02:17:30 -05:00
Amaan Qureshi 5a4765ca01 style: clippy lints 2024-02-29 02:17:30 -05:00
Amaan Qureshi 60a935139b refactor!: remove top-level corpus dir for tests 
It's confusing to have tests in two different top-level directories when working between different grammars, and most of them use `test/corpus` which is more fitting, so time to go.
 2024-02-29 02:17:30 -05:00
Amaan Qureshi 4ff33e81bb fix(assertions): case shouldn't matter for comment node detection 2024-02-29 02:17:30 -05:00
Amaan Qureshi 8c4861c186 feat(test): add attributes for corpus tests 2024-02-29 02:17:30 -05:00
ObserverOfTime b3c14c9295 build(bindings): make everything c11 2024-02-29 01:50:04 -05:00
ObserverOfTime b4b2d9cecc refactor: remove extern/const where possible 2024-02-29 01:50:04 -05:00
Amaan Qureshi d92a7a9690 test: add test case for parse stack merging with incorrect error cost bug 2024-02-28 08:51:28 -05:00
Segev Finer e87cf7ef15
refactor(bindings)!: convert node bindings to NAPI 
Co-authored-by: Boris Verkhovskiy <boris.verk@gmail.com>
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
 2024-02-27 17:06:25 -05:00
Amaan Qureshi eb332ffa4d style(bindings): consistent wording 2024-02-27 16:07:38 -05:00
ObserverOfTime 502dacf220 refactor: rename TS_REUSE_ALLOCATOR flag 
TREE_SITTER_REUSE_ALLOCATOR is more consistent
 2024-02-27 15:54:38 -05:00
ObserverOfTime 46520eda58 build(bindings): metadata improvements 2024-02-27 15:54:38 -05:00
Amaan Qureshi f7c13406c9 build: add uninstall command 2024-02-27 15:54:38 -05:00
ObserverOfTime 799833f9cf build: use c11 everywhere 
And improve the makefiles
 2024-02-27 15:54:38 -05:00
ObserverOfTime 412a6a774e refactor(bindings): remove npmignore 
Replace with package.json files
 2024-02-27 06:22:00 -05:00
Amaan Qureshi 8ae3870b51 fix(generate): add .npmignore, populate Swift's exclude list 2024-02-26 14:30:48 -05:00
ObserverOfTime 721eca2394 fix(bindings): editorconfig and setup.py fixes 2024-02-26 12:00:16 -05:00
Amaan Qureshi f1eecf9786 feat: expose the allocator and array header files for external scanners 2024-02-26 09:43:13 -05:00
Amaan Qureshi 9e5bf6591f feat: improve language bindings 
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
 2024-02-25 19:06:01 -05:00
Max Brunsfeld 29d3583bdf Use workspace dependencies for internal crates 2024-02-25 11:14:29 -08:00
Amaan Qureshi 32c23b6c90 fix: wrap || comparison in parenthesis when && is used 2024-02-24 01:35:40 -05:00
Max Brunsfeld 1f751bbd5f
Merge pull request #2841 from segevfiner/patch-1 
Make Node.js language bindings context aware
 2024-02-23 11:23:11 -08:00
Will Lillis f894a5350c
feat(cli): add optional config-path argument 2024-02-21 12:02:12 -05:00
Amaan Qureshi 58a4fcc792 chore: get rid of github_issue_test file 2024-02-21 12:01:25 -05:00
Amaan Qureshi 38efefd8bd style: cleaner cast 2024-02-21 12:01:25 -05:00
Amaan Qureshi 167855b289 fix(test): edge case when parsing UNEXPECTED/MISSING nodes with an indentation level greater than 0 2024-02-21 12:01:25 -05:00
Amaan Qureshi a1a3903c10 fix(test): allow writing updates to tests without erroneous nodes instead of denying all of them if a single error is found 2024-02-21 12:01:25 -05:00
Amaan Qureshi b97208704b
fix(cli): don't use long for grammar_path 
Co-authored-by: buckynbrocko <77247638+buckynbrocko@users.noreply.github.com>
 2024-02-19 16:05:08 -05:00
Amaan Qureshi b40839cd72 style: prefer turbofish syntax where possible 2024-02-19 16:00:50 -05:00
Amaan Qureshi fd91404ab0 style: tidying 2024-02-19 16:00:50 -05:00
Amaan Qureshi d95fcc83b9 fix: remove redundant imports 2024-02-19 06:36:48 -05:00
Amaan Qureshi f526be8061 test: update html tests 2024-02-19 06:36:48 -05:00
Amaan Qureshi 5b299eafe5
fix: don't throw an error if the user uses map in the grammar 2024-02-18 00:26:14 -05:00
Amaan Qureshi da1f890752 chore: error out when multiple arguments are passed to token/token.immediate 2024-02-17 01:26:44 -05:00
Amaan Qureshi 8dd65ccbc0 refactor: &PathBuf -> &Path 2024-02-17 00:45:28 -05:00
Amaan Qureshi ef1b714e08 fix(cli): don't update tests automatically if parse errors are detected 2024-02-16 20:23:39 -05:00
Amaan Qureshi e32a7f3998 chore: clippy lints 2024-02-16 16:32:07 -05:00
Amaan Qureshi 4342efd57e feat: allow specifying an external scanner's files 2024-02-16 16:31:56 -05:00
Amaan Qureshi b6c75ccec1 chore: update relevant rust tests 2024-02-16 12:34:12 -05:00
Amaan Qureshi e996c32108 refactor!: remove the apply-all-captures flag, make last-wins precedence the default 2024-02-16 12:34:12 -05:00
Nickolay 6895b7a1e1
Add some documentation to the playground page 
The minimal UI of the Playground could benefit from some documentation to make it easier for the newer users to understand what's going on. Also added a link to the new documentation from the local playground.

Closes https://github.com/tree-sitter/tree-sitter/issues/1305
 2024-02-15 16:24:21 +01:00
Amaan Qureshi d80d101e34 build: move common Cargo.toml keys into the workspace and inherit them 2024-02-14 14:56:23 -05:00
Amaan Qureshi 73f56bffa9 fix: sexp format edge case with quoted closed parenthesis 2024-02-13 18:31:28 -05:00
Amaan Qureshi 4303ab99c9 fix: properly handle Query.matches when filtering out results 2024-02-13 16:40:05 -05:00
Amaan Qureshi 7e0dd7b9c1 feat(cli): add an optional grammar-path argument for the playground 2024-02-13 16:28:54 -05:00
Amin Yahyaabadi 51c147053e
feat: error out if an empty string is in the extras array 
This prevents never-ending loops in the parser
 2024-02-13 15:00:48 -05:00
Amaan Qureshi 5ea0dbf77a chore: some more clippy lints 2024-02-13 03:33:07 -05:00
Amaan Qureshi a07f988905 refactor: extract regex check into a function and lower its precedence 2024-02-13 03:33:07 -05:00
Bedis Nbiba 7dd096c5f7
feat: implement first-line-regex 2024-02-13 01:51:41 -05:00
Amaan Qureshi 21f25a5305 feat: improve error message for files with an unknown grammar path 2024-02-12 17:17:37 -05:00
Amaan Qureshi 3da79ba2b6 fix: update schema for regex flags 2024-02-12 15:39:14 -05:00
Amaan Qureshi 39df8e2833 chore(test): use different languages for async tests 2024-02-12 06:30:36 -05:00
Amaan Qureshi e6d67ecde0 chore(wasm): use SIDE_MODULE=2 to silence warning 2024-02-11 20:18:30 -05:00
Amaan Qureshi 4149ed4149 chore: provide a CLI flag to open log.html 
On macOS, this was done by default regardless of what the user wants.
This was also not done on Windows or Linux. Instead, we now provide a
`--open-log` flag to open the log file in the default browser, and it
works on all platforms.
 2024-02-11 02:28:34 -05:00
Amaan Qureshi ce5f5fffc1 chore(generate): dedup and warn about duplicate or invalid rules 2024-02-11 01:28:31 -05:00
Amaan Qureshi 78ccc39ca9
fix: properly error out when the word property is an invalid rule 2024-02-11 00:37:23 -05:00
Amaan Qureshi cca814afd2 chore: add java and tsx to corpus tests 2024-02-10 18:25:35 -05:00
Amaan Qureshi 3d68ca4583 fix: inherited grammar generation 2024-02-10 18:25:35 -05:00
Amaan Qureshi dcb34b25ec fix: prettify xml output and add node position info 2024-02-08 18:09:10 -05:00
Amaan Qureshi a969fc428c chore: move dependencies into the workspace and inherit from there 2024-02-08 15:24:09 -05:00
Amaan Qureshi 7980669267 feat: support dot graph output on windows 2024-02-08 15:21:47 -05:00
Amaan Qureshi 214e27083a feat(cli): upgrade to clap v4 2024-02-08 05:39:41 -05:00
dundargoc df1fe842eb docs: various fixes 
Closes https://github.com/tree-sitter/tree-sitter/issues/1317.
Closes https://github.com/tree-sitter/tree-sitter/issues/1752.
Closes https://github.com/tree-sitter/tree-sitter/issues/2439.

Co-authored-by: Simon Hengel <sol@typeful.net>
Co-authored-by: Akash Yadav <itsaky01@gmail.com>
Co-authored-by: Sebastiaan Speck <12570668+sebastiaanspeck@users.noreply.github.com>
Co-authored-by: Laytan Laats <laytanlaats@hotmail.com>
 2024-02-08 00:08:59 +01:00
Amaan Qureshi 767db31114
fix: remove redundant code 2024-02-07 07:13:03 -05:00
Amaan Qureshi 993ee75d8c
fix: don't add extraneous exports to the json and add missing rules to the schema 2024-02-07 07:10:51 -05:00
Daumantas Kavolis 08d211a546
feat: improve time reports 2024-02-07 03:19:02 -05:00
Amaan Qureshi 59be1edaa1
refactor: swap &Vec[T] with &[T] where appropriate 2024-02-07 02:50:31 -05:00
buckynbrocko 8b38ab67de
chore(cli): reword test diff legend 2024-02-07 01:33:06 -05:00
buckynbrocko a4a7edc5af
feat(cli): add include/exclude options to the test subcommand 2024-02-07 01:04:54 -05:00
dundargoc c8bd6705cf
chore: clippy 2024-02-06 23:34:14 -05:00
Amaan Qureshi f601c87091
fix: do not skip over spaces that are surrounded by quotes 2024-02-06 19:00:43 -05:00
Amaan Qureshi a0ddae9d1a
Merge pull request #2724 from plugwash/master 
cli - update to regex-syntax 0.8
 2024-02-06 19:00:11 -05:00
Max Brunsfeld f4788b49c0 Merge branch 'master' into wasm-stdlib 2024-02-05 11:55:58 -08:00
Max Brunsfeld ad9091afe0 Fix wasm build errors due to builtin symbols 2024-02-05 10:19:34 -08:00
root b7d48b48fd
chore(cli): update to regex-syntax 0.8 
regex-syntax has restructured the Ast enum a bit, rather than having a Class
member it now has seperate members for different types of Class.
 2024-02-05 01:22:09 -05:00
Amaan Qureshi 000c2f4dec
feat: allow the grammar to log to stdout 2024-02-05 00:46:34 -05:00
Amaan Qureshi 55afb4efaf
ci: remove unnecessary targets 2024-02-04 04:19:09 -05:00
Amaan Qureshi 04ff704bca
chore(cli): apply clippy fixes 2024-02-04 04:18:48 -05:00
Max Brunsfeld e054de4191 Return an error from build-wasm if scanner uses unavailable functions 2024-02-02 12:04:49 -08:00
Amaan Qureshi 9f2dc9d6b5
fix: rework parser.h includes for test grammars and multi-grammar repos 2024-02-02 10:42:39 -05:00
Amaan Qureshi 422e74fbdb
chore: update javascript-relevant tests 2024-02-02 08:58:22 -05:00
Max Brunsfeld 1d8975319c
Merge pull request #2840 from tree-sitter/language-reference-count 
Introduce APIs for managing the lifetimes of languages, allow WASM languages to be deleted
 2024-01-30 10:24:37 -08:00
Max Brunsfeld d351f81b4a Release stale instances from wasm store when languages are dropped 2024-01-29 10:17:49 -08:00
Steven Kalt d35efd4608
feat(cli): support building WASM via podman 
Previously, `tree-sitter build-wasm` had the ability to build WASM
by using docker to pull in an image with a complete emscripten toolchain.
This commit adds the ability to use podman to do the same thing.

Using podman requires two notable changes:
1. Using the fully-qualified image name. Docker defaults to prepending
    `docker.io` to the image name, but podman does not.
2. Podman will mount the `/src/` volume as belonging to root unless
  `--userns=keep-id` is passed. I think podman's different
  volume-ownership is related to podman's daemonless execution and
  `--uidmap` functionality, but I'm not 100% sure.

To test, I ran
```sh
script/fetch-fixtures
script/generate-fixtures
script/generate-fixtures-wasm # <- the important one!
```

which worked as well as the docker version.
 2024-01-29 00:50:32 -05:00
Max Brunsfeld c4142737ec Merge branch 'master' into language-reference-count 2024-01-25 12:43:56 -08:00
Max Brunsfeld 3c66dd2d6c Add missing language return in ts_query_new 
Co-authored-by: Antonio <antonio@zed.dev>
Co-authored-by: Mikayla <mikayla@zed.dev>
Co-authored-by: Joseph <joseph@zed.dev>
 2024-01-25 11:06:54 -08:00
Amaan Qureshi b26e0a8ec0
fix: account for grammars in subdirectories, add typescript to test corpus 2024-01-25 14:02:12 -05:00
Max Brunsfeld 3139760fa9 Record allocations in wasm language error test 2023-12-31 10:05:06 -08:00
Segev Finer 4a42c6d5f1
Make Node.js language bindings context aware 
They don't have any dynamic global data, so all it takes is just declaring them as such
 2023-12-28 16:39:19 +02:00
Max Brunsfeld 82c1b5ead3 Fix compile error in parser hang test 2023-12-27 15:10:52 -08:00
Max Brunsfeld 4a8e4b1963 Allow wasm languages to be deleted 2023-12-27 15:00:16 -08:00
Max Brunsfeld da16cb1459 Introduce language ref-count management C APIs, remove Copy impl for Language in Rust 2023-12-27 14:59:16 -08:00
Andrew Hlynskyi d56b51a11d
Revert "Alt #2454" 2023-11-29 11:20:05 +02:00
Max Brunsfeld 034f0d0280
Merge pull request #1864 from tree-sitter/wasm-language 
Add optional WASM feature to the native library, allowing it to run wasm-compiled parsers via wasmtime
 2023-11-28 12:08:47 -08:00
Max Brunsfeld 13dd76e444 Return an informative error on failing to construct a WasmStore 2023-11-27 17:48:29 -08:00
Max Brunsfeld ac29eab5f9 Remove tests for LookaheadIterator compilation failures 
These add noise when running tests. Let's not add unit tests for
compilation failurs.
 2023-11-27 10:32:25 -08:00
Max Brunsfeld 6fd7a1e44e Return informative error when load_language fails 2023-11-26 12:15:05 -08:00
Max Brunsfeld e9289d3b30 Fix error when wrong language name is passed to load_language 2023-11-24 20:25:28 -08:00
Max Brunsfeld b59716600a Add a test that loads multiple languages in a wasm store 2023-11-24 20:03:54 -08:00
Amaan Qureshi 0ff28346be
Merge pull request #2714 from MrPrezident/test_point_range_captures_fix 
fix "test_point_range_captures not working"
 2023-11-12 21:09:09 -08:00
MrPrezident 3c4957e940 fix "test_point_range_captures not working" 
Fix for https://github.com/tree-sitter/py-tree-sitter/issues/105
 2023-11-12 20:58:51 -06:00
Max Brunsfeld d56d1a32ac Fix test grammar loading in unit test 2023-11-04 12:26:00 -07:00
Max Brunsfeld 473b3c8208 Add a CLI feature flag for wasm runtime support 2023-10-31 16:56:11 -07:00
Max Brunsfeld 3380e0eed7 Fix hang when printing dot graphs from the CLI 2023-10-31 15:59:37 -07:00
Max Brunsfeld f4e2f68f14 Merge branch 'master' into wasm-language 2023-10-27 12:11:43 +01:00
Daumantas Kavolis e26e23fd0e
Fix formatting 2023-10-25 10:24:35 +03:00
Daumantas Kavolis e265929f90
Fix goto_previous_sibling with padded first children 2023-10-25 10:19:03 +03:00
Andrew Hlynskyi a5a7564818 chore(rust): add error reporting for position funcs 2023-10-05 17:24:01 +03:00
Andrew Hlynskyi c63f1680ad chore(rust): improve perf for position funcs 2023-10-05 17:24:01 +03:00
Andrew Hlynskyi d95836eb35 cicd: add en extra check for non matchable skips 2023-10-05 14:39:36 +03:00
Andrew Hlynskyi 82ddb3ddcc cicd: add skips for fragile corpus tests 2023-10-05 07:59:35 +03:00
Amaan Qureshi ef9cabd4b5
fix: update javascript tests and use cpp/javascript master for fixtures 2023-09-20 11:31:53 -04:00
Andrew Hlynskyi 524bf7e2c6
Merge pull request #2607 from ahlinc/fix-zero-deallocs 
fix: `dealloc` calls on zero pointers
 2023-09-03 09:45:52 +03:00
Andrew Hlynskyi 9cc1daafca chore(ffi): remove enum name prefixes from all C enum values 2023-09-03 07:38:27 +03:00
Andrew Hlynskyi 7f7084c2cb chore(test): panic on zero pointer deallocs for alloc tracked scopes 2023-09-03 07:12:11 +03:00
Andrew Hlynskyi 67a5dbdd93 fix: dealloc calls on zero pointers 2023-09-03 06:53:18 +03:00
Andrew Hlynskyi 08ac19086b chore: simplify test case after query state improvements 2023-09-02 23:12:28 +03:00
Andrew Hlynskyi 52f7eaff31 chore(rust): make Query extra predicates state fully immutable 2023-09-02 23:12:28 +03:00
Amaan Qureshi dc5ec1cf53
refactor: remove &Option<T> where used 2023-09-01 17:23:39 -04:00
Amaan Qureshi c5cb27e522
docs: improve predicate docs 2023-08-31 16:01:37 -04:00
Amaan Qureshi 09ac28c77d
feat!: properly handle predicates used on quantified captures 2023-08-31 16:01:36 -04:00
Andrew Hlynskyi 0862921537 doc: Include README as top-level module documentation for all crates 2023-08-28 23:09:37 +03:00
Andrew Hlynskyi cbce87dab1 chore: add an extra notes for hang tests 2023-08-28 14:16:36 +03:00
Andrew Hlynskyi a9c4965dd6 cicd: exclude hang tests for exotic arches and ASAN 2023-08-28 05:39:57 +03:00
Andrew Hlynskyi b205a1f5a2 Run parser hang test in a killable subprocess 2023-08-27 21:00:26 +03:00
Amaan Qureshi a4ea4737ac fix: do not increment current_included_range_index past included_range_count in __do_advance 2023-08-27 14:16:18 +03:00
Andrew Hlynskyi 60779cc1ac fix(gen): parser.c should include parser.h relatively 2023-08-26 20:57:08 +03:00
Amaan Qureshi 6d41d99990
Merge pull request #2559 from amaanq/silent-ignore-u 
fix: ignore regex u flag
 2023-08-25 19:25:15 -04:00
Amaan Qureshi 6969766664 fix: ignore regex u flag 
It's needed for certain regex patterns in JS, and we can just silently ignore it.
 2023-08-25 19:23:08 +03:00
Andrew Hlynskyi b3fef28a10 chore(gen): add parser.c enum names to be better discoverable 2023-08-25 19:11:42 +03:00
Andrew Hlynskyi fbfa58edc8 chore(gen): move external scanner stuff closer to the end of parser.c 2023-08-25 19:11:42 +03:00
Amaan Qureshi 1dbb986515 chore: add a test for an aliased anonymous symbol with flags 2023-08-23 16:55:12 +03:00
Andrew Hlynskyi 683fe442e4 fix(gen): cycle between aliases and anonymous symbols 
An example of an error cycle in a `parser.c`:

```
static const TSSymbol ts_symbol_map[] = {
  ...
  [anon_sym_RBRACE] = anon_sym_RBRACE2,
  [anon_sym_RBRACE2] = anon_sym_RBRACE,
  ...
};
```
 2023-08-23 16:51:05 +03:00
Amaan Qureshi e3a5863287
feat: add @injection.parent to inject an injection's node with the parent language 2023-08-21 01:08:17 -04:00
Andrew Hlynskyi ffae7d6115 fix: mark helper Rust funcs that receive raw pointers as unsafe 2023-08-20 02:12:51 +03:00
Amaan Qureshi ac579be788
fix(safety): mark functions that potentially deref a raw pointer as unsafe 2023-08-19 18:59:11 -04:00
Amaan Qureshi 45313e0dfb
feat: add injection.self to inject a node with itself 2023-08-18 18:37:57 -04:00
Amaan Qureshi ce4a9ef4de
feat: allow specifying an input query-paths argument 2023-08-17 22:21:20 -04:00
Amaan Qureshi 9971e5d5f5
feat: allow specifying an input captures-path argument 2023-08-17 22:20:57 -04:00
Amaan Qureshi 458b5de0fc
feat: add any-of predicate 2023-08-17 22:20:57 -04:00
Amaan Qureshi 8eb92d1c64
feat: add an --apply-all-captures argument to highlight & test 2023-08-17 22:20:57 -04:00
Andrew Hlynskyi 6616d7deeb Change edit dumps format and add indexes for all corpus tests 
The new dumped edits format is `edit.<SEED>.<TEST_INDEX>.<SEED_INDEX>.<TEST_NAME>`

To cleanup edits:
- Instead of the standard:
    > rm target/scratch/edit.*

- Use the following shell command:
    > find target/scratch/ -name 'edit.*' -delete
 2023-08-18 00:02:21 +03:00
Andrew Hlynskyi caaa533b8d test: add TREE_SITTER_DUMP_EDITS env var to dump corpus edits 
It dumps edits to the `target/scratch` folder in a format like `<SEED>.edit`
 2023-08-17 20:55:28 +03:00
Andrew Hlynskyi 0dd85c8452 Add TREE_SITTER_GRAMMAR_DEBUG env var to use debug grammars in tests 2023-08-17 18:16:41 +03:00
Amaan Qureshi e0434327d0
fix(render): only output SPACE for strings that are just a space 2023-08-16 13:44:44 -04:00
Amaan Qureshi 2d0a3d647b
Merge pull request #2491 from amaanq/sanitize 
fix: replace & sanitize more characters
 2023-08-13 20:15:20 -04:00
Amaan Qureshi 0b1b0d2fb7
fix: replace & sanitize more characters 2023-08-13 19:29:37 -04:00
Amaan Qureshi 318d4a705a fix: don't add flags if they're not present 2023-08-13 16:28:03 -04:00
Amaan Qureshi dee98e06e9
Merge pull request #2474 from amaanq/case-insensitive 
feat!: support the case-insensitive regex flag
 2023-08-11 23:12:15 -04:00
Andrew Hlynskyi 4a007259fc Fix warning from #2454 in more clear way 2023-08-10 03:59:34 +03:00
Brian Strauch d0afe1ad0a fix: pad hex numbers with zeros when writing colors 
This solves a bug where a hex color with a zero can get truncated
 2023-08-08 09:04:15 -04:00
Amaan Qureshi f35752e3ac
feat!: support the case-insensitive regex flag 2023-08-06 23:40:29 -04:00
Amaan Qureshi 96919cae5d
fix(generate): add Vertical Tab and Form Feed to the whitespace class 2023-08-06 20:41:09 -04:00
Andrew Hlynskyi e024653cb0
Merge pull request #2403 from jakesarjeant/master 
feat(cli): add option to select JS runtime other than node
 2023-08-03 22:29:22 +03:00
Jake Sarjeant 61b70943b1 feat(cli): add option to select JS runtime other than node 2023-08-03 21:34:47 +03:00
Eddie Hatfield 14aa23b1e1 feat: run highlight tests in subdirectories 2023-08-03 19:11:28 +01:00
Andrew Hlynskyi 485d192880 binding_rust: set_max_start_depth accepts optional to reset limit 2023-08-03 11:45:36 +03:00
Samuel Moelius 7b01d8ee05 Add test_query_max_start_depth_more 2023-08-03 11:45:34 +03:00
Samuel Moelius 41ec8b4166 Partially revert d4d5e29 2023-08-03 11:41:50 +03:00
Andrew Hlynskyi 7816397026 Add compile fail tests for LookaheadIterator 2023-08-03 03:49:57 +03:00
Andrew Hlynskyi 4eb22ed3ae Fix UB for LookaheadIterator 2023-08-03 03:49:57 +03:00
Andrew Hlynskyi 4f28ce4e3b Add async context tests for Node and TreeCursor 2023-08-03 03:03:03 +03:00
Michael Davis 3e499d675a
CLI: Re-use highlight cancellation flag for HTML highlights 
The `html` highlight function created its own cancellation flag which
conflicts with the Ctrl-c handler set up in the CLI's `main` block
for `tree-sitter highlight`. We can re-use the cancellation flag from
that block to avoid a panic that happens when using `tree-sitter
highlight -H <file>`

    thread 'main' panicked at 'Error setting Ctrl-C handler: MultipleHandlers', cli/src/util.rs:31:6

This change also aligns the parameters that `highlight::ansi` and
`highlight::html` take.
 2023-08-02 08:56:26 -05:00
Amaan Qureshi acef7a3cdf chore: add tests 2023-08-02 10:47:28 +01:00
Amaan Qureshi b8fe5fe21b fix: do not allow eof to advance states if the new state is the same state 2023-08-02 10:47:27 +01:00
Andrew Hlynskyi a08c61235d Fix, reference by value for Language as everywhere 2023-08-02 00:04:24 +03:00
Daumantas Kavolis ec90c215ae Add tests for bidirectional cursor 2023-08-02 00:02:33 +03:00
Daumantas Kavolis 1cb378b445 Add rust tests for lookahead iterator 2023-08-02 00:02:33 +03:00
Andrew Hlynskyi 19e665ef79 Add tests for TextProvider 2023-08-01 09:12:45 +03:00
Andrew Hlynskyi a2f834d846 More error contexts + conv panics to errors with context 2023-07-30 21:16:45 +03:00
Amaan Qureshi f4e788b28e
feat: warn when unused conflicts are present in a grammar 2023-07-28 00:23:28 -04:00
Amaan Qureshi b8f7645ae2
Merge pull request #2434 from amaanq/test-update 
fix: retain header and divider delimiter lengths when updating tests
 2023-07-27 13:40:53 -04:00
Amaan Qureshi 36c6c8aadf
Merge pull request #2423 from amaanq/nonzero-field-id 
feat!: use `Option<NonZeroU16>` for TSFieldIds
 2023-07-27 13:04:34 -04:00
Amaan Qureshi 0787c24a14
Merge pull request #2430 from amaanq/negative-assertions 
Negative assertions
 2023-07-27 12:44:12 -04:00
Amaan Qureshi 99366f3b5d
ci(sparc64): increase timeout time 2023-07-27 12:26:25 -04:00
Amaan Qureshi 192793a6ab
fix: avoid panic if input is empty 2023-07-27 03:37:02 -04:00
Amaan Qureshi ee399e67d4
fix: retain header and divider delimiter lengths when updating tests 
This helps to prevent visual bloat in diffs when updating tests where the delimiter is not 80 chars long
 2023-07-27 03:31:46 -04:00
Amaan Qureshi 68b1006a3b
feat: add negative assertions, remove duplicate code 2023-07-25 00:27:34 -04:00
Amaan Qureshi c9fd357c06
fix: avoid infinite loop bug 
If the current tag is the last tag and the test fails, the j loop neverends. This prevents that by checking that in the j loop
 2023-07-24 23:43:25 -04:00
Amaan Qureshi a809274009
Merge pull request #2426 from amaanq/sort-tests 
fix(tests): sort categories alphabetically
 2023-07-24 23:21:57 -04:00
Amaan Qureshi 36deb567c8
fix(tests): sort categories alphabetically 2023-07-24 03:05:09 -04:00
Amaan Qureshi 75e1bcf70a
refactor!: rename proc_macro to avoid conflicts when building std with -Z build-std 2023-07-24 00:44:44 -04:00
Amaan Qureshi c521e9c18e
chore: improve error message in some spots loading grammar.json 2023-07-24 00:44:44 -04:00
Amaan Qureshi 9e99789e4c
Merge pull request #2407 from amaanq/bump-deps 
Bump deps
 2023-07-19 17:05:10 -04:00
Amaan Qureshi fc1514d0cd
Merge pull request #2401 from amaanq/fix-inline-bug 
fix: disallow inlining the first rule
 2023-07-19 17:04:17 -04:00
Amaan Qureshi 5fba369c4a
fix: disallow inlining the first rule 
This prevents a panic when indexing symbol_ids during the generation process
 2023-07-19 16:14:58 -04:00
Samuel Moelius d9d4116105
Add test_consecutive_zero_or_modifiers 2023-07-19 03:27:43 -04:00
Amaan Qureshi 0e53b66931
chore: remove useless use 2023-07-19 03:12:13 -04:00
Max Brunsfeld 8310fd423e Update node unit test to reflect C grammar change 2023-07-18 17:58:19 -07:00
Max Brunsfeld 40f7b2ec97 Fix parsing of queries that start with repetitions followed by alternatives 2023-07-18 17:57:52 -07:00
Amaan Qureshi e77405841e
refactor: make a subdirectory for the scratch dir depending on the machine's features 2023-07-18 20:14:21 -04:00
Amaan Qureshi 76f5f81aee
chore: update test 2023-07-18 19:29:40 -04:00
Amaan Qureshi c436897662
test: bump start_time limit for sparc64 2023-07-18 19:07:51 -04:00
Andrew Hlynskyi e1ea3328f4
Merge pull request #2394 from amaanq/hmmcipt2 
Add more architectures to CICD
 2023-07-16 17:23:26 +03:00
Amaan Qureshi 975f3f4686 refactor!: use AtomicUsize for AllocationRecorder for portability 2023-07-16 05:14:11 -04:00
Amaan Qureshi 2a277879ab
Merge pull request #2340 from tree-sitter/highlights-conformance 
feat: add --check flag to `tree-sitter highlight`
 2023-07-16 04:57:23 -04:00
Andrew Hlynskyi 0558f1e0bf test: add retries for all timeout tests 2023-07-16 05:05:36 +03:00
Patrick Thomson cb58bc593f
Add --check flag to tree-sitter highlight. 
Recently I've been pulling a lot of grammars into GitHub's highlighting backend,
replacing legacy language support with tree-sitter highlighting queries.
Our backend systems have a standard set of highlight captures we expect, very
similar to the standard tagging captures we expect. Though end-user applications
are free to choose whatever tagging nomenclature they want, I think it's nice to
include a checking stage that will help us ensure that we know whether a capture
might be recognized or not. It will also help us figure out where we need to
expand our standard set of captures (see #1539).
 2023-07-15 20:49:03 -04:00
Andrew Hlynskyi 0b0cc6c429 Fix rustc 1.71.0 warnings 2023-07-13 17:50:04 +03:00
Amaan Qureshi 9d669abac4 feat: add encoding flag and automatically check if a file might be utf-16 2023-07-10 16:43:35 -04:00
Max Brunsfeld 356f68293a Fix false positive query match bug, introduced in #2085 2023-07-10 16:12:59 -04:00
Amaan Qureshi 3504aa3260 fix(deps): bump proc-macro2 as an unstable feature became stabilized 2023-07-10 16:12:19 -04:00
Amaan Qureshi b192200008 fix: update tests from python grammar changes 2023-07-10 14:40:18 -04:00
Max Brunsfeld 4922602369
Merge pull request #2316 from tree-sitter/node-descendant-index 
Add APIs for retrieving descendant counts, seeking to nodes by their descendant index
 2023-06-12 15:13:43 -07:00
Max Brunsfeld 3375527a89 Fix bug in ts_tree_cursor_goto_descendant 2023-06-12 14:45:30 -07:00
Max Brunsfeld 9dd725b4e3 Start work on exposing APIs for node descendant counts and indices 2023-06-09 08:51:47 -07:00
Samuel Moelius 67f4f2328c Add test_query_error_does_not_oob 2023-05-19 21:31:01 -04:00
Andrew Hlynskyi da269ac350 test: add a test case to demonstrate the #2107 issue 2023-05-13 19:40:49 +03:00
Andrew Hlynskyi ceac2380ed chore(test): reorganize test helpers 2023-05-12 19:29:51 +03:00
Andrew Hlynskyi e966c5ad5b fix: use SIGINT instead of stdin for interruption to don't mess up piped commands 2023-05-02 10:43:46 +03:00
Andrew Hlynskyi 85a588b5b0 Fix tests, adapt to tree-sitter-json 40a81c01a40ac48744e0c8ccabbaba1920441199 2023-04-22 12:08:50 +03:00
Andrew Hlynskyi d4d5e29c91 feat(lib): ts_query_cursor_set_max_start_depth - use 0 to reset 2023-04-17 11:16:04 +03:00
Lewis Russell 1e81a1b67f feat(lib): add ts_query_cursor_set_max_start_depth query API 
This allows configuring cursors from traversing too deep into a tree.
 2023-04-17 11:15:13 +03:00
Joel Spadin 10178ade35 fix: Use / paths when building WASM 
Changed the build-wasm command to always use forward slashes in paths,
since using Windows style paths breaks if the build is run with Docker.

Fixes #532
 2023-04-07 13:07:45 -05:00
Andrew Hlynskyi 0d326824d2 test: add a reproducing test for #2162 2023-04-06 04:49:39 +03:00
Andreas Deininger 0751736d17 docs: convert various links to https protocol 2023-04-04 18:05:46 +03:00
Andrew Hlynskyi c1f784498f chore: nit picking in internal proc_macro crate 2023-04-04 13:51:10 +03:00
Andrew Hlynskyi da894afef5 cargo: specify minimum supported Rust version 2023-04-04 03:42:55 +03:00
Andrew Hlynskyi ddb0af9509 test: use random SEED numbers 
This is needed to omit occurrences of the same seed in a sequence of
following seeds due to the reason of that two initial seed are very
close if based on unix epoch seconds.
 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi 3aeef44eb6 test: run all corpus tests with 10 retries 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi 588549c093 test: run test_parsing_with_a_timeout with 10 retries 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi 62f8c431ae test: add retry and test_with_seed proc macros 2023-03-16 15:17:21 +02:00
Andrew Hlynskyi ca152a93dd ci: fix tests 2023-03-14 20:06:31 +02:00
Max Brunsfeld 8dcf851739 Add unit test for querying within a range of a long top-level repetition 2023-02-16 12:03:51 -08:00
Max Brunsfeld 837899e456 Add API for checking if a pattern in a query is non-local 2023-02-16 11:59:34 -08:00
Max Brunsfeld 40703f110c Fix bug in maintenance of query cursor's tree depth 2023-02-16 11:59:34 -08:00
Max Brunsfeld bd63fb2a0d Tweak query tests 2023-02-16 11:59:34 -08:00
Max Brunsfeld ff2436a6f8 Add --row-range, --quiet, and --time flags to query subcommand 2023-02-14 14:41:25 -08:00
Max Brunsfeld 125503ff3b Fix CLI build on windows 2023-02-13 23:40:06 -08:00
Max Brunsfeld 97fd990822 Add --dot flag to parse subcommand, for printing tree as DOT graph 2023-02-13 12:33:34 -08:00
Andrew Hlynskyi 6f42c78f2d
Merge pull request #2019 from verhovsky/error-message 
cli: make error message more specific for building in `docker`
 2023-01-27 11:45:04 +02:00
Max Brunsfeld 555277a102 Allow testing highlight and tag queries when testing wasm build 
Replace non-mutating `ts_parser_wasm_store` function with
`ts_parser_take_wasm_store`, which removes and returns the wasm
store, in order to facilitate single ownership.
 2023-01-23 11:46:59 -08:00
Andrew Hlynskyi c7d431b53e Add setting TREE_SITTER_DEBUG in the test subcommand 
Were asked in #1218
 2023-01-10 10:47:24 +02:00
Boris Verkhovskiy 61b85b2664 Make error message more specific 2023-01-08 08:10:14 -07:00
Andrew Hlynskyi 108d0ecede loader: add TREE_SITTER_LIBDIR; cli: add --libdir to tree-sitter generate 
Closes #1336
 2023-01-06 10:28:47 +02:00
Andrew Hlynskyi 5088781ef9 cli: add -b, --build flags for tree-sitter generate 2023-01-06 10:28:47 +02:00
Andrew Hlynskyi ad45f5cd2c Remove unused no-minimize arg for the generate command 2023-01-06 10:28:47 +02:00
Andrew Hlynskyi 870fb8772f
Merge pull request #1953 from tree-sitter/fix/playground-port-bind-racing 
fix(cli): Racing on playground webserver port binding
 2022-11-16 06:28:43 +02:00
Max Brunsfeld 51720beeb0 Start work on a --wasm flag for the test subcommand 2022-11-15 17:14:33 -08:00
Max Brunsfeld d47713ee4a Integrate WASM compilation into the CLI's Loader 2022-11-15 17:14:33 -08:00
Max Brunsfeld 3f1a7f9cd4 Start work on ability to load wasm languages from native lib, via wasmtime 2022-11-15 17:14:33 -08:00
Max Brunsfeld 8883d43bee
Merge pull request #1941 from mliszcz/master 
Fix test output formatting for rules starting with M/U
 2022-11-15 17:13:39 -08:00
Jonathan Arnett 813216be9a Configure compiled WASM grammars to not catch rejections 2022-11-15 16:39:17 -08:00
mliszcz 7bf2484d81 Fix test output formatting for rules starting with M/U 
Previously the rule names could not begin with an uppercase M or U
because the test output formatter assumed that they represent special
tokens: MISSING or UEXPECTED.

Fixes #1940.
 2022-11-15 20:20:44 +01:00
Max Brunsfeld 8e3dc7cd7a Add tests that randomly edit files with disjoint included ranges 2022-11-14 16:04:37 -08:00
Max Brunsfeld d07f864815 Fix parse error when reusing a node at the end of an included range 2022-11-11 16:34:57 -08:00
Andrew Hlynskyi a33e9141a8 fix(cli): Racing on playground webserver port binding 2022-11-09 04:56:52 +02:00
Max Brunsfeld da6e24de17 Fix adjustment of trees' included ranges on edits 
Previously, when an included range started or ended *inside* of
an edit, that range did not get updated correctly.

Co-authored-by: Mikayla Maki <mikayla@zed.dev>
 2022-11-07 16:52:19 -08:00
Max Brunsfeld f0177f216e Use library-configured allocator to free intermediate included ranges 2022-11-07 10:48:32 -08:00
Max Brunsfeld e2fe380a08 Generate parsers with ABI version 14 by default 2022-09-02 12:31:29 -07:00
Max Brunsfeld 1b86ccecc8
Merge pull request #1845 from tree-sitter/root-node-with-offset 
Add API for applying a positional offset when accessing a tree's nodes
 2022-08-25 10:51:20 -07:00
Max Brunsfeld 6b87326470
Merge pull request #1787 from kianmeng/fix-typos 
Fix typos
 2022-08-25 10:25:39 -07:00
Max Brunsfeld be442a8f3a
Merge pull request #1831 from shaded-enmity/addr-port-env-vars 
Add env vars for port and address
 2022-08-25 10:23:27 -07:00
Max Brunsfeld 53ed4cf037 Tolerate tree edits whose old range extends beyond the end of the tree 2022-08-25 10:20:40 -07:00
Max Brunsfeld 477b667753 Add ts_tree_root_node_with_offset API 2022-08-18 13:48:47 -07:00
Nat Mote 4e3179fbc0
Avoid extracting default alias for extras 
Fixes #1834
 2022-08-10 07:27:34 -07:00
Pavel Odvody d6c451dc60
Add env vars for port and address 
Make it possible to pass port and address via
pair of environmental variables TREE_SITTER_PLAYGROUND_PORT
and TREE_SITTER_PLAYGROUND_ADDR, respectively.

Signed-off-by: Pavel Odvody <pavel@redhat.com>
 2022-08-02 14:54:45 +02:00
Max Brunsfeld 548c12fb88 Fix bug where patterns with top-level alternatives were not considered 'rooted' 2022-07-07 17:53:54 -07:00
Max Brunsfeld 1401767689 query: Don't attempt to match top-level sibling patterns directly in ERROR nodes 
Co-authored-by: Keith Simmons <keith@zed.dev>
 2022-07-07 15:27:00 -07:00
Kian-Meng Ang b8552ec6c4 Fix typos 2022-06-28 19:57:42 +08:00
Max Brunsfeld 8b5ff2ec8f Loosen randomized changed range assertions on newline characters 
For now, don't error if the scope changes for a newline character.
 2022-06-25 17:13:46 -07:00
Max Brunsfeld 58b719541b Fix failure to match queries with wildcard at root with range restrictions 2022-06-22 15:54:06 -07:00
Aleksei Bavshin b27b4665ac
test: fix incorrect uses of i8 instead of c_char 
Fixes build on aarch64, ppc64le and other platforms that have c_char
defined as u8.
 2022-03-23 00:00:26 -07:00
Max Brunsfeld 7170ec7c96 Improve randomized testing setup 
* Allow iterations to be specified via an env var
* Randomly decide the edit count, with a maximum
  specified via an env var.
* Instead of separate env vars for starting seed + trial, just accept a seed
* Remove some noisy output
 2022-03-02 17:12:25 -08:00
Max Brunsfeld 4b93326898 Don't generate primary states array if it will be unused due to abi version setting 2022-03-02 14:57:59 -08:00
Max Brunsfeld a494d6aa28
Merge pull request #1668 from tree-sitter/remember-lookahead-bytes-on-error-detection 
Remember lookahead bytes on error detection
 2022-02-26 14:59:40 -08:00
Max Brunsfeld c697ebfb27 Add explicit unit test for error detection lookahead bug 2022-02-26 14:33:09 -08:00
Max Brunsfeld 9866674cf8
Merge pull request #1660 from alex-pinkus/expanded-regex-support 
Expand regex support to include emojis and binary ops
 2022-02-24 17:14:23 -08:00
Max Brunsfeld 5eb0a3090f
Merge pull request #1547 from the-mikedavis/md-test-tags 
test tags queries in 'tree-sitter test'
 2022-02-24 15:22:15 -08:00
Alex Pinkus 8fadf18655 Expand regex support to include emojis and binary ops 
The `Emoji` property alias is already present, but the actual property
is not available since it lives in a new file. This adds that file to
the `generate-unicode-categories-json`.

The `emoji-data` file follows the same format as the ones we already
consume in `generate-unicode-categories-json`, so adding emoji support
is fairly easy. his, grammars would need to hard-code a set of
unicode ranges in their own regex. The Javascript library `emoji-regex`
cannot be used because of #451.

For unclear reasons, the characters #, *, and 0-9 are marked as
`Emoji=Yes` by `emoji-data.txt`. Because of this, a grammar that wishes
to use emojis is likely to want to exclude those characters. For that
reason, this change also adds support for binary operations in regexes,
e.g. `[\p{Emoji}&&[^#*0-9]]`.

Lastly (and perhaps controversially), this change introduces new
variables available at grammar compile time, for the major, minor, and
patch versions of the tree-sitter CLI used to compile the grammar. This
will allow grammars to conditionally adopt these new regex features
while remaining backward compatible with older versions of the CLI.
Without this part of the change, grammar authors who do not precompile
and check-in their `grammar.json` would need to wait for downstream
systems to adopt a newer tree-sitter CLI version before they could begin
to use these features.
 2022-02-19 11:41:36 -08:00
Max Brunsfeld cb4317ba8e Change goto_first_child_for_{byte,point} to compare nodes' ranges inclusively 
Co-Authored-By: Antonio Scandurra <me@as-cii.com>
 2022-02-04 12:38:33 -08:00
Max Brunsfeld 994cb61f2c Always generate parser.h, regardless of chosen ABI version 
For some ABI changes, we may need to make changes to the parser.h in order
to restore a previous binary format, but for the current range of supported
ABI versions (13 + 14), the current parser.h is fine.

Refs #1599
 2022-01-23 10:29:52 -08:00
Max Brunsfeld 82ceebc10d 🎨 Use base struct syntax to clean up grammar expectations 2022-01-20 17:17:46 -08:00
Max Brunsfeld fce23d63b3
Merge pull request #1602 from the-mikedavis/md-ignore-future-matches-for-non-local-patterns 
prevent future captures for `#is-not? local` matches
 2022-01-19 16:40:30 -08:00
Michael Davis 02abc2a063
add test for removals in eager query matches 2022-01-18 20:54:55 -06:00
Alex Pinkus 858ea5782b Fix back compat by moving primary_field_ids to the end 
Due to an oversight in #1589, I added `primary_field_ids` into the
`TSLanguage` struct in a place that wasn't the end. This is not actually
backwards compatible and causes downstream failures :(
 2022-01-17 17:23:02 -08:00
Max Brunsfeld 516fd6f6de Add --abi flag to generate command, generate version 13 by default 2022-01-17 14:50:47 -08:00
Max Brunsfeld aaf4572727
Merge pull request #1589 from alex-pinkus/deduplicate-core-ids 
Ignore duplicate states when initializing subgraphs in `ts_query__analyze_patterns`
 2022-01-17 13:54:31 -08:00
Alex Pinkus eaf9b170f1 Don't start with duplicate states in ts_query__analyze_patterns 
This change exposes a new `primary_state_ids` field on the `TSLanguage`
struct, and populates it by tracking the first encountered state with a
given `core_id`. (For posterity: the initial change just exposed
`core_id` and deduplicated within `ts_analyze_query`).

With this `primary_state_ids` field in place, the
`ts_query__analyze_patterns` function only needs to populate its
subgraphs with starting states that are _primary_, since non-primary
states behave identically to primary ones. This leads to large savings
across the board, since most states are not primary.
 2022-01-16 11:17:47 -08:00
Hendrik van Antwerpen ec9b00e5c6 Handle multiple top-level alternations correctly 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen a1a241b013 Expose quantifiers per pattern, instead of merging for all patterns in a query 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen 1f1a449c76 Improve capture quantifier computation 
Compute quantifiers in a bottom-up manner, which allows more precise
results for alternations, where the quantifiers are now precisly joined.
 2022-01-11 18:33:36 +01:00
Max Brunsfeld 622359b400 Simplify allocation-recording in test suite using new ts_set_allocator API 2021-12-30 16:09:07 -08:00
Max Brunsfeld 1d77b9ea47 cli: Fix parsing of test files with newlines in test names 
Fixes #1527
 2021-12-23 14:14:27 -08:00
Michael Davis e7f0cc4e24
add python test case for tag testing 2021-12-18 11:07:46 -06:00
Michael Davis dba7a13808
refactor out get_tag_positions helper function 2021-12-18 11:07:42 -06:00
Michael Davis 47dc88b647
add tags config fixture test helper 2021-12-18 11:07:18 -06:00
Michael Davis 067f742ad3
run test_tag in 'tree-sitter test' when tags test dir exists 2021-12-18 11:07:18 -06:00
Michael Davis 7df82c825f
add module for testing tags 2021-12-18 11:07:11 -06:00
Patrick Thomson f5d1c0b860 Support @module as a highlight key. 
Some languages have the notion of modules, and to represent those
we've started to use a `@module` tag, as discussed in
https://github.com/elixir-lang/tree-sitter-elixir/issues/15.
Because historically we've used the constructor highlight color for
modules in JS/Ruby, it's defined to map to the same color.
 2021-12-15 17:33:57 -05:00
Romanos Skiadas 4784ecaf0a fix crash on empty sexp in test, () 
Now this no longer crashes:

=====
a test
=====
-----
()

...

tree-sitter test:

1 failure:

expected / actual

  1. a test:

    (source_file)

    ()

fixes #1537
 2021-12-15 19:10:26 +02:00
Max Brunsfeld 836d753c73 Fix include_bytes error when building cli outside of source tree 
Refs #1523
 2021-12-13 14:22:33 -08:00
Max Brunsfeld 9f535f95bd Always include playground.html in the CLI binary 
Fixes #1523
 2021-12-06 16:52:14 -08:00
Max Brunsfeld 65746afeff playground: Make query error underlines render in safari 2021-11-22 16:04:45 -08:00
Max Brunsfeld 86b408412c Use serde's derive feature everywhere 2021-11-21 13:39:30 -08:00
Max Brunsfeld 26dac9b2dd Fix query bugs revealed by randomized tests 
* Fix bugs related to named wildcard patterns vs regular wildcard patterns.
* Fix handling of extra nodes during query analysis. Previously, the
expected child_index was updated incorrectly after an extra node,
leading to false "impossible pattern" errors.
* Refine logic for avoiding unnecessary state-splitting due to fallible steps.
Compute *two* different analysis results related to step fallibility:
  * `root_pattern_guaranteed` which, like before, summarizes whether the
    entire pattern is guaranteed to match once this step is reached.
  * `parent_pattern_guaranteed` - which just indicates whether the
    immediate parent pattern is guaranteed. This is now used when
    deciding whether it's necessary to split a match state.
 2021-11-21 12:02:58 -08:00
Max Brunsfeld 142f4b6438 Rename Query::step_is_definite -> is_pattern_guaranteed_at_step 2021-11-21 11:37:52 -08:00
Max Brunsfeld f69c4861c3 Add a randomized test for query matching 2021-11-21 11:29:36 -08:00
Max Brunsfeld a0c085bbec Return an error when trying to inline a token 
Fixes #1420
 2021-11-19 13:02:04 -08:00
Max Brunsfeld 2f9b7ac465 Fix logging of seed in randomized test 2021-11-19 12:16:54 -08:00
Jostein Kjønigsen d0559ca43f Fix whitespace issues when using paths-file. 2021-10-25 23:51:01 +02:00
Max Brunsfeld d05c665863 Convert some of the fixture grammars from JSON to JS 
These tests are easier to write and maintain if the grammars are just JS,
like grammars normally are. It doesn't slow the tests down significantly
to shell out to `node` for each of these grammars.
 2021-10-22 18:47:23 -06:00
Max Brunsfeld ddb12dc0c6 query: Return error on unclosed tree pattern in alternation 
Fixes #1436
 2021-10-12 09:20:43 -07:00
Max Brunsfeld e78413832b Restructure test suite's allocation recording so that tests can run in parallel 2021-10-11 17:24:37 -07:00
Razze 956705a23d Update to unicode standard 14 2021-10-10 16:40:31 +02:00
Max Brunsfeld 21fd9ee36c
Merge pull request #1412 from ahelwer/windows-test-failures 
Fixed some test failures due to CRLF line endings
 2021-10-05 14:13:26 -07:00
Amin Yahyaabadi 8a5620ce15 Use serde's derive feature 2021-10-05 02:38:18 -05:00
Andrew Helwer 993e93eeee Fix linux build 2021-09-28 19:06:05 -04:00
Andrew Helwer 3b5f9cb642 Fixed cargo fmt 2021-09-28 18:53:24 -04:00
Andrew Helwer 8eef790a61 Build rust with warnings=errors 2021-09-28 18:51:09 -04:00
Andrew Helwer e8e4dafa92 Fixed cargo fmt suggestions 2021-09-28 18:31:04 -04:00
Andrew Helwer 3d1f760e31 Simplified corpus test output comparisons 2021-09-28 18:27:25 -04:00
Andrew Helwer 0c3fb6e6a2 Fixed cargo fmt issues 2021-09-24 22:49:25 -04:00
Andrew Helwer 0dc1bd806f Fixed some test failures due to CRLF line endings 2021-09-24 22:42:38 -04:00
Max Brunsfeld 0e3a2780cd Avoid dynamic regex construction when parsing test files 2021-09-21 21:35:14 -07:00
Andrew Hlynskyi 16470bc0b1 chore(cli): Rename all internal web_ui stuff to playground 2021-09-22 04:28:08 +03:00
Andrew Hlynskyi 2abd6bc318 fix(cli): Fix playground opening in a browser, regression in #1304 2021-09-22 01:38:52 +03:00
Andrew Hlynskyi e2de738026 cli(query): Improve and unify query subcommand output 2021-09-21 23:10:00 +03:00
Andrew Hlynskyi 62d79b80e1 feat(cli): add a flag to compile a parser in debug mode with -O0 C/C++ compiler flag 2021-09-13 13:48:02 +03:00
Max Brunsfeld 4fa5ab8a52
Merge pull request #1348 from tlaplus-community/testfile-separator 
Add support for arbitrary suffixes in test file headers and separators
 2021-09-07 22:58:05 -07:00
Max Brunsfeld 23b28f6f36 Fix 'include!' error when building the CLI outside of the repo 2021-09-03 13:45:13 -07:00
Max Brunsfeld 8fa875b1a4 Fix possible infinite loop when running syntax highlighting tests 
Fixes #1347
 2021-08-29 15:04:23 -07:00
Andrew Helwer 03b6a00bb8 Support for suffixes in test file separators 
Some languages use the non-suffixed separators in their syntax
Fixes #982
 2021-08-19 09:42:46 -04:00
FnControlOption e030434ca7 Handle aliases in unicode property escapes in regexes 2021-08-18 22:22:46 -07:00
Paul Gey a533e4d7bb Remove unnecessary borrows 
This produces an `unused_must_use` warning on nightly:
https://github.com/rust-lang/rust/pull/86426
 2021-08-14 15:44:24 +02:00
Max Brunsfeld c6dd5da5e6
Merge pull request #1329 from narpfel/improve-performance 
Improve performance of `tree-sitter generate`
 2021-08-11 16:08:23 -07:00
Paul Gey 965e3c9e5e Generator::add_parse_table: Store entries in hash map 
This avoids a quadratic behaviour due to repeatedly using `find` on a
growing `Vec`.
 2021-08-08 21:45:43 +02:00
Paul Gey cf69a2c94c Use IndexMap and FxHash for some hot hash maps 2021-08-08 21:45:43 +02:00
Andrew Hlynskyi 533073cdb5 fix(cli): Remove tree-sitter grammar ./... call limitation 2021-08-06 02:11:35 +03:00
Kolja Lampe c21bec3716 Always print where the playground is running 2021-08-06 01:58:24 +03:00
Kolja Lampe 99243ddaf2 Correct quiet description for playground 2021-08-06 01:58:24 +03:00
Andrew Hlynskyi bc13d6e504
Merge pull request #1319 from ahlinc/fix/dead-code-in-logger 
fix(cli): allow dead code in Logger; fixes CI failure
 2021-08-06 01:08:58 +03:00
Andrew Hlynskyi d8ac8f5d16 fix(cli): allow dead code in Logger 2021-08-04 03:38:58 +03:00
Max Brunsfeld c7998d93d0
Merge pull request #1278 from CyberShadow/pull-20210719-104342 
feat(cli): Make "test" output more readable
 2021-07-29 15:06:18 -07:00
Max Brunsfeld 59cd1c3962
Merge pull request #1235 from ahlinc/feat/cli-help-messages 
feat(cli): Add a lot of help messages for CLI options
 2021-07-19 11:32:32 -07:00
Vladimir Panteleev 505bc5e1af
feat(cli): Make "test" output more readable 2021-07-19 10:56:21 +00:00
Andrew Hlynskyi 7f538170bf fix(parser): count rows in the debug log from 0 2021-07-15 11:47:14 +03:00
Max Brunsfeld 9cb732859f Update unit test to reflect HTML grammar change 2021-07-12 12:21:10 -07:00
Andrew Hlynskyi b4208ee307 feat(cli): Add a lot of help messages for CLI options 2021-07-03 16:14:57 +03:00
Andrew Hlynskyi bc94c0cc2f fix(cli): fix theme key loading from config.json, closes #1232 2021-07-03 03:38:24 +03:00
Andrew Hlynskyi 60f0953b65 feat(cli): Set TREE_SITTER_DEBUG env var on 'tree-sitter parse -d' 2021-06-30 18:51:43 +03:00
Max Brunsfeld 190fee3625
Merge pull request #1177 from forbisc/master 
update set_included_ranges to modify extent if the current position is at the very beginning of the range.
 2021-06-30 08:38:59 -07:00
Max Brunsfeld c512a0eed7
Merge pull request #1194 from ahlinc/fix/1032 
Close #1032 - fix all weirdness in the generated Cargo.toml
 2021-06-29 16:48:23 -07:00
Max Brunsfeld bd1f7f6881
Merge pull request #1196 from ahlinc/feat/independent-binding-files-gen 
feat(cli): Independent binding files generation
 2021-06-29 16:28:46 -07:00
Max Brunsfeld af0ed73f76
Merge branch 'master' into fix/1032 2021-06-29 15:59:15 -07:00
Max Brunsfeld 23433f4826
Merge pull request #1210 from ahlinc/fix/init-config-experience 
fix(cli): init-config experience, it shouldn't silently overwrite existing config file
 2021-06-29 15:53:36 -07:00
Max Brunsfeld e7d3b29470
Merge pull request #1180 from CyberShadow/pull-20210618-110411 
Pin emscripten/emsdk Docker version
 2021-06-29 15:45:24 -07:00
Vladimir Panteleev b14ea51e3d
Refactor emscripten/emsdk version to a single file 2021-06-29 21:39:12 +00:00
Andrew Hlynskyi f22d62393b fix(cli): actual Rust binding version in generated Cargo.toml 2021-06-30 00:36:11 +03:00
Cameron Forbis 1ea797fbd2 run fmt 2021-06-29 13:59:21 -07:00
Cameron Forbis ae5340901e address code review comments 2021-06-29 13:49:43 -07:00
Andrew Hlynskyi 202a37f1a1 chore(cli): Use DeriveDisplayOrder Clap's setting 
To show sub commands in the defined order that looks more accurate
and have more meaning for sub commands use order.
 2021-06-27 03:37:56 +03:00
Andrew Hlynskyi 2e2b239c42 fix(cli): Don't silently overwrite existing config file 2021-06-27 03:15:35 +03:00
Vladimir Panteleev 5adf7bdcde
cli/src/main.rs: Fix panic with empty file list to parse 
E.g.: tree-sitter parse --paths /dev/null
 2021-06-26 18:11:30 +00:00
Vladimir Panteleev 725f3f7f2b
Pin emscripten/emsdk Docker version 
Fixes issues caused by incompatible changes in Emscripten since the
time that tree-sitter was built.
 2021-06-26 18:07:12 +00:00
Max Brunsfeld d7746bec99
Merge pull request #1198 from ahlinc/feat/color-and-aliases 
feat(cli): colored help and frequent command aliases
 2021-06-24 09:12:48 -07:00
Max Brunsfeld cf8e7bd413
Merge pull request #1199 from dcreager/edit-file 
cli: Allow `$` in `tree-sitter parse --edit`
 2021-06-24 09:07:38 -07:00
Andrew Hlynskyi bd0e60240e fix(cli): Change -h -> -H for the highlight subcommand to don't clash with the help flag 2021-06-24 12:09:00 +03:00
Andrew Hlynskyi 650ee94800 fix(cli): disable help subcommand, -h, --help are enough 2021-06-24 12:04:12 +03:00
Andrew Hlynskyi 1220ec3852 feat(cli): Rename web-ui -> playground with fallback alias 2021-06-24 12:04:12 +03:00
Andrew Hlynskyi 80ee33b073 feat(cli): Add short aliases for frequent subcommands 2021-06-24 11:55:50 +03:00
Andrew Hlynskyi 04badf8a8e feat(cli): Enable clap colored help 2021-06-24 11:55:50 +03:00
Douglas Creager 71c43a869b Allow $ in tree-sitter parse --edit 
You can now specify `$` as the position to apply an edit, signifying the
end of the file.  (That prevents you from having to calculate the size
of the file yourself.)
 2021-06-23 10:04:29 -04:00
Andrew Hlynskyi 421815a68d feat(cli): Make more clearer sub command selection 2021-06-23 10:51:55 +03:00
Andrew Hlynskyi d3527109a8 Updating of binding.gyp should depend on its content instead of bindings/node folder 2021-06-23 02:42:48 +03:00
Andrew Hlynskyi 22d63338a2 Use double quoted patterns for more precise pattern matching in the binding.gyp files 2021-06-23 02:41:30 +03:00
Andrew Hlynskyi 86b8137457 Add create_path_else fn to handle creation or modification 2021-06-23 02:40:32 +03:00
Andrew Hlynskyi 797c7668c1 feat(cli): Independant language binding files generation 2021-06-23 02:39:38 +03:00
Andrew Hlynskyi 4578e58794 fix(cli): close #1032 - fix repository template url generation in cargo.toml 2021-06-23 01:02:29 +03:00
Cameron Forbis 9182ebef86 update set_included_ranges to modify extent if the current position is at the very beginning of the included range 2021-06-17 16:42:25 -07:00
Edgar 9feca80b08
Make HighlightConfiguration::configure use into iterator instead of a Vec<String> 2021-06-16 19:14:45 +02:00
Andrew Hlynskyi 318b00e437 fix(cli): Don't print 'Error:' prefix 2021-06-14 10:46:39 +03:00
Douglas Creager b44270efab cli: Missing config file shouldn't be an error 
Just fall back on the default values for each configuration option.
 2021-06-10 10:22:20 -04:00
Douglas Creager e841fcfa1b cli: Extract CLI configuration into separate crate 
This patch adds the `tree-sitter-config` crate, which manages
tree-sitter's configuration file.  This new setup allows different
components to define their own serializable configuration types, instead
of having to create a single monolithic configuration type.  But the
configuration itself is still stored in a single JSON file.

Before, the default location for the configuration file was
`~/.tree-sitter/config.json`.  This patch updates the default location
to follow the XDG Base Directory spec (or other relevant platform-
specific spec).  So on Linux, for instance, the new default location is
`~/.config/tree-sitter/config.json`.  We will look in the new location
_first_, and fall back on reading from the legacy location if we can't
find anything.
 2021-06-10 10:21:22 -04:00
Douglas Creager 66c30648c2 rust: Extract runtime language detection into separate crate 
This patch adds a new `tree-sitter-loader` crate, which holds the CLI's
logic for finding and building local grammar definitions at runtime.
This allows other command-line tools to use this logic too!
 2021-06-09 17:42:30 -04:00
Douglas Creager 75da247317 cli: Ignore BrokenPipe errors again 
With the change to anyhow in the previous commit, we stopped ignoring
BrokenPipe errors.  Now we do again, not as a core part of our error
type, but as part of the `main` functions reaction to any error that
occurs.
 2021-06-09 16:17:23 -04:00
Douglas Creager d2d01e77e3 cli: Use anyhow and thiserror for errors 
This patch updates the CLI to use anyhow and thiserror for error
management.  The main feature that our custom `Error` type was providing
was a _list_ of messages, which would allow us to annotate "lower-level"
errors with more contextual information.  This is exactly what's
provided by anyhow's `Context` trait.

(This is setup work for a future PR that will pull the `config` and
`loader` modules out into separate crates; by using `anyhow` we wouldn't
have to deal with a circular dependency between with the new crates.)
 2021-06-09 16:17:23 -04:00
Max Brunsfeld 9d77561c43
Merge pull request #1150 from ahlinc/fix/panic-on-broken-pipe 
Stylistic fix for #1135 - Panic on broken pipe
 2021-06-08 11:21:04 -07:00
Andrew Hlynskyi 8bbf27b82b chore(cli): Panic on broken pipe; stylistic fix 2021-06-08 21:04:26 +03:00
Max Brunsfeld 22620dee73
Merge pull request #1135 from ahlinc/fix/panic-on-broken-pipe 
fix(cli): Panic on broken pipe
 2021-06-07 11:15:37 -07:00
Andrew Hlynskyi ce2bb561b4 chore(cli): Panic on broken pipe; review fixes 2021-06-07 21:11:49 +03:00
Andrew Hlynskyi dbdda25224 fix(cli): Panic on broken pipe; use conditional match arm with ErrorKind 2021-06-07 20:22:12 +03:00
Max Brunsfeld f3ea60e23f Merge branch 'master' into query-cursor-api 2021-06-02 11:51:26 -07:00
Douglas Creager cd96552448 query: Allow configurable match limit 
The default is now a whopping 64K matches, which "should be enough for
everyone".  You can use the new `ts_query_cursor_set_match_limit`
function to set this to a lower limit, such as the previous default of
32.
 2021-06-02 11:30:55 -04:00
Andrew Hlynskyi 65dc922944 fix(cli): Panic on broken pipe for: 'tree-sitter parse ... | head' 
Introduce a concept of ignored errors when the Error wrapper struct
contains None instead of Vec of String messages.
 2021-05-31 18:34:09 +03:00
Max Brunsfeld d72771a19f Make ::set_{byte,point}_range methods take a Range 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 14:21:11 -07:00
Max Brunsfeld 851f55afce Report non-rooted matches that intersect cursor's range restriction 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 11:58:38 -07:00
Max Brunsfeld 919e9745a6 Add ts_tree_cursor_goto_first_child_for_point function 
This function (and the similar `ts_tree_cursor_goto_first_child_for_byte`)
allows you to efficiently seek the tree cursor to a given position,
exploiting the tree's internal balancing, without having to visit
all of the preceding siblings of each node.
 2021-05-27 12:30:19 -07:00
Max Brunsfeld fda35894d4 Stop matching new patterns past the end of QueryCursor's range 
This restores the original signatures of the `set_byte_range` and
`set_point_range` functions. Now, the QueryCursor will properly report
matches that intersect, but are not fully contained by its range.

Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-25 18:02:35 -07:00
Max Brunsfeld f597cc6a75 Preserve matches that contain the QueryCursor's start byte 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
Co-Authored-By: Antonio Scandurra <me@as-cii.com>
 2021-05-25 13:06:24 -07:00
Max Brunsfeld a61f25bc58 Add APIs for advancing a QueryCursor to an arbitrary position 2021-05-24 21:07:59 -07:00
Douglas Creager 78010722a4 query: Allow unlimited pending matches 
Well, not completely unlimited — we're still using a 16-bit counter to
keep track of them.  But we longer have a static maximum of 32 pending
matches when executing a query.
 2021-05-24 11:02:58 -04:00
Max Brunsfeld 8c3d1466ec Allow QueryCursor's text callback to return an iterator 2021-05-23 21:05:26 -07:00
Andrew Hlynskyi 0ae70289ae fix(cli): Correct fallback on docker compilation for all platforms, fixes #1124 2021-05-23 17:04:34 +03:00
Niranjan Hasabnis c0e2b43d34 Fixing Rust formatting issue 2021-05-21 15:43:10 +00:00
Niranjan Hasabnis c31acb8fec Changing API name; Adding unit test and Rust bindings 2021-05-21 01:50:10 +00:00
Andrew Hlynskyi 3c0152a331 chore(fmt): Apply 'cargo fmt' to the whole code base 2021-05-19 23:21:43 +03:00
Markus F.X.J. Oberhumer cc519b3121 cli: Improve const-correctness of the generated parsers (part 2 of 2). 
This is a follow-up to my previous commit 1badd131f9 .

I've made this an extra patch as it requires a minor
API change in <tree_sitter/parser.h>.

This commit moves the remaining generated tables into
the read-only segment.

Before:
  $ for f in bash c cpp go html java javascript jsdoc json php python ruby rust; do \
       gcc -o $f.o -O2 -Ilib/include -c test/fixtures/grammars/$f/src/parser.c; \
    done
  $ size --totals *.o
      text    data     bss     dec     hex filename
   5353477   24472       0 5377949  520f9d (TOTALS)

After:
  $ for f in bash c cpp go html java javascript jsdoc json php python ruby rust; do \
       gcc -o $f.o -O2 -Ilib/include -c test/fixtures/grammars/$f/src/parser.c; \
    done
  $ size --totals *.o
   5378147       0       0 5378147  521063 (TOTALS)
 2021-05-19 12:49:57 +02:00
Max Brunsfeld fe5a29a1c6
Merge pull request #1081 from ahlinc/feat/fail-ci-on-compiler-warnings 
feat: Fail CI on compiler warnings
 2021-05-01 09:26:44 -07:00