libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-11 07:56:28 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
5714 commits 20 branches 114 tags 51 MiB
Commit graph

1363 commits

Author SHA1 Message Date
Quentin LE DILAVREC 9610a84600
fix(lib): restrict pattern_map optimization when a wildcard step has an immediate first child 
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
 2024-07-05 03:35:24 -04:00
Amaan Qureshi 6ec478c1e9 fix: do not "absorb" rules that consist of a single terminal if the rule is hidden 2024-06-21 20:20:02 -04:00
Amaan Qureshi 252e2a4bc0 fix: intern a sequence or choice of a single element the same as the element itself 2024-06-21 20:20:02 -04:00
Amaan Qureshi a73191bf0e fix: better error when a supertype rule is invalid 2024-06-03 06:35:27 -04:00
Amaan Qureshi e553578696 feat: add fuzz subcommand 2024-05-30 23:00:45 -04:00
Amaan Qureshi fc146ad510 refactor: remove difference dependency 2024-05-25 23:04:28 -04:00
Amaan Qureshi c440f2a7c6 refactor: remove ansi_term dependency 2024-05-25 21:09:28 -04:00
Max Brunsfeld 38137c71b2 feat!: introduce tree-sitter-language crate for grammar crates to depend on 
Co-authored-by: Conrad <conrad@zed.dev>
Co-authored-by: Marshall <marshall@zed.dev>
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
 2024-05-24 21:54:07 -04:00
ObserverOfTime 04b9eb5700 feat(bindings): update npm scripts 2024-05-24 17:35:12 -04:00
ObserverOfTime 629395fd58 feat(bindings): add node, python, swift tests 2024-05-24 16:55:05 -04:00
ObserverOfTime 055d0cbd34
feat(dsl)!: support other JS runtimes 2024-05-24 16:53:33 -04:00
ObserverOfTime d77279d2e3 style(bindings): fix indent & line endings 
- Use 2 spaces for parser.c
- Don't force LF line endings
 2024-05-24 15:45:48 -04:00
ObserverOfTime fd0e1c7b5b
feat(bindings): add query constants to python 2024-05-24 15:31:38 -04:00
Amaan Qureshi a88fc9cf63 fix: always reset to the first language when iterating over language attributes 2024-05-19 19:40:47 -04:00
ObserverOfTime 636801770e feat(cli): add debug build flag 2024-05-17 16:12:56 -04:00
Amaan Qureshi 8e8648afa9 fix(test): multi-grammar corpus tests are now in the repo root 2024-05-06 15:25:42 -04:00
Amaan Qureshi b1fd3214db feat(cli): bump unicode data to v15.1.0 2024-05-06 15:25:42 -04:00
WillLillis 577d333849 feat(cli): add test listing and allow users to parse a specific test number 2024-05-05 16:35:17 -04:00
Amaan Qureshi 33045ee898
fix(test): allow newlines in between test name and attribute 2024-05-05 15:19:42 -04:00
Amaan Qureshi 1e219ff9eb fix(cli): properly account for multi-grammar repos when using docker to build a wasm parser 2024-05-05 14:56:42 -04:00
ObserverOfTime 572cdaa7c0 ci: fix address sanitizer step 2024-05-05 13:06:45 -04:00
ObserverOfTime a2d2da314d fix(cli): keep default cc flags in build 2024-05-05 13:06:45 -04:00
Amaan Qureshi 67a043911c
test: temporarily disable php test 2024-05-04 03:05:58 -04:00
Amaan Qureshi 4c083252ec fix(lib): advance the lookahead end byte by 4 when there's an invalid code point 
This helps in the case where an edit was made in the middle of a code
point, but bytes 1-3 are valid, thus we could advance by at most 4 bytes
 2024-04-30 20:55:43 -04:00
Amaan Qureshi 61d0395543 fix(lib): do not return field names for extras 2024-04-30 19:23:11 -04:00
WillLillis 946acfd70f feat(cli): support NO_COLOR 
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>

This allows users to avoid colored output when NO_COLOR is set to 1.
 2024-04-29 20:25:41 -04:00
Amaan Qureshi 627617edb4 refactor(tests): migrate remaining grammar.json tests to grammar.js 2024-04-29 00:01:35 -04:00
Amaan Qureshi 26fa3a76a5 fix(generate): take AsRef<Path> for the path parameter to avoid clones 2024-04-27 17:09:36 -04:00
ObserverOfTime 5aad87a74c fix(generate): don't check arbitrarily named dirs 
i.e. check `bindings/swift` rather than `bindings/swift/TreeSitterFoo`
since the dir name may have different casing (e.g. `TreeSitterFOO`)
 2024-04-27 17:09:36 -04:00
Amaan Qureshi 41308888fc fix: don't error out on package.json lookup errors if --no-bindings is passed 2024-04-27 16:29:26 -04:00
Amaan Qureshi 517e61b3e6 fix: improve error message when the tree-sitter field is malformed 2024-04-27 16:29:26 -04:00
vanaigr 90e0e28b95
feat: reverse iteration through node parents (#3214) 2024-04-23 15:19:57 +01:00
ObserverOfTime 83b6010461 feat(grammar): add "inherits" field if available 2024-04-14 15:59:01 -04:00
Max Brunsfeld 295344b142 fix: Avoid generating unused character set constants 2024-04-14 09:57:10 -07:00
Amaan Qureshi 9ba8f6dfe6 fix(cli): test parsing on windows 2024-04-12 21:09:28 -04:00
Max Brunsfeld 63fa0f23f2 Include 2-character ranges in array-based state transitions 2024-04-12 16:40:04 -07:00
Max Brunsfeld 056237f18b Fix sorting of transitions within a lex state 2024-04-12 16:21:57 -07:00
Max Brunsfeld 7ec40b0ab4 Implement single-char state transitions using a static array and for loop 
This reduces compile time, compared to generating many individual if statements.
 2024-04-12 14:40:11 -07:00
Max Brunsfeld 3210c7e21f Avoid using a large character set constant when it doesn't reduce code size 2024-04-12 12:01:23 -07:00
Max Brunsfeld 3498498449 Merge branch 'master' into simpler-large-char-set-code 2024-04-12 10:03:46 -07:00
Max Brunsfeld 15fe07a20e Clean up code generation for lexer state transitions 2024-04-12 09:02:33 -07:00
Amaan Qureshi 96d18408a3 refactor(js): misc fixes & tidying 2024-04-11 22:35:43 -04:00
Amaan Qureshi abc7910381 refactor(rust): misc fixes & tidying 2024-04-11 22:35:43 -04:00
Amaan Qureshi 5825e24d56 style: wrap comments 2024-04-11 22:35:43 -04:00
Amaan Qureshi b35efa8f33 style: format imports 2024-04-11 22:35:43 -04:00
Amaan Qureshi a48054f1ae
fix: wrong flag check in build.rs 2024-04-11 22:31:10 -04:00
Max Brunsfeld 1f0707e1ac Fix clippy warnings 2024-04-11 16:29:59 -07:00
Max Brunsfeld b8701fcf18 Check EOF when checking a large char set that contains the null character 2024-04-11 16:19:21 -07:00
Max Brunsfeld be6e6d3708 Merge branch 'master' into simpler-large-char-set-code 2024-04-11 16:03:20 -07:00
Max Brunsfeld 18ea74ee12
Merge pull request #3280 from ObserverOfTime/reduce 
refactor(parser): make REDUCE macro non-variadic
 2024-04-11 15:39:28 -07:00
Ron Panduwana c051a90ac5 fix: insert "tree-sitter" section in current directory's package.json if it exists 
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
 2024-04-11 17:09:35 -04:00
ObserverOfTime 63babea301 fix: proper function prototypes 2024-04-11 16:28:21 -04:00
ObserverOfTime 50dfd409f6 fix(bindings): generate parser.c if missing 2024-04-11 16:28:21 -04:00
ObserverOfTime 7830877f63 fix(bindings): add utf-8 flag to python & node 2024-04-11 16:27:50 -04:00
ObserverOfTime 818cd8c291
refactor(parser): make REDUCE macro non-variadic 2024-04-11 20:47:08 +03:00
Amaan Qureshi 8bfe4e1f6b fix: add back build-wasm temporarily 2024-04-11 10:48:30 -04:00
Ron Panduwana 33de5ef849 fix: default output directory for build --wasm should use current_dir 2024-04-11 10:48:30 -04:00
Amaan Qureshi 895c7680e7 fix(windows): add /utf-8 flag for parsers using unicode symbols 2024-04-11 09:44:51 -04:00
Sebastian Lackner 5dc62cc828
fix(cli): fix mismatched parenthesis when accounting for && 2024-04-11 09:01:56 -04:00
Max Brunsfeld 3d088888f5 Derive large character sets from lex states for individual tokens 2024-04-10 16:53:39 -07:00
Max Brunsfeld be34bc9430 Identify large char sets for lexer using NFA transitions 2024-04-09 17:53:37 -07:00
Amaan Qureshi 21c06101aa fix(bindings): remove required platforms for swift 2024-04-08 17:56:05 -04:00
Amaan Qureshi 0d2dea9e0e fix(test): allow colons in test names 2024-04-08 17:56:05 -04:00
Amaan Qureshi a9172e0caa fix: add a semicolon after SKIP macros 2024-04-08 17:56:05 -04:00
Amaan Qureshi abed43a169 chore: clippy fix 2024-04-08 17:56:05 -04:00
ObserverOfTime 78b6067a5d fix(parser): fix variadic macro 2024-04-02 03:18:05 -04:00
Max Brunsfeld 39be6972fe Use static arrays and a fixed binary search for large char set checks 2024-03-29 23:00:48 -07:00
Amaan Qureshi 62cd13a3de test: add regression test for node parent + string bug 2024-03-20 19:23:08 -04:00
Max Brunsfeld 09b18fad5b
Merge pull request #3181 from tree-sitter/handle-wasm-oom 
When loading languages via WASM, gracefully handle memory errors and leaks in external scanners
 2024-03-18 13:15:06 -07:00
Max Brunsfeld fed9b98d6c Clean up wasm store construction, avoid hard-coding initial memory size 
Retrieve the initial memory size from the stdlib's memory import type.
 2024-03-17 10:20:18 -07:00
Max Brunsfeld 7a9b3076ef Handle memory errors occurring in wasm scanners 
* In WASM, use a custom, simple malloc implementation that lets us
  expicitly reset the heap with a new start location.
* When a WASM call traps or errors, propagate that as a parse failure.
* Reset the WASM heap after every parse.

Co-authored-by: Conrad <conrad@zed.dev>
 2024-03-17 10:19:42 -07:00
ObserverOfTime 4bbaee2f56 fix(lib): allow hiding symbols 2024-03-17 07:21:06 -04:00
ObserverOfTime 4b12eab927 feat(lib): implement Display for Node 2024-03-17 06:12:28 -04:00
Amaan Qureshi e8dbe011d3 test: fix header writes 2024-03-17 05:36:30 -04:00
Amaan Qureshi 754aa62e89 chore: turbofish styling 2024-03-17 05:36:30 -04:00
Amaan Qureshi 99b93d83a1 feat(cli)!: add a separate build command to compile parsers 
This allows users to build parsers without having to run `test` or
`parse` to invoke the compilation process, and allows them to output the
object file to wherever they like. The `build-wasm` command was merged
into this by just specifying the `--wasm` flag.
 2024-03-17 05:36:30 -04:00
ObserverOfTime 1ff40f5571 fix(lib): makefile installation 2024-03-12 18:14:58 -04:00
Amaan Qureshi 81d90c7cd8 fix(lib): makefile installation on macOS 2024-03-12 15:53:31 -04:00
Amaan Qureshi ad07fa8a9e feat(bindings/rust): expose Parser::included_ranges 2024-03-12 02:29:35 -04:00
Amaan Qureshi 0a5a564ea7
0.22.0 2024-03-10 17:15:08 -04:00
ObserverOfTime e9b3f65ceb fix(bindings): fix template oversights 2024-03-10 15:48:23 -04:00
Christopher Durham 78cc77e7b2 refactor!: remove redundant escape regex & curly brace regex preprocessing 
The regex-syntax crate now natively supports literal escapes for all
ASCII characters except those in [0-9A-Za-z<>].
 2024-03-10 14:47:11 -04:00
ObserverOfTime b60b2489aa feat(bindings): remove dsl types file 2024-03-10 10:06:17 -04:00
ObserverOfTime 072865e450 feat(bindings): add prebuildify to node 
Co-Authored-By: Amaan Qureshi <amaanq12@gmail.com>
 2024-03-10 10:06:17 -04:00
ObserverOfTime 69cf13bc05 feat(bindings): add peerDependencies for npm 2024-03-10 10:06:17 -04:00
Amaan Qureshi 17e50c4c57 fix: allow the regex v flag 2024-03-10 09:50:29 -04:00
Amaan Qureshi 4b578a3649 fix(generate): camel case name in Cargo.toml description 2024-03-06 19:53:51 -05:00
ObserverOfTime 2d652c90a7 fix(cli): only output the sources with --no-bindings 2024-03-06 13:56:40 -05:00
Amaan Qureshi 92675117a6 fix(generate): extern allocator functions for the template don't need to be "exported" 2024-03-05 11:19:06 -05:00
Amaan Qureshi 54a31069af fix: parsers should export the language function on windows 2024-03-05 11:19:06 -05:00
Amaan Qureshi 304f8b7c04 fix: don't use __declspec(dllexport) on windows 2024-03-04 13:23:06 -05:00
Amaan Qureshi f8c41f74f8 fix(bindings): insert types after main if it exists 2024-03-03 11:17:30 -05:00
Amaan Qureshi 57d01335c0 chore: delete binding_files.rs 2024-03-03 10:46:37 -05:00
Amaan Qureshi 27f52266ac feat: replace nan with node-addon-api and conditionally print logs 2024-03-03 10:46:37 -05:00
Segev Finer 30bb44f48a feat: add typings for the node bindings 
Fixes #3072
 2024-03-03 10:46:37 -05:00
geekvest b43e8ce902
docs: remove duplicate the's 
Signed-off-by: geekvest <cuimoman@sohu.com>
 2024-03-03 10:30:21 -05:00
Segev Finer 1dd35fe0e0
feat(node): type tag the language 
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
 2024-03-02 20:59:20 -05:00
ObserverOfTime 924c2b4425 chore(bindings): ignore more artifacts 2024-03-02 20:13:39 -05:00
ObserverOfTime f7c6dcadaa feat(bindings): add more make targets 2024-03-02 20:13:39 -05:00
Amaan Qureshi c66ed6a141
fix: and binding.gyp... 2024-03-01 11:02:33 -05:00
Amaan Qureshi 274d0cda0a fix: binding.cc overwrite should replace PARSER_NAME 2024-03-01 10:59:25 -05:00
Amaan Qureshi 6b2879144f build(deps): bump deps & lockfile 2024-02-29 02:17:30 -05:00
Amaan Qureshi 5a4765ca01 style: clippy lints 2024-02-29 02:17:30 -05:00
Amaan Qureshi 60a935139b refactor!: remove top-level corpus dir for tests 
It's confusing to have tests in two different top-level directories when working between different grammars, and most of them use `test/corpus` which is more fitting, so time to go.
 2024-02-29 02:17:30 -05:00
Amaan Qureshi 4ff33e81bb fix(assertions): case shouldn't matter for comment node detection 2024-02-29 02:17:30 -05:00
Amaan Qureshi 8c4861c186 feat(test): add attributes for corpus tests 2024-02-29 02:17:30 -05:00
ObserverOfTime b3c14c9295 build(bindings): make everything c11 2024-02-29 01:50:04 -05:00
ObserverOfTime b4b2d9cecc refactor: remove extern/const where possible 2024-02-29 01:50:04 -05:00
Amaan Qureshi d92a7a9690 test: add test case for parse stack merging with incorrect error cost bug 2024-02-28 08:51:28 -05:00
Segev Finer e87cf7ef15
refactor(bindings)!: convert node bindings to NAPI 
Co-authored-by: Boris Verkhovskiy <boris.verk@gmail.com>
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
 2024-02-27 17:06:25 -05:00
Amaan Qureshi eb332ffa4d style(bindings): consistent wording 2024-02-27 16:07:38 -05:00
ObserverOfTime 502dacf220 refactor: rename TS_REUSE_ALLOCATOR flag 
TREE_SITTER_REUSE_ALLOCATOR is more consistent
 2024-02-27 15:54:38 -05:00
ObserverOfTime 46520eda58 build(bindings): metadata improvements 2024-02-27 15:54:38 -05:00
Amaan Qureshi f7c13406c9 build: add uninstall command 2024-02-27 15:54:38 -05:00
ObserverOfTime 799833f9cf build: use c11 everywhere 
And improve the makefiles
 2024-02-27 15:54:38 -05:00
ObserverOfTime 412a6a774e refactor(bindings): remove npmignore 
Replace with package.json files
 2024-02-27 06:22:00 -05:00
Amaan Qureshi 8ae3870b51 fix(generate): add .npmignore, populate Swift's exclude list 2024-02-26 14:30:48 -05:00
ObserverOfTime 721eca2394 fix(bindings): editorconfig and setup.py fixes 2024-02-26 12:00:16 -05:00
Amaan Qureshi f1eecf9786 feat: expose the allocator and array header files for external scanners 2024-02-26 09:43:13 -05:00
Amaan Qureshi 9e5bf6591f feat: improve language bindings 
Co-authored-by: ObserverOfTime <chronobserver@disroot.org>
 2024-02-25 19:06:01 -05:00
Max Brunsfeld 29d3583bdf Use workspace dependencies for internal crates 2024-02-25 11:14:29 -08:00
Amaan Qureshi 32c23b6c90 fix: wrap || comparison in parenthesis when && is used 2024-02-24 01:35:40 -05:00
Max Brunsfeld 1f751bbd5f
Merge pull request #2841 from segevfiner/patch-1 
Make Node.js language bindings context aware
 2024-02-23 11:23:11 -08:00
Will Lillis f894a5350c
feat(cli): add optional config-path argument 2024-02-21 12:02:12 -05:00
Amaan Qureshi 58a4fcc792 chore: get rid of github_issue_test file 2024-02-21 12:01:25 -05:00
Amaan Qureshi 38efefd8bd style: cleaner cast 2024-02-21 12:01:25 -05:00
Amaan Qureshi 167855b289 fix(test): edge case when parsing UNEXPECTED/MISSING nodes with an indentation level greater than 0 2024-02-21 12:01:25 -05:00
Amaan Qureshi a1a3903c10 fix(test): allow writing updates to tests without erroneous nodes instead of denying all of them if a single error is found 2024-02-21 12:01:25 -05:00
Amaan Qureshi b97208704b
fix(cli): don't use long for grammar_path 
Co-authored-by: buckynbrocko <77247638+buckynbrocko@users.noreply.github.com>
 2024-02-19 16:05:08 -05:00
Amaan Qureshi b40839cd72 style: prefer turbofish syntax where possible 2024-02-19 16:00:50 -05:00
Amaan Qureshi fd91404ab0 style: tidying 2024-02-19 16:00:50 -05:00
Amaan Qureshi d95fcc83b9 fix: remove redundant imports 2024-02-19 06:36:48 -05:00
Amaan Qureshi f526be8061 test: update html tests 2024-02-19 06:36:48 -05:00
Amaan Qureshi 5b299eafe5
fix: don't throw an error if the user uses map in the grammar 2024-02-18 00:26:14 -05:00
Amaan Qureshi da1f890752 chore: error out when multiple arguments are passed to token/token.immediate 2024-02-17 01:26:44 -05:00
Amaan Qureshi 8dd65ccbc0 refactor: &PathBuf -> &Path 2024-02-17 00:45:28 -05:00
Amaan Qureshi ef1b714e08 fix(cli): don't update tests automatically if parse errors are detected 2024-02-16 20:23:39 -05:00
Amaan Qureshi e32a7f3998 chore: clippy lints 2024-02-16 16:32:07 -05:00
Amaan Qureshi 4342efd57e feat: allow specifying an external scanner's files 2024-02-16 16:31:56 -05:00
Amaan Qureshi b6c75ccec1 chore: update relevant rust tests 2024-02-16 12:34:12 -05:00
Amaan Qureshi e996c32108 refactor!: remove the apply-all-captures flag, make last-wins precedence the default 2024-02-16 12:34:12 -05:00
Nickolay 6895b7a1e1
Add some documentation to the playground page 
The minimal UI of the Playground could benefit from some documentation to make it easier for the newer users to understand what's going on. Also added a link to the new documentation from the local playground.

Closes https://github.com/tree-sitter/tree-sitter/issues/1305
 2024-02-15 16:24:21 +01:00
Amaan Qureshi d80d101e34 build: move common Cargo.toml keys into the workspace and inherit them 2024-02-14 14:56:23 -05:00
Amaan Qureshi 73f56bffa9 fix: sexp format edge case with quoted closed parenthesis 2024-02-13 18:31:28 -05:00
Amaan Qureshi 4303ab99c9 fix: properly handle Query.matches when filtering out results 2024-02-13 16:40:05 -05:00
Amaan Qureshi 7e0dd7b9c1 feat(cli): add an optional grammar-path argument for the playground 2024-02-13 16:28:54 -05:00
Amin Yahyaabadi 51c147053e
feat: error out if an empty string is in the extras array 
This prevents never-ending loops in the parser
 2024-02-13 15:00:48 -05:00
Amaan Qureshi 5ea0dbf77a chore: some more clippy lints 2024-02-13 03:33:07 -05:00
Amaan Qureshi a07f988905 refactor: extract regex check into a function and lower its precedence 2024-02-13 03:33:07 -05:00
Bedis Nbiba 7dd096c5f7
feat: implement first-line-regex 2024-02-13 01:51:41 -05:00
Amaan Qureshi 21f25a5305 feat: improve error message for files with an unknown grammar path 2024-02-12 17:17:37 -05:00
Amaan Qureshi 3da79ba2b6 fix: update schema for regex flags 2024-02-12 15:39:14 -05:00
Amaan Qureshi 39df8e2833 chore(test): use different languages for async tests 2024-02-12 06:30:36 -05:00
Amaan Qureshi e6d67ecde0 chore(wasm): use SIDE_MODULE=2 to silence warning 2024-02-11 20:18:30 -05:00
Amaan Qureshi 4149ed4149 chore: provide a CLI flag to open log.html 
On macOS, this was done by default regardless of what the user wants.
This was also not done on Windows or Linux. Instead, we now provide a
`--open-log` flag to open the log file in the default browser, and it
works on all platforms.
 2024-02-11 02:28:34 -05:00
Amaan Qureshi ce5f5fffc1 chore(generate): dedup and warn about duplicate or invalid rules 2024-02-11 01:28:31 -05:00
Amaan Qureshi 78ccc39ca9
fix: properly error out when the word property is an invalid rule 2024-02-11 00:37:23 -05:00
Amaan Qureshi cca814afd2 chore: add java and tsx to corpus tests 2024-02-10 18:25:35 -05:00
Amaan Qureshi 3d68ca4583 fix: inherited grammar generation 2024-02-10 18:25:35 -05:00
Amaan Qureshi dcb34b25ec fix: prettify xml output and add node position info 2024-02-08 18:09:10 -05:00
Amaan Qureshi a969fc428c chore: move dependencies into the workspace and inherit from there 2024-02-08 15:24:09 -05:00
Amaan Qureshi 7980669267 feat: support dot graph output on windows 2024-02-08 15:21:47 -05:00
Amaan Qureshi 214e27083a feat(cli): upgrade to clap v4 2024-02-08 05:39:41 -05:00
dundargoc df1fe842eb docs: various fixes 
Closes https://github.com/tree-sitter/tree-sitter/issues/1317.
Closes https://github.com/tree-sitter/tree-sitter/issues/1752.
Closes https://github.com/tree-sitter/tree-sitter/issues/2439.

Co-authored-by: Simon Hengel <sol@typeful.net>
Co-authored-by: Akash Yadav <itsaky01@gmail.com>
Co-authored-by: Sebastiaan Speck <12570668+sebastiaanspeck@users.noreply.github.com>
Co-authored-by: Laytan Laats <laytanlaats@hotmail.com>
 2024-02-08 00:08:59 +01:00
Amaan Qureshi 767db31114
fix: remove redundant code 2024-02-07 07:13:03 -05:00
Amaan Qureshi 993ee75d8c
fix: don't add extraneous exports to the json and add missing rules to the schema 2024-02-07 07:10:51 -05:00
Daumantas Kavolis 08d211a546
feat: improve time reports 2024-02-07 03:19:02 -05:00
Amaan Qureshi 59be1edaa1
refactor: swap &Vec[T] with &[T] where appropriate 2024-02-07 02:50:31 -05:00
buckynbrocko 8b38ab67de
chore(cli): reword test diff legend 2024-02-07 01:33:06 -05:00
buckynbrocko a4a7edc5af
feat(cli): add include/exclude options to the test subcommand 2024-02-07 01:04:54 -05:00
dundargoc c8bd6705cf
chore: clippy 2024-02-06 23:34:14 -05:00
Amaan Qureshi f601c87091
fix: do not skip over spaces that are surrounded by quotes 2024-02-06 19:00:43 -05:00
Amaan Qureshi a0ddae9d1a
Merge pull request #2724 from plugwash/master 
cli - update to regex-syntax 0.8
 2024-02-06 19:00:11 -05:00
Max Brunsfeld f4788b49c0 Merge branch 'master' into wasm-stdlib 2024-02-05 11:55:58 -08:00
Max Brunsfeld ad9091afe0 Fix wasm build errors due to builtin symbols 2024-02-05 10:19:34 -08:00
root b7d48b48fd
chore(cli): update to regex-syntax 0.8 
regex-syntax has restructured the Ast enum a bit, rather than having a Class
member it now has seperate members for different types of Class.
 2024-02-05 01:22:09 -05:00
Amaan Qureshi 000c2f4dec
feat: allow the grammar to log to stdout 2024-02-05 00:46:34 -05:00
Amaan Qureshi 55afb4efaf
ci: remove unnecessary targets 2024-02-04 04:19:09 -05:00
Amaan Qureshi 04ff704bca
chore(cli): apply clippy fixes 2024-02-04 04:18:48 -05:00
Max Brunsfeld e054de4191 Return an error from build-wasm if scanner uses unavailable functions 2024-02-02 12:04:49 -08:00
Amaan Qureshi 9f2dc9d6b5
fix: rework parser.h includes for test grammars and multi-grammar repos 2024-02-02 10:42:39 -05:00
Amaan Qureshi 422e74fbdb
chore: update javascript-relevant tests 2024-02-02 08:58:22 -05:00
Max Brunsfeld 1d8975319c
Merge pull request #2840 from tree-sitter/language-reference-count 
Introduce APIs for managing the lifetimes of languages, allow WASM languages to be deleted
 2024-01-30 10:24:37 -08:00
Max Brunsfeld d351f81b4a Release stale instances from wasm store when languages are dropped 2024-01-29 10:17:49 -08:00
Steven Kalt d35efd4608
feat(cli): support building WASM via podman 
Previously, `tree-sitter build-wasm` had the ability to build WASM
by using docker to pull in an image with a complete emscripten toolchain.
This commit adds the ability to use podman to do the same thing.

Using podman requires two notable changes:
1. Using the fully-qualified image name. Docker defaults to prepending
    `docker.io` to the image name, but podman does not.
2. Podman will mount the `/src/` volume as belonging to root unless
  `--userns=keep-id` is passed. I think podman's different
  volume-ownership is related to podman's daemonless execution and
  `--uidmap` functionality, but I'm not 100% sure.

To test, I ran
```sh
script/fetch-fixtures
script/generate-fixtures
script/generate-fixtures-wasm # <- the important one!
```

which worked as well as the docker version.
 2024-01-29 00:50:32 -05:00
Max Brunsfeld c4142737ec Merge branch 'master' into language-reference-count 2024-01-25 12:43:56 -08:00
Max Brunsfeld 3c66dd2d6c Add missing language return in ts_query_new 
Co-authored-by: Antonio <antonio@zed.dev>
Co-authored-by: Mikayla <mikayla@zed.dev>
Co-authored-by: Joseph <joseph@zed.dev>
 2024-01-25 11:06:54 -08:00
Amaan Qureshi b26e0a8ec0
fix: account for grammars in subdirectories, add typescript to test corpus 2024-01-25 14:02:12 -05:00
Max Brunsfeld 3139760fa9 Record allocations in wasm language error test 2023-12-31 10:05:06 -08:00
Segev Finer 4a42c6d5f1
Make Node.js language bindings context aware 
They don't have any dynamic global data, so all it takes is just declaring them as such
 2023-12-28 16:39:19 +02:00
Max Brunsfeld 82c1b5ead3 Fix compile error in parser hang test 2023-12-27 15:10:52 -08:00
Max Brunsfeld 4a8e4b1963 Allow wasm languages to be deleted 2023-12-27 15:00:16 -08:00
Max Brunsfeld da16cb1459 Introduce language ref-count management C APIs, remove Copy impl for Language in Rust 2023-12-27 14:59:16 -08:00
Andrew Hlynskyi d56b51a11d
Revert "Alt #2454" 2023-11-29 11:20:05 +02:00
Max Brunsfeld 034f0d0280
Merge pull request #1864 from tree-sitter/wasm-language 
Add optional WASM feature to the native library, allowing it to run wasm-compiled parsers via wasmtime
 2023-11-28 12:08:47 -08:00
Max Brunsfeld 13dd76e444 Return an informative error on failing to construct a WasmStore 2023-11-27 17:48:29 -08:00
Max Brunsfeld ac29eab5f9 Remove tests for LookaheadIterator compilation failures 
These add noise when running tests. Let's not add unit tests for
compilation failurs.
 2023-11-27 10:32:25 -08:00
Max Brunsfeld 6fd7a1e44e Return informative error when load_language fails 2023-11-26 12:15:05 -08:00
Max Brunsfeld e9289d3b30 Fix error when wrong language name is passed to load_language 2023-11-24 20:25:28 -08:00
Max Brunsfeld b59716600a Add a test that loads multiple languages in a wasm store 2023-11-24 20:03:54 -08:00
Amaan Qureshi 0ff28346be
Merge pull request #2714 from MrPrezident/test_point_range_captures_fix 
fix "test_point_range_captures not working"
 2023-11-12 21:09:09 -08:00
MrPrezident 3c4957e940 fix "test_point_range_captures not working" 
Fix for https://github.com/tree-sitter/py-tree-sitter/issues/105
 2023-11-12 20:58:51 -06:00
Max Brunsfeld d56d1a32ac Fix test grammar loading in unit test 2023-11-04 12:26:00 -07:00
Max Brunsfeld 473b3c8208 Add a CLI feature flag for wasm runtime support 2023-10-31 16:56:11 -07:00
Max Brunsfeld 3380e0eed7 Fix hang when printing dot graphs from the CLI 2023-10-31 15:59:37 -07:00
Max Brunsfeld f4e2f68f14 Merge branch 'master' into wasm-language 2023-10-27 12:11:43 +01:00
Daumantas Kavolis e26e23fd0e
Fix formatting 2023-10-25 10:24:35 +03:00
Daumantas Kavolis e265929f90
Fix goto_previous_sibling with padded first children 2023-10-25 10:19:03 +03:00
Andrew Hlynskyi a5a7564818 chore(rust): add error reporting for position funcs 2023-10-05 17:24:01 +03:00
Andrew Hlynskyi c63f1680ad chore(rust): improve perf for position funcs 2023-10-05 17:24:01 +03:00
Andrew Hlynskyi d95836eb35 cicd: add en extra check for non matchable skips 2023-10-05 14:39:36 +03:00
Andrew Hlynskyi 82ddb3ddcc cicd: add skips for fragile corpus tests 2023-10-05 07:59:35 +03:00
Amaan Qureshi ef9cabd4b5
fix: update javascript tests and use cpp/javascript master for fixtures 2023-09-20 11:31:53 -04:00
Andrew Hlynskyi 524bf7e2c6
Merge pull request #2607 from ahlinc/fix-zero-deallocs 
fix: `dealloc` calls on zero pointers
 2023-09-03 09:45:52 +03:00
Andrew Hlynskyi 9cc1daafca chore(ffi): remove enum name prefixes from all C enum values 2023-09-03 07:38:27 +03:00
Andrew Hlynskyi 7f7084c2cb chore(test): panic on zero pointer deallocs for alloc tracked scopes 2023-09-03 07:12:11 +03:00
Andrew Hlynskyi 67a5dbdd93 fix: dealloc calls on zero pointers 2023-09-03 06:53:18 +03:00
Andrew Hlynskyi 08ac19086b chore: simplify test case after query state improvements 2023-09-02 23:12:28 +03:00
Andrew Hlynskyi 52f7eaff31 chore(rust): make Query extra predicates state fully immutable 2023-09-02 23:12:28 +03:00
Amaan Qureshi dc5ec1cf53
refactor: remove &Option<T> where used 2023-09-01 17:23:39 -04:00
Amaan Qureshi c5cb27e522
docs: improve predicate docs 2023-08-31 16:01:37 -04:00
Amaan Qureshi 09ac28c77d
feat!: properly handle predicates used on quantified captures 2023-08-31 16:01:36 -04:00
Andrew Hlynskyi 0862921537 doc: Include README as top-level module documentation for all crates 2023-08-28 23:09:37 +03:00
Andrew Hlynskyi cbce87dab1 chore: add an extra notes for hang tests 2023-08-28 14:16:36 +03:00
Andrew Hlynskyi a9c4965dd6 cicd: exclude hang tests for exotic arches and ASAN 2023-08-28 05:39:57 +03:00
Andrew Hlynskyi b205a1f5a2 Run parser hang test in a killable subprocess 2023-08-27 21:00:26 +03:00
Amaan Qureshi a4ea4737ac fix: do not increment current_included_range_index past included_range_count in __do_advance 2023-08-27 14:16:18 +03:00
Andrew Hlynskyi 60779cc1ac fix(gen): parser.c should include parser.h relatively 2023-08-26 20:57:08 +03:00
Amaan Qureshi 6d41d99990
Merge pull request #2559 from amaanq/silent-ignore-u 
fix: ignore regex u flag
 2023-08-25 19:25:15 -04:00
Amaan Qureshi 6969766664 fix: ignore regex u flag 
It's needed for certain regex patterns in JS, and we can just silently ignore it.
 2023-08-25 19:23:08 +03:00
Andrew Hlynskyi b3fef28a10 chore(gen): add parser.c enum names to be better discoverable 2023-08-25 19:11:42 +03:00
Andrew Hlynskyi fbfa58edc8 chore(gen): move external scanner stuff closer to the end of parser.c 2023-08-25 19:11:42 +03:00
Amaan Qureshi 1dbb986515 chore: add a test for an aliased anonymous symbol with flags 2023-08-23 16:55:12 +03:00
Andrew Hlynskyi 683fe442e4 fix(gen): cycle between aliases and anonymous symbols 
An example of an error cycle in a `parser.c`:

```
static const TSSymbol ts_symbol_map[] = {
  ...
  [anon_sym_RBRACE] = anon_sym_RBRACE2,
  [anon_sym_RBRACE2] = anon_sym_RBRACE,
  ...
};
```
 2023-08-23 16:51:05 +03:00
Amaan Qureshi e3a5863287
feat: add @injection.parent to inject an injection's node with the parent language 2023-08-21 01:08:17 -04:00
Andrew Hlynskyi ffae7d6115 fix: mark helper Rust funcs that receive raw pointers as unsafe 2023-08-20 02:12:51 +03:00
Amaan Qureshi ac579be788
fix(safety): mark functions that potentially deref a raw pointer as unsafe 2023-08-19 18:59:11 -04:00
Amaan Qureshi 45313e0dfb
feat: add injection.self to inject a node with itself 2023-08-18 18:37:57 -04:00
Amaan Qureshi ce4a9ef4de
feat: allow specifying an input query-paths argument 2023-08-17 22:21:20 -04:00
Amaan Qureshi 9971e5d5f5
feat: allow specifying an input captures-path argument 2023-08-17 22:20:57 -04:00
Amaan Qureshi 458b5de0fc
feat: add any-of predicate 2023-08-17 22:20:57 -04:00
Amaan Qureshi 8eb92d1c64
feat: add an --apply-all-captures argument to highlight & test 2023-08-17 22:20:57 -04:00
Andrew Hlynskyi 6616d7deeb Change edit dumps format and add indexes for all corpus tests 
The new dumped edits format is `edit.<SEED>.<TEST_INDEX>.<SEED_INDEX>.<TEST_NAME>`

To cleanup edits:
- Instead of the standard:
    > rm target/scratch/edit.*

- Use the following shell command:
    > find target/scratch/ -name 'edit.*' -delete
 2023-08-18 00:02:21 +03:00
Andrew Hlynskyi caaa533b8d test: add TREE_SITTER_DUMP_EDITS env var to dump corpus edits 
It dumps edits to the `target/scratch` folder in a format like `<SEED>.edit`
 2023-08-17 20:55:28 +03:00
Andrew Hlynskyi 0dd85c8452 Add TREE_SITTER_GRAMMAR_DEBUG env var to use debug grammars in tests 2023-08-17 18:16:41 +03:00
Amaan Qureshi e0434327d0
fix(render): only output SPACE for strings that are just a space 2023-08-16 13:44:44 -04:00
Amaan Qureshi 2d0a3d647b
Merge pull request #2491 from amaanq/sanitize 
fix: replace & sanitize more characters
 2023-08-13 20:15:20 -04:00
Amaan Qureshi 0b1b0d2fb7
fix: replace & sanitize more characters 2023-08-13 19:29:37 -04:00
Amaan Qureshi 318d4a705a fix: don't add flags if they're not present 2023-08-13 16:28:03 -04:00
Amaan Qureshi dee98e06e9
Merge pull request #2474 from amaanq/case-insensitive 
feat!: support the case-insensitive regex flag
 2023-08-11 23:12:15 -04:00
Andrew Hlynskyi 4a007259fc Fix warning from #2454 in more clear way 2023-08-10 03:59:34 +03:00
Brian Strauch d0afe1ad0a fix: pad hex numbers with zeros when writing colors 
This solves a bug where a hex color with a zero can get truncated
 2023-08-08 09:04:15 -04:00
Amaan Qureshi f35752e3ac
feat!: support the case-insensitive regex flag 2023-08-06 23:40:29 -04:00
Amaan Qureshi 96919cae5d
fix(generate): add Vertical Tab and Form Feed to the whitespace class 2023-08-06 20:41:09 -04:00
Andrew Hlynskyi e024653cb0
Merge pull request #2403 from jakesarjeant/master 
feat(cli): add option to select JS runtime other than node
 2023-08-03 22:29:22 +03:00
Jake Sarjeant 61b70943b1 feat(cli): add option to select JS runtime other than node 2023-08-03 21:34:47 +03:00
Eddie Hatfield 14aa23b1e1 feat: run highlight tests in subdirectories 2023-08-03 19:11:28 +01:00
Andrew Hlynskyi 485d192880 binding_rust: set_max_start_depth accepts optional to reset limit 2023-08-03 11:45:36 +03:00
Samuel Moelius 7b01d8ee05 Add test_query_max_start_depth_more 2023-08-03 11:45:34 +03:00
Samuel Moelius 41ec8b4166 Partially revert d4d5e29 2023-08-03 11:41:50 +03:00
Andrew Hlynskyi 7816397026 Add compile fail tests for LookaheadIterator 2023-08-03 03:49:57 +03:00
Andrew Hlynskyi 4eb22ed3ae Fix UB for LookaheadIterator 2023-08-03 03:49:57 +03:00
Andrew Hlynskyi 4f28ce4e3b Add async context tests for Node and TreeCursor 2023-08-03 03:03:03 +03:00
Michael Davis 3e499d675a
CLI: Re-use highlight cancellation flag for HTML highlights 
The `html` highlight function created its own cancellation flag which
conflicts with the Ctrl-c handler set up in the CLI's `main` block
for `tree-sitter highlight`. We can re-use the cancellation flag from
that block to avoid a panic that happens when using `tree-sitter
highlight -H <file>`

    thread 'main' panicked at 'Error setting Ctrl-C handler: MultipleHandlers', cli/src/util.rs:31:6

This change also aligns the parameters that `highlight::ansi` and
`highlight::html` take.
 2023-08-02 08:56:26 -05:00
Amaan Qureshi acef7a3cdf chore: add tests 2023-08-02 10:47:28 +01:00
Amaan Qureshi b8fe5fe21b fix: do not allow eof to advance states if the new state is the same state 2023-08-02 10:47:27 +01:00
Andrew Hlynskyi a08c61235d Fix, reference by value for Language as everywhere 2023-08-02 00:04:24 +03:00
Daumantas Kavolis ec90c215ae Add tests for bidirectional cursor 2023-08-02 00:02:33 +03:00
Daumantas Kavolis 1cb378b445 Add rust tests for lookahead iterator 2023-08-02 00:02:33 +03:00
Andrew Hlynskyi 19e665ef79 Add tests for TextProvider 2023-08-01 09:12:45 +03:00
Andrew Hlynskyi a2f834d846 More error contexts + conv panics to errors with context 2023-07-30 21:16:45 +03:00
Amaan Qureshi f4e788b28e
feat: warn when unused conflicts are present in a grammar 2023-07-28 00:23:28 -04:00
Amaan Qureshi b8f7645ae2
Merge pull request #2434 from amaanq/test-update 
fix: retain header and divider delimiter lengths when updating tests
 2023-07-27 13:40:53 -04:00
Amaan Qureshi 36c6c8aadf
Merge pull request #2423 from amaanq/nonzero-field-id 
feat!: use `Option<NonZeroU16>` for TSFieldIds
 2023-07-27 13:04:34 -04:00
Amaan Qureshi 0787c24a14
Merge pull request #2430 from amaanq/negative-assertions 
Negative assertions
 2023-07-27 12:44:12 -04:00
Amaan Qureshi 99366f3b5d
ci(sparc64): increase timeout time 2023-07-27 12:26:25 -04:00
Amaan Qureshi 192793a6ab
fix: avoid panic if input is empty 2023-07-27 03:37:02 -04:00
Amaan Qureshi ee399e67d4
fix: retain header and divider delimiter lengths when updating tests 
This helps to prevent visual bloat in diffs when updating tests where the delimiter is not 80 chars long
 2023-07-27 03:31:46 -04:00
Amaan Qureshi 68b1006a3b
feat: add negative assertions, remove duplicate code 2023-07-25 00:27:34 -04:00
Amaan Qureshi c9fd357c06
fix: avoid infinite loop bug 
If the current tag is the last tag and the test fails, the j loop neverends. This prevents that by checking that in the j loop
 2023-07-24 23:43:25 -04:00
Amaan Qureshi a809274009
Merge pull request #2426 from amaanq/sort-tests 
fix(tests): sort categories alphabetically
 2023-07-24 23:21:57 -04:00
Amaan Qureshi 36deb567c8
fix(tests): sort categories alphabetically 2023-07-24 03:05:09 -04:00
Amaan Qureshi 75e1bcf70a
refactor!: rename proc_macro to avoid conflicts when building std with -Z build-std 2023-07-24 00:44:44 -04:00
Amaan Qureshi c521e9c18e
chore: improve error message in some spots loading grammar.json 2023-07-24 00:44:44 -04:00
Amaan Qureshi 9e99789e4c
Merge pull request #2407 from amaanq/bump-deps 
Bump deps
 2023-07-19 17:05:10 -04:00
Amaan Qureshi fc1514d0cd
Merge pull request #2401 from amaanq/fix-inline-bug 
fix: disallow inlining the first rule
 2023-07-19 17:04:17 -04:00
Amaan Qureshi 5fba369c4a
fix: disallow inlining the first rule 
This prevents a panic when indexing symbol_ids during the generation process
 2023-07-19 16:14:58 -04:00
Samuel Moelius d9d4116105
Add test_consecutive_zero_or_modifiers 2023-07-19 03:27:43 -04:00
Amaan Qureshi 0e53b66931
chore: remove useless use 2023-07-19 03:12:13 -04:00
Max Brunsfeld 8310fd423e Update node unit test to reflect C grammar change 2023-07-18 17:58:19 -07:00
Max Brunsfeld 40f7b2ec97 Fix parsing of queries that start with repetitions followed by alternatives 2023-07-18 17:57:52 -07:00
Amaan Qureshi e77405841e
refactor: make a subdirectory for the scratch dir depending on the machine's features 2023-07-18 20:14:21 -04:00
Amaan Qureshi 76f5f81aee
chore: update test 2023-07-18 19:29:40 -04:00
Amaan Qureshi c436897662
test: bump start_time limit for sparc64 2023-07-18 19:07:51 -04:00
Andrew Hlynskyi e1ea3328f4
Merge pull request #2394 from amaanq/hmmcipt2 
Add more architectures to CICD
 2023-07-16 17:23:26 +03:00
Amaan Qureshi 975f3f4686 refactor!: use AtomicUsize for AllocationRecorder for portability 2023-07-16 05:14:11 -04:00
Amaan Qureshi 2a277879ab
Merge pull request #2340 from tree-sitter/highlights-conformance 
feat: add --check flag to `tree-sitter highlight`
 2023-07-16 04:57:23 -04:00
Andrew Hlynskyi 0558f1e0bf test: add retries for all timeout tests 2023-07-16 05:05:36 +03:00
Patrick Thomson cb58bc593f
Add --check flag to tree-sitter highlight. 
Recently I've been pulling a lot of grammars into GitHub's highlighting backend,
replacing legacy language support with tree-sitter highlighting queries.
Our backend systems have a standard set of highlight captures we expect, very
similar to the standard tagging captures we expect. Though end-user applications
are free to choose whatever tagging nomenclature they want, I think it's nice to
include a checking stage that will help us ensure that we know whether a capture
might be recognized or not. It will also help us figure out where we need to
expand our standard set of captures (see #1539).
 2023-07-15 20:49:03 -04:00
Andrew Hlynskyi 0b0cc6c429 Fix rustc 1.71.0 warnings 2023-07-13 17:50:04 +03:00