547 commits

Author	SHA1	Message	Date
Amaan Qureshi	350fff24bf	fix(lib): simplify edge cases with zero-width tokens	2024-10-28 13:57:29 +00:00
Amaan Qureshi	40606dd632	feat: add version subcommand for versioning grammars	2024-10-18 22:08:57 -04:00
Amaan Qureshi	38e3e51fca	feat(rust): add Language::node_kind_is_supertype	2024-10-14 17:53:22 -04:00
Amaan Qureshi	538a197976	fix(lib): correct unexpected side effect in get_column when the lexer is at EOF	2024-10-08 23:27:42 -04:00
Amaan Qureshi	9c08edb066	build: configure clippy lints at the workspace level	2024-10-08 21:07:41 -04:00
Will Lillis	5c6445edea	chore: misc clippy lints	2024-10-06 17:55:00 -04:00
Amaan Qureshi	8943983df6	feat!: properly handle UTF-16 endianness encoding	2024-10-05 21:12:48 -04:00
Amaan Qureshi	c611e15a31	chore(cli): minor correction in comments	2024-10-05 01:41:05 -04:00
Riley Bruins	0683136ca0	feat(api): expose function to check if symbol represents a supertype	2024-09-30 14:44:13 -04:00
Amaan Qureshi	934a2814fd	fix: deprecate child_containing_descendant and add child_with_descendant instead	2024-09-30 13:57:28 -04:00
Amaan Qureshi	ea3846a2c5	feat: move tree-sitter configuration to dedicated file (#3700)	2024-09-30 11:11:23 -04:00
Ron Panduwana	a83b893016	fix: handle more cases of editing subtrees that depend on column values	2024-09-29 20:36:08 -04:00
Jeong, Hun	bbc1370dd5	feat(lib)!: treat nodes' end ranges exclusively in goto_first_child_for_{byte,point} ... This goes back on #1640, because now cursors are bi-directional, and going to the previous sibling is simple.	2024-09-29 17:45:07 -04:00
Riley Bruins	b36ef4b7f4	fix(lib)!: child_containing_descendant now returns direct children ... Previously, `child_containing_descendant` would return `null` when called on a node's direct parent. In my opinion, this doesn't make much sense; it seems like a node would contain itself. This (breaking) commit changes the function so that it can return direct children.	2024-09-29 17:44:37 -04:00
Lukas Seidel	6b1ebd3d29	feat!: implement StreamingIterator instead of Iterator for QueryMatches and QueryCaptures ... This fixes UB when either `QueryMatches` or `QueryCaptures` had collect called on it. Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>	2024-09-29 17:34:48 -04:00
Amaan Qureshi	871e8966c5	build: bump deps	2024-09-28 13:58:58 -04:00
Amaan Qureshi	31f24395b4	feat: move generate logic to its own crate	2024-09-27 17:04:21 -04:00
Amaan Qureshi	90efa34608	chore: clippy fixes	2024-09-27 16:32:48 -04:00
Amaan Qureshi	b2359e4020	feat!: move generation of grammar files to an init command ... The generate subcommand should stick to solely generating a parser and headers.	2024-09-27 16:06:57 -04:00
Amaan Qureshi	0c43988a5e	fix(lib): correct descendant-for-range behavior with zero-width tokens	2024-09-27 00:37:13 -04:00
Samuel Tardieu	1aa28e04ee	style(tests): do not use .as_bytes().len() on strings	2024-09-22 20:27:38 -04:00
Amaan Qureshi	6f050f0da5	fix: properly handle utf8 code points for highlight and tag assertions	2024-09-22 01:03:59 -04:00
Amaan Qureshi	0a85744eba	fix(lib): peek at the next sibling when iterating to find the child that contains a given descendant ... This issue shows up when we have a zero-width token that is the target descendant node, previously the previous sibling would be returned as the child that contains the descendant, which is incorrect.	2024-09-17 04:37:43 +02:00
Amaan Qureshi	7e3f572655	feat: add field_name_for_named_child	2024-09-09 11:05:29 -04:00
Amaan Qureshi	10e474f488	feat!: remove filter flag from commands in favor of include and exclude	2024-09-07 20:53:04 -04:00
Amaan Qureshi	ee06325f67	fix(lib): correct extra node creation from non-zero root-alias cursors	2024-08-31 23:58:19 -04:00
Amaan Qureshi	3f424c0121	feat: add an API to time out query executions ... Currently, if a predicate is hard to match on the Rust side, a sizable query against a very large file can take forever, and ends up hanging. This commit adds an API function `ts_query_cursor_set_timeout_micros` to limit how long query execution is allowed to take, thereby negating the chance of a hang to occur.	2024-08-31 14:33:28 -04:00
Amaan Qureshi	6ef76858c0	fix!: revert interning of a sequence or choice of a single rule ... This reverts commit 252e2a4bc0	2024-08-24 19:43:04 -04:00
Ryan Patterson	b5e4ef6d9a	clone wasm store engine (#3542) ... This resolves https://github.com/tree-sitter/tree-sitter/issues/3454. This brings the usage of wasmtime::Engine in line with how wasmtime intends it to be used. All wasmtime functions that receive an Engine always receive an `&Engine`, never an owned `Engine`. They are always responsible for cloning the reference if they need it. This brings the usage of wasmtime::Engine in line with how TSParser treats TSLanguages: when setting a language to the parser, the parser is responsible for cloning the reference to the TSLanguage. It is counterintuitive for TSParser to have different behavior when receiving wasmtime_engine_t. C API users also expect this behavior, see "Memory Management" [here](https://docs.wasmtime.dev/c-api/wasm_8h.html). Talking about the C API: without this change, failing to clone the `wasmtime_engine_t` (which, again, is never something API users need to do in wasmtime) and then reusing the engine in multiple TSLanguages results in a use after free. With this change, failing to call `wasm_engine_delete` on your owned Engine results in a memory leak. Memory leaks are safer than use-after-free.	2024-08-22 08:01:37 -07:00
Amaan Qureshi	6dd459b4ab	fix(lib): an empty root node should not precede an empty range ... The problem is, given an empty file, the root node of this file spans 0 bytes. As such, the logic for determining whether or not the node precedes the range fails, and is true when it should be false.	2024-08-17 17:05:26 -04:00
Ron Panduwana	2bb20fe2fe	feat: allow external scanners to use the logger ... Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>	2024-08-17 14:46:28 -04:00
Ryan Patterson	779566f588	Reset language when resetting wasm store (#3495) ... * Reset language when resetting wasm store * test behavior of language copying	2024-07-31 10:30:58 -07:00
ObserverOfTime	3950dddfde	fix(rust): fix new clippy warnings	2024-07-28 10:12:32 +03:00
Amaan Qureshi	25c7189180	feat(lib): add ts_query_end_byte_for_pattern	2024-07-07 20:29:09 -04:00
Quentin LE DILAVREC	9610a84600	fix(lib): restrict pattern_map optimization when a wildcard step has an immediate first child ... Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>	2024-07-05 03:35:24 -04:00
Amaan Qureshi	252e2a4bc0	fix: intern a sequence or choice of a single element the same as the element itself	2024-06-21 20:20:02 -04:00
Amaan Qureshi	e553578696	feat: add fuzz subcommand	2024-05-30 23:00:45 -04:00
Amaan Qureshi	8e8648afa9	fix(test): multi-grammar corpus tests are now in the repo root	2024-05-06 15:25:42 -04:00
ObserverOfTime	572cdaa7c0	ci: fix address sanitizer step	2024-05-05 13:06:45 -04:00
Amaan Qureshi	67a043911c	test: temporarily disable php test	2024-05-04 03:05:58 -04:00
Amaan Qureshi	4c083252ec	fix(lib): advance the lookahead end byte by 4 when there's an invalid code point ... This helps in the case where an edit was made in the middle of a code point, but bytes 1-3 are valid, thus we could advance by at most 4 bytes	2024-04-30 20:55:43 -04:00
Amaan Qureshi	61d0395543	fix(lib): do not return field names for extras	2024-04-30 19:23:11 -04:00
WillLillis	946acfd70f	feat(cli): support NO_COLOR ... Co-authored-by: Amaan Qureshi <amaanq12@gmail.com> This allows users to avoid colored output when NO_COLOR is set to 1.	2024-04-29 20:25:41 -04:00
Amaan Qureshi	627617edb4	refactor(tests): migrate remaining grammar.json tests to grammar.js	2024-04-29 00:01:35 -04:00
vanaigr	90e0e28b95	feat: reverse iteration through node parents (#3214)	2024-04-23 15:19:57 +01:00
Amaan Qureshi	abc7910381	refactor(rust): misc fixes & tidying	2024-04-11 22:35:43 -04:00
Amaan Qureshi	5825e24d56	style: wrap comments	2024-04-11 22:35:43 -04:00
Amaan Qureshi	b35efa8f33	style: format imports	2024-04-11 22:35:43 -04:00
Amaan Qureshi	62cd13a3de	test: add regression test for node parent + string bug	2024-03-20 19:23:08 -04:00
Max Brunsfeld	09b18fad5b	Merge pull request #3181 from tree-sitter/handle-wasm-oom ... When loading languages via WASM, gracefully handle memory errors and leaks in external scanners	2024-03-18 13:15:06 -07:00
Max Brunsfeld	fed9b98d6c	Clean up wasm store construction, avoid hard-coding initial memory size ... Retrieve the initial memory size from the stdlib's memory import type.	2024-03-17 10:20:18 -07:00
Max Brunsfeld	7a9b3076ef	Handle memory errors occurring in wasm scanners ... * In WASM, use a custom, simple malloc implementation that lets us expicitly reset the heap with a new start location. * When a WASM call traps or errors, propagate that as a parse failure. * Reset the WASM heap after every parse. Co-authored-by: Conrad <conrad@zed.dev>	2024-03-17 10:19:42 -07:00
Amaan Qureshi	e8dbe011d3	test: fix header writes	2024-03-17 05:36:30 -04:00
Amaan Qureshi	754aa62e89	chore: turbofish styling	2024-03-17 05:36:30 -04:00
Amaan Qureshi	99b93d83a1	feat(cli)!: add a separate build command to compile parsers ... This allows users to build parsers without having to run `test` or `parse` to invoke the compilation process, and allows them to output the object file to wherever they like. The `build-wasm` command was merged into this by just specifying the `--wasm` flag.	2024-03-17 05:36:30 -04:00
Amaan Qureshi	ad07fa8a9e	feat(bindings/rust): expose Parser::included_ranges	2024-03-12 02:29:35 -04:00
Amaan Qureshi	92675117a6	fix(generate): extern allocator functions for the template don't need to be "exported"	2024-03-05 11:19:06 -05:00
Amaan Qureshi	6b2879144f	build(deps): bump deps & lockfile	2024-02-29 02:17:30 -05:00
Amaan Qureshi	5a4765ca01	style: clippy lints	2024-02-29 02:17:30 -05:00
ObserverOfTime	b4b2d9cecc	refactor: remove extern/const where possible	2024-02-29 01:50:04 -05:00
Amaan Qureshi	d92a7a9690	test: add test case for parse stack merging with incorrect error cost bug	2024-02-28 08:51:28 -05:00
Amaan Qureshi	f1eecf9786	feat: expose the allocator and array header files for external scanners	2024-02-26 09:43:13 -05:00
Amaan Qureshi	58a4fcc792	chore: get rid of github_issue_test file	2024-02-21 12:01:25 -05:00
Amaan Qureshi	b40839cd72	style: prefer turbofish syntax where possible	2024-02-19 16:00:50 -05:00
Amaan Qureshi	fd91404ab0	style: tidying	2024-02-19 16:00:50 -05:00
Amaan Qureshi	f526be8061	test: update html tests	2024-02-19 06:36:48 -05:00
Amaan Qureshi	e32a7f3998	chore: clippy lints	2024-02-16 16:32:07 -05:00
Amaan Qureshi	4342efd57e	feat: allow specifying an external scanner's files	2024-02-16 16:31:56 -05:00
Amaan Qureshi	b6c75ccec1	chore: update relevant rust tests	2024-02-16 12:34:12 -05:00
Amaan Qureshi	e996c32108	refactor!: remove the apply-all-captures flag, make last-wins precedence the default	2024-02-16 12:34:12 -05:00
Amaan Qureshi	d80d101e34	build: move common Cargo.toml keys into the workspace and inherit them	2024-02-14 14:56:23 -05:00
Amaan Qureshi	4303ab99c9	fix: properly handle Query.matches when filtering out results	2024-02-13 16:40:05 -05:00
Amaan Qureshi	5ea0dbf77a	chore: some more clippy lints	2024-02-13 03:33:07 -05:00
Amaan Qureshi	a07f988905	refactor: extract regex check into a function and lower its precedence	2024-02-13 03:33:07 -05:00
Bedis Nbiba	7dd096c5f7	feat: implement first-line-regex	2024-02-13 01:51:41 -05:00
Amaan Qureshi	39df8e2833	chore(test): use different languages for async tests	2024-02-12 06:30:36 -05:00
Amaan Qureshi	4149ed4149	chore: provide a CLI flag to open log.html ... On macOS, this was done by default regardless of what the user wants. This was also not done on Windows or Linux. Instead, we now provide a `--open-log` flag to open the log file in the default browser, and it works on all platforms.	2024-02-11 02:28:34 -05:00
Amaan Qureshi	cca814afd2	chore: add java and tsx to corpus tests	2024-02-10 18:25:35 -05:00
Amaan Qureshi	a969fc428c	chore: move dependencies into the workspace and inherit from there	2024-02-08 15:24:09 -05:00
dundargoc	df1fe842eb	docs: various fixes ... Closes https://github.com/tree-sitter/tree-sitter/issues/1317. Closes https://github.com/tree-sitter/tree-sitter/issues/1752. Closes https://github.com/tree-sitter/tree-sitter/issues/2439. Co-authored-by: Simon Hengel <sol@typeful.net> Co-authored-by: Akash Yadav <itsaky01@gmail.com> Co-authored-by: Sebastiaan Speck <12570668+sebastiaanspeck@users.noreply.github.com> Co-authored-by: Laytan Laats <laytanlaats@hotmail.com>	2024-02-08 00:08:59 +01:00
Amaan Qureshi	59be1edaa1	refactor: swap &Vec[T] with &[T] where appropriate	2024-02-07 02:50:31 -05:00
dundargoc	c8bd6705cf	chore: clippy	2024-02-06 23:34:14 -05:00
Max Brunsfeld	f4788b49c0	Merge branch 'master' into wasm-stdlib	2024-02-05 11:55:58 -08:00
Amaan Qureshi	55afb4efaf	ci: remove unnecessary targets	2024-02-04 04:19:09 -05:00
Amaan Qureshi	04ff704bca	chore(cli): apply clippy fixes	2024-02-04 04:18:48 -05:00
Max Brunsfeld	e054de4191	Return an error from build-wasm if scanner uses unavailable functions	2024-02-02 12:04:49 -08:00
Amaan Qureshi	9f2dc9d6b5	fix: rework parser.h includes for test grammars and multi-grammar repos	2024-02-02 10:42:39 -05:00
Amaan Qureshi	422e74fbdb	chore: update javascript-relevant tests	2024-02-02 08:58:22 -05:00
Max Brunsfeld	d351f81b4a	Release stale instances from wasm store when languages are dropped	2024-01-29 10:17:49 -08:00
Max Brunsfeld	c4142737ec	Merge branch 'master' into language-reference-count	2024-01-25 12:43:56 -08:00
Max Brunsfeld	3c66dd2d6c	Add missing language return in ts_query_new ... Co-authored-by: Antonio <antonio@zed.dev> Co-authored-by: Mikayla <mikayla@zed.dev> Co-authored-by: Joseph <joseph@zed.dev>	2024-01-25 11:06:54 -08:00
Amaan Qureshi	b26e0a8ec0	fix: account for grammars in subdirectories, add typescript to test corpus	2024-01-25 14:02:12 -05:00
Max Brunsfeld	3139760fa9	Record allocations in wasm language error test	2023-12-31 10:05:06 -08:00
Max Brunsfeld	82c1b5ead3	Fix compile error in parser hang test	2023-12-27 15:10:52 -08:00
Max Brunsfeld	4a8e4b1963	Allow wasm languages to be deleted	2023-12-27 15:00:16 -08:00
Max Brunsfeld	da16cb1459	Introduce language ref-count management C APIs, remove Copy impl for Language in Rust	2023-12-27 14:59:16 -08:00
Max Brunsfeld	034f0d0280	Merge pull request #1864 from tree-sitter/wasm-language ... Add optional WASM feature to the native library, allowing it to run wasm-compiled parsers via wasmtime	2023-11-28 12:08:47 -08:00
Max Brunsfeld	13dd76e444	Return an informative error on failing to construct a WasmStore	2023-11-27 17:48:29 -08:00
Max Brunsfeld	ac29eab5f9	Remove tests for LookaheadIterator compilation failures ... These add noise when running tests. Let's not add unit tests for compilation failurs.	2023-11-27 10:32:25 -08:00
Max Brunsfeld	6fd7a1e44e	Return informative error when load_language fails	2023-11-26 12:15:05 -08:00