libs-code/tree-sitter.tree-sitter
1
0
Fork 0
mirror of https://github.com/tree-sitter/tree-sitter.git synced 2026-09-14 09:26:23 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
5714 commits 19 branches 114 tags 52 MiB
Commit graph

316 commits

Author SHA1 Message Date
Andrew Hlynskyi af42cfccd6 rust: fix ffi impl, some structs are stack allocated 2023-07-13 23:48:38 +03:00
Amaan Qureshi 2f2e416031 chore: update Rust bindings 2023-07-13 11:39:33 -04:00
Andrew Hlynskyi f01c4f8376 Restore Rust bindings generation with newer bindgen 0.65.1 2023-07-13 17:34:32 +03:00
Max Brunsfeld 4922602369
Merge pull request #2316 from tree-sitter/node-descendant-index 
Add APIs for retrieving descendant counts, seeking to nodes by their descendant index
 2023-06-12 15:13:43 -07:00
Max Brunsfeld a2119cb691 Add APIs for retrieving tree cursor's depth and descendant index 2023-06-12 11:50:44 -07:00
Max Brunsfeld 9dd725b4e3 Start work on exposing APIs for node descendant counts and indices 2023-06-09 08:51:47 -07:00
Samuel Moelius f461d953a6 Fix OOB in Query::new 2023-05-19 21:31:26 -04:00
Andrew Hlynskyi cc6596be82 chore(bindgen): update bindgen to 0.65.1 and regenerate bindings 2023-04-17 11:24:05 +03:00
Lewis Russell 1e81a1b67f feat(lib): add ts_query_cursor_set_max_start_depth query API 
This allows configuring cursors from traversing too deep into a tree.
 2023-04-17 11:15:13 +03:00
Andrew Hlynskyi 613382c70a docs: update badges; fix markdown lint complains 
Linter config `.vscode/settings.json`:
```json
{
    "[markdown]": {
        "files.trimTrailingWhitespace": false,
    },
    "markdownlint.config": {
        "default": true,
        // "ul-style": {
        //     "style": "asterisk"
        // },
        "MD001": false,
        "MD024": false,
        "MD025": false,
        "MD033": false,
        "MD041": false,
        "MD053": false,
    },
}
```
 2023-04-16 23:39:08 +03:00
Andrew Hlynskyi c38f78345e binding(rust): update script/generate-bindings to use latest rust-bindgen 0.64.0 version 2023-04-04 22:16:27 +03:00
Andrew Hlynskyi d56506b727 ref(rust_bindings): keep Query construction logic in a one place 2023-04-04 15:08:53 +03:00
Vhyrro e5357dc5ad ref(rust_bindings): add from_raw() functions 2023-04-04 15:08:53 +03:00
Vhyrro 6c2957c8d3 ref(rust_bindings): move into_raw() functions into the ffi module 2023-04-04 15:08:53 +03:00
Vhyrro 576e4c7d06 fix(rust_bindings): use-after-free for Language::into_raw() 2023-04-04 15:08:53 +03:00
Vhyrro 3d396e120b feat(rust_bindings): provide into_raw() functions for treesitter structs 2023-04-04 15:08:53 +03:00
Andrew Hlynskyi ec2af46f6f chore: remove all dead badges 2023-03-16 18:00:02 +02:00
Max Brunsfeld 837899e456 Add API for checking if a pattern in a query is non-local 2023-02-16 11:59:34 -08:00
Max Brunsfeld 97fd990822 Add --dot flag to parse subcommand, for printing tree as DOT graph 2023-02-13 12:33:34 -08:00
Léo Gaspard 3964651fc6
Derive Hash for Language 
Given Language implements Eq, it can probably implement Hash too.

This makes it easy to derive a random identifier from a `Language`
 2023-02-09 03:35:42 +01:00
Max Brunsfeld 555277a102 Allow testing highlight and tag queries when testing wasm build 
Replace non-mutating `ts_parser_wasm_store` function with
`ts_parser_take_wasm_store`, which removes and returns the wasm
store, in order to facilitate single ownership.
 2023-01-23 11:46:59 -08:00
Max Brunsfeld 042e6f9d57 Restucture wasm module, return a normal TSLanguage from load_language 2022-11-15 17:14:33 -08:00
Max Brunsfeld 3f1a7f9cd4 Start work on ability to load wasm languages from native lib, via wasmtime 2022-11-15 17:14:33 -08:00
Max Brunsfeld 618146260e Add doc comments for tree included ranges getter 2022-11-08 17:47:57 -08:00
Max Brunsfeld f0177f216e Use library-configured allocator to free intermediate included ranges 2022-11-07 10:48:32 -08:00
Max Brunsfeld 1848d0bc36 Add tree included ranges getter 
Co-authored-by: Nathan Sobo <nathan@zed.dev>
 2022-11-04 15:24:07 -07:00
Max Brunsfeld 477b667753 Add ts_tree_root_node_with_offset API 2022-08-18 13:48:47 -07:00
Max Brunsfeld 548c12fb88 Fix bug where patterns with top-level alternatives were not considered 'rooted' 2022-07-07 17:53:54 -07:00
rhysd 08899428f3 Add C APIs as document aliases 2022-05-30 21:36:11 +09:00
Aleksei Bavshin fe33599f46
lib: fix incorrect int ptr cast on big-endian architectures 
`*usize` -> `*u32` conversion on 64-bit big-endian machine takes high
halfword of the value. As a consequence, any result returned via
`count` is unexpectedly shifted left:

    u32   = 00 00 00 01             // 1
    usize = 00 00 00 01 00 00 00 00 // 4294967296

Fixes following test failure:
```
$ cargo test -- tests::corpus_test
<...>
running 13 tests
memory allocation of 206158430208 bytes failed
error: test failed, to rerun pass '--lib'
```
 2022-03-23 00:47:01 -07:00
Max Brunsfeld aaf4572727
Merge pull request #1589 from alex-pinkus/deduplicate-core-ids 
Ignore duplicate states when initializing subgraphs in `ts_query__analyze_patterns`
 2022-01-17 13:54:31 -08:00
Alex Pinkus eaf9b170f1 Don't start with duplicate states in ts_query__analyze_patterns 
This change exposes a new `primary_state_ids` field on the `TSLanguage`
struct, and populates it by tracking the first encountered state with a
given `core_id`. (For posterity: the initial change just exposed
`core_id` and deduplicated within `ts_analyze_query`).

With this `primary_state_ids` field in place, the
`ts_query__analyze_patterns` function only needs to populate its
subgraphs with starting states that are _primary_, since non-primary
states behave identically to primary ones. This leads to large savings
across the board, since most states are not primary.
 2022-01-16 11:17:47 -08:00
Hendrik van Antwerpen acd3d32c36 Remove reference to strings from quantifier-only function 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen e338726cde Prefix globally visible TSquantifier values 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen 36f2440369 Complete comment 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen a1a241b013 Expose quantifiers per pattern, instead of merging for all patterns in a query 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen 1f1a449c76 Improve capture quantifier computation 
Compute quantifiers in a bottom-up manner, which allows more precise
results for alternations, where the quantifiers are now precisly joined.
 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen 9bac066330 Deal with quantifiers appearing on capture's enclosing patterns 
- Use a proper enum type for quantifiers.
- Drop quantifiers from `TSQueryStep`, which was not used.
- Keep track of the captures introduced during a pattern parse, and
  apply the quantifier for the pattern to the captures that were
  introduced by the pattern or any sub patterns.
- Use 'quantifier' instead of 'suffix'.
 2022-01-11 18:33:36 +01:00
Hendrik van Antwerpen ae7869d1a6 Expose capture suffixes in queries 2022-01-11 18:33:36 +01:00
Max Brunsfeld 622359b400 Simplify allocation-recording in test suite using new ts_set_allocator API 2021-12-30 16:09:07 -08:00
Max Brunsfeld 8de8c64c95 Remove unnecessary types from binding.rs 2021-12-09 21:02:15 -08:00
Max Brunsfeld 142f4b6438 Rename Query::step_is_definite -> is_pattern_guaranteed_at_step 2021-11-21 11:37:52 -08:00
Max Brunsfeld e78413832b Restructure test suite's allocation recording so that tests can run in parallel 2021-10-11 17:24:37 -07:00
Max Brunsfeld fe29bc8c19 Tweak comment formatting in api.h so bindgen doesn't create a doctest 2021-10-11 17:23:44 -07:00
Wilfred Hughes f17f08bbef Add missing word in doc comment 2021-09-30 21:53:41 -07:00
Max Brunsfeld 561dfe3a2d
Merge pull request #1394 from tree-sitter/fix/query-predicates-with-alternations 
fix(cli): Panic on queries containing alternation with predicates
 2021-09-24 08:56:23 -07:00
Andrew Hlynskyi d973527964 binding(rust): Mark set_cancellation_flag self as mutable 2021-09-21 23:03:50 +03:00
Andrew Hlynskyi b324d0802a fix(cli): Panic on queries containing alternation with predicates 
`QuearyMatch::satisfies_text_predicates()` was changed to pass
captures that don't relate to a checked predicate. This allows
predicates in inner alternations for queries.

Refs #1392
 2021-09-21 19:51:30 +03:00
Andrew Hlynskyi ae12ff81b5 feat(rust): Add an id() method for QueryMatch 
Refs #1372
 2021-09-14 19:08:29 +03:00
Andrew Hlynskyi 52e6c900c3 fix(lib): fix segfault on ts_query_new with incompatible grammar version, close #1318 2021-09-03 14:24:18 +03:00
Max Brunsfeld f3ea60e23f Merge branch 'master' into query-cursor-api 2021-06-02 11:51:26 -07:00
Max Brunsfeld 8416894923 Use std::iter::Once in impl TextProvider for [u8] 
Co-Authored-By: Douglas Creager <dcreager@dcreager.net>
 2021-06-02 09:15:04 -07:00
Douglas Creager cd96552448 query: Allow configurable match limit 
The default is now a whopping 64K matches, which "should be enough for
everyone".  You can use the new `ts_query_cursor_set_match_limit`
function to set this to a lower limit, such as the previous default of
32.
 2021-06-02 11:30:55 -04:00
Max Brunsfeld d72771a19f Make ::set_{byte,point}_range methods take a Range 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 14:21:11 -07:00
Max Brunsfeld 7f4eb9a222 Provide ::set_{byte,point}_range on both query iterators 2021-05-28 14:07:54 -07:00
Max Brunsfeld 97dfee6325 Add QueryMatch::nodes_for_capture_index 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 12:38:30 -07:00
Max Brunsfeld dab11134c2 Add Query::capture_index_for_name method 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-28 12:27:50 -07:00
Max Brunsfeld 919e9745a6 Add ts_tree_cursor_goto_first_child_for_point function 
This function (and the similar `ts_tree_cursor_goto_first_child_for_byte`)
allows you to efficiently seek the tree cursor to a given position,
exploiting the tree's internal balancing, without having to visit
all of the preceding siblings of each node.
 2021-05-27 12:30:19 -07:00
Max Brunsfeld 036aceed57 In script/generate-bindings, add flags for latest bindgen 2021-05-25 18:02:39 -07:00
Max Brunsfeld fda35894d4 Stop matching new patterns past the end of QueryCursor's range 
This restores the original signatures of the `set_byte_range` and
`set_point_range` functions. Now, the QueryCursor will properly report
matches that intersect, but are not fully contained by its range.

Co-Authored-By: Nathan Sobo <nathan@zed.dev>
 2021-05-25 18:02:35 -07:00
Max Brunsfeld f597cc6a75 Preserve matches that contain the QueryCursor's start byte 
Co-Authored-By: Nathan Sobo <nathan@zed.dev>
Co-Authored-By: Antonio Scandurra <me@as-cii.com>
 2021-05-25 13:06:24 -07:00
Max Brunsfeld a61f25bc58 Add APIs for advancing a QueryCursor to an arbitrary position 2021-05-24 21:07:59 -07:00
Max Brunsfeld 8c3d1466ec Allow QueryCursor's text callback to return an iterator 2021-05-23 21:05:26 -07:00
Max Brunsfeld 0e445c47fa rust: Parser and QueryCursor are Sync 
These objects are not generally intended to be shared betwen threads,
but they meet the Rust definition of Sync: all of their methods are
that take a shared reference to `self` are safe to call from multiple
threads simultaneously. In other words, there is no interior
mutability.
 2021-05-21 21:14:24 -07:00
Niranjan Hasabnis c31acb8fec Changing API name; Adding unit test and Rust bindings 2021-05-21 01:50:10 +00:00
Max Brunsfeld 399b5e4daf Remove interior mutability for parent-node caching in Tree 
In Rust binding, mark Tree as Sync
 2021-05-20 13:56:26 -07:00
Max Brunsfeld b5ec720696
Merge pull request #1071 from ahlinc/feat/rust-binding-additional-error-traits 
feat(binding_rust): implement Display and Error traits for TS errors to be convertible
 2021-04-22 21:24:07 -07:00
Andrew Hlynskyi 3c0a49289c binding_rust: Improve implementation of fmt::Display for QueryError trait, avoid multiple format!() calls 2021-04-23 06:45:04 +03:00
Max Brunsfeld 0bbacebf67
Merge pull request #1068 from ahlinc/fix/cargo-build-warinings 
Resolve syntax warnings for cargo build
 2021-04-22 13:38:03 -07:00
Andrew Hlynskyi de23c9219a binding_rust: Unbind nodes lifitime from a source for QueryCursor.captures 2021-04-22 21:00:58 +03:00
Andrew Hlynskyi 9c91affe56 binding_rust: Unbind nodes lifitime from a source for QueryCursor.matches 2021-04-22 21:00:58 +03:00
Andrew Hlynskyi e7b1d84a83 binding_rust: implement Display and Error traits for TS errors to be convertible 2021-04-22 17:50:12 +03:00
Andrew Hlynskyi 2094a80022 fix(cli): resolve syntax warnings for cargo build 2021-04-22 16:23:00 +03:00
Max Brunsfeld e1a1006b3d
Merge pull request #866 from rhysd/patch-1 
Fix typo in document
 2021-03-11 15:21:45 -08:00
Max Brunsfeld 5ecdfce122
Merge pull request #850 from berkus/patch-1 
Fix typo
 2021-03-09 15:07:48 -08:00
Max Brunsfeld db6e1d9bdc Add ts_query_cursor_did_exceed_match_limit API 2021-03-05 15:02:49 -08:00
Max Brunsfeld 774ae5e3d5 In parse tables, store production ids as 16 bits 
Also remove the use of bitfields from the parse table format.
In all cases, bitfields were not necessary to achieve the
current binary sizes. Avoiding them makes the binaries more
portable.

There was no way to make this change backward-compatible,
so we have finally dropped support for parsers generated
with an earlier version of Tree-sitter.

At some point, when Atom adopts this version of Tree-sitter,
this change will affect Atom users who have installed packages
using third-party Tree-sitter parsers. The packages will need
to be updated to use a regenerated version of the parsers.
 2021-02-25 16:12:31 -08:00
Douglas Creager a29c8d9264 Move allocation tracking into lib crate 
We have several test cases defined in the `cli` crate that depend on the
`lib` crate's `allocation-tracking` feature.  The implementation of the
actual allocation tracker used to live in the `cli` crate, close to the
test cases that use it.  The `allocation-tracking` feature in the `lib`
crate was just used to tell the tree-sitter implementation to expect
that the allocation tracker exists, and to use it.

That pattern meant that we had a circular dependency: `cli` depends on
`lib`, but `lib` required some code that was implemented in `cli`.
That, in turn, caused linker errors — but only when compiling in certain
configurations! [1]

This patch moves all of the allocation tracking implementation into the
`lib` crate, gated on the existing `allocation-tracking` feature, which
fixes the circular dependency.

Note that this patch does **not** fix the fact that feature unification
causes the `lib` crate to be built with the `allocation-tracking`
feature enabled, even though it's not a default.  Fixing that depends on
the forthcoming version 2 feature resolver [2], or using the `dev_dep`
workaround [3] in the meantime.

[1] https://github.com/tree-sitter/tree-sitter/issues/919
[2] https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-resolver-version-2
[3] https://github.com/tree-sitter/tree-sitter/issues/919#issuecomment-777107086
 2021-02-23 09:16:37 -05:00
Max Brunsfeld 6dbe6a3a90 Merge branch 'master' into actions-ci 2021-02-05 10:19:05 -08:00
Michał Muskała 9f8ffe0164 Implement Clone for TreeCursor 
Cloning was supported by the C library, but not exposed in Rust bindings
 2021-02-03 17:16:13 +00:00
Linda_pp bb47ef0d78
Fix typo in document 2021-01-01 00:17:22 +09:00
Berkus Decker 0ceb2af38e
Fix typo 2020-12-17 02:41:25 +02:00
Max Brunsfeld b661050a61 Simplify setup for enabling/disabling allocation recording in the C lib 2020-12-02 15:35:13 -08:00
Max Brunsfeld 7aca288330 Move LossyUtf8 struct out of util module 2020-10-14 11:59:56 -07:00
Max Brunsfeld 0a46033391 Remove duplication of LossyUtf8 helper 2020-10-14 11:35:50 -07:00
Max Brunsfeld 87fd2f5ca3 rust: Detect debug builds using PROFILE env var in build script 
Fixes #757
 2020-10-12 11:53:16 -07:00
Max Brunsfeld 857a9ed07b query: Handle captured wildcard nodes at the root of patterns 2020-10-08 12:34:08 -07:00
Max Brunsfeld d1c95193c1 query: Fix invalid use of slice::from_raw_parts 2020-10-05 12:08:53 -07:00
Max Brunsfeld 2a3c2ad6b9 Implement Send for QueryCursor 2020-09-30 21:19:32 -07:00
Max Brunsfeld ba239ce4ab Make query error line numbers consistently display 1-indexed 2020-09-24 15:03:51 -07:00
Max Brunsfeld 518916f221 Return correct path and line in query errors from the CLI 2020-09-24 13:47:27 -07:00
Max Brunsfeld 9daec9cb22 Tweak impossible pattern error messages 2020-08-20 13:35:11 -07:00
Max Brunsfeld 4301110c12 query: Indicate specific step that's impossible 2020-08-20 13:06:38 -07:00
Max Brunsfeld 1ea29053e1 Merge branch 'master' into query-pattern-is-definite 2020-08-14 09:31:55 -07:00
Max Brunsfeld 1ae5cbc851 query: Handle #not-match? in rust, wasm bindings 2020-07-24 12:15:23 -07:00
Max Brunsfeld cc37da7457 Query analysis: fix propagation of uncertainty from later siblings 2020-06-26 16:46:12 -07:00
Max Brunsfeld 7f955419a8 Start work on recognizing impossible patterns 2020-06-25 15:06:27 -07:00
Max Brunsfeld 4c2f36a07b Mark steps as definite on query construction 
* Add a ts_query_pattern_is_definite API, just for debugging this
* Store state_count on TSLanguage structs, to allow for scanning parse tables
 2020-06-25 15:06:27 -07:00
Max Brunsfeld f38f5d1d22 Tweak readmes 2020-05-12 16:16:48 -07:00
Max Brunsfeld 40262483a9 Change query syntax for predicates 
Signed-off-by: Patrick Thomson <patrickt@github.com>
 2020-05-11 12:35:51 -07:00
Max Brunsfeld 21175142af
Merge pull request #583 from tree-sitter/tags 
Add a 'tags' crate, for computing ctags-style code navigation tags
 2020-04-03 11:20:51 -07:00
Alberto González Palomo bc0d53d4f0
Clarify where to put cc dependency for build script. (#586) 2020-03-30 09:43:41 -07:00
Max Brunsfeld 8eac81b8df Merge branch 'master' into tags 2020-03-25 12:49:15 -07:00
Max Brunsfeld f453178ca2 rust: Remove unnecessary dependencies from core library 2020-03-16 14:46:05 -07:00
Luca Barbato 21de99de87
Remove non-building doctests (#578) 2020-03-16 14:35:27 -07:00
Max Brunsfeld 0457736766 rust: add handling of arbitrary predicate operators 2020-03-13 13:02:34 -07:00
Max Brunsfeld 90cacca040 rust: Simplify and generalize handling of set! and is? predicates 2020-03-10 10:42:22 -07:00
Max Brunsfeld 048b8c87a7
Merge pull request #561 from ubolonton/fix-out-of-bounds-access 
Fix out-of-bounds array access in `Language` APIs
 2020-02-27 09:27:18 -08:00
Tuấn-Anh Nguyễn 5f68fba61c Mark the return value of TreeCursor::field_name static 2020-02-27 22:36:13 +07:00
Tuấn-Anh Nguyễn 23261c4f6f Make ts_language_symbol_name return NULL for out-of-bound ids 2020-02-27 22:24:00 +07:00
Tuấn-Anh Nguyễn c719e24a45 Make ts_language_field_name_for_id return NULL for out-of-bound id 2020-02-27 21:19:08 +07:00
Max Brunsfeld 33492ca9df query: Add not-eq? predicate in rust & wasm binding 2020-02-19 21:36:30 -08:00
Max Brunsfeld b2e79f6438 rust: In queries, allow set! w/ capture and constant 2020-02-14 20:50:07 -08:00
Max Brunsfeld 56e52c76f9 rust: Add Hash for Point, Range and Node 2020-02-13 16:03:02 -08:00
Max Brunsfeld 631710bada Implement Default for Point, Eq for Node 2020-02-11 15:58:35 -08:00
Max Brunsfeld a1c3de1b99 rust: Add Language::node_kind_is_visible API 2020-01-31 12:04:18 -08:00
Max Brunsfeld 64996976f4 rust: Add Language::id_for_node_kind method 2020-01-29 16:48:08 -08:00
Max Brunsfeld e232ed04d1 Fix lifetimes on QueryCursor methods 
Closes #530
 2020-01-28 09:14:09 -08:00
Max Brunsfeld 7de36a33eb Remove halt_on_error API 2020-01-27 15:36:09 -08:00
Max Brunsfeld 9f63139a10 Fix error when set_included_ranges is called with an invalid range list 2020-01-17 10:31:28 -08:00
Max Brunsfeld f3747863df Add ts_query_disable_pattern API 2020-01-15 17:08:55 -08:00
Tuấn-Anh Nguyễn 8e6220a0c5 Allow QueryCursor::matches' text_callback to return any AsRef<[u8]> (#488) 2019-11-17 10:40:06 -08:00
Max Brunsfeld ce633a85c6 Improve ts_language_symbol_for_name function 2019-11-15 14:21:13 -08:00
Max Brunsfeld b4a2f4ed60 Fix debug/release conditional compilation flags for free() 2019-11-15 10:09:22 -08:00
Max Brunsfeld 967da88371 Avoid unnecessary recompiles between debug & test builds 
This makes development much quicker when switching back and forth
between compiling with RLS while editing and running tests with
`cargo test`.
 2019-11-14 13:34:25 -08:00
Max Brunsfeld 1909ae2cee binding_rust: Improve node children APIs 
Closes #480
 2019-11-08 14:20:10 -08:00
Max Brunsfeld c6d1bd2b04 Fix a few rustdoc links 2019-11-08 12:43:26 -08:00
Max Brunsfeld d68dfa78b2 Add misssing docs to rust binding 2019-11-08 12:29:26 -08:00
Max Brunsfeld a7824cf59d Fix docs for ts_node_is_extra 
Fixes #446
 2019-11-05 09:49:16 -08:00
Max Brunsfeld e14e285a10 cli: Check queries when running tree-sitter test 2019-10-18 14:44:16 -07:00
Max Brunsfeld b79295e1a0 🔥 Everything related to property sheets 2019-10-17 11:00:31 -07:00
Max Brunsfeld 324c259cbb rust: Create readable messages for query syntax errors 2019-10-14 12:30:22 -07:00
Max Brunsfeld a45dc67390 Loosen unnecessary lifetime restriction in rust QueryCursor APIs 2019-10-14 12:30:22 -07:00
Max Brunsfeld f490befcde Add ts_query_disable_capture API 2019-10-14 12:30:22 -07:00
Matthew Krupcale ee9a3c0ebb lib: remove utf8proc dependency (#436) 
* Remove dependency on utf8proc

This removes the only external dependency on utf8proc for UTF-8 decoding. It does so by implementing its own UTF-8 decoder. This decoder is both faster and has a simpler API.

 * .gitmodules: remove utf8proc submodule
 * docs/section-2-using-parsers.md: remove requirement for utf8proc submodule
 * docs/section-6-contributing.md: likewise
 * lib/Cargo.toml: remove utf8proc subdirectory package include
 * lib/README.md: remove utf8proc subdirectory description
 * lib/binding_rust/build.rs: remove utf8proc compiler include directory
 * lib/src/lexer.c: remove utf8proc dependencies and types
 * lib/src/lib.c: remove utf8proc dependency
 * lib/src/unicode.h: define types for Unicode decoders
 * lib/src/utf16.{c,h}: implement more readable UTF-16 decoder
 * lib/src/utf8.{c,h}: implement fast UTF-8 decoder
 * lib/utf8proc: remove utf8proc submodule directory
 * script/build-lib: remove utf8proc compiler include directory
 * script/build-wasm: likewise

* Optimize ts_lexer__get_lookahead.

Try to favor non-failure code path and assign lookahead values directly to lexer

 * lib/src/lexer.c: optimize for non-failure code path

* Fix some compiler errors

 * lib/src/lexer.c: cast from signed to unsigned for decode_next result
 * lib/src/utf16.c: fix non-constant initializers for older compilers

* Remove some missed remnants of utf8proc

 * docs/section-2-using-parsers.md: only two include paths necessary now
 * lib/src/lib.c: no need to define UTF8PROC_STATIC

* Use ICU's utf8 and utf16 decoding routines

* Remove unnecessary casts when calling icu macros

* Check buffer length before attempting to decode a unicode character

* Use new unicode function when parsing Queries

Co-Authored-By: Matthew Krupcale <mkrupcale@matthewkrupcale.com>

* Mark libicu files as vendored for GitHub's stats
 2019-10-14 11:18:39 -07:00
Max Brunsfeld 015be68c94 rust: Generalize text_callback type for query methods 2019-10-03 12:53:14 -07:00
Max Brunsfeld 9872a083b7 rust: Change QueryCursor::captures to expose the full match 2019-10-03 12:45:58 -07:00
Max Brunsfeld 27149902f8 Handle is?, is-not?, and set! predicate functions in queries 2019-09-25 10:21:20 -07:00
Max Brunsfeld b15e90bd26 Handle set! predicate function in queries 2019-09-24 11:54:24 -07:00
Max Brunsfeld 070f11b8bf Use ptr::NonNull in Rust bindings 2019-09-23 17:00:52 -07:00
Max Brunsfeld d9b8bae629 rust: Include pattern indices when iterating query captures 2019-09-18 14:22:55 -07:00
Max Brunsfeld 82955759c0 Add an API for getting a pattern's start offset in the source code 2019-09-17 16:19:58 -07:00
Max Brunsfeld 2d1ca8bc9f Fix match return order fom ts_query_cursor_next_match 2019-09-17 14:52:27 -07:00
Max Brunsfeld d4d554b2ae Add wasm bindings for predicates 2019-09-16 10:25:44 -07:00
Max Brunsfeld 096126d039 Allow predicates in queries, to match on nodes' text 2019-09-15 22:06:51 -07:00
Max Brunsfeld a1fec71b19 Tweak QueryCursor to allow iterating either matches or captures 
For syntax highlighting, we want to iterate over all of the captures in 
order, and don't care about grouping the captures by pattern.
 2019-09-13 15:19:04 -07:00
Max Brunsfeld c71de5bd81 Tweak query interface 
* Rename TSQueryContext -> TSQueryCursor
* Remove the permanent association between the cursor and its query. The 
cursor can now be used again for a different query.
 2019-09-11 17:33:48 -07:00
Max Brunsfeld c8c75782e3 Allow tree queries to execute within a limited range 2019-09-11 16:49:29 -07:00
Max Brunsfeld fe7c74e7aa Start work on an API for querying trees 2019-09-10 20:53:57 -07:00
Max Brunsfeld 09a2755399 Store parse states with few lookahead symbols in a more compact way 2019-08-29 15:52:23 -07:00
Max Brunsfeld b3ab2e07a2 binding_rust: Generalize the interface to callback-based parse methods 
Fixes #386
 2019-08-29 10:39:51 -07:00
Tuấn-Anh Nguyễn f0999d7ac0 Make Rust functions return ExactSizeIterator instead of just Iterator (#438) 
* Add CBufferIter::size_hint

* Make Rust functions return ExactSizeIterator instead of just Iterator

* Fix wrong CBufferIter::size_hint
 2019-08-28 09:28:47 -07:00
Tuấn-Anh Nguyễn d96ba09391 Make Tree::changed_ranges return an Iterator instead of a Vec (#437) 
* Make Tree::changed_ranges return an Iterator instead of a Vec

* Remove CBufferIter.free parameter
 2019-08-26 19:36:16 -07:00
Tuấn-Anh Nguyễn a073fc403a Add some missing APIs in Rust binding (#435) 2019-08-24 14:28:58 -07:00
Gregor 105b0ba52c Add build instruction to rust binding README (#432) 
Fixes #431
 2019-08-21 11:59:37 -07:00
Rob Donnelly b1a4cc044d Fix Rust bindings example (#415) 
`Parser::parse` returns `Option<&Tree>`.  An `unwrap()` or similar is required to get the actual `Tree`.
 2019-08-08 08:35:23 -07:00
Max Brunsfeld 93f7de03e2 cli: Handle multi-parser repos, content-regex property 
Prompted by tree-sitter/tree-sitter-typescript#68
 2019-08-07 17:41:45 -07:00
Max Brunsfeld f4740a1beb Make properties generation aware of which nodes are leaves 2019-07-22 09:30:05 -07:00
Max Brunsfeld f505613af3 Fix bug introduced when optimizing property sheet JSON files 2019-07-17 17:00:25 -07:00
Max Brunsfeld def1e1e91a Reduce bloat in generated property sheet JSON files 2019-07-17 14:28:44 -07:00
Max Brunsfeld bd466febb4 highlight: Fix panic when cancelled before parsing a nested document 2019-07-16 14:34:24 -07:00
Max Brunsfeld 28011b1e60 Add ts_node_is_extra API 2019-06-19 15:58:29 -07:00
Max Brunsfeld a7d02e7276 Add support for highlight properties that track local variables 2019-05-09 20:34:34 -07:00
Max Brunsfeld 31c2086273 🎨 Refactor TreePropertyCursor 2019-05-09 09:40:15 -07:00
Max Brunsfeld 572f290ec0 Fix highlighting in the presence of node fields 2019-05-07 11:17:34 -07:00
Max Brunsfeld 3fc459a84b Reorganize language bindings 
* Move rust binding: lib/binding -> lib/binding_rust
* Move wasm bindinig: lib/web -> lib/binding_web
* Add wasm readme
 2019-05-07 10:41:49 -07:00