mirror of
https://github.com/tree-sitter/tree-sitter.git
synced 2026-09-10 07:36:22 -04:00
fad0a62a35
Every (state, terminal) parse-table entry stored its action list inline as a 32-byte `ParseTableEntry`, but across a grammar those lists are ~98-99% _duplicates_. The number of distinct lists is a few thousand regardless of grammar size, while total entries scale into the hundreds of thousands. Store each unique action list once in a shared `ActionListPool` (a flat arena of actions plus `(offset, len)` ranges) and replace the inline entry with a 4-byte `ActionListId` (a pool index with the `reusable` flag packed into the high bit). - intern_table converts the freshly built `ParseTable<ParseTableEntry>` into `ParseTable<ActionListId>`. - minimize carries and operates on the 4-byte ids. The three global state renumberings rewrite Shift targets once at the pool level (`remap_terminal_references`), while the per-state unit-reduction redirects copy the changed list into a new slot (COW). `mark_fragile_tokens` becomes a free bit flip on the id. - `canonicalize` dedups and compacts the pool once before `render`, dropping the dead and duplicate slots the remaps and COW leave behind. `render` assigns the output action-list offsets directly. Yields ~7% wall time reduction, ~12% peak rss reduction. |
||
|---|---|---|
| .. | ||
| cli | ||
| config | ||
| generate | ||
| highlight | ||
| language | ||
| loader | ||
| tags | ||
| xtask | ||