diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 21cf78de..434d60e9 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -10,7 +10,7 @@ on: # pull_request: # No need to trigger on PR, cargo-dist already does push: branches: - - master + - master concurrency: group: ${{ github.workflow }}-${{ github.ref }} @@ -23,15 +23,15 @@ jobs: matrix: build: [linux, macos, windows] include: - - build: linux - os: ubuntu-latest - target: x86_64-unknown-linux-musl - - build: macos - os: macos-latest - target: x86_64-apple-darwin - - build: windows - os: windows-latest - target: x86_64-pc-windows-msvc + - build: linux + os: ubuntu-latest + target: x86_64-unknown-linux-musl + - build: macos + os: macos-latest + target: x86_64-apple-darwin + - build: windows + os: windows-latest + target: x86_64-pc-windows-msvc steps: - name: "[linux] Install dependencies" run: | @@ -81,7 +81,6 @@ jobs: echo "old: $base" cat "$base" diff "$base" "$new_snap" - else fi done shell: bash diff --git a/AGENTS.md b/AGENTS.md index a3648f74..ccfa0779 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,7 +3,7 @@ ## Build/Test/Lint Commands - Build: `cargo build [--release]` - Run: `cargo run [--release]` -- Test (all): `cargo nextest` +- Test (all): `cargo nextest run` - Test (single): `cargo nextest test_name` - Integration/E2E tests: `cargo nextest --tests` (will need tmux under the hood) - Memory leak detection: `cargo nextest run --profile valgrind` @@ -24,10 +24,16 @@ - Follow the existing structure for new modules (see src/engine/ or src/model/) - Implement relevant traits (SkimItem, etc.) for new types when needed -## Project Structure -- Core functionality in `skim/src/` -- Common utilities in `skim-common/` -- Task automation in `xtask/` +## Architecture Documentation +- `ARCHITECTURE.md` documents the full architecture: data flow, operating modes, subsystems, threading model, and public API. +- **Update `ARCHITECTURE.md` whenever you make structural changes**, including: + - Adding, removing, or renaming modules, structs, or traits + - Changing the data flow between subsystems (reader → pool → matcher → TUI) + - Adding new operating modes or modifying existing ones + - Changing the threading model or synchronization primitives + - Adding or removing public API surface (`SkimItem`, `SkimOptions`, `SkimOutput`, etc.) + - Changing the event/action system or key binding infrastructure +- Keep call-site line numbers in the cross-reference table up to date when the referenced functions move. ## Testing diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000..d480efd9 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,1197 @@ +# Skim Architecture + +## Table of Contents + +1. [High-Level Overview](#high-level-overview) +2. [Workspace & Crate Layout](#workspace--crate-layout) +3. [Entry Points](#entry-points) +4. [Core Data Flow](#core-data-flow) +5. [Operating Modes](#operating-modes) + - [Normal Interactive Mode](#normal-interactive-mode) + - [Filter Mode (`--filter`)](#filter-mode---filter) + - [Interactive / Command Mode (`--interactive`)](#interactive--command-mode---interactive) + - [Select-1 / Exit-0 / Sync Modes](#select-1--exit-0--sync-modes) + - [ANSI Mode (`--ansi`)](#ansi-mode---ansi) + - [Tmux Mode (`--tmux`)](#tmux-mode---tmux) +6. [Item Ingestion Pipeline](#item-ingestion-pipeline) +7. [The Matching Subsystem](#the-matching-subsystem) + - [Match Engine Hierarchy](#match-engine-hierarchy) + - [Fuzzy Algorithms](#fuzzy-algorithms) + - [Parallel Matching](#parallel-matching) + - [Ranking & Sorting](#ranking--sorting) +8. [TUI Subsystem](#tui-subsystem) + - [Backend & Terminal Setup](#backend--terminal-setup) + - [Event Loop](#event-loop) + - [App State](#app-state) + - [Widget System](#widget-system) + - [Layout Engine](#layout-engine) +9. [Individual Widgets](#individual-widgets) + - [Input Widget](#input-widget) + - [ItemList Widget](#itemlist-widget) + - [ItemRenderer](#itemrenderer) + - [Preview Widget](#preview-widget) + - [Header Widget](#header-widget) + - [StatusLine / Info](#statusline--info) +10. [Key Bindings & Action System](#key-bindings--action-system) +11. [Preview System](#preview-system) +12. [Output & Result Collection](#output--result-collection) +13. [IPC / Listen Socket](#ipc--listen-socket) +14. [Theming](#theming) +15. [History](#history) +16. [Pre-Selection](#pre-selection) +17. [Threading Model](#threading-model) +18. [Important Call Sites (Cross-Reference)](#important-call-sites-cross-reference) +19. [Public Library API](#public-library-api) + +--- + +## High-Level Overview + +Skim (`sk`) is a terminal fuzzy-finder written in Rust, equivalent in spirit to `fzf`. It can operate as a standalone CLI binary or as an embedded library crate. At runtime it orchestrates four concurrent activities: + +``` +stdin / command + │ + ▼ + ┌──────────┐ batched items ┌──────────────┐ + │ Reader │──────────────────▶│ ItemPool │ + └──────────┘ │ (Arc<…>) │ + └──────┬───────┘ + │ take() + ▼ + ┌──────────────┐ + │ Matcher │◀── query string + │ (parallel) │ + └──────┬───────┘ + │ ProcessedItems + ▼ + ┌──────────────────────────────────┐ + │ TUI │ + │ ┌────────┐ ┌────────┐ │ + │ │ Input │ │Preview │ │ + │ ├────────┤ ├────────┤ │ + │ │ItemList│ │ Header │ │ + │ └────────┘ └────────┘ │ + └──────────────────────────────────┘ + │ + ▼ + SkimOutput +``` + +The **Reader** pulls raw text from stdin or a shell command and converts it into `Arc` batches, depositing them into the shared `ItemPool`. +The **Matcher** picks items up from the pool, evaluates every item against the current query string using a configured engine, and writes ranked `MatchedItem` results into `ProcessedItems`. +The **TUI** renders four composable widgets (Input, ItemList, Preview, Header), drives a `crossterm`-based event loop, and converts user keystrokes into typed `Action` values that are dispatched back to the `App` state machine. + +--- + +## Workspace & Crate Layout + +``` +skim/ ← workspace root +├── src/ ← single `skim` crate (lib + binary) +│ ├── bin/ +│ │ └── main.rs ← `sk` binary entry point +│ ├── lib.rs ← library root; re-exports public types +│ ├── skim.rs ← Skim orchestrator +│ ├── options.rs ← SkimOptions (all CLI / library options) +│ ├── output.rs ← SkimOutput (returned to callers) +│ ├── reader.rs ← Reader + ReaderControl + CommandCollector trait +│ ├── matcher.rs ← Matcher + MatcherControl (parallel worker dispatcher) +│ ├── item.rs ← ItemPool, MatchedItem, Rank, RankBuilder +│ ├── skim_item.rs ← SkimItem trait +│ ├── binds.rs ← KeyMap, parse_key, parse_action_chain +│ ├── theme.rs ← ColorTheme, named palettes +│ ├── thread_pool.rs ← ThreadPool + parallel_work_queue +│ ├── field.rs ← field range parsing (--nth / --with-nth) +│ ├── spinlock.rs ← lightweight SpinLock +│ ├── util.rs ← printf helper, misc utilities +│ ├── tmux.rs ← tmux popup integration +│ ├── prelude.rs ← convenience re-exports +│ ├── manpage.rs ← man-page generation (cli feature) +│ ├── shell.rs ← shell completion generation (cli feature) +│ ├── engine/ ← match engine implementations +│ │ ├── mod.rs +│ │ ├── factory.rs ← ExactOrFuzzyEngineFactory, AndOrEngineFactory, RegexEngineFactory +│ │ ├── andor.rs ← AndEngine, OrEngine +│ │ ├── exact.rs ← ExactEngine (prefix/postfix/inverse/exact string) +│ │ ├── fuzzy.rs ← FuzzyEngine + FuzzyAlgorithm enum +│ │ ├── all.rs ← MatchAllEngine (match-all / empty query) +│ │ ├── normalized.rs ← NormalizedEngine (Unicode normalization wrapper) +│ │ ├── regexp.rs ← RegexEngine (regex-mode) +│ │ ├── split.rs ← SplitMatchEngine (--split-match) +│ │ └── util.rs ← normalization helpers +│ ├── fuzzy_matcher/ ← raw fuzzy scoring algorithms +│ │ ├── mod.rs ← FuzzyMatcher trait, MatchIndices type alias +│ │ ├── skim.rs ← SkimMatcherV2 +│ │ ├── clangd.rs ← ClangdMatcher +│ │ ├── fzy.rs ← FzyMatcher +│ │ ├── frizbee.rs ← FrizbeeMatcher (typo-resistant) +│ │ └── arinae/ ← ArinaeMatcher (default; Smith-Waterman based) +│ ├── helper/ ← higher-level item helpers +│ │ ├── mod.rs +│ │ ├── item.rs ← DefaultSkimItem (ANSI parsing, field transforms) +│ │ ├── item_reader.rs ← SkimItemReader + SkimItemReaderOption (stdin/cmd → items) +│ │ ├── selector.rs ← DefaultSkimSelector (pre-selection) +│ │ └── macros.rs ← helper macros +│ └── tui/ ← terminal UI +│ ├── mod.rs ← Size, Direction, BorderType, re-exports +│ ├── app.rs ← App struct + render + event dispatch (central state machine) +│ ├── backend.rs ← Tui (ratatui terminal wrapper + crossterm event pump) +│ ├── event.rs ← Event enum, Action enum, ActionCallback, parse_action +│ ├── widget.rs ← SkimWidget trait + SkimRender result type +│ ├── input.rs ← Input widget (query box + cursor + status info) +│ ├── item_list.rs ← ItemList widget (scrollable match result list) +│ ├── item_renderer.rs ← ItemRenderer (per-item ANSI/highlight rendering) +│ ├── preview.rs ← Preview widget (PTY or plain text preview pane) +│ ├── header.rs ← Header widget (--header / --header-lines) +│ ├── statusline.rs ← InfoDisplay enum (status bar mode) +│ ├── layout.rs ← LayoutTemplate + AppLayout (pre-computed areas) +│ ├── options.rs ← TuiLayout enum, PreviewLayout struct +│ └── util.rs ← cursor helpers, style merging +├── tests/ ← integration & snapshot tests +│ ├── common/ +│ │ └── insta.rs ← snap! / insta_test! macros for TUI snapshot testing +│ ├── snapshots/ ← committed .snap files +│ ├── ansi.rs ← ANSI rendering tests +│ ├── options.rs ← option coverage tests +│ ├── preview.rs ← preview pane tests +│ └── … +├── benches/ ← criterion benchmarks +└── Cargo.toml +``` + +The single crate exports: +- A **library** (`lib`): all types under `skim::*`, suitable for embedding. +- A **binary** (`sk`, requires feature `cli`): the `clap`-based CLI. + +The `cli` feature gates `clap`, `clap_complete`, `shlex`, `env_logger`, and `clap_mangen`. + +--- + +## Entry Points + +### Binary (`src/bin/main.rs`) + +``` +main() + │ + ├─ SkimOptions::from_env() ← parses argv via clap (feature=cli) + ├─ opts.build() ← applies defaults, loads history files + │ + ├─ if opts.shell → generate_completions() ← early exit + ├─ if opts.man → manpage::generate() ← early exit + ├─ if opts.remote → IPC relay mode ← early exit + │ + ├─ sk_main(opts) + │ ├─ SkimItemReader::new(reader_opts) ← configure stdin reader + │ ├─ opts.cmd_collector = cmd_collector + │ │ + │ ├─ if --tmux && TMUX is set → tmux::run_with(&opts) + │ │ + │ └─ else: + │ ├─ if stdin not a TTY (piped) → cmd_collector.of_bufread(stdin) + │ └─ Skim::run_with(opts, rx_item?) + │ + └─ print output / write history / exit +``` + +### Library (`src/lib.rs` + `src/skim.rs`) + +Two public entry points exist on `Skim`: + +| Method | Use case | +|---|---| +| `Skim::run_with(options, source)` | Takes a `SkimItemReceiver` channel (or `None` to use the configured command collector). The canonical entry point. | +| `Skim::run_items(options, items)` | Convenience wrapper: accepts any `IntoIterator`, batches them through a bounded channel, and calls `run_with`. | + +Both return `Result`. + +--- + +## Core Data Flow + +### Initialisation sequence + +``` +Skim::run_with(options, source) + │ + ├─ Skim::init(options, source) + │ ├─ parse height (Size enum) + │ ├─ ColorTheme::init_from_options(&options) + │ ├─ Reader::from_options(&options).source(source) + │ ├─ resolve cmd / expand initial_cmd (interactive mode) + │ └─ App::from_options(options, theme, cmd) + │ ├─ Input::from_options(…) + │ ├─ Preview::from_options(…) + │ ├─ Header::from_options(…) + │ ├─ ItemList::from_options(…) + │ ├─ ItemPool::from_options(…) + │ ├─ Matcher::from_options(…) + │ └─ LayoutTemplate::from_options(…) + │ + ├─ Skim::start() + │ ├─ reader.collect(item_pool, initial_cmd) ← spawns reader thread(s) + │ └─ app.restart_matcher(force=true) ← kicks off first match pass + │ + ├─ Skim::should_enter() → decides whether to open TUI + │ (handles filter / select-1 / exit-0 / sync blocking) + │ + ├─ if should_enter: + │ ├─ Skim::init_tui() → Tui::new_with_height(height) + │ ├─ Skim::enter() → tui.enter() [raw mode + mouse + event task] + │ └─ Skim::run() → async event loop (tick()) + │ + └─ Skim::output() ← collect results + kill reader +``` + +### Steady-state loop (`Skim::tick()`) + +Each call to `tick()` runs a `tokio::select!` on four concurrent futures: + +| Branch | Source | Action | +|---|---|---| +| `tui.next()` | crossterm keyboard/mouse/resize/paste events | Dispatch to `app.handle_event()` | +| `matcher_interval.tick()` | 10 ms periodic timer | `app.restart_matcher(false)` | +| `items_available.notified()` | `Notify` set by `ItemPool::append` | `app.restart_matcher(false)` | +| `listener.accept()` | IPC socket (when `--listen`) | Parse RON-encoded `Action`, push to event queue | + +--- + +## Operating Modes + +### Normal Interactive Mode + +The default mode. The TUI is shown in full. Items arrive from stdin or a command, are matched against the live query, and displayed in the list. The user navigates with keyboard/mouse and presses Enter to accept. + +**Key files:** `src/skim.rs`, `src/tui/app.rs`, `src/tui/backend.rs` + +### Filter Mode (`--filter`) + +When `--filter ` is set, skim never opens the TUI. + +`Skim::should_enter()` enters a busy-wait loop: +``` +loop { + if matcher.stopped() && reader.is_done() && pool.num_not_taken() == 0 { + break; + } + sleep(1ms); + app.restart_matcher(false); +} +``` +Then `app.item_list.items` is populated from `processed_items` and `output()` is called immediately. The matched items are printed to stdout by the binary, one per line (or null-delimited with `--print0`). + +In filter mode the `FuzzyEngine` is built with `filter_mode = true`, which uses `fuzzy_match_range` instead of `fuzzy_indices` to skip the per-character index computation and run faster. + +**Key files:** `src/skim.rs` (`should_enter()`), `src/engine/fuzzy.rs` (`filter_mode` fast path), `src/bin/main.rs` (output loop) + +### Interactive / Command Mode (`--interactive`) + +When `--interactive` is set together with `--cmd