--- title: "RTK Compression" version: 3.8.2 lastUpdated: 2026-05-13 --- # RTK Compression RTK compression is OmniRoute's command-aware compression engine for terminal and tool output. It is designed for coding-agent sessions where most context growth comes from test logs, build output, package manager noise, shell transcripts, Docker output, git output, and stack traces. RTK can run directly with `defaultMode: "rtk"` or as the first step in a stacked pipeline, usually: ```txt rtk -> caveman ``` That order compresses noisy machine output first, then lets Caveman condense remaining prose. Upstream RTK reports `60-90%` command-output savings. Its README sample session goes from `~118,000` standard tokens to `~23,900` RTK tokens, which is `79.7%` saved (`~80%`). OmniRoute uses that upstream average for the stacked savings calculation with Caveman input compression: ```txt RTK average: 80% saved Caveman input: 46% saved Stacked: 1 - (1 - 0.80) * (1 - 0.46) = 89.2% saved Range: 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6% ``` ## What It Compresses The built-in catalog currently ships 49 filters across these categories: | Category | Examples | | --------- | ------------------------------------------------------------- | | `git` | `git status`, `git branch`, `git diff`, `git log` | | `test` | Vitest, Jest, Pytest, Playwright, Go tests, Cargo tests | | `build` | TypeScript, ESLint, Biome, Prettier, Vite, Webpack, Turbo, Nx | | `package` | `npm install`, `npm audit`, `pip`, `uv sync`, Poetry, Bundler | | `shell` | `ls`, `find`, `grep`, generic shell logs | | `docker` | `docker ps`, Docker logs | | `infra` | Terraform, OpenTofu, `systemctl status` | | `generic` | JSON output, stack traces, generic output fallback | The detector in `open-sse/services/compression/engines/rtk/commandDetector.ts` classifies output before filter selection. Filters can also match by command pattern or output regex when a command class is not enough. ## Filter Resolution RTK loads filters in this order: 1. Project filters from `.rtk/filters.json`, only when trusted. 2. Global filters from `DATA_DIR/rtk/filters.json`. 3. Built-in filters from `open-sse/services/compression/engines/rtk/filters/`. Project filters are intentionally trust-gated because regex filters can change how tool output is shown to agents. A project filter file is accepted when one of these is true: - `rtkConfig.trustProjectFilters` is `true`. - `OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1` is set. - `.rtk/trust.json` contains the SHA-256 hash of `.rtk/filters.json`. Trust file example: ```json { "filtersSha256": "0123456789abcdef..." } ``` Custom filters can be one filter object or an array of filter objects. Invalid custom filters are skipped and reported by `/api/context/rtk/filters` diagnostics. Invalid built-in filters fail fast. ## Filter DSL Filters use the JSON schema described in [Compression Rules Format](./COMPRESSION_RULES_FORMAT.md). The runtime applies these stages in order: ```txt stripAnsi -> filterStderr -> replace -> matchOutput -> drop/include lines -> truncateLineAt -> head/tail/maxLines -> onEmpty ``` Important fields: | Field | Purpose | | ---------------------------- | -------------------------------------------------------------- | | `rules.stripAnsi` | Remove terminal color/control sequences before matching | | `rules.filterStderr` | Normalize common stderr prefixes before matching/filtering | | `rules.replace` | Apply ordered regex replacements | | `rules.matchOutput` | Return a compact summary when output matches a known condition | | `rules.matchOutput[].unless` | Skip the shortcut when an error/failure pattern is present | | `rules.dropPatterns` | Remove noisy lines | | `rules.includePatterns` | Prefer actionable lines | | `rules.collapsePatterns` | Collapse repeated matching lines | | `rules.truncateLineAt` | Unicode-safe per-line truncation | | `rules.onEmpty` | Fallback message if all lines are filtered out | | `tests[]` | Inline samples used by the verify gate | Built-in filters are expected to include inline `tests[]` samples. Custom filters should include them too, especially when they are shared across projects. ## Configuration Global settings are available through `/api/settings/compression`. RTK-specific settings are also available through `/api/context/rtk/config`. ```json { "defaultMode": "stacked", "autoTriggerMode": "stacked", "autoTriggerTokens": 32000, "stackedPipeline": [ { "engine": "rtk", "intensity": "standard" }, { "engine": "caveman", "intensity": "full" } ], "rtkConfig": { "enabled": true, "intensity": "standard", "applyToToolResults": true, "applyToCodeBlocks": false, "applyToAssistantMessages": false, "enabledFilters": [], "disabledFilters": [], "maxLinesPerResult": 120, "maxCharsPerResult": 12000, "deduplicateThreshold": 3, "customFiltersEnabled": true, "trustProjectFilters": false, "rawOutputRetention": "never", "rawOutputMaxBytes": 1048576 } } ``` `enabledFilters` and `disabledFilters` use filter ids, for example `test-vitest` or `git-diff`. ## API | Route | Method | Purpose | | ---------------------------------- | ------ | -------------------------------------------- | | `/api/context/rtk/config` | GET | Read RTK config | | `/api/context/rtk/config` | PUT | Update RTK config | | `/api/context/rtk/filters` | GET | List filter catalog and load diagnostics | | `/api/context/rtk/test` | POST | Preview RTK compression for one text payload | | `/api/context/rtk/raw-output/[id]` | GET | Read retained redacted raw output | | `/api/compression/preview` | POST | Preview any compression mode | RTK test payload: ```json { "command": "npm test", "text": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed", "config": { "intensity": "standard" } } ``` Compression preview payload: ```json { "mode": "stacked", "messages": [ { "role": "tool", "content": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed" } ], "config": { "rtkConfig": { "rawOutputRetention": "failures" } } } ``` Management routes require dashboard management auth or the matching API-key policy. ## Raw Output Recovery RTK normally returns only compressed text. For debugging, `rawOutputRetention` can retain redacted raw output: | Value | Behavior | | ---------- | ------------------------------------------------------- | | `never` | Do not retain raw output | | `failures` | Retain only likely failure output | | `always` | Retain every compressed RTK raw output, after redaction | Retained files are written under: ```txt DATA_DIR/rtk/raw-output/ ``` Secrets are redacted before persistence, including common bearer tokens, API keys, Slack tokens, AWS access keys, and assignment-style `token=...`, `secret=...`, `password=...` values. Analytics stores only the pointer id, size, and hash metadata. ## Verify Gate The focused verify gate runs built-in inline filter tests without shelling out to external commands: ```bash node --import tsx/esm --test tests/unit/compression/rtk-verify.test.ts ``` The broader RTK gate is: ```bash node --import tsx/esm --test \ tests/unit/compression/rtk-*.test.ts \ tests/unit/compression/pipeline-integration.test.ts \ tests/unit/compression/context-compression-api.test.ts ``` Run the broad compression gate before release: ```bash node --import tsx/esm --test \ tests/unit/compression/*.test.ts \ tests/golden-set/*.test.ts \ tests/integration/compression-pipeline.test.ts \ tests/unit/api/compression/compression-api.test.ts ``` ## Extending RTK 1. Add or update a filter JSON file. 2. Include at least one `tests[]` sample that proves the important behavior. 3. Add a fixture under `tests/unit/compression/fixtures/rtk/` for new command families. 4. Add command detection coverage when introducing a new output class. 5. Run the verify and broad RTK gates. 6. If the filter is project-local, commit `.rtk/filters.json` and refresh `.rtk/trust.json` only after review.