--- title: "Compression Engines" version: 3.8.2 lastUpdated: 2026-05-13 --- # Compression Engines OmniRoute compression is built around engine contracts. A mode can run one engine directly (`caveman` or `rtk`) or a deterministic stacked pipeline that executes multiple engines in order. ## Modes | Mode | Engine path | Intended input | | ------------ | ---------------------------------- | -------------------------------------------- | | `off` | none | Exact prompt preservation | | `lite` | Caveman lite helpers | Low-risk always-on cleanup | | `standard` | Caveman | Natural-language prompt condensation | | `aggressive` | Caveman + history/tool summarizers | Long chat sessions | | `ultra` | Caveman + pruning helpers | Context-limit recovery | | `rtk` | RTK | Terminal, shell, build, test, and git output | | `stacked` | Pipeline, default `rtk -> caveman` | Mixed tool logs and prose, max savings | ## Engine Registry The registry lives in `open-sse/services/compression/engines/registry.ts`. Engines expose a shared contract: - `id`: stable engine id such as `caveman` or `rtk` - `apply(text, config)`: legacy execution path used by stacked pipelines - `compress(input, config)`: primary execution path returning text + stats - `getConfigSchema()`: returns the JSON-Schema-like shape of valid config - `validateConfig(config)`: returns `{ valid, errors[] }` Registration uses `registerCompressionEngine(engine)` (or `registerEngine` for advanced cases), which calls `assertValidEngine()` and `validateConfig(defaultConfig)` before accepting. Use `unregisterCompressionEngine(id)` to remove an engine at runtime. `strategySelector.ts` registers the built-in engines before compression runs. This lets preview, runtime compression, stacked mode, tests, and future engines use the same execution path. ### MCP description compression (related) A separate registry compresses MCP tool description metadata at registry-level — see `open-sse/mcp-server/descriptionCompressor.ts` and [MCP-SERVER.md](../frameworks/MCP-SERVER.md). It reuses Caveman rules but operates on tool metadata, not request payloads. ## Caveman Caveman mode focuses on semantic condensation of normal prose: - preserves code blocks, URLs, JSON, paths, and structured data - removes filler, hedging, repeated context, and verbose connective phrasing - supports language-aware file rule packs in `open-sse/services/compression/rules/` - remains available through the legacy `standard`, `aggressive`, and `ultra` modes The dashboard surface is `Dashboard -> Context & Cache -> Caveman`. Caveman upstream reports `~75%` fewer output tokens, `65%` average output savings in benchmarks with a `22-87%` range, and a `~46%` input-compression tool. OmniRoute uses the Caveman input-side number when documenting stacked prompt/context savings; Caveman output mode remains a separate response-behavior feature. ## RTK RTK mode focuses on command and tool output: - detects output classes such as `git status`, `git branch`, `git diff`, Vitest/Jest/Pytest, Cargo/Go tests, TypeScript/Vite/Webpack builds, ESLint, npm audit/installs, Docker logs, shell `find`/`grep`, stack traces, and generic logs - applies 49 JSON filters from `open-sse/services/compression/engines/rtk/filters/` - supports the RTK-style declarative pipeline: ANSI stripping, replace, match-output short-circuit, strip/keep lines, per-line truncation, head/tail/max-line truncation, and on-empty fallback - supports trust-gated project filters in `.rtk/filters.json` and global filters in `DATA_DIR/rtk/filters.json` - strips ANSI sequences, progress noise, repeated lines, and unhelpful boilerplate - preserves actionable failures, warnings, summaries, changed files, and tail context - can optionally retain redacted raw output for recovery/debugging through authenticated management routes The dashboard surface is `Dashboard -> Context & Cache -> RTK`. Operational details for custom filters, trust, verify, and raw-output recovery live in [`RTK_COMPRESSION.md`](./RTK_COMPRESSION.md). RTK upstream reports `60-90%` savings for command-output compression. Its README example shows a 30-minute Claude Code session going from `~118,000` tokens to `~23,900`, or `79.7%` saved. ## Stacked Pipelines Stacked mode runs pipeline steps in order. The default is: ```txt rtk -> caveman ``` Use this for coding-agent sessions where a prompt combines command output with human or assistant prose. RTK reduces noisy tool logs first, then Caveman compresses remaining natural language. Pipeline steps are configured with `stackedPipeline` in compression settings or through compression combos. When both engines reduce the same eligible payload, savings compound: ```txt combined = 1 - (1 - RTK savings) * (1 - Caveman input savings) average = 1 - (1 - 0.80) * (1 - 0.46) = 89.2% range = 1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6% ``` ## MCP Accessibility Tree Filter The MCP accessibility-tree smart filter is a post-execution compression layer that runs on MCP **tool results**, not on prompts or context. It targets the verbose accessibility-tree and browser snapshot payloads returned by tools like Playwright, computer-use, and browser-automation MCP servers. ### What it does 1. **Noise stripping** — removes empty generic/text entries (`- generic:`, `- text: ""`) 2. **Sibling collapse** — when ≥ `collapseThreshold` (default 30) consecutive lines are structural repeats, collapses them into the first `collapseKeepHead` (default 10) lines + a count summary + the last `collapseKeepTail` (default 5) lines 3. **Ref preservation** — `[ref=eXX]` anchors required by Playwright/computer-use are never touched 4. **Hard truncation** — if the text after collapse still exceeds `maxTextChars` (default 50,000), truncates with a navigation hint so the agent can continue working ### Engine location ```txt open-sse/services/compression/engines/mcpAccessibility/ index.ts ← smartFilterText() entry point collapseRepeated.ts ← sibling-collapse algorithm constants.ts ← DEFAULT_MCP_ACCESSIBILITY_CONFIG ``` ### Configuration Controlled by `compression.mcpAccessibility` in global settings (migration 056). Default config: ```json { "enabled": true, "maxTextChars": 50000, "collapseThreshold": 30, "collapseKeepHead": 10, "collapseKeepTail": 5, "minLengthToProcess": 2000 } ``` The filter is only applied to tool-result payloads whose `type` is `"text"` and whose length exceeds `minLengthToProcess`. It does not affect prompt compression or request payloads. ### Expected savings 60–80% on browser snapshot tool results, depending on page complexity. The collapse algorithm is O(n) in line count and adds negligible latency. ### This filter vs the compression engines above | Aspect | Caveman / RTK / Stacked | MCP accessibility filter | | ----------- | ------------------------- | -------------------------------------- | | Target | Request prompts / context | MCP tool results | | Trigger | Compression mode setting | `compression.mcpAccessibility.enabled` | | Scope | All SSE messages | Tool results only | | Ref anchors | N/A | Preserved unconditionally | --- ## Compression Combos Compression combos are named compression profiles that can be assigned to routing combos: - `compression_combos`: stores mode, pipeline, RTK config, language config, and default marker - `compression_combo_assignments`: maps a compression combo to a routing combo - runtime integration resolves an assigned compression combo before generic combo overrides - analytics include `compression_combo_id` and `engine` Dashboard surface: `Dashboard -> Context & Cache -> Compression Combos`. ## API Surface | Route | Purpose | | -------------------------------------- | ---------------------------------------------------------------- | | `/api/settings/compression` | Global compression settings (includes `mcpAccessibility` config) | | `/api/compression/preview` | Preview any compression mode | | `/api/compression/language-packs` | List available Caveman language packs | | `/api/context/caveman/config` | Caveman settings alias | | `/api/context/rtk/config` | RTK defaults and settings | | `/api/context/rtk/filters` | RTK filter catalog | | `/api/context/rtk/test` | RTK preview/test endpoint | | `/api/context/rtk/raw-output/[id]` | Authenticated redacted raw-output recovery | | `/api/context/combos` | Compression combo CRUD | | `/api/context/combos/[id]/assignments` | Routing-combo assignment CRUD | | `/api/context/analytics` | Compression analytics alias | Management routes require management authentication or API-key policy checks. ## MCP Tools Compression exposes five MCP tools: | Tool | Scope | Purpose | | ----------------------------------- | ------------------- | -------------------------------- | | `omniroute_compression_status` | `read:compression` | Settings, analytics, cache stats | | `omniroute_compression_configure` | `write:compression` | Update global settings | | `omniroute_set_compression_engine` | `write:compression` | Set mode and optional pipeline | | `omniroute_list_compression_combos` | `read:compression` | List compression combos | | `omniroute_compression_combo_stats` | `read:compression` | Read combo/engine analytics | ## Validation The focused gates for this area are: ```bash node --import tsx/esm --test tests/unit/compression/rtk-*.test.ts tests/unit/compression/pipeline-integration.test.ts tests/unit/compression/context-compression-api.test.ts node --import tsx/esm --test tests/unit/compression/*.test.ts tests/golden-set/*.test.ts tests/integration/compression-pipeline.test.ts tests/unit/api/compression/compression-api.test.ts node --import tsx/esm --test tests/unit/compression/mcpAccessibility*.test.ts npm run typecheck:core ```