Benchmarks & Performance

Performance metrics, latency, and context compression benchmarks for ContextOS.

Implementation Sourcescripts/run-benchmarks.ts

Compression Benchmarks

ContextOS solves this by aggressively stripping out noise: omitting unchanged imports, stripping documentation strings, and extracting only the exact code blocks necessary to satisfy the query.

The figures below are from a recorded Redis 7.x checkout run. They are corpus-, query-, and configuration-specific measurements rather than release guarantees.

Standard 100-Query Benchmark

Average Tokens / Query
589

Average token usage per query across the 100-query benchmark.

Retrieval Latency
< 50ms

P99 Latency for retrieving from local SQLite database (including BM25 and Vector matching).

Exact Function Accuracy
98%

Successfully pinpointing the exact AST node (function/class) required to answer the query.

Conceptual Accuracy
96%

Successfully mapping the semantic intent of the query to correct codebase domains.

Reproduce the recorded harness

Build the CLI, provide a Redis checkout, and keep embedding downloads out of the fixture comparison when you want deterministic keyword-only results:

npm run build
CONTEXTOS_EMBEDDINGS=0 CONTEXTOS_REDIS_REPO=/path/to/redis \
  node scripts/bench/run-redis-bench.mjs --output-dir "$TMPDIR/contextos-redis-results"

The generated redis-results.json reports aggregateexpectedFileRecallPercent, anyHitRatePercent, and targeted/generic token averages. The checked-in fixture runner is separate: npm run bench -- --json copies theretrieval-examples/ fixtures into temporary projects and reports passRatePercent, averageRecallPercent,expectedFileRecallPercent, and anyHitRatePercent.

Throughput & Scalability

The backend Daemon handles indexing asynchronously. Rather than blocking your LLM queries, indexing happens passively in the background using SQLite WAL mode.

  • 1,000,000 files
    Hard limit on total files parsed per workspace to prevent Out-Of-Memory (OOM) errors and sandbox escapes.
  • ~2ms / file
    Average parsing time using the Tree-sitter AST parser, enabling rapid initial indexing.
  • pLimit(5)
    Concurrency cap on file watchers, preventing CPU starvation during massive branch swaps (e.g., git checkout).