Benchmarks & Performance
Performance metrics, latency, and context compression benchmarks for ContextOS.
Compression Benchmarks
ContextOS solves this by aggressively stripping out noise: omitting unchanged imports, stripping documentation strings, and extracting only the exact code blocks necessary to satisfy the query.
The figures below are from a recorded Redis 7.x checkout run. They are corpus-, query-, and configuration-specific measurements rather than release guarantees.
Standard 100-Query Benchmark
Average token usage per query across the 100-query benchmark.
P99 Latency for retrieving from local SQLite database (including BM25 and Vector matching).
Successfully pinpointing the exact AST node (function/class) required to answer the query.
Successfully mapping the semantic intent of the query to correct codebase domains.
Reproduce the recorded harness
Build the CLI, provide a Redis checkout, and keep embedding downloads out of the fixture comparison when you want deterministic keyword-only results:
npm run build
CONTEXTOS_EMBEDDINGS=0 CONTEXTOS_REDIS_REPO=/path/to/redis \
node scripts/bench/run-redis-bench.mjs --output-dir "$TMPDIR/contextos-redis-results"The generated redis-results.json reports aggregateexpectedFileRecallPercent, anyHitRatePercent, and targeted/generic token averages. The checked-in fixture runner is separate: npm run bench -- --json copies theretrieval-examples/ fixtures into temporary projects and reports passRatePercent, averageRecallPercent,expectedFileRecallPercent, and anyHitRatePercent.
Throughput & Scalability
The backend Daemon handles indexing asynchronously. Rather than blocking your LLM queries, indexing happens passively in the background using SQLite WAL mode.
- 1,000,000 filesHard limit on total files parsed per workspace to prevent Out-Of-Memory (OOM) errors and sandbox escapes.
- ~2ms / fileAverage parsing time using the Tree-sitter AST parser, enabling rapid initial indexing.
- pLimit(5)Concurrency cap on file watchers, preventing CPU starvation during massive branch swaps (e.g.,
git checkout).