Retrieval Pipeline

Pipeline Overview

The ContextOS retrieval pipeline consists of four distinct stages: Keyword Matching, Graph Expansion, Intent Scoring (Ranking), and Context Compression.

1. Query Execution: A natural language query or specific symbol search is initiated.
2. FTS5 BM25 Lookup: The local SQLite index performs a full-text search across AST chunk bodies, file paths, and symbol signatures.
3. Graph Expansion: ContextOS uses the top hits as seeds for a BFS traversal over the dependency graph.
4. Scoring & Fusion: All retrieved chunks are merged and scored via Reciprocal Rank Fusion (RRF), boosted by Layer logic (Workspace > Global).
5. Compression: The final list of chunks is packed sequentially into the requested token budget.

1. Keyword Matching (BM25 Proxy)

Instead of loading a heavy embedding model (which is slow locally and prone to semantic contamination), ContextOS relies on SQLite's highly optimized FTS5 virtual tables using the porter stemmer.

We sanitize incoming queries to strip punctuation and build a valid FTS match expression. This guarantees exact matches for technical symbols (e.g. AuthenticationService) while falling back to stemmed matches for natural language (e.g. "authentication").

2. Graph Expansion

Once the initial text search returns a list of candidate files, we execute Graph Expansion.

If the LLM is asking about a function that relies on an interface from another file, FTS alone won't find it. ContextOS traverses the relationships table (derived from AST imports) to pull in those critical peripheral definitions.

3. Scoring (RRF)

ContextOS ranks results by merging three separate signals:

• Text Relevance: The FTS5 bm25() score.
• Graph Relevance: The depth-decayed score from graph expansion.
• Layer Boosts: Entities found in the current workspace get a 10x score multiplier compared to global fallbacks.

These signals are combined using a deterministic Reciprocal Rank Fusion (RRF) algorithm to ensure stable, predictable sorting across identical queries.

4. Compression

After sorting, ContextOS must fit the results into a strict token budget. It uses the gpt-tokenizer to measure exact token boundaries. We compress results at the AST-chunk level, aggressively stripping out irrelevant file sections.

If a file exceeds the budget, only the matched functions/classes are included, while the rest of the file is dropped, protecting the LLM's context window.