Implementation Sourcesrc/core/indexer/index.ts

The Pipeline

ContextOS operates in two distinct phases: Compile-Time (indexing) and Runtime (retrieval). The architecture ensures that all heavy computation—parsing, graph building, and embedding generation—happens asynchronously during compilation, keeping runtime latencies strictly under 50ms.

Compile Phase

Repository Scanner // Traverses workspace, dirty-checks hash

Tree-sitter Parser // Generates AST, extracts symbols & chunks

Graph Builder // Extracts internal symbol relations and file imports

Background Embedding // Non-blocking generation of dense vectors

SQLite Storage // Atomic commit to local .contextos DB

Runtime Phase

RRF Hybrid Retriever // BM25 + Vector Search + Intent Multipliers

Graph Expansion // BFS traversal through dependency edges

Compression Engine // Aggressive token reduction (stripping docs/imports)

Compile-Time Subsystems

Parser (Tree-sitter) & Chunking

The parser converts raw string buffers into a structured AST. We use tree-sitter because it guarantees incremental parsing and fault tolerance. During extraction, the AST is shredded into independent Chunk objects (representing functions, interfaces, or classes), retaining exact byte offsets and parent symbol mappings.

Graph Builder

Once chunks are generated, the system maps internal and external dependencies. extractImportRelationships walks the import statements, mapping logical connections between file stems. These generate edges with specific traversal weights.

SQLite Storage & Foreign Keys

ContextOS guarantees transactional integrity through SQLite's WAL mode and cascading foreign keys. If a file is deleted, this.filesRepo.deleteByPath(filePath) instantly purges all associated chunks, relationships, and embeddings via ON DELETE CASCADE.

Runtime Subsystems

RRF Hybrid Retrieval

During a query, ContextOS does not rely solely on dense vectors. It runs a Reciprocal Rank Fusion (RRF) algorithm combining exact keyword matches from FTS5, semantic matches from the embedding store, and feedback_signals multipliers.

Compression Engine

After the graph expander collects the relevant nodes, the Compression Engine ensures the payload fits within the LLM's token budget. It utilizes hierarchical dropping—purging imports, docstrings, and low-priority chunks before it resorts to truncating business logic.