Initialization Sequence
The end-to-end process executed by the 'contextos init' command, powered by the core Indexer.
Purpose
The contextos init command prepares the repository and schedules the Indexer in the background. The indexer scans the repository, parses source code into an AST, chunks it, builds dependency edges, generates embeddings when available, and writes the local SQLite database. Initialization can return while a large repository is still being indexed; use contextos status to observe fullIndexCompleted.
The Indexer Pipeline
The core logic lives inside src/core/indexer/index.ts. For every file discovered in the workspace, the indexFile method is invoked.
1. Validation & Safety Caps
Before any file is read into memory, we execute safety checks. We immediately skip files larger than 2MB (MAX_FILE_BYTES) or binary files (detected via null bytes) to prevent catastrophic OOM crashes during AST parsing.
const MAX_FILE_BYTES = 2 * 1024 * 1024; // 2MB
// Cap file size before read
if (stat.size > MAX_FILE_BYTES) return emptyStats;
// Check for binary
if (content.indexOf('\0') !== -1) return emptyStats;2. Dirty Checking
ContextOS only compiles what has changed. We hash the file buffer and check it against the filesRepo.
const hash = hashContent(content);
if (!this.filesRepo.isChanged(filePath, hash)) {
return emptyStats;
}3. AST Generation & Chunking
The file extension routes the buffer to the appropriate tree-sitter parser (Code, Config, or Markdown). The AST is then walked to extract independent chunks and imports.
if (isCode) {
const parsed = await parseCode(filePath, content);
imports = parsed.imports || [];
chunks = chunkCode(parsed, { layer, workspaceName });
}4. Database Transaction
State updates run in SQLite transactions. If a file changed, old chunks and relationships are removed before the new chunks are inserted. The vector table is maintained separately because sqlite-vec does not provide foreign-key cascades, so stale vectors are explicitly deleted during replacement and cleanup.
// Update file record first for FK constraints
this.filesRepo.upsert({ path: filePath, hash, ... });
// Cleanup old chunks and their relationships
this.chunksRepo.deleteBySource(filePath);
// Remove vectors explicitly (vec0 has no FK cascade)
embeddingsStore.deleteByChunkIds(oldChunkIds);
// Bulk insert new chunks
this.chunksRepo.bulkUpsert(chunks);5. Graph Edges & Non-Blocking Embeddings
Embeddings generation is the slowest part of the pipeline. To ensure init remains fast, embedding generation is strictly non-blocking. If the model fails or times out, the pipeline continues, as ContextOS falls back seamlessly to BM25 FTS5.
try {
await indexChunkEmbeddings(this.chunksRepo.getDatabase(), chunks);
} catch {
// continue without embeddings
}
// Extract and bulk insert graph edges
const allRels = chunks.flatMap(c => extractRelationships(c));
this.relsRepo.bulkUpsert(allRels);Where F = number of files, and S = average size of files in bytes.
Failure Modes
- OOM (Out of Memory): Highly unlikely due to the hard
MAX_FILE_BYTEScutoff, but an extremely complex 1.9MB TypeScript file could theoretically cause the tree-sitter WASM heap to panic. - Corrupted SQLite state: Impossible. If the process is hard-killed (SIGKILL) mid-transaction, SQLite's WAL handles recovery on the next boot.