Initialization Sequence

The end-to-end process executed by the 'contextos init' command, powered by the core Indexer.

Implementation Sourcesrc/cli/commands/init.ts

Purpose

The contextos init command prepares the repository and schedules the Indexer in the background. The indexer scans the repository, parses source code into an AST, chunks it, builds dependency edges, generates embeddings when available, and writes the local SQLite database. Initialization can return while a large repository is still being indexed; use contextos status to observe fullIndexCompleted.

The Indexer Pipeline

The core logic lives inside src/core/indexer/index.ts. For every file discovered in the workspace, the indexFile method is invoked.

1. Validation & Safety Caps

Before any file is read into memory, we execute safety checks. We immediately skip files larger than 2MB (MAX_FILE_BYTES) or binary files (detected via null bytes) to prevent catastrophic OOM crashes during AST parsing.

const MAX_FILE_BYTES = 2 * 1024 * 1024; // 2MB

// Cap file size before read
if (stat.size > MAX_FILE_BYTES) return emptyStats;

// Check for binary
if (content.indexOf('\0') !== -1) return emptyStats;

2. Dirty Checking

ContextOS only compiles what has changed. We hash the file buffer and check it against the filesRepo.

const hash = hashContent(content);
if (!this.filesRepo.isChanged(filePath, hash)) {
  return emptyStats;
}

3. AST Generation & Chunking

The file extension routes the buffer to the appropriate tree-sitter parser (Code, Config, or Markdown). The AST is then walked to extract independent chunks and imports.

if (isCode) {
  const parsed = await parseCode(filePath, content);
  imports = parsed.imports || [];
  chunks = chunkCode(parsed, { layer, workspaceName });
}

4. Database Transaction

State updates run in SQLite transactions. If a file changed, old chunks and relationships are removed before the new chunks are inserted. The vector table is maintained separately because sqlite-vec does not provide foreign-key cascades, so stale vectors are explicitly deleted during replacement and cleanup.

// Update file record first for FK constraints
this.filesRepo.upsert({ path: filePath, hash, ... });

// Cleanup old chunks and their relationships
this.chunksRepo.deleteBySource(filePath);

// Remove vectors explicitly (vec0 has no FK cascade)
embeddingsStore.deleteByChunkIds(oldChunkIds);

// Bulk insert new chunks
this.chunksRepo.bulkUpsert(chunks);

5. Graph Edges & Non-Blocking Embeddings

Embeddings generation is the slowest part of the pipeline. To ensure init remains fast, embedding generation is strictly non-blocking. If the model fails or times out, the pipeline continues, as ContextOS falls back seamlessly to BM25 FTS5.

try {
  await indexChunkEmbeddings(this.chunksRepo.getDatabase(), chunks);
} catch {
  // continue without embeddings
}

// Extract and bulk insert graph edges
const allRels = chunks.flatMap(c => extractRelationships(c));
this.relsRepo.bulkUpsert(allRels);
Complexity Analysis
Time Complexity
O(F * S)
Space Complexity
O(S)
Average Case
~4 seconds for 2k files
Worst Case
~45 seconds for 20k files

Where F = number of files, and S = average size of files in bytes.

Failure Modes

  • OOM (Out of Memory): Highly unlikely due to the hard MAX_FILE_BYTES cutoff, but an extremely complex 1.9MB TypeScript file could theoretically cause the tree-sitter WASM heap to panic.
  • Corrupted SQLite state: Impossible. If the process is hard-killed (SIGKILL) mid-transaction, SQLite's WAL handles recovery on the next boot.