Initialization Sequence

Purpose

The contextos init command is the entrypoint to the system. It delegates immediately to the Indexer class, which is responsible for scanning the repository, parsing all source code into an AST, chunking it, building the dependency graph, generating embeddings, and seeding the local SQLite database. It effectively acts as the "compiler" phase of ContextOS.

The Indexer Pipeline

The core logic lives inside src/core/indexer/index.ts. For every file discovered in the workspace, the indexFile method is invoked.

1. Validation & Safety Caps

Before any file is read into memory, we execute safety checks. We immediately skip files larger than 2MB (MAX_FILE_BYTES) or binary files (detected via null bytes) to prevent catastrophic OOM crashes during AST parsing.

const MAX_FILE_BYTES = 2 * 1024 * 1024; // 2MB

// Cap file size before read
if (stat.size > MAX_FILE_BYTES) return emptyStats;

// Check for binary
if (content.indexOf('\0') !== -1) return emptyStats;

2. Dirty Checking

ContextOS only compiles what has changed. We hash the file buffer and check it against the filesRepo.

const hash = hashContent(content);
if (!this.filesRepo.isChanged(filePath, hash)) {
  return emptyStats;
}

3. AST Generation & Chunking

The file extension routes the buffer to the appropriate tree-sitter parser (Code, Config, or Markdown). The AST is then walked to extract independent chunks and imports.

if (isCode) {
  const parsed = await parseCode(filePath, content);
  imports = parsed.imports || [];
  chunks = chunkCode(parsed, { layer, workspaceName });
}

4. Database Transaction

All state updates run in a highly-optimized SQLite transaction to preserve foreign keys. If a file changed, we first execute an ON DELETE CASCADE purge of its old chunks and relationships, then bulk insert the new ones.

// Update file record first for FK constraints
this.filesRepo.upsert({ path: filePath, hash, ... });

// Cleanup old chunks (cascades to relationships)
this.chunksRepo.deleteBySource(filePath);

// Bulk insert new chunks
this.chunksRepo.bulkUpsert(chunks);

5. Graph Edges & Non-Blocking Embeddings

Embeddings generation is the slowest part of the pipeline. To ensure init remains fast, embedding generation is strictly non-blocking. If the model fails or times out, the pipeline continues, as ContextOS falls back seamlessly to BM25 FTS5.

try {
  await indexChunkEmbeddings(this.chunksRepo.getDatabase(), chunks);
} catch {
  // continue without embeddings
}

// Extract and bulk insert graph edges
const allRels = chunks.flatMap(c => extractRelationships(c));
this.relsRepo.bulkUpsert(allRels);

Where F = number of files, and S = average size of files in bytes.

Failure Modes

• OOM (Out of Memory): Highly unlikely due to the hard MAX_FILE_BYTES cutoff, but an extremely complex 1.9MB TypeScript file could theoretically cause the tree-sitter WASM heap to panic.
• Corrupted SQLite state: Impossible. If the process is hard-killed (SIGKILL) mid-transaction, SQLite's WAL handles recovery on the next boot.