01

1. Establish source identity

Give every source a stable identifier independent of its display name. Record repository, path, content hash, access-policy version, modified time, and discovery time. The content hash determines whether parsing is necessary; the access-policy version determines whether index permissions must change.

Use an idempotency key derived from source identity and parsing policy when submitting the job. A retry should return the existing work, not create a second copy.

02

2. Parse and validate before chunking

Wait for a successful terminal job and validate the schema version, non-empty content, page coverage, and warning policy. Route empty or degraded documents to remediation instead of indexing them as if they succeeded.

Keep Markdown for text and DocIR for structure. The source hash, parser version, route, and options hash become the parsing version of the document.

source@hash → parse job → schema validation → quality policy → normalized document version
03

3. Build structure-aware chunks

Walk headings and ordered blocks. Combine related paragraphs until the token budget is reached, keep compact tables intact, and attach heading path and page references. Assign stable chunk identifiers derived from the document version and block range.

Do not discard the original block mapping after embedding. It is needed for citations, re-ranking explanations, evaluation, and selective reprocessing.

04

4. Version and evaluate the index

Write new chunks under an index version, embed them, run retrieval checks, and only then make the version active. Keep the previous version until the cutover is verified. Deleting a source should deactivate its chunks even if retained parser artifacts follow a different schedule.

Evaluate questions that cover exact facts, tables, multi-section synthesis, and negative controls. Store retrieved chunk IDs and expected evidence so a regression can be attributed to parsing, chunking, embedding, or ranking.

  • Coverage: every authorized source has an active or explained failure state.
  • Freshness: active index matches source content and access version.
  • Traceability: every answer citation reaches a source page or block.
  • Recoverability: index activation and purge are separate reversible steps.

Frequently asked questions

Questions teams ask before building

What order should RAG ingestion stages run in?

Identify and fetch the source, parse, validate, chunk, embed, evaluate, and then activate the new index version.

How can RAG ingestion be idempotent?

Use stable source IDs, content hashes, policy versions, and request idempotency keys. Reuse unchanged parse and embedding artifacts when allowed.

When should an index update become visible?

Only after parsing, validation, chunking, embedding, and retrieval checks succeed. Switch an active version pointer atomically.