Build a RAG ingest layer on document structure, not cleanup scripts

Put parsing before chunking

Chunking cannot recover structure that parsing already destroyed. If two columns are interleaved or a table is flattened, a semantic splitter receives the wrong sequence. The ingest layer should first produce a normalized document with headings, blocks, page references, and warnings; only then should it decide chunk size and overlap.

DocParse provides Markdown for the text payload and DocIR for boundaries and metadata. A practical chunk record stores the text, heading path, page range, source identifier, parser route, parser version, and any warning that may affect trust.

• Hash the source and options for idempotent ingestion.
• Chunk by heading and block boundaries before applying token limits.
• Keep tables intact or create a table-specific representation.
• Attach page and block identifiers to every vector record.

Make citations traceable

A citation should lead back to evidence a user can inspect. Store the job ID, source name, page index, and block identifiers with each chunk. When an answer cites a chunk, the application can show the original page or extracted block and explain which parser route created it.

This also improves evaluation. When retrieval misses, distinguish between parsing loss, chunking loss, embedding mismatch, and ranking failure. Without provenance, every error looks like a model problem.

A production ingestion sequence

Start with an inline parse on a representative file. Validate Markdown and DocIR. In production, submit asynchronously with an idempotency key, wait for a successful terminal event, validate the schema version, build chunks, embed them, and only then switch the active index pointer. Keep the previous index version until the new one passes retrieval checks.

Purge policy should match the product contract. If raw files must be deleted, preserve only the metadata and derived artifacts the customer has authorized. DocParse exposes explicit purge rather than treating retention as an undocumented side effect.

source → parse job → validated DocIR → structure-aware chunks → embeddings → index swap → retrieval evaluation