Turn mixed files into a maintainable knowledge base

Normalize before indexing

A knowledge base usually receives files created by many teams over many years. File extension is not a reliable quality signal: some PDFs contain clean text, others are scans; spreadsheets carry structure that should not become prose; presentations rely on spatial grouping.

A shared parsing API isolates this variation. The indexer receives Markdown and a versioned document representation, while routing and format-specific dependencies remain behind the service boundary.

Preserve source identity and freshness

Store the source content hash, modified timestamp, parser version, and options hash with the indexed document. Skip unchanged sources, and rebuild only when source or parsing policy changes. Use idempotency keys during synchronization so duplicate delivery does not create duplicate work.

Do not update the active knowledge item until the new parse and index pass validation. Keeping the previous version makes refreshes recoverable.

• Stable source IDs independent of file names.
• Page and heading metadata on every indexed chunk.
• Explicit status for parsing, indexing, active, failed, and superseded versions.
• Purge rules aligned with the knowledge-base retention contract.

Make failures visible to operators

A zero-length parse, repeated OCR warning, or unsupported format should enter a remediation queue instead of silently disappearing from search. Show the affected source, route, parser version, and actionable error to the content owner.

Quality dashboards should segment coverage by source type and collection. A single document count does not reveal that all slide decks or scans failed.