Build cohorts before metrics
Sample the documents users actually submit. Separate digital single-column PDFs, multi-column reports, scans, tables, presentations, spreadsheets, images, and legacy files. Include clean, average, and degraded examples for each important cohort.
Keep a holdout set that is not used for parser tuning. Record confidentiality and retention rules before sending documents to any external service.
Score more than text similarity
Character or token overlap misses reading order and structure. Label headings, paragraphs, lists, tables, page boundaries, and required facts. For tables, compare headers, row and column alignment, merged cells, and numeric values—not merely whether all words appear.
Score unsupported input, empty success, timeout, retry behavior, and partial output. A parser that returns an explicit failure can be safer than one that returns plausible but incomplete Markdown.
- Text completeness and corruption.
- Reading order and section hierarchy.
- Table and list structure.
- Page, bounding-box, and citation accuracy.
- Failure classification, latency distribution, and unit cost.
Measure the downstream outcome
For RAG, run retrieval and citation questions. For extraction, validate fields against a schema and ground truth. For review workflows, measure correction time. A structural difference matters only if it changes the product outcome or operator cost.
Inspect disagreements manually and classify root cause. The same parser may be excellent for digital reports and poor for photographed forms, which suggests routing rather than a universal winner.
Make the comparison reproducible
Pin product plan, API version, model name, options, date, and region where relevant. Save source hashes, raw responses when policy permits, normalized outputs, timings, errors, and scoring code.
Re-run a smaller regression set when a provider model or internal parser changes. Comparison pages and purchasing decisions should state their verification date and avoid presenting old results as current truth.
Frequently asked questions
Questions teams ask before building
How many documents are needed for a parser evaluation?
There is no universal number. Cover every material cohort with enough variation to expose failures; use confidence intervals or repeatability checks when comparing close results.
Can edit distance measure parsing quality?
It helps with text recognition but misses hierarchy, order, tables, pages, and downstream usefulness. Use multiple metrics and manual error classification.
Should cost be measured per page or document?
Measure both provider billing units and the workload’s real document distribution, including retries, routing, storage, and operational overhead.