01

Build cohorts before metrics

Sample the documents users actually submit. Separate digital single-column PDFs, multi-column reports, scans, tables, presentations, spreadsheets, images, and legacy files. Include clean, average, and degraded examples for each important cohort.

Keep a holdout set that is not used for parser tuning. Record confidentiality and retention rules before sending documents to any external service.

02

Score more than text similarity

Character or token overlap misses reading order and structure. Label headings, paragraphs, lists, tables, page boundaries, and required facts. For tables, compare headers, row and column alignment, merged cells, and numeric values—not merely whether all words appear.

Score unsupported input, empty success, timeout, retry behavior, and partial output. A parser that returns an explicit failure can be safer than one that returns plausible but incomplete Markdown.

  • Text completeness and corruption.
  • Reading order and section hierarchy.
  • Table and list structure.
  • Page, bounding-box, and citation accuracy.
  • Failure classification, latency distribution, and unit cost.
03

Measure the downstream outcome

For RAG, run retrieval and citation questions. For extraction, validate fields against a schema and ground truth. For review workflows, measure correction time. A structural difference matters only if it changes the product outcome or operator cost.

Inspect disagreements manually and classify root cause. The same parser may be excellent for digital reports and poor for photographed forms, which suggests routing rather than a universal winner.

04

Make the comparison reproducible

Pin product plan, API version, model name, options, date, and region where relevant. Save source hashes, raw responses when policy permits, normalized outputs, timings, errors, and scoring code.

Re-run a smaller regression set when a provider model or internal parser changes. Comparison pages and purchasing decisions should state their verification date and avoid presenting old results as current truth.

05

The file to send

Pick one labeled file from a single cohort, such as digital two-column PDFs. Send that file. Score the Markdown and DocIR against the page you labeled.

Write result.manifest.decision.route and result.manifest.parserVersion next to the score. A later run is comparable when those two fields match. Record the score for that file, and keep a separate score for each cohort. Write the expected heading, one table cell, and the page order down before you open the Markdown, then mark each one against result.document.

  • Score one heading, one table cell, and the page order.
  • One showcase file is one sample.
  • Stay under 20 MB so the trial accepts the upload.
export DOCPARSE_URL="https://docparse.genedai.me"
export DOCPARSE_API_KEY="your_key"

curl -fsS -X POST \
  "$DOCPARSE_URL/v1/parse?wait=true" \
  -H "Authorization: Bearer $DOCPARSE_API_KEY" \
  -F "file=@document.pdf" \
  -F "mode=both" \
  -o result.json

jq -r '.result.markdown' result.json
jq '.result.document.schemaVersion, .result.document.pages[0].blockIds' result.json
jq '.result.manifest.decision.route, .result.manifest.parserVersion' result.json

Frequently asked questions

Questions this guide answers

How many documents are needed for a parser evaluation?

There is no universal number. Cover every material cohort with enough variation to expose failures; use confidence intervals or repeatability checks when comparing close results.

Can edit distance measure parsing quality?

It helps with text recognition but misses hierarchy, order, tables, pages, and downstream usefulness. Use multiple metrics and manual error classification.

Should cost be measured per page or document?

Measure both provider billing units and the workload’s real document distribution, including retries, routing, storage, and operational overhead.