Parse, then cut
A window cut on the raw PDF string slices sentences, headings, and table rows at arbitrary points. Retrieval then returns a fragment that cannot be read.
Run the parse, check the warnings, and walk the ordered blocks. The Markdown and DocIR from that job are the input to chunking.
Boundaries that match the document
Start a chunk at a heading. Add the following paragraphs until the token budget is reached. Keep a small table intact, and repeat its header if you must split a long one.
Do not mix two unrelated sections just to fill a window.
Ids that survive reprocessing
Derive the chunk id from the document version and the block range. When the source hash changes, replace that version. When only the chunker changes, you can rebuild chunks without reparsing if DocIR is still retained.
An answer that cites a chunk should be able to open the page. Store the page index on the chunk, not only in a debug log.
Where the chunk boundaries already are
Block ids are assigned in order, starting at b000001. pages[].blockIds is the sequence for that page. A heading block is the start of a section. The paragraphs and lists that follow it, until the next heading, are the section body.
A table block that already passed the grid checks should stay whole. Split it only when the row count exceeds the token budget, and repeat the first row on each piece. Store pageIndex from the block. For a PDF, the file position you show a reviewer is pageIndex plus 1.
Walk the blocks in pages[].blockIds order. Do not cut the Markdown string and then guess the block ids. A block id looks like b000001 and increases in assignment order. Copy those ids onto the chunk. Stop the chunk before the next heading block. If metadata.warnings says the digital PDF text-layer fast path was used, keep that warning on the chunk so a later answer can say the layout path did not run.
- Parse, read warnings, then cut.
- Start a chunk at a heading block.
- Keep a small table block intact.
- Put source hash, parserVersion, pageIndex, and block ids on the chunk.
The file to send
Parse one text-based PDF, then walk result.document.pages[].blockIds. Start a chunk at block.type heading and stop before the next heading.
Keep a small table whole. Repeat the first row only when you split a long table. Copy pageIndex, parser version, and the block ids onto the chunk. Show the PDF page as pageIndex + 1.
- Cut on the block list, not on a raw Markdown window.
- Keep the text-layer fast-path warning on the chunk when it is present.
- One trial job is enough to check the boundaries.
export DOCPARSE_URL="https://docparse.genedai.me"
export DOCPARSE_API_KEY="your_key"
curl -fsS -X POST \
"$DOCPARSE_URL/v1/parse?wait=true" \
-H "Authorization: Bearer $DOCPARSE_API_KEY" \
-F "file=@document.pdf" \
-F "mode=both" \
-o result.json
jq -r '.result.markdown' result.json
jq '.result.document.schemaVersion, .result.document.pages[0].blockIds' result.json
jq '.result.manifest.decision.route, .result.manifest.parserVersion' result.json
Frequently asked questions
Questions this guide answers
Should I chunk the PDF before parsing?
Parse into heading, paragraph, list, and table blocks first. Chunk those blocks. A fixed window on the raw PDF string cuts sentences and table rows.
What should a RAG chunk store besides text?
Store the heading path, pageIndex, source hash, parserVersion, and the block ids. For a PDF, display pageIndex plus 1 as the file position. A printed footer number can differ from that position.
How big should a chunk be?
Fill the token budget with one heading and the blocks that follow it. Keep a table block whole unless the row count exceeds the budget, and then repeat its first row.