Provenance-Constrained Multi-Stage LLM Extraction from Heterogeneous Product Catalogs: System Design and Benchmark Protocol

1. Valid output, uncertain origin

A supplier catalog is more than a set of strings. Its rows encode a product, brand, presentation, sale format, unit, price, and sometimes a discount or wholesale condition. Meaning is distributed across page headers, merged cells, visual groups, footnotes, and abbreviations. Spreadsheet files expose cells but may still contain decorative rows and multi-line records; PDFs may preserve appearance while losing table semantics. The extraction task must recover both business fields and a defensible link to the source row.

The inspected Cuqui implementation decomposes the task into document metadata, row reconstruction, and product normalization [1–3]. The decomposition is technically plausible: early stages preserve layout and source identity; later stages work on bounded context and a strict schema. Plausibility is not comparative evidence. The research question is whether that extra structure reduces omissions, hallucinations, and review cost relative to a single-pass prompt when model family, source documents, and scoring are held constant.

• Specify the extraction task at row, field, and provenance levels.
• Audit the implemented stage boundaries, validation rules, retry behavior, and review gates.
• Reproduce the repository's deterministic software-test result at a pinned revision.
• Design a controlled comparison against a single-pass baseline without claiming an unrun accuracy result.
• Define release artifacts sufficient for independent scoring and error analysis.

2. Related work and task positioning

Document AI research has approached visually rich documents through OCR-dependent multimodal models and OCR-free image-to-sequence systems. LayoutLMv3 jointly pretrains text and image representations for document understanding tasks [8]. Donut removes the external OCR stage and directly maps document images to structured output [7]. Nougat focuses on converting scientific pages into markup while preserving semantics lost in PDF [10]. These systems are learned document models; the present system instead orchestrates a general-purpose multimodal model through explicit stages and business schemas.

DocILE is the closest benchmark reference because it separates key information localization/extraction from line-item recognition and provides annotated business documents, baselines, and layout variation [9]. Product catalogs differ from invoices in ontology and pricing conventions, but the methodological lesson transfers: line items must be identified before field-level extraction can be scored, and evaluation should include unseen layouts rather than random rows from the same template.

3. Task definition and unit of analysis

Let a document D contain an ordered set of source rows R. A gold annotation maps each valid product row r to zero or one normalized product y, with fields such as canonical name, brand, category, packaging, price type, amount, unit, and review status. Decorative headers, totals, and terms are labelled non-product. A prediction must first identify a source row, then assign fields. This avoids rewarding a system that produces plausible products without traceable origin.

The benchmark unit is a gold source row, nested within a document. Row-level metrics should be macro-averaged by document as well as micro-averaged across rows so that one large spreadsheet does not dominate the result. Field accuracy is computed only after predicted and gold rows are matched by provenance. Unmatched predictions count as hallucinated rows; unmatched gold rows count as omissions.

4. Implemented three-stage pipeline

The pipeline uploads a validated file to the Gemini Files API, waits until the file is active, and then executes two document-wide stages before batch normalization [2]. Stage 1 and Stage 2 use gemini-3.1-pro; Stage 3 uses gemini-3.1-flash-lite-preview at the recorded revision. Every generation requests application/json and supplies a JSON Schema. Each model call has a 60-second timeout and up to three attempts with increasing one-second backoff [2]. Model identifiers are version-sensitive and must be pinned again at experiment time.

Stage 2 is the structural hinge. Its flattened row count must equal totalRowCount and every rowId must be unique. Stage 3 partitions the rows into groups of ten. A deterministic batchId includes the batch index plus first and last row IDs; the model must return the same ID, index, total batch count, ordered row IDs, and page numbers. Any item whose sourceRowId is outside the current batch rejects the batch result [2,3].

metadata = stage1(file)
rows = stage2(file, metadata)
assert unique(rows.rowId)
assert flatten(rows).length == rows.totalRowCount
for batch in chunk(rows, 10):
    result = stage3(metadata, batch)
    assert result.context == expected_context(batch)
    assert every(result.item.sourceRowId in batch.rowIds)
    normalize_or_route_to_review(result.items)

5. Provenance constraints and human-review gates

Schema-valid JSON is necessary but insufficient. The implementation adds relational checks that the schema alone cannot express: unique row IDs, exact batch identity, ordered row membership, page coverage, and sourceRowId containment [2,3]. These checks prevent several silent failure classes, including a model answering from an earlier batch, returning a structurally valid but misaligned row, or inventing a product with no source member in the current context.

The 0.5 confidence threshold is a policy constant, not a calibrated probability. Model self-confidence may be poorly calibrated and can drift across versions. The benchmark must measure selective accuracy: among items auto-approved at a threshold, how many are correct, and what fraction of all items is deferred? This makes the review gate an evaluated decision rule rather than a decorative score.

6. Resumability, duplicate control, and operational state

The ingestion action validates file magic bytes and size, computes SHA-256, and rejects recent duplicates for the same provider [2]. It stores file identity, metadata JSON, row JSON, progress, batch counters, errors, and results in an ingestion-run record. Stage 3 can resume from the first batch whose row IDs are not already present. Temporary upload files are deleted in a finally path. These controls address long-running job reliability but also create experimental variables: retries, resumed batches, and duplicate rejection must appear in result metadata.

7. Reproduced software evidence

The public repository was cloned and detached at a5d3aadbfa049f4f6d77a2f144f557fa9bf7cffb. Under Node 24.14.0 and npm 11.9.0, npm ci completed and npm test exited zero. Vitest reported 20 passing test files and 569 passing tests in 1.06 seconds, with 693 ms attributed to test execution. This count supersedes the README statement of 409 tests for the pinned revision [1,5].

git checkout --detach a5d3aadbfa049f4f6d77a2f144f557fa9bf7cffb
npm ci
npm test

The separate performance file also passed five tests. Its timed operation generates 10,000 mock products and pushes identifiers into an in-memory array while mirroring object-spread overhead [6]. It does not call Convex, perform network I/O, invoke Gemini, validate an uploaded document, or persist products. The repository decision document's multi-million-products-per-second interpretation is therefore not used here [4]. The defensible result is only that the simulation and its assertions completed locally.

8. Controlled benchmark design

The proposed pilot uses a versioned corpus of at least 60 permission-cleared catalogs: 20 PDF, 20 XLS, and 20 XLSX. Sampling should stratify by layout complexity, row count, multi-line density, packaging ambiguity, mixed units, discounts, and image/scanned content. Documents from the same supplier template must remain in the same split to avoid layout leakage. A frozen development subset is used for prompt iteration; the final test subset is opened once.

Each document-condition pair should run at least three times because generative extraction can vary. The primary comparison is paired by document. Model identifiers, prompts, schemas, temperatures, retry policy, and pricing must be archived. A failure after all retries remains a failed document; silently excluding it would bias accuracy and latency upward.

1. Create annotation guidelines and label rows before inspecting model output.
2. Double-label at least 20% of documents; adjudicate row boundaries and normalized fields.
3. Freeze splits, prompts, schemas, models, and scoring code.
4. Execute every condition with identical source documents and record all retries.
5. Publish redacted gold labels, raw predictions, match decisions, aggregates, and failure examples.

9. Metrics and statistical analysis

Report document-level macro means with bootstrap 95% confidence intervals. For paired binary row outcomes, use a paired test such as McNemar where matching is valid; for document-level continuous measures, use paired bootstrap differences. Multiple field comparisons should be treated as a family and interpreted with effect sizes, not selected p-values. Error examples should be sampled by a predeclared taxonomy rather than chosen only for visual impact.

10. Error taxonomy, validity, and ethics

• Construct validity: JSON validity and test count are not proxies for extraction correctness.
• Internal validity: stage models differ, so gains cannot automatically be attributed only to decomposition.
• External validity: Argentine food catalogs may not represent other sectors, languages, or document conventions.
• Temporal validity: preview model behavior and pricing can change after the recorded revision.
• Annotation validity: row boundaries and packaging normalization require documented adjudication.

Catalogs may contain supplier identities, negotiated prices, contact details, or commercial terms. Public benchmarking requires permission or irreversible redaction. Hashes, raw files, model uploads, logs, and released examples need a retention and access policy. Human annotators should see only the minimum necessary data, and the released corpus should avoid reconstructable confidential combinations.

11. Conclusion

The inspected system contains a thoughtful extraction architecture: explicit stage boundaries, schema-constrained generation, row and batch identities, deterministic provenance checks, forced review rules, retries, and resumable state. The complete local software suite is reproducible at 569 passing tests. Those are legitimate engineering results.

The central research claim remains open. There is no labelled corpus or controlled single-pass comparison, so improved extraction reliability has not been demonstrated. The protocol in this paper defines the missing experiment and prevents software-test volume or in-memory throughput from being substituted for model quality. Running and releasing that benchmark is the next evidence-producing step.