A practical business guide

How to test document extraction against a small labeled sample

Prepare a checked answer key, compare extracted fields, and record the errors that matter to your workflow.

Download the worksheet ↧

Test on documents with answers you checked first

Label a small set of documents before you run the extractor. Compare its output with the labeled answers field by field. Count correct, wrong, missing, and extra values separately. Decide which fields matter enough to block release if the tool gets them wrong.

Start with the document types the workflow will receive. Include a clean form, a scan, a missing optional field, and a layout variant. A small sample can expose specific failures. It cannot establish the error rate for all future documents. Use synthetic documents or material you are authorized to evaluate. If the documents contain private values, decide where they may be stored and analyzed before sending them to a tool.

Write the answer key and matching rules

  1. Assign a document ID and save the input version. Record the page or region where the expected value appears.
  2. Label each field with the verified value, absent, or ambiguous. When the source has two plausible values, record the ambiguity before running the tool. Use those cases to check escalation separately.
  3. Write the matching rule for each field. A date may match after a defined conversion to an ISO date. An order ID may require exact spelling and all leading zeros. State when a field is allowed to be blank.
  4. Run the same inputs through each candidate configuration. Record the model or processor version, the extraction settings, and the raw output. Keep the scored documents separate from examples used to tune the extractor.

If you adjust a prompt, field definition, or parser after seeing the errors, rerun the original cases to check the repair. Evaluate the changed version on other labeled documents to test whether the improvement carries over. Google Document AI describes how to compare predictions with annotations on test documents. Microsoft documents the use of labeled examples and evaluation before and after training. The worksheet below uses simple work records you can prepare independently of either product.

Score the fields individually

For a field that occurs at most once per document, compare the labeled answer and the predicted value after applying your documented matching rule. If an expected ID was misread, mark it as wrong. If the field had a value and the extractor gave no value, mark it as missing. If the field was absent and the extractor returned something, mark it as extra. Record a correctly empty field separately. Leave ambiguous examples out of the exact-match score and track whether they were sent to a person.

When you report precision and recall for single-value fields, a wrong value counts as both an unmatched prediction and an unmatched reference. Precision is correct matches divided by predictions. Recall is correct matches divided by labeled expected values. A field with no expected examples has no recall estimate. For repeating line items, first match the same item across the answer key and output, using a stable line ID when possible. Score the values for that item after matching the rows.

Worked example with four fictional delivery forms

Copper Mesa Couriers is fictional. It compares the order ID and optional dock code on four invented delivery forms. The outputs below were written to illustrate scoring. No extractor generated them. The team requires an exact order ID before the workflow can update its delivery tracker.

Answer key and hypothetical predictions
DocumentExpected order ID / dock codePredicted order ID / dock codeField outcomes
CM-01 · Clean PDF0042 / D70042 / D7ID correct, dock correct
CM-02 · Scan0043 / D2043 / D2ID wrong, dock correct
CM-03 · Blank dock field0044 / absent0044 / D9ID correct, dock extra
CM-04 · Alternate layout0045 / D4absent / absentID missing, dock missing

The order ID has two matches out of three predictions and two matches out of four expected IDs. Its precision is 2/3 and its recall is 2/4. The dock code has two matches out of three predictions and two matches out of three expected codes. Its precision and recall are each 2/3. The missing ID and the changed leading zeros fail the tracker requirement. Keep automatic updates off for those cases while you test the repair. If the expected value is unreadable or conflicting on a new form, use the field review guide to assign an owner and present the evidence.

Use a worksheet you can rerun

DOCUMENT EXTRACTION EVALUATION
Sample version / date / allowed storage:
Document ID / type / layout / page evidence:
Field / critical or optional / matching rule:
Expected value / absent / ambiguous:
Processor version and configuration:
Prediction and source region:
Outcome: correct / wrong / missing / extra / correctly empty:
Exclusion reason and review routing result:
Per field: expected count / prediction count / correct count:
Decision, repair owner, rerun and fresh sample:

Use one worksheet row per document and field. Save the labeled inputs and rerun the same cases after a change. For help turning the worksheet into a repeatable internal check, bring one synthetic or redacted document to an Internal Tools fit call.

Put the plan to work

Start with one useful change.

Bring the process, the people involved, and a redacted example. A free 30-minute fit call can establish the next step.