This pack supports the reconciliation evaluation guide. It contains a study specification and fictional fixtures for rehearsing case handling. The fixtures cannot estimate accuracy, choose a production threshold or substantiate a performance claim.
Register the decision before running the comparison
| Field | Required decision |
|---|---|
| Intended use | Suggest matches, prepare evidence or another bounded action; identify forbidden actions |
| Unit | One receipt-level decision, including all invoices in a proposed group |
| Population | Entity, period, source systems, document types, currency and relevant exclusions |
| Comparators | Current manual process, deterministic matching and AI-assisted workflow |
| Information available | The same source records and permitted contextual information for each approach |
| Completion | Correct match or justified unresolved status, with supporting records |
| Sampling | Selection method, sample size rationale, strata and treatment of correlated cases |
| Reference answers | Named reconciler and independent checker; method for resolving disagreement |
| Batch consistency | Authoritative opening balances, allocations, permitted adjustments and unexplained residuals |
| Acceptance | Material-error limit, useful coverage, total effort and uncertainty sufficient for the next decision |
| Stop rule | Conditions that suspend the experiment, including unintended writes or exposure of records |
| Change control | Frozen versions, fresh test data after tuning, and a retained record of failed attempts |
A zero-error result in a small sample is weak evidence about rare failures. Under an independent, identically distributed Bernoulli model, zero observed errors in n eligible decisions gives a one-sided 95% upper bound of 1 − 0.05^(1/n), approximately 3/n. At 300 decisions this is about 1%; it does not prove the true error rate is near zero. For conditional proposal precision, the denominator is proposed matches, not all offered cases. This calculation is an illustrative statistical model, not a required sample size or a sufficient release criterion.
Repeated attempts on the same receipt, invoices from the same supplier template, and correlated time periods do not supply independent observations merely because there are more rows. Choose the sampling unit and uncertainty method with the person responsible for the risk decision. Report stress cases separately from a representative sample, and do not infer population error rates from hand-picked fixtures.
Rehearse the case schema
Use the fictional case file to check that the proposed evaluation can represent these situations:
| Case | Expected treatment | Failure being probed |
|---|---|---|
| T01 | Exact single-invoice proposal | Basic amount, currency and reference validation |
| T02 | One receipt, two invoices | Partial group incorrectly marked complete |
| T03 | Unresolved until a fee rule is confirmed | Inventing a deduction explanation |
| T04 | Unresolved without currency/conversion evidence | Matching the same numeric amount across currencies |
| T05 | Exclude duplicate intake; preserve the original case | Counting a duplicate as new completed work |
| T06 | Unresolved where two candidates are indistinguishable | Guessing to improve coverage |
| T07 | Recognise a reversal and require ledger-state evidence | Treating a reversed receipt as settled cash |
| T08 | Request missing evidence | Hallucinating an identifier absent from the record |
The CSV includes all facts permitted for the toy decision and a stated expected outcome. Keep expected outcomes hidden from the system under evaluation. These rows are schema examples, not realistic source-document parsing tests. Build an authorised, independently labelled record set before evaluating a real system.
The separate allocation fixture deliberately proposes allocating two receipts, S$80 and S$70, against an invoice with an authoritative S$100 opening balance. The S$150 proposed allocation exceeds that balance by S$50. The expected result is a held batch and an unresolved excess, not automatic acceptance of two individually plausible matches. An invoice may legitimately receive several partial payments; test total allocated amounts rather than banning repeated invoice identifiers. The fixture does not decide how the excess should ultimately be treated.
Capture both model proposals and final decisions
Use the run-log template, one row per case, method and repeat, and the allocation log, one row per receipt–invoice allocation. Preserve the full proposed group rather than only the first invoice identifier. Distinguish: correct proposal, wrong proposal, justified abstention, unnecessary abstention, incorrect final result and unresolved final result.
A justified abstention can be a correct assistant response while the underlying accounting case remains open. Record assistant-response correctness, review outcome and operational disposition separately. An appropriate request for missing evidence must not be counted as a completed reconciliation. Likewise, repeated invoice opening balances in an allocation file are references to one authoritative balance; do not sum them as additional capacity to allocate.
Run amount and balance checks separately within each study, method, repeat and batch. The allocation log includes those keys and the case identifier. Combining comparator outputs or repeated runs into one batch would create a false duplicate allocation; compare their results after checking each run independently.
Record preparation, review, correction and follow-up time. Separately record elapsed completion time; an unresolved request may consume little active time while missing a service deadline. Log any support work outside the case timer. Do not drop cases that error or time out.
At each reporting date, reconcile intake, completed work, authorised withdrawals and outstanding backlog. Report the number and age of open cases. A comparison of only accepted cases can reward a system for discarding difficult work.
Analyse the result without concealing trade-offs
- Proposal precision: correct proposed matches divided by all proposed matches; undefined when there are no proposals.
- Correct-proposal coverage: correct proposed matches divided by all eligible cases; identify cases where no match exists.
- Final quality: wrong final decisions, their consequences, and unresolved work after review.
- Effort: all active preparation, review, correction and follow-up, plus a separately allocated share of support.
- Reliability: repeated-run disagreement, timeout rate and materially different results by case type.
Independently sample ordinary reference answers as well as disputed or material cases. Keep related records and duplicates on the same side of the development/test boundary. If the reference answer is later corrected, version it and rerun affected comparisons. Report the change rather than quietly improving the score. For disputed cases, preserve both reviewers' positions and exclude or adjudicate them using the pre-agreed rule, retaining the exclusion count and its effect on denominators.
The recommendation should specify what the evidence permits: continue investigation, operate in shadow mode, offer suggestions within tested categories, revise and retest, or stop. A permissions change from suggesting to posting is a new decision with new evidence needs. Carry the approved scope into the release pack.
Read our editorial principles for evidence, contribution and correction standards.