How to test AI-assisted reconciliation before relying on it

An incorrect match can make a reconciliation queue look healthier while leaving the books wrong. A useful evaluation therefore measures the accuracy of proposed matches, unresolved exceptions and the staff effort required to reach a correct final result.

This protocol compares three ways of handling the same bounded task: the existing manual process, a rules-based approach and an AI-assisted approach. It is a method for running an experiment. No benchmark has been executed and the numerical example below is illustrative.

Define the task precisely. For example, identify the invoice or invoice group corresponding to each receipt in one entity's statement, then prepare the supporting evidence. The experiment should use isolated records and produce proposals; it should not alter the ledger or execute payments.

Singapore's Model AI Governance Framework for Agentic AI discusses baseline testing, accountability and controlled access. Those principles support the boundaries of this experiment. They do not prescribe the sample size or acceptable error rate for your reconciliation process.

Build a reference set the business can defend

Choose historical records that the organisation is authorised to use. Have an experienced reconciler determine the expected result, then obtain an independent review of disputed or material cases. Preserve the evidence behind each accepted answer. Where the records cannot support a definitive match, the correct answer may be “unresolved”. Forcing a match would reward the wrong behaviour.

Define the unit of evaluation before counting cases. One receipt matched to three invoices is one reconciliation decision if the task is defined around receipts. Evaluate the full group, including amount and currency, rather than awarding success for finding only one invoice. Document partial payments, fees, reversals, duplicates and currency differences explicitly.

Also check the completed batch. Track amounts allocated from every receipt to each invoice, including permitted adjustments. Combined allocations must reconcile to authoritative balances; flag duplicate allocation and preserve unexplained residuals. Individually plausible matches can still over-allocate an invoice across several receipts.

Separate development records from a held-back test set. The builder may use development cases to tune extraction, prompts and rules. Once final testing starts, freeze the held-back records and expected answers. If the design changes after a failure, retain the failure and use a fresh final test where possible. Repeated adjustment against the same answers overstates what has been learned about unseen work.

Independently check a documented sample of ordinary reference answers as well as disputed or material cases. Retain disagreements and their adjudication. Keep linked receipts, invoice groups and duplicates on the same side of the development/test boundary so that different row identifiers do not conceal the same decision.

Maintain two views of the data: a representative sample of normal work and a deliberately difficult set. The latter should include ambiguous names, missing identifiers, near-duplicate amounts and out-of-scope documents. Report the groups separately. A stress set is useful for exposing failure, but its deliberately unusual case mix does not estimate normal operating prevalence.

Set sample size and precision requirements with the acceptance criteria. Related suppliers, templates and periods can share a failure mode; explain how the sample covers those clusters. Repeated runs test variability, not additional independent business cases. If the evidence cannot resolve the agreed error limit, record an inconclusive result and narrow the proposed use or obtain more evidence. The evaluation pack includes a conditional zero-error calculation to make this limit concrete.

Keep the comparison fair

Give each approach the same source information and require the same completed outcome. Record preparation time. If one method needs a person to clean a file or locate a missing reference, include that effort rather than starting its clock afterwards.

Use comparable reviewer conditions. Randomise case order and, where feasible, separate reviewers or case batches so someone does not remember the answer from an earlier run. Record experience and training. Review time measured by the builder may not reflect the intended operator's performance.

For systems with variable outputs, repeat selected cases and inspect whether the conclusion changes. Record the model, configuration, rules, software versions and test date. Keep operational data protected when storing evaluation records; use approved environments and restrict access to source documents.

Measure the proposal and the final result

Proposal precision is the number of correct proposed matches divided by all proposed matches. Correct-proposal coverage is the number of correct proposed matches divided by all cases. Abstention is a decision to leave a case unresolved for human handling. Together, these measures show whether high precision was achieved by solving useful work or by declining most cases.

Then assess the final outcome after review. Record errors that reviewers missed, additional errors introduced during correction and the time required to complete the case. Separate material wrong matches from less consequential formatting or explanation defects. The dollar exposure and downstream consequence of an error may matter more than its contribution to a percentage.

Account for every incoming case, including timeouts, rejected records and work transferred to another queue. Reconcile opening backlog and arrivals to completions, authorised withdrawals and closing backlog. Report overdue unresolved cases alongside final quality so that apparent efficiency does not hide accumulating work.

An assistant's confidence score is not an observed probability of correctness unless that relationship has been established on suitable data. Choose routing thresholds using measured results and business consequences, rather than accepting a reassuring number displayed by the tool.

Read an illustrative result correctly

Suppose a hypothetical test contains 1,000 cases. An assistant proposes matches for 800; 792 are correct and eight are wrong. It leaves 200 cases unresolved.

The headline “99% accurate” would conceal useful information. Eight wrong matches could carry unacceptable consequences if acted on automatically. The same proposals might still be useful to trained reviewers if the evidence helps them identify mistakes efficiently. That possibility requires testing the reviewed outcome, including missed errors.

For workload illustration only, assume current handling averages four minutes per case: 4,000 minutes, or 66.7 hours. If the 800 proposed cases average one minute each for all checking and correction, and the 200 unresolved cases still take four minutes, assisted handling totals 1,600 minutes, or 26.7 hours. The apparent capacity reduction is 40 hours.

That calculation assumes the one-minute average includes correcting the eight wrong proposals and that final quality is equivalent. Both assumptions need measurement. Add preparation, supervision, support and usage cost before drawing an economic conclusion. Use the full-cost worksheet to distinguish released capacity from expenditure the organisation can actually avoid.

Record the experiment before running it

The following record makes the result reviewable and reduces the temptation to redefine success afterwards.

Review difficult cases before the aggregate score. A repeatable error on duplicate payments or multi-invoice groups may justify excluding that category even when the overall average is attractive. A small test with no observed material errors cannot prove that the risk is zero.

Take the completed experiment record to the process owner. If the proposal is to introduce live use, carry its tested limits into the release evidence sheet, including permissions, review capacity and recovery. The test should define the next safe and economically credible operating decision, rather than produce a percentage detached from the work.