Methodology
Where our accuracy numbers come from, and what they are not.
Accuracy claims in this industry are usually a single number with no definition behind it. This page defines ours. Everything published on the benchmarks page comes from the run described here, version 2026.Q3, last refreshed on 2026-07-27.
The dataset
The benchmark dataset is a curated set of anonymized statements donated by users under an internal test license, plus synthetic statements generated from public bank templates. It spans checking, savings, credit card, and business accounts across US, UK, EU, IN, AU, and AE banks.
Current size: 1284 documents. The set is deliberately unbalanced toward the hard cases: scanned PDFs are over-represented relative to real traffic, because that is where extraction fails and where a benchmark is worth running.
The environment
Runs execute on a single Chromium instance with 4 vCPU and 8 GB RAM. Parsing happens in-browser via the same WASM pipeline shipped to end users. OCR times include model warm-up on the first page.
What each metric means
- Parsing success rate. The share of documents where the pipeline produced a transaction table at all. A document that yields zero rows, or that we refuse to process because it is encrypted or empty, counts as a failure.
- Row-level accuracy. Of the rows produced, the share where date, description and signed amount all match the human-verified reference exactly. One wrong field fails the whole row. There is no partial credit.
- Median and p95 runtime. Wall-clock time from file selection to a rendered transaction table, including OCR model warm-up on the first page.
How confidence is calculated in the product
The confidence figure shown after an upload is not the benchmark number. It is computed per document from three signals: coverage, which is the share of candidate transaction lines that produced a parsed row; arithmetic reconciliation, which checks the opening balance plus every debit and credit against the closing balance; and field-level quality, which flags rows with a missing or ambiguous date or amount.
When the running balance reconciles to the penny and every candidate line produced a row, the extraction is structurally complete and confidence reports as full. When it does not reconcile, the discrepancy is surfaced rather than hidden, because a silent rounding error in a statement export is exactly the kind of thing that should stop you before it reaches a tax return.
Current published results
| Format | Parsing success | Row accuracy | Sample |
|---|---|---|---|
| CSV | 99.7% | 99.9% | 412 |
| XLSX | 99.2% | 99.8% | 208 |
| PDF (digital) | 96.4% | 98.1% | 486 |
| PDF (scanned) | 84.2% | 92.3% | 138 |
| Receipt PDF | 95.8% | 97.2% | 40 |
Limitations
- These are our own measurements on our own dataset. They are not independently audited.
- A benchmark set is not your statements. A bank we have never seen may parse worse than the numbers suggest.
- Row accuracy is measured against a human reference that is itself occasionally wrong, which biases results slightly upward.
- Runtime depends heavily on machine and browser. Treat the figures as relative, not as a guarantee.
Refresh cadence
Quarterly. Next scheduled run: 2026-10-15.
Numbers are never edited between runs. If a figure on the site disagrees with what you measure, tell us at support@karchu.com and we will investigate it in the next run. Our broader content rules are on the editorial standards page.