We published the benchmark. Including where we lose.
Not whether a value was read — whether it can be defended. Open method. We scored ourselves first.
Read is not the same as reliable.
Parsing benchmarks check the characters came off the page. That was 2023's problem.
- A right value with a wrong citation still passes
- Averages hide errors that went through unreviewed
- Saying “I don't know” earns no credit
Extraction score
- Value read pass
- Citation right? not asked
- Declined when unsure? not asked
- Trail rebuildable? not asked
Passes
Attestation benchmark
- Value read pass
- Citation right? wrong page
- Declined when unsure? yes
- Trail rebuildable? yes
Fails citation correctness
Illustrative example
Why publish something we can be beaten on?
A measure means something only if its author can lose on it. If somebody beats us, the benchmark is working.
Each tied to a question asked under oath.
Per field, per family — and no headline number. A composite lets a weak measure hide.
Citation correctness
Does the cited page and region contain the value?
Gate precision
Error rate among values that went through unreviewed.
Abstention correctness
Fraction of abstained values that would have been wrong.
Reviewer agreement
Fraction of steward corrections that agree with the labelled ground truth.
Evidence completeness
Can the full trail be rebuilt with no human help?
Case-file completeness
Does each sampled decision's case file have every element?
Real numbers — but they invite a run that clears the gate by lowering it. We report them in the evaluation report.
Reproduced, not trusted.
What we release — and the limits we state rather than hide.
The document set
One labelled reference document for each of the five packs that have one. A starting point: your ground-truth set is built in the pilot.
Labels and labelling guide
Field-level ground truth, with the adjudication rules behind it.
The scoring harness
Emit fields and citations in the neutral format, and you can be scored.
We wrote it. It is v0.1.
One reference document per pack where it exists, one author. Credible once others run it and argue.
It is not your estate
It makes vendors comparable. Your documents tell you what you'd get.
Frozen and dated
Every result carries its version. Scores are never restated.
Gothink AIR v1.1.0 against v0.1.
Self-run on each pack’s labelled reference document — gothink measuring gothink. The right amount of trust is “enough to check”.
| Measure | Wealth Management | Insurance Submissions | Where this is weakest |
|---|---|---|---|
| Citation correctness | — | — | Multi-page tables where a value is restated in a summary |
| Gate precision | — | — | Handwritten amendments that clear the gate on a clean scan |
| Abstention correctness | — | — | Over-abstains on legible but unusual clause layouts |
| Reviewer agreement | — | — | Fields where two reviewers disagree with each other first |
| Evidence completeness | — | — | Records that crossed a schema version boundary mid-run |
| Case-file completeness | — | — | New measure; decisions recorded from other vendors carry only what their logs hold |
Publish the run before the number
Figures are withheld until the first run is complete and independently witnessed — then published with raw outputs.
No competitor scores. Ever.
We'll link to any published v0.1 run — including ones where Gothink AIR comes second.
Three ways to use it.
Put it in the RFP
Ask each vendor for six numbers and raw outputs.
Request the corpus and harnessRun it and publish
Permissive licence, even to market against us. Publish the version and raw outputs.
Tell us you're running itScore what you have
Most in-house pipelines never measured gate precision. Find the missing layers.
The five layersThe benchmark shortlists. Your documents decide.
Down to two? Run twenty-five of your own documents through each, on these measures.