Evaluation & impact
Every number on this page is computed by code from the labelled synthetic invoice book or an evaluation file. Nothing is hardcoded.
1 · Can Resolve avoid the wrong chase?
Resolve blocked 10 of 14 wrong chases in this labelled synthetic invoice book.
Resolve's decisions come only from observable evidence. Hidden ground truth is read solely by this evaluation harness. The classifier here is demo-lexicon-v1 (deterministic; developed alongside these synthetic messages) — this measures the system design on synthetic data, not model accuracy.
Blind reminders are an easy baseline. A competent rules setup — ledger check, strict bank matching (amount + invoice reference), and keyword suppression for payment / dispute / document words — is compared on the same book.
| Approach | Wrong chases blocked | Unnecessary pauses | Avoidable follow-up handled | Chooses the next action? |
|---|---|---|---|---|
| Reminder-only | 0 of 14 | 0 | 0 of 20 | No |
| Rules baseline | 8 of 14 | 0 | 12 of 20 | No — only suppresses |
| Resolve | 10 of 14 | 2 | 16 of 20 | Yes — wait, plan, correction, review |
Read this honestly: on this book, simple rules capture most of the suppression value, and Resolve pauses 2 more invoices unnecessarily. The keyword list was written by the same author as the synthetic messages, so the rules baseline is likely optimistic. Resolve's difference is mostly in what happens next — a dated wait with a timer, an approved part-payment plan, a routed correction — which suppression alone cannot do, and which the pilot must measure.
- SCP-1006Sabarmati Agro IndustriesAlready paidPaid by the buyer's parent entity net of TDS with no invoice reference. The matching rule needs an exact amount and a known payer or invoice reference.A signal existed but did not meet the deterministic rule.
- SCP-1007Konkan Cashew ProcessorsAlready paidBuyer had already paid in cash at the merchant's office, but no observable payment signal existed.Resolve could not detect this because no bank, payment or buyer signal was available.
- SCP-1008Suryoday Ayurveda LtdAlready paidCheque handed to the sales rep; not yet deposited, so no bank or payment signal existed.Resolve could not detect this because no bank, payment or buyer signal was available.
- SCP-1028Chambal Chemicals LtdActive disputeQuality complaint was raised verbally with the sales rep and never reached any channel Resolve can see.Resolve could not detect this because no bank, payment or buyer signal was available.
| SCP-1053 | disputed | Needs review |
| SCP-1042 | paid | Paid but unmatched |
| SCP-1001 | paid | Needs review |
| SCP-1002 | paid | Paid but unmatched |
| SCP-1003 | paid | Needs review |
| SCP-1004 | paid | Needs review |
| SCP-1005 | paid | Needs review |
| SCP-1025 | disputed | Commercial dispute |
| SCP-1026 | disputed | Commercial dispute |
| SCP-1027 | disputed | Commercial dispute |
- SCP-1009 (Needs review) — Credit was for an earlier invoice (SCP-0978) of the same amount; this invoice is simply unpaid.
- SCP-1037 (Needs review) — Stalling; no real blocker.
2 · Classifier evaluation
No independently authored held-out set has been imported, so no model accuracy is shown. Held-out cases must be written after the prompt is frozen, by someone other than the classifier developer — see evaluation/HELDOUT_INSTRUCTIONS.md.
| State | n | Precision | Recall | F1 |
|---|---|---|---|---|
| Cash constrained | 3 | 100% | 100% | 1.00 |
| Commercial dispute | 3 | 100% | 100% | 1.00 |
| Follow-up active | 3 | 40% | 67% | 0.50 |
| Needs review | 16 | 69% | 56% | 0.62 |
| Paid but unmatched | 2 | 67% | 100% | 0.80 |
| Paperwork blocked | 5 | 83% | 100% | 0.91 |
| Promise to pay | 15 | 86% | 80% | 0.83 |
The frozen prompt case-classifier-v1 was run unchanged through Groq. The model was not tuned on these cases, but they were written by the same author as the prompt, so treat this as a sanity check — not generalisation.
47 of 47 development cases pass. These were written while building the parser — passing them is a regression check, not an accuracy claim.
case-classifier-v1ba530b3adeb983d8c73e…Once a held-out set has been evaluated, changing the prompt requires a new version and ideally a fresh held-out set.
| Dispute → Follow-up | HIGH (3) |
| Paperwork → Follow-up | HIGH (3) |
| Dispute → Cash constraint | HIGH (3) |
| Follow-up → Paperwork | MODERATE (2) |
| Needs review → Promise | MODERATE (2) |
| Follow-up → Needs review | LOW (1) |
No rupee values are attached to errors.
3 · Pilot metric
The prototype does not simulate this result. Whether a buyer would pay after a resolution action depends on real behaviour, which synthetic data cannot honestly model.