Iteration and evaluation: accuracy, latency, cost, satisfaction

Three ways to see it

1. 1

   Four metrics cover most LLM systems. Accuracy: does the output match the expected answer? For an FBR summariser, accuracy is judged by an expert review of whether each summary correctly identifies the flagged irregularities. Latency: how long from request to response? For interactive tools, 95th percentile under three seconds is the target. For batch tools, the limit is the working-day cycle. Cost: PKR per call, tracked daily, with alerts when it drifts. Satisfaction: a one-tap rating from the user after each output. Each metric is necessary; none is sufficient on its own. A fast cheap tool with low accuracy is useless. A perfect tool that costs ten lakh a day is unaffordable.

2. 2

   Building the eval set. Start with 50 to 100 real inputs that span the variety your system must handle. For the FBR case: ten happy paths (clear filer/non-filer cases), ten edge cases (partial data, ambiguous amounts), ten adversarial cases (mixed-language, malformed PDFs), ten high-impact cases (high-revenue filers where the wrong answer is expensive), ten Urdu-dominant cases. For each input, capture the expected behaviour as either a gold-standard reference output written by an expert, or as a checklist of must-have facts. The set is your living artefact; it grows every time a real user finds a case the system fails on.

3. 3

   Three evaluation methods. Manual review by a domain expert is the gold standard but expensive and slow. Pairwise model judging is the middle ground: a second LLM compares the output to the reference and scores it. Cheap, fast, biased in ways you must measure. Production metrics are the third: thumbs-up/down ratios, time-to-completion changes, downstream error rates. All three should be combined. Build a small dashboard that surfaces all four metrics across the eval set and across production for every prompt or model change. The dashboard becomes the artefact for monthly governance meetings.

The why-tree

Try this with Claude

AI-edge prompt to try with Claude or ChatGPT: 'Design an evaluation harness in Python that runs my 50-input CSV through Claude Sonnet, captures the four metrics (accuracy via LLM judge, latency in ms, cost in PKR, user thumb), and writes a single Markdown report comparing the current run to the previous. Output the code plus the Markdown template. Comments in Urdu.' Run, audit, refine.