AI disasters: when algorithms hurt citizens

Three ways to see it

1. 1

   Reduce a public-sector AI disaster to its first principles and you find a recurring pattern in three steps. Step one: a real efficiency goal that everyone supports. Step two: an automation that compresses many human decisions into a single algorithmic decision, with human review removed in the name of scale. Step three: a feedback signal that is too slow or too quiet to surface harm before it accumulates. The disaster is not in any one step. It is in the combination, and the combination is what civil servants are uniquely placed to spot.

2. 2

   Robodebt is the textbook case but it is far from alone. The Netherlands' SyRI welfare-fraud system was struck down by a court in 2020 for disproportionately targeting low-income immigrant neighbourhoods. The UK Post Office's Horizon system, while not strictly AI, used software-generated discrepancies to prosecute over 700 sub-postmasters, ruining lives and livelihoods, in what is now considered the largest miscarriage of justice in modern British history. Michigan's MiDAS unemployment fraud detection system falsely accused over 40,000 people of fraud in 2013 to 2015 with a 93 percent error rate. France's CAF welfare fraud scoring has been challenged in court for embedding social discrimination. Each of these had Robodebt's pattern.

3. 3

   Pakistan's analogues are easier to imagine than to admit. Imagine an AI-driven flagship at FBR that auto-issues notices to 100,000 small traders flagged by an evasion model that was never validated against the kiryana sector. Imagine a BISP exit decision based on family vehicle ownership, when the registered vehicle is actually a deceased father's that the family cannot legally re-register. Imagine an AI that ranks domicile applications and silently deprioritises Hazara names because the training data underrepresented them. Each scenario is plausible. Each is preventable, but only by an officer in the chain who recognises the Robodebt pattern early.

The why-tree

Try this with Claude

AI-edge prompt: 'Read the published findings of the Australian Robodebt Royal Commission. Distill its top ten lessons for a Pakistani additional secretary about to launch a flagship AI-driven citizen-services programme. For each lesson, give one specific paragraph the additional secretary should add to the cabinet summary, and one bureaucratic guardrail that survives a change of minister.'