Proposed amendment to legal presumption about the reliability of computers
A long-running UK scandal over the Post Office’s Horizon IT system, which wrongly showed financial shortfalls and helped send hundreds of sub‑postmasters to prison, is prompting calls to change how courts treat computer-generated evidence. Commenters describe how UK law has effectively presumed computers to be reliable, making it hard for defendants to challenge flawed or manipulated systems, and argue this undermines the principle of “innocent until proven guilty.” Many advocate stronger regulation, transparency, and professional accountability for critical software, especially as similar issues loom with AI and other opaque digital systems.
Background: UK Post Office / Horizon scandal
- Fujitsu’s Horizon system for UK Post Offices produced incorrect balances, leading to thousands of prosecutions, convictions, financial ruin, and suicides over ~15 years.
- Bugs were numerous and fundamental (transactions, distributed systems, lack of proper ledger/accounting design, Forex mis-handling).
- Management and Post Office prosecutors knew of bugs and remote “backdoor” interventions, yet maintained the system was robust, hid evidence, and continued prosecutions.
- Scandal is framed as both a software failure and, more importantly, a political/legal/ethical cover‑up and abuse of power.
Legal presumption that computers are reliable
- UK law evolved from presuming mechanical instruments correct, to briefly requiring proof of computer correctness (1984), then back to presumption of correctness (1999) after a hearsay review.
- Several argue this effectively shifted burden of proof onto defendants, clashing with “innocent until proven guilty.”
- Others note the intent was to avoid endless challenges (speed cameras, tax, tickets) and that courts still can question computer evidence, but didn’t in Horizon.
- Proposed amendment is seen as an improvement but criticized as too weak if prior government “certification” still creates a strong presumption.
Responsibility: engineers vs management vs justice system
- One view: primary blame lies with management, executives, and prosecutors who ignored reports, suppressed evidence, and lied; software bugs are inevitable.
- Counter‑view: both management and engineers share responsibility; basic properties like idempotent financial transactions were missing, and some technical witnesses allegedly misled courts.
- Many emphasize this was ultimately a justice‑system failure: courts and prosecutors treated computer output as near‑infallible evidence.
Regulation, “engineering” status, and liability
- Strong current for treating critical software like civil/aerospace engineering: licensing, standards, personal/professional liability, and insurance for safety‑critical and financial systems.
- Others warn this could entrench incumbents, stifle innovation, shift blame onto individual coders, and be hard to design in a field lacking stable standards.
- Debate over regulating the “engineer” title, mandating certified components, and tiered accreditation for critical vs trivial systems.
Transparency, open source, and evidence
- Calls for: open‑sourcing publicly funded systems, stronger audit trails (full calculation steps, logs), mandatory disclosure of known bugs, and security/process documentation when software evidence is used in court.
- Some argue that without access to source or rigorous independent audits, any right to challenge software in court is hollow.
AI and future risks
- Several draw parallels to generative AI: fear that courts or institutions might presume AI outputs reliable, despite non‑determinism and hallucinations.
- Widespread agreement that presuming correctness of opaque, complex systems is dangerous, especially as they gain legal or administrative power.