The legal rule that computers are presumed to be operating correctly (2022) [pdf]
Courts in the UK and elsewhere often operate on a legal presumption that computer systems are functioning correctly when their outputs are used as evidence, a stance critics argue is dangerously out of step with how complex and error-prone software actually is. Commenters point to scandals like the Post Office Horizon case, hardware faults, cosmic-ray-induced bit flips, and the ease of tampering or misdesign as reasons why digital records should be scrutinized at least as much as human testimony or paper documents. Several suggest reforms such as requiring disclosure of bug logs, validation evidence, and system design details, and warn that extending the same presumption of reliability to opaque systems or AI models would be especially problematic.
Scope of the Legal Presumption
- Many see the “computers are presumed correct” rule as deeply misguided, especially given modern complex, bespoke systems.
- Others argue it’s a practical starting point: courts can’t relitigate hardware/software reliability for every digital artifact.
- Critics note that for human witnesses we don’t instruct juries to presume correctness in the same way.
Post Office Horizon Scandal as Central Example
- Horizon prosecutions allegedly relied on a presumption that system outputs were correct, effectively forcing defendants to disprove the computer.
- Known-error logs and reliability documents existed but were not disclosed; proposed reforms would require such disclosure.
- Commenters stress the core injustice: defendants couldn’t access the very evidence needed to challenge the system.
Alternatives and Reform Proposals
- Suggestions include:
- Treating computer systems like witnesses: subject to examination, cross-checking, and credibility assessment.
- Requiring disclosure of bug trackers, validation documents, and reliability records for systems used as evidence.
- Possibly invalidating evidence when systems and data can’t be reproduced or audited years later.
- Some worry this raises barriers to justice; others say it rightly shifts responsibility to powerful institutions.
Computers vs Paper and General Reliability
- Debate over whether paper records are more or less error-prone than digital ones.
- Courts already handle paper fallibility via jury judgment; many feel similar skepticism should apply to software and logs.
- Multiple anecdotes about hardware faults, bit flips, and subtle bugs underline that real systems are only “statistically correct.”
Cosmic Rays, Hardware Errors, and ECC
- Extended subthread on bit flips: experiments failing to detect flips, clustered ECC errors in servers, and uncertainty over whether cosmic rays or hardware defects dominate.
- Consensus that large deployments will inevitably see rare faults; ECC and defensive design are seen as necessary in many contexts.
Broader Concerns: ML, Black Boxes, and Trust
- Extending presumptive correctness to LLMs or opaque algorithms is viewed as dangerous.
- Some emphasize societal over-trust in computers, shaped by decades of marketing and mystique.
- Hacking and insider tampering are raised as additional reasons to be skeptical of digital evidence.