The legal rule that computers are presumed to be operating correctly (2022) [pdf]

Scope of the Legal Presumption

• Many see the “computers are presumed correct” rule as deeply misguided, especially given modern complex, bespoke systems.
• Others argue it’s a practical starting point: courts can’t relitigate hardware/software reliability for every digital artifact.
• Critics note that for human witnesses we don’t instruct juries to presume correctness in the same way.

Post Office Horizon Scandal as Central Example

• Horizon prosecutions allegedly relied on a presumption that system outputs were correct, effectively forcing defendants to disprove the computer.
• Known-error logs and reliability documents existed but were not disclosed; proposed reforms would require such disclosure.
• Commenters stress the core injustice: defendants couldn’t access the very evidence needed to challenge the system.

Alternatives and Reform Proposals

• Suggestions include:
  • Treating computer systems like witnesses: subject to examination, cross-checking, and credibility assessment.
  • Requiring disclosure of bug trackers, validation documents, and reliability records for systems used as evidence.
  • Possibly invalidating evidence when systems and data can’t be reproduced or audited years later.
• Some worry this raises barriers to justice; others say it rightly shifts responsibility to powerful institutions.

Computers vs Paper and General Reliability

• Debate over whether paper records are more or less error-prone than digital ones.
• Courts already handle paper fallibility via jury judgment; many feel similar skepticism should apply to software and logs.
• Multiple anecdotes about hardware faults, bit flips, and subtle bugs underline that real systems are only “statistically correct.”

Cosmic Rays, Hardware Errors, and ECC

• Extended subthread on bit flips: experiments failing to detect flips, clustered ECC errors in servers, and uncertainty over whether cosmic rays or hardware defects dominate.
• Consensus that large deployments will inevitably see rare faults; ECC and defensive design are seen as necessary in many contexts.

Broader Concerns: ML, Black Boxes, and Trust

• Extending presumptive correctness to LLMs or opaque algorithms is viewed as dangerous.
• Some emphasize societal over-trust in computers, shaped by decades of marketing and mystique.
• Hacking and insider tampering are raised as additional reasons to be skeptical of digital evidence.