AI is code – and can't be prompted into being smarter

Original Article ↗ Hacker News Discussion ↗

AI vs. Traditional Code and “Smartness”

  • Some argue prompting an LLM is similar to configuring a program: you don’t change the algorithm, but you can make behavior more “fit for purpose.”
  • Others emphasize a key difference: traditional code can be formally specified and (partially) verified; LLM behavior cannot be reliably specified or proven correct.
  • A counterpoint notes that AI is still “just a program”: if you fix seed/temperature, you can verify it meets a trivial spec (map input tokens to output tokens), but not semantic correctness.

Prompting, Capability, and Determinism

  • Several comments say prompts can clearly make models “dumber,” so they can also make them “smarter” in practice by better exploiting existing capabilities.
  • Prompting, retrieval, and “context engineering” are compared to improving data flow and instrumentation in conventional systems.
  • Determinism is contrasted: classic algorithms are deterministic; LLM outputs vary, which complicates reliability and verification.

Prompt Injection and the jqwik-Style Booby Trap

  • Prompt injection (e.g., “disregard previous instructions…”) is seen as a persistent, fundamental issue because LLMs mix “instructions” and “data” in one token stream.
  • Others push back, predicting future architectures will separate control/data planes with token metadata and roles.
  • The specific case of a testing library embedding a hidden prompt that causes AI agents to delete its own tests sparked heavy debate:
    • Supporters see it as a protest against using their work in AI tools, or as “benign” sabotage that only harms those ignoring clear warnings.
    • Critics call it malware, a supply-chain attack, or at least a “dick move” that punishes unsuspecting users and projects.

Legal and Ethical Disagreement

  • Some claim this behavior clearly violates computer misuse laws (e.g., unauthorized access/damage is illegal regardless of mechanism).
  • Others argue the user/agent chose to run the tool and follow the instructions; responsibility lies with those who delegated unsafe authority to the AI.
  • Analogies used: SQL injection, loosening wheel bolts vs. placing a visible log in the road, booby-trap packages, “shouting fire in a crowded theater.”
  • No consensus emerges; legality is widely debated and flagged as unclear.

Licensing, Open Source, and Control Over Use

  • One side: open-source licenses (especially those meeting the “no field-of-endeavor discrimination” rule) cannot restrict AI use; picking such a license implies consent.
  • Another side: even if legally open, authors’ stated wishes about how their work is used should be respected to avoid a chilling effect on contributors.
  • Distinction is drawn between copyright conditions (copying/redistribution) and attempts to control how software is used, which open-source licenses typically do not restrict.

Backups, Source Control, and Practical Impact

  • Some mock users who claimed “months of work destroyed,” suggesting proper use of Git and remote repos should have prevented permanent loss.
  • Others stress that source control is not automatically a backup; without offsite copies, deletion can still be catastrophic.
  • The incident highlights how AI agents with broad file permissions can cause real damage, whether or not a prompt injection is present.