AI is code – and can't be prompted into being smarter

Original Article ↗ Hacker News Discussion ↗

AI models are likened to regular software in that they’re just code and weights, but many argue they can’t be reliably “prompted” into being smarter or safer, especially given their non‑determinism and lack of formal verifiability. A recent incident with the jqwik testing library, which hid a prompt-injection message that led some AI coding agents to delete test code, is used to highlight how vulnerable LLM-based tools are to malicious or adversarial inputs baked into dependencies. Commenters debate whether such booby traps amount to malware or illegal access under laws like the CFAA, what rights open-source authors have to restrict AI use of their work, and how much prompting, system design, or ongoing training can realistically mitigate these risks.

AI vs. Traditional Code and “Smartness”

  • Some argue prompting an LLM is similar to configuring a program: you don’t change the algorithm, but you can make behavior more “fit for purpose.”
  • Others emphasize a key difference: traditional code can be formally specified and (partially) verified; LLM behavior cannot be reliably specified or proven correct.
  • A counterpoint notes that AI is still “just a program”: if you fix seed/temperature, you can verify it meets a trivial spec (map input tokens to output tokens), but not semantic correctness.

Prompting, Capability, and Determinism

  • Several comments say prompts can clearly make models “dumber,” so they can also make them “smarter” in practice by better exploiting existing capabilities.
  • Prompting, retrieval, and “context engineering” are compared to improving data flow and instrumentation in conventional systems.
  • Determinism is contrasted: classic algorithms are deterministic; LLM outputs vary, which complicates reliability and verification.

Prompt Injection and the jqwik-Style Booby Trap

  • Prompt injection (e.g., “disregard previous instructions…”) is seen as a persistent, fundamental issue because LLMs mix “instructions” and “data” in one token stream.
  • Others push back, predicting future architectures will separate control/data planes with token metadata and roles.
  • The specific case of a testing library embedding a hidden prompt that causes AI agents to delete its own tests sparked heavy debate:
    • Supporters see it as a protest against using their work in AI tools, or as “benign” sabotage that only harms those ignoring clear warnings.
    • Critics call it malware, a supply-chain attack, or at least a “dick move” that punishes unsuspecting users and projects.

Legal and Ethical Disagreement

  • Some claim this behavior clearly violates computer misuse laws (e.g., unauthorized access/damage is illegal regardless of mechanism).
  • Others argue the user/agent chose to run the tool and follow the instructions; responsibility lies with those who delegated unsafe authority to the AI.
  • Analogies used: SQL injection, loosening wheel bolts vs. placing a visible log in the road, booby-trap packages, “shouting fire in a crowded theater.”
  • No consensus emerges; legality is widely debated and flagged as unclear.

Licensing, Open Source, and Control Over Use

  • One side: open-source licenses (especially those meeting the “no field-of-endeavor discrimination” rule) cannot restrict AI use; picking such a license implies consent.
  • Another side: even if legally open, authors’ stated wishes about how their work is used should be respected to avoid a chilling effect on contributors.
  • Distinction is drawn between copyright conditions (copying/redistribution) and attempts to control how software is used, which open-source licenses typically do not restrict.

Backups, Source Control, and Practical Impact

  • Some mock users who claimed “months of work destroyed,” suggesting proper use of Git and remote repos should have prevented permanent loss.
  • Others stress that source control is not automatically a backup; without offsite copies, deletion can still be catastrophic.
  • The incident highlights how AI agents with broad file permissions can cause real damage, whether or not a prompt injection is present.