AI can't stop making up software dependencies and sabotaging everything

Trust, Responsibility, and “Black Boxes”

• Strong consensus that AI assistants must not be blindly trusted, especially for safety/security-relevant work.
• Several people liken LLMs to unreliable junior developers: useful, but everything must be reviewed, tested, and staged.
• Others push back that humans are also “black boxes,” but replies emphasize existing human processes (code review, tests, CVEs) vs the hype-driven trust often placed in AI.
• A recurring theme: if you ship AI-generated code or dependencies without understanding them, the failure is fundamentally a process/discipline problem.

Vibe Coding and Dependency Culture

• “Vibe coding” (accepting whatever the AI suggests) is widely criticized as dangerous and lazy, especially in today’s dependency-heavy ecosystems.
• Even “trivial” projects (e.g., personal sites) can become security-sensitive if they pull in crypto miners, exfiltration code, or shady packages.
• Several commenters broaden “safety/security critical” to nearly all production software: dependency choices are almost always a security decision.
• Some go further, calling dependency-driven development an anti-pattern and blaming modern package ecosystems for fragility; others note total self-reliance (“write everything yourself”) is unrealistic.

Supply Chain Security and Mitigations

• Many see AI as a new amplifier for an existing problem: package supply-chain insecurity and lack of trustworthy dependency models.
• Proposed mitigations include:
  • Capability-based security or effects systems restricting what dependencies can do.
  • Sandbox/process isolation, WebAssembly components, OS-level tools like pledge, and revived permission systems (à la Java Security Manager).
  • Curated or allow-listed package sets, company-maintained whitelists, and social code review systems.
  • SBOMs, provenance/attestation, and automated scanning—though people warn about cost, false positives/negatives, and liability.
• Skeptics argue that registry-side malware scanning or full curation is expensive, hard to keep current, and may give a false sense of safety.

Nature and Value of Hallucinations

• Multiple comments argue that “hallucination” is not an edge case but the primary mode of generative models: statistical mashups that often miss unstated rules.
• Debate over terminology: some prefer “hallucination,” others “lying,” “bullshit,” or analogies like Runge spikes and cargo-culting.
• A minority sees upside: hallucinated APIs often mirror patterns across many libraries and might point to abstractions that “should exist,” useful for design inspiration—if clearly flagged as speculative.
• Others counter that in practice these hallucinations create concrete risk: imaginary packages, internal-only endpoints, bogus CLI flags, and wrong options that can break systems or open attack surfaces.

Real-World Experiences Using AI for Code

• Many anecdotes describe LLMs:
  • Inventing PowerShell cmdlets, Python packages, config keys, and CLI options, complete with fake “documentation.”
  • Looping between incompatible or incorrect fixes and reintroducing earlier errors.
  • Persistently suggesting nonexistent APIs despite explicit instructions not to.
• Coping patterns:
  • Treating AI as a code-snippet generator for small, local problems, not whole features.
  • Restarting sessions rather than iteratively “fixing” prior outputs in one conversation.
  • Enforcing dependency review (history, popularity, behavior) just as for human PRs.
• Some report that newer models are better, especially on well-known stacks and long-context codebase Q&A; others find agentic IDEs intrusive, unreliable, and harmful to focus and skill development.

Broader Reflections on Software Practice

• Several commenters lament that instead of investing in stronger abstractions, formal methods, and better languages, the industry is embracing probabilistic code generators that demand even more verification.
• Others argue that delegating low-level coding to LLMs effectively forces developers up the abstraction ladder (interfaces, permissions, architecture)—if they’re capable of thinking that way.
• There’s concern that over-reliance on AI will stunt junior developers’ growth: they’ll ship more features faster but understand less, making them less able to recognize or correct AI mistakes when it really matters.