Anthropic's open-source framework for AI-powered vulnerability discovery

Original Article ↗ Hacker News Discussion ↗

Anthropic has released an open-source Python “harness” for using its Claude models to find security vulnerabilities in code, prompting debate over how effective and maintainable such tooling can be and whether it mainly serves to upsell a commercial “Claude Security” offering. Commenters focus on the heavy token usage and resulting costs, comparing them to traditional security engineers, SAST tools, and formal audits, and warning of an emerging “pay-to-win” or “loot box” dynamic where better-funded attackers and defenders simply brute-force more scans. Many see AI-assisted vulnerability discovery as genuinely powerful but argue that its value depends on high-quality, custom harnesses, careful triage of false positives, and a broader shift toward personal or organizational “shop jig” tooling rather than generic, one-size-fits-all products.

Project status and purpose

  • The repo is explicitly “not maintained” and positioned more as a reference harness than a supported tool.
  • Several commenters see it mainly as marketing for a commercial “Claude Security” / managed scanning offering.
  • Others frame it as a pattern library: useful to study, then recreate or adapt internally rather than adopt directly.

Costs and scalability

  • Token use per agent is high; full-codebase scans, especially with top-tier models (Opus/Mythos), are perceived as expensive.
  • Some argue you’d only run full scans periodically and diffs in CI, but others note many orgs ship every sprint, making recurring cost large.
  • Cost comparisons vary: some say a dedicated security hire might be cheaper; others cite reports claiming AI-based scanning can match many engineers, making it “pennies on the dollar” versus traditional audits.
  • External calculators are referenced showing multi-million–dollar annual token spends for 100+ dev teams if used heavily.

Effectiveness, false positives, and workflow

  • Without a well-designed harness, people report poor results and many false positives (“vibe auditing”).
  • Even with a harness, findings still need expert review and triage, otherwise developers drown in noise, similar to today’s linters and SAST.

Impact on attackers, defenders, and existing tools

  • Some see this as a potential existential threat or eventual feature for traditional SAST vendors.
  • Others note attackers can use the same models, turning vulnerability discovery into a “proof-of-work” arms race, but emphasize the bugs already existed.
  • Concerns include a flood of high-severity reports overwhelming maintainers and bug-bounty programs.

AI-generated code and security lifecycle

  • Many note it now takes far more tokens to secure code than to generate it.
  • There’s skepticism about AI vendors effectively charging first to generate “sloppy” code and then to scan/fix it.
  • Some argue models should be trained to emit secure code, but others reply that serious bugs often span large, dependency-heavy codebases beyond what can be fully reasoned about on each edit.

Business model and ecosystem debates

  • Debate over why vendors sell raw tokens vs vertical SaaS:
    • One side claims if tokens were truly magical, vendors would hoard them and dominate industries directly.
    • Others counter that selling infrastructure (like fabs or tractors) can still be optimal, and building end-user SaaS is a different, distraction-heavy business.
  • Some see security harnesses as part of a broader shift: AI companies turning domain-specific harnesses (design, security, etc.) into packaged products.

Open source harnesses and “shop jig” tools

  • Multiple alternative or similar tools are mentioned; some trip antivirus and are mainly for practitioners comfortable with that.
  • A recurring analogy likens these frameworks to “shop jigs”: custom tooling tuned to a team or individual’s workflow, often better built in-house than used off-the-shelf.
  • Commenters discuss making such harnesses portable across jobs, or shared within organizations to raise the team-wide productivity floor.
  • There’s a broader theme that AI makes bespoke tooling so cheap that generalized libraries/harnesses are increasingly used as inspiration rather than as direct dependencies.

Trust, naming, and practical concerns

  • Confusion around the GitHub account name (“Anthropics” vs “Anthropic”) recurs.
  • Some dismiss the project as “open-source glue to an LLM blob” or criticize that it’s unmaintained and closed to contributions.
  • A few express distrust of sending source code to remote LLMs or being gated by specific browsers/search engines.