Show HN: Skyvern – Browser automation using LLMs and computer vision
A new open-source project called Skyvern uses large language models and computer vision to control web browsers, aiming to automate tedious workflows like form-filling, RPA tasks, and testing without site-specific scripting. Commenters are intrigued by its potential for accessibility, virtual assistants, and enterprise automation, but raise concerns about cost, CAPTCHA bypassing, spam, and the broader trend of relying on brittle GUI automation instead of stable APIs. The creators describe plans for API-first usage, cost optimizations, support for more models, and features like caching or generating reusable automation logic, while explicitly avoiding abusive use cases.
Overall Use Cases and Positioning
- Tool is seen as an LLM + vision-based browser automation / RPA system, not just a testing tool.
- Highlighted use cases: repetitive form-filling (e.g., government sites, insurance quotes), data entry in enterprise workflows, collecting tax documents, and potentially pen‑testing and virtual assistants.
- Some see strongest value where a small number of complex workflows are repeated at scale (e.g., accountants handling many clients).
Cost, Performance, and Architecture
- Current cost estimates: roughly $0.05–$0.20 per page; one user incurred a few dollars on small experiments, finding it “pricey”.
- Complaints that calling GPT‑4V on every step is expensive and slow; several suggest using LLMs only to discover selectors or generate scripts, then replay cheaply.
- Project authors mention: max-step limits, per-step cost visibility, future “memorization/caching” of workflows, and an LLM router with support for models like Claude 3 and local LLaMA.
- Technically, it combines DOM parsing with vision and heuristics rather than pure pixel-based approaches, which are reported as unreliable.
Spam, Abuse, and CAPTCHAs
- Strong concern that tools like this escalate botting and spam (forms, voting, credit card testing).
- CAPTCHAs are already often bypassed via cheap human‑in‑the‑loop or AI; this project can solve some CAPTCHAs but that part is not open-sourced and is “gated” by use case.
- Proposed mitigations discussed: charging money/credit-card verification, manual onboarding, and even eID‑style rate‑limited identity tokens—though some argue identity binding doesn’t solve spam.
Impact on the Web, APIs, and Jobs
- Some fear a cat‑and‑mouse arms race: more DRM, anti‑bot measures, unscreenshotable UIs, and a web less friendly to humans or accessibility.
- Others frame it as a “unified API” where agents can use existing UIs when formal APIs are absent, including government sites that don’t want to expose APIs.
- There’s speculation that such automation will accelerate back-office job automation.
Testing, Scraping, and Code Generation
- Multiple people want this for frontend/e2e testing: asserting conditions, generating Cypress/Playwright/Selenium code, and producing reusable scrapers.
- A recurring suggestion: treat the LLM as a “compiler” that outputs automation code or an action plan, only re-running it when UIs change, instead of using LLMs at runtime on every run.
Ethics, Security, and Licensing
- Serious concern about automating medical billing/EHR workflows due to fraud and loss of human accountability.
- Interest in using it defensively (e.g., password rotation flows) and in studying prompt‑injection and security models.
- AGPLv3 licensing is a blocker for some enterprises, despite appreciation for open-sourcing.