Using GPT-4 Vision with Vimium to browse the web

Original Article ↗ Hacker News Discussion ↗

Using GPT‑4 Vision with the Vimium browser extension, an open source project shows how an AI agent can visually interpret web pages and control them via keyboard-style “click” hints, effectively browsing on a user’s behalf. Commenters see major potential for accessibility, robotic process automation, and large‑scale web scraping, but raise concerns about privacy, robustness on real‑world sites, costs, and environmental impact. The thread also explores technical refinements—like feeding accessibility trees or DOM data instead of pure screenshots—and anticipates broader effects on ad tracking, manual data entry jobs, and test automation.

Overall Reaction

  • Many commenters find the demo “insane” and exciting, especially as a proof-of-concept for multimodal agents that can drive a browser.
  • Others question direct user value today, arguing it’s slower than just typing and clicking, and mostly interesting as a glimpse of what more advanced agents could do.

Accessibility & Privacy

  • Strong optimism that GPT-4 Vision–style tools will hugely improve web accessibility and may enable an end‑to‑end AI screen reader in a few years.
  • Blind and visually-impaired users are particularly hopeful but worry about privacy when screen content is sent to large external models.
  • Open‑source vision models (e.g., CogVLM, LLaVA) are suggested as more private alternatives.

Automation, RPA & Legacy Systems

  • Many see this as the next step in Robotic Process Automation: automating data entry, copying between legacy GUIs, and brittle web workflows.
  • Stories are shared of manual copying, auto-clicker/VBA hacks, and complex payroll/tax procedures that resist traditional API/ETL solutions.
  • Some are building generic “GUI automation” layers and co‑browsing infrastructure meant to pair with GPT‑4V-like models.

Technical Approaches & Limitations

  • Debate over using pure vision vs. also feeding DOM/accessibility trees or full HTML/text; several report better reliability with text structures.
  • The system currently relies on screenshots with overlayed Vimium-style labels; suggestions include improving label styling and sending both annotated and clean images.
  • Vision API lacks built-in JSON mode/function-calling; workarounds include post-processing its output with another model.
  • CAPTCHAs are explicitly blocked by OpenAI, though some users report partial success in other setups.
  • Cost and latency are recurring concerns; token limits and image caps make large-scale or always-on use expensive.

Web Scraping, Ads, and Bots

  • People foresee powerful scraping and “anything API” layers that work from visual output, bypassing anti-scraping DOM tricks.
  • Some imagine agents that fetch content without exposing users to ads or tracking, though screenshots still contain ads.
  • Concerns are raised about bot traffic becoming indistinguishable from human browsing and about environmental/energy costs.

Risks, Skepticism & Future Impact

  • Worries include enshittification via pricing, job impacts (e.g., QA, data entry), and “Chinese Room automation” of online labor.
  • Some fear “beginning of the end” scenarios or unsupervised agents, while others see this as the “Windows 95” stage of a long adoption curve.