Show HN: Open-source macOS AI copilot using vision and voice

Original Article ↗ Hacker News Discussion ↗

An open-source macOS “copilot” app that uses Electron, OpenAI’s Vision and voice APIs, and screen capture to help users interact with whatever is on their screen is drawing both enthusiasm and skepticism. Supporters like its practical utility for tasks such as debugging, music production, and learning complex software, and appreciate that it’s easy to extend or adapt to local models. Critics raise concerns about Electron’s performance, reliance on OpenAI’s cloud (cost, privacy, and corporate policy issues), and lack of text-only input, prompting suggestions for native Swift implementations, local multimodal models, and tighter OS integration.

Overall reception

  • Many commenters find the app “very cool” and a good “Show HN”–style MVP, especially for learning complex software faster (e.g., Ableton Live).
  • Others are skeptical of real usefulness, noting some demos look generic and Eliza-like, with slow responses and limited apparent understanding of content.
  • Several praise the concept of “LLM as interface” and foresee voice/vision assistants becoming normal across devices and operating systems.

Technology stack (Electron vs native)

  • Some criticize the use of Electron for a macOS-specific tool, citing performance and OS integration concerns.
  • Others argue Electron is a pragmatic choice for a first project and for quick MVPs; stack choice is framed as secondary to shipping and learning.
  • Suggestions include Swift/SwiftUI, AppKit, or alternatives like Tauri for smaller, more native-feeling apps.
  • There is mention that Windows support might be possible with relatively few code changes.

Privacy, security, and corporate use

  • Multiple comments warn that sending arbitrary screenshots to a third-party cloud (OpenAI Vision API) is unacceptable in many corporate or regulated environments.
  • Others counter that:
    • This risk is similar to cloud-based screen-sharing tools.
    • Users who can configure API keys should understand offsite data risks and corporate policies.
    • OpenAI claims API data is not used for training, though trust in that claim is debated.
  • Some projects mentioned implement PII-scrubbing as a mitigation strategy.

OpenAI dependence vs local models

  • Several commenters dislike reliance on OpenAI and remote vision models and want:
    • A fully local, offline version using open models (e.g., LLaVA, Whisper, local multimodal setups).
    • An OpenAI-compatible local API, so the app can just point to localhost.
  • It’s noted that, because the project is open source, calls to OpenAI can in principle be swapped out for self-hosted models, though this is nontrivial for vision.

Features, UX, and extensions

  • Popular feature requests:
    • Text input/output instead of, or in addition to, voice (for quiet environments or devices without mics).
    • Streaming text responses rather than only TTS.
    • Better window behavior (auto-hide, configurability).
    • Cost/prompt estimators due to Vision API pricing and rate limits.
  • The author adds a text-input mode in response to feedback.
  • Ideas proposed but not (yet) implemented:
    • Integration with macOS accessibility APIs to read text or perform actions.
    • Letting the agent click/type and manipulate the UI directly via a driver.
    • Context-aware prompts based on current app, terminal history, or OCR.
    • Using it as a model for car/real-world assistants combining maps, audio, and vision.

Comparisons and related tools

  • Commenters reference similar tools:
    • Command-line AI assistants for terminals.
    • Local voice/vision assistants.
    • macOS GPT clients and web-based wrappers.
  • Some see this project as a prototype for OS-level copilots likely to be shipped by major vendors in the future.