Ferret: A Multimodal Large Language Model
Apple’s Ferret project showcases a multimodal large language model that can precisely refer to and reason about regions within images, hinting at future on-device AI capabilities for tasks like accessibility, visual grounding, and richer assistants. Commenters note that Ferret is a research-only finetune built atop existing open models (LLaMA, Vicuna, LLaVA) and was trained on Nvidia A100 GPUs, underscoring both Apple’s reliance on standard AI infrastructure and its focus on privacy-friendly, device-centric inference. Much of the conversation centers on what this implies for Siri, iOS 18, and the broader AI landscape, with many expecting local “good enough” models on phones to erode the advantage of cloud-based services like ChatGPT, even as true GPT‑4–level performance remains out of reach for mobile hardware in the near term.
Definition & Naming of MLLM / Ferret
- MLLM is discussed as “Multimodal Large Language Model,” handling multiple input modalities (e.g., text + images).
- Some debate over wording (MLLM vs LLMM, what is “language” if inputs are non-linguistic), but consensus sticks with established “MLLM.”
- Ferret’s name is seen as an animal-themed model with a figurative sense of “diligent searcher,” not clearly an acronym.
What Ferret Actually Is
- Paper describes Ferret as a multimodal LLM focused on spatial referring / grounding in images: pointing to arbitrary regions, shapes, or coordinates, and describing them accurately.
- Uses a hybrid region representation and a spatial-aware visual sampler; trained on a curated GRIT dataset with hard negatives to reduce hallucinations.
- Some see the abstract as buzzword-heavy; others regard the architectural tweaks (region handling) as the real innovation.
- Several note it’s a finetune on top of Vicuna/LLaMA and LLava, not a ground-up base model; excitement is tempered by this.
Licensing & Openness
- Released “for research use only,” with non-commercial data (CC BY-NC 4.0) and restrictions inherited from LLaMA, Vicuna, and GPT‑4.
- Some disappointment that this is not truly open-source; speculation that licensing constraints on underlying data/models force the research-only stance.
Apple’s AI Strategy & Moats
- Many see this as part of Apple “filling the moat” against cloud LLM providers by enabling on-device multimodal models and toolkits.
- View that LLMs are commoditizing; Apple’s real moat is hardware + installed base and deep platform integration.
- Others argue OpenAI and others can copy the approach but cannot match OS-level integration on iPhone.
On-Device LLMs, Siri, and iOS
- Strong desire for a significantly improved, LLM-powered Siri and iOS 18 “AI-focused” features, mixing local small models with cloud fallbacks.
- Opinions split on realism of GPT‑4‑class models on phones in the near term; consensus that smaller models plus server-side help are more likely.
- Some argue even a 7B model would dramatically outperform current Siri; others say 7B/13B models are still “dumb” and unreliable.
Safety, Control, and Apple’s Caution
- Several expect Apple to heavily constrain any on-device LLM due to safety, liability, and brand concerns, possibly limiting its generality.
- Concern that truly unpredictable, fully open-ended assistants are incompatible with Apple’s tightly controlled user experience.
Hardware, Training, and CUDA
- Noted that Ferret was trained on 8×Nvidia A100 GPUs; discussion that Apple still relies on standard Linux/Nvidia clusters for training.
- Debate over whether this undermines Apple Silicon’s ML branding; counterargument that training and inference are different, and data centers are a commodity domain.