Phi-2: The surprising power of small language models

Original Article ↗ Hacker News Discussion ↗

Microsoft’s new Phi‑2 model highlights how “small” language models with 2.7B parameters can rival or beat larger systems like Mistral 7B and Gemini Nano on many benchmarks, while still being just about small enough to run locally with quantization. Commenters probe its training recipe—1.4T tokens with heavy use of high‑quality synthetic data, possible test leakage, and a modest reported training cost—as an example of how data quality and distillation from larger models may matter more than sheer parameter count. Others focus on practical limits: the roughly 10GB weight size and research‑only license, questions about commercial use and redistribution, and what advances like Phi‑2 mean for on‑device AI, interpretability research, and the wider competitive landscape in AI tooling.

Model size, performance, and deployment

  • Phi‑2 has 2.7B parameters and reportedly outperforms some larger models (e.g., Mistral 7B, Gemini Nano 2) on benchmarks in the post.
  • Stored model is ~10GB (32‑bit weights); commenters note this implies ~12GB+ VRAM for comfortable use, but say 4–5 bit quantization is common and keeps degradation modest.
  • Some readers were excited by a “small model” that could run on consumer hardware; others note that “size” is usually counted in parameters, not bytes, which complicates comparisons with heavily quantized models like Gemini Nano.

Training cost, data, and distillation

  • Training reportedly took 14 days on 96 A100 GPUs; one estimate pegs this at roughly car-level cost, seen as democratizing.
  • Others argue data generation (likely from GPT‑3.5/4, per earlier Phi papers) is much more expensive than the training itself.
  • Discussion centers on synthetic, “textbook-quality” data and “teacher–student” distillation: large models generating high‑quality training sets for smaller ones.
  • Some see this as the start of a positive feedback loop for rapid model improvement; others doubt models trained largely on LLM outputs can surpass their teachers in generality.

Benchmarks, leakage, and evaluation

  • Some express suspicion that benchmark test data may have leaked into training.
  • Others point out the paper claims de‑duplication, test‑set removal, and use of private evaluation sets.
  • One commenter notes that embedding similarity search alone is insufficient to rule out leakage.

Access, licensing, and local use

  • Initial confusion about access via Azure AI Studio and authentication issues; later clarified that weights are also on Hugging Face.
  • License is “research use only” and forbids redistributing weights, which sparked debate about enforceability and copyright status of model weights.
  • Commercial usability is raised but not clearly answered in the thread (marked as unclear).

Comparisons to human learning

  • Several comments compare 1.4T training tokens to rough estimates of ~30M “token-equivalents” for human infants.
  • Others argue this is an apples‑to‑oranges comparison: babies get rich multimodal, interactive, and feedback‑driven input, not just text.
  • Debate over whether such comparisons are insightful or misleading; some see them as useful for exploring “systems that learn language,” not human cognition per se.

Theoretical questions: Turing‑completeness and minimal models

  • One subthread asks about the smallest model that can implement basic arithmetic and control flow to be Turing‑complete.
  • Responses disagree: some say small transformers plus an external loop/memory could trivially be Turing‑complete; others stress that a plain, single‑pass LLM with fixed context is not.
  • General consensus: Turing‑completeness is mostly orthogonal to practical LLM capability and not necessarily desirable.

Ethics, competition, and mental health tangent

  • One commenter denounces perceived anticompetitive clauses by major AI/cloud vendors and calls for boycotts; another questions whether this behavior meets legal thresholds for monopoly or conspiracy.
  • A long tangent emerges where several participants speculate about that commenter’s mental health and urge seeking professional help; others push back on public “diagnosis” while supporting destigmatized discussion of mental health.

Research value and safety

  • Multiple comments highlight that a strong 2.7B‑parameter model is an excellent testbed for:
    • mechanistic interpretability,
    • safety/alignment techniques,
    • cheap fine‑tuning (e.g., LoRA on consumer GPUs),
    • and experimentation with specialized/domain‑specific models.
  • Some mock the notion of a “safety score,” but others see safety and interpretability research on small powerful models as a major upside.