Ask HN: Is anyone doing anything cool with tiny language models?

Hacker News Discussion ↗

Tiny language models—ranging from tens of millions to a few billion parameters—are increasingly being used for focused, on-device tasks where privacy, latency, and cost matter more than raw capability. Commenters describe applications such as code autocomplete, SMS spam trolling, on-device Excel-style classification, medical triage support, UI automation, browser-based cookie-banner detection, and game NPC dialog, often fine-tuned for narrow domains. A recurring theme is that small, specialized models can outperform or be more practical than large general models for classification, rewriting, and simple reasoning, especially when they run locally on consumer hardware.

Overview

  • Thread surveys many real-world uses of “tiny” and small LMs (tens of millions to a few billion params), mostly running locally on CPUs or modest GPUs.
  • Strong interest in privacy, low latency, and narrow, specialized tasks rather than general chat.

Training & Deployment of Tiny Models

  • 0.1B–0.125B models (GPT‑2 scale, JetBrains’ autocomplete) are reported as trainable on consumer hardware; rules of thumb like ~8× params in VRAM are mentioned.
  • Examples include training ~0.125B models on ~1B tokens in minutes on rented H100s using NanoGPT variants.
  • Disk sizes around ~70MB zipped per model are seen as small enough for static web apps and browser-based inference (WebGPU, transformers.js, web-llm).
  • Several projects use llama.cpp, Ollama, or custom C++ microservices to self-host small models with low latency.

Practical Use Cases

  • Developer tooling: code completion, bash/sed/awk one-liners, git commit messages, variable naming, address and job parsing, cookie-banner detection, Excel-like formula suggestion/repair, email or SMS agents, and structured extraction (e.g., nutrition labels, job attributes).
  • Productivity and research: Excel add‑ins and small classifiers to filter or prioritize scientific abstracts, detect urgent maternal-health messages, or categorize playlist songs/titles.
  • System integration: wake-word detection on microcontrollers, Android text firewalls, on-device “activity analysis” assistants, robot/voice command interfaces, and Raspberry Pi personal RAG assistants.

Creative & Playful Uses

  • Story generators on tiny displays, endless “office gossip” audio streams, Magic card generators, Tic‑Tac‑Toe opponents, NPC dialogue and bargaining in games, and persona-based rewriting (e.g., hiding personal style or imitating fictional characters).
  • Local LMs used to anonymize code/questions before sending to large remote models, then reinsert real identifiers afterward.

Capabilities, Limits & Skepticism

  • Consensus that small, fine‑tuned models excel at narrow, text-classification or transformation tasks (summarization, paraphrasing, filtering, translation in some cases).
  • Reported weaknesses: complex logic, math, temporal expressions, truly diverse creativity, and safety-critical reasoning.
  • Some argue specialized small models are more useful than giant generalists; others worry comparisons to legacy >100B models can be misleading.
  • Noted gaps: fine‑tuning expertise, good training data, and a way to package and share small, task-specific models (plus prompts and parsers) cleanly.

Privacy, Security & Ethics

  • Strong focus on keeping data on-device (HIPAA-sensitive EHR queries, SMS and call filtering, private editors).
  • Applications to detect prompt injection, rewrite toxic or abusive text, or classify unethical stock suggestions.
  • Some concern that playful uses (e.g., engaging spam) may inadvertently strengthen adversaries or raise new privacy risks.