Build your own decision model

Original Article ↗ Hacker News Discussion ↗

Zero-shot “decision models” like Jev are drawing intense interest as a fast, cheap way to turn LLM-style intelligence into reliable multiple-choice classifiers without training task-specific models. Commenters contrast this with traditional ML pipelines, debating whether the hype ignores long-standing issues like calibration, overfitting to benchmarks, and the need for proper evaluation, while also acknowledging that developer-friendly APIs and low latency open up many practical uses from email filtering to game logic. Several point to open-source clones and small CPU-friendly models as evidence that this capability is rapidly commoditizing and likely to become a standard building block inside larger AI systems and everyday apps.

Scope of “Decision Models” vs Traditional Classifiers and LLMs

  • Several commenters see Jev-style “decision models” as essentially classifiers with better UX: zero-shot, no training, simple API (like if/switch).
  • Frustration from ML practitioners: LLMs have long been usable as zero-shot classifiers; the new hype feels like repackaging.
  • Others argue timing and ergonomics matter: devs now have multi-step LLM pipelines and really need fast, reliable, structured choices.
  • Debate over “low data regimes”:
    • One side claims synthetic data + distillation makes low-data less relevant.
    • Others insist that real low-data settings still exist and synthetic data is not a trivial replacement.

Calibration, Benchmarks, and Methodology Concerns

  • Questioning whether post-hoc temperature tuning to match benchmarks is essentially p-hacking.
  • More experienced voices stress proper train/validation/test splits and warn that tuning prompts or calibration on test data is overfitting.
  • Skepticism about claims of “calibrated probabilities”: unclear what ground truth they’re calibrated to.

Use Cases and Practical Benefits

  • Reported applications include:
    • Email categorization, research paper tagging, dangerous command detection.
    • Token-saving pre-filters before handing text to full LLMs.
    • Game-related tools (Pac-Man/maze players, Jeopardy-style games, Oregon Trail, Nintendo 3DS testing).
    • Local autocorrect and browser-based decision helpers.
  • Key practical advantages cited: speed (often ~100–300 ms), low cost, simplicity, and ease of experimenting without big ML teams.

DIY Models, Clones, and Tooling

  • Multiple open projects replicate or extend the idea (e.g., small CPU models, browser-only models, Qwen/Gemma/ModernBERT-based variants).
  • Some argue fixed, well-defined tasks are better served by small, ad-hoc classifiers once enough labeled data exists.
  • Others foresee an ecosystem where general decision models bootstrap tasks, then hand off to specialized classifiers.

Terminology, Hype, and Valuation

  • Debate over whether “decision model” and “system one model” are meaningful distinctions or marketing rebrands.
  • Some see genuine novelty in broad zero-shot text classification; others liken it to glorified constrained generation.
  • Concerns are raised about very high valuations and possible AI bubble dynamics, given that many devs can now build similar models.