Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

Original Article ↗ Hacker News Discussion ↗

Small open-weight “decision models” like Jeff aim to replicate Jev’s zero-shot classification capabilities while running cheaply and quickly on local hardware, using fine‑tuned Qwen and Gemma models that output calibrated option probabilities in a single pass. Commenters weigh Jeff’s strengths in speed, offline deployment, and task-specific fine-tuning against Jev’s stronger out‑of‑the‑box accuracy and proprietary data advantage, especially on harder reasoning benchmarks. Much of the debate centers on when to use these models versus traditional classifiers or full LLMs, with trade‑offs in latency, cost, required training data, and the importance of true zero‑shot generalization.

Overview of Jeff Decision Models

  • Jeff provides small open-weight decision models (0.8B and 2B) fine-tuned from Qwen3.5 and Gemma for zero-shot classification.
  • Models take a situation plus options and output calibrated probabilities in one pass, with no text generation.
  • Key goals: very low latency (~30 ms for 0.8B on M4 Max), local deployment, Jev-compatible API, and easy further fine-tuning.

Performance, Benchmarks, and Games

  • On a mixed benchmark panel Jeff 2B roughly matches published Jev scores overall, but lags badly on multi-step reasoning (e.g., BBH, JevBench hard tier).
  • For classification-style tasks (e.g., Financial PhraseBank, RAGTruth), Jeff reports scores above Jev.
  • Some users report strong performance; others find Jeff far below Jev (e.g., 70% vs 94% accuracy on their own job-ad classifier).
  • Game tests (Doom, Frogger, Pac-Man) are used as fun zero-shot evaluations; Jeff 0.8B matches or exceeds hand-coded bots in some scenarios, but others share much better Doom results using tiny bespoke classifiers.

Zero‑Shot vs Fine‑Tuning vs Classic ML

  • Strong debate on value of zero-shot generalists (Jev/Jeff) versus:
    • Fine-tuned small models (e.g., ModernBERT, Gemma).
    • Simple pipelines: embedding model + logistic regression / SVM / MLP, or even bag‑of‑words + linear models.
  • Several claim these classic setups match or beat Jev/Laya on many standard text classification datasets and run in <1 ms.
  • Counterpoint: zero-shot removes data, ML expertise, and infra barriers, enabling quick trials across many use cases.

Local Deployment, Cost, and Product Positioning

  • Many appreciate that Jeff is Apache-licensed, runs locally, and can be fine-tuned quickly on a single GPU.
  • Others argue renting GPUs or using cheap APIs (e.g., Jev pricing) is more practical at scale.
  • Discussion on whether needing fine-tuning changes the product category versus Jev’s promise of “good enough” zero-shot.

Jev-Style Models vs General LLMs

  • Some argue standard LLMs with structured outputs, logprobs, and constrained decoding can substitute Jev, but:
    • Jev/Jeff-style models are seen as dramatically faster/cheaper for classification-like tasks.
  • Multiple comments stress that these are “System 1” judgment models, not planners: great for fast decisions, weak at deep reasoning.

Architecture, Alternatives, and Concerns

  • Speculation that Jev is an encoder-only or BERT-like model, trained with heavy synthetic data; details are not public.
  • Other Jev-like projects (e.g., Laya, small Doom agents) are discussed as task-specific, fine-tune-heavy alternatives.
  • Some worry such fast classifiers are ideal for surveillance and autonomous weapons, and see their optimization and spread as concerning.