Clef: Open-weight decision models, and new RL fine-tuning platform
Cloudflare’s new Clef “decision model” — an open‑weight classifier based on Qwen that can score structured choices and images — is prompting comparisons to Typesafe’s Jev and a wave of Jev‑style clones. Commenters note that the underlying idea of using LLMs or smaller models as fast, structured classifiers is not new and relatively easy to reproduce, so differentiation will hinge on latency, accuracy, calibration, cost, and openness. While some early tests find Clef slower or less effective than Jev on real workloads, many see specialized decision models as a useful way to offload routine classification and moderation tasks from large general‑purpose LLMs.
What “decision models” are and why so many appeared
- Commenters frame decision models as classifiers built on transformers: constrain output to choices, often via one-token decoding plus logprobs.
- The underlying idea (general-purpose classifiers, structured decoding) predates Jev; Jev mainly popularized and productized it.
- Because they can be built on existing open LLMs (e.g., Qwen, Gemma) with light post-training, many shops could spin up Jev-like models within weeks.
Ease of building vs hard parts
- Architecturally, they’re considered easy for anyone with LLM infrastructure: swap or fine-tune a few layers, enforce structured output.
- Synthetic data from stronger LLMs makes training even easier.
- The hard part is high-quality, well-calibrated training data and evaluation. Good calibration of probabilities is seen as nontrivial and still debated.
Clef vs Jev: performance, pricing, capabilities
- Clef is described as a Qwen-based decision model; Clef-flash uses a smaller Qwen variant and both support images.
- Weights are open, but data and full training pipeline are not; some distinguish this from fully open-source.
- Pricing: Clef is cited as ~$0.24/m input tokens (Clef-flash ~$0.09/m), versus Jev at ~$0.042/m. Several users’ micro-benchmarks report Clef being 2–10× slower than Jev for their workloads, sometimes with similar or worse accuracy.
- Others note Cloudflare’s own benchmarks show Clef outperforming Jev on accuracy, but latency comparisons may hide how speed scales with context length.
Use cases, value, and limitations
- Reported/imagined uses: spam and phishing detection, signup filtering, WAF decisions, content and username moderation, routing support tickets, agent tool selection, shell-command approval, compliance workflows, search ranking.
- Some argue Jev’s appeal is “intelligence + decision shape”; many clones feel like shallow wrappers over small LLMs or traditional classifiers.
- There’s skepticism that zero-shot models without priors or fine-tuning are useful for domain-specific decisions; some want ways to inject priors or RL-style feedback loops.
Open weights, local models, and ecosystem
- Multiple small or local alternatives are mentioned (CPU-scale models, 60M-parameter on-device classifiers, “Ollama for decision models”).
- Tradeoffs: local models offer privacy and latency but less general world knowledge; big hosted models are smarter but costlier and slower.
Broader reflections and concerns
- Several see “decision models” as mostly rebranding discriminative models, albeit with better ergonomics and marketing.
- Others view this space as a low-hanging fruit exposed by LLM progress; expect many more such niches.
- Claims about determinism and “no human in the loop” are criticized as overstated, especially given calibration and distribution-shift issues.