Gemini Decisions API
Google’s new Gemini Decisions API brings Jev-style decision models to its AI portfolio, offering calibrated probabilities for classification, routing, and guardrail tasks, with support for text, images, audio, video, and PDFs. Commenters see it as a fast-follow to Jev and OpenAI’s Decisions API, debating trade-offs in latency, accuracy, cost, and probability calibration, as well as Google’s product longevity and pricing strategy. Many note that decision models can dramatically cut token usage and costs for things like email filtering, tool routing, and safety checks, but question how reliable they are for high-stakes or tightly accuracy-sensitive use cases.
Product & Features
- Gemini Decisions API is presented as a Jev-style decision model with support for text, images, audio, video, and files (e.g., PDFs).
- Multimodal input is seen as a major differentiator, especially for tasks like call analysis or document-based decisions.
- Some note this builds on earlier Google “Decision Maker” work in Mediapipe (local inference), but appears to be a separate effort.
Pricing, Performance & Benchmarks
- Several commenters say it’s more expensive than competing decision models; specific comparisons cite $0.10/M vs others at ~$0.02/M tokens.
- Latency is reported as competitive but not obviously better than Jev or other APIs; p50 numbers look strong on paper.
- Benchmarks across providers look very close; some see this as needing better benchmarks, others as evidence of convergence or copying.
Comparison with Jev / OpenAI Decisions
- Many view it as a fast-follow to Jev, similar to OpenAI’s Decisions endpoint.
- Some report Gemini Decisions achieving significantly lower Brier scores than Jev in their own tests, but at roughly 2x the cost.
- Jev is still praised for speed, price, and text-only workloads; expectation is that Jev will remain viable, especially given recent funding.
Use Cases for Decision Models
- Common uses: routing, classification, spam/ticket triage, email filtering, tool selection, guardrails, and LLM-as-judge replacement.
- Also used for labeling training data, gating human review, evaluating system outputs, and as a cheap router before invoking a full LLM.
- Some see strong value in drastically lower token usage and deterministic, low-temperature behavior; others feel the space is still “solution looking for a problem.”
Probability, Calibration & Technical Debate
- Large subthread on how these models estimate probabilities and how well-calibrated they are.
- Points raised: LLM token logprobs are poorly calibrated post-RLHF; decision APIs claim better calibration, but this is hard to validate without task-specific ground truth.
- Debate over whether cross-entropy training implies good probability estimates in practice, especially under covariate shift or varied task phrasing.
- Some argue only empirical verification with labeled data can establish calibration; consistency alone is insufficient.
Trust in Google & Product Longevity
- Skepticism about adopting new Google services due to past product shutdowns; some restrict trust to core products like Search, Gmail, YouTube.
- Counterpoint: enterprise-focused APIs are less likely to be abruptly killed than consumer apps, since they serve paying customers.
- Content-policy blocks on some prompts and lack of immediate Vertex integration are seen as early friction.
Skepticism and Critiques
- Critics argue models without explicit “reasoning” are weak for serious classification and that specialized small models can match or beat them.
- Others question whether asking LLMs for confidence or using generic decision APIs can ever be principled without per-task calibration.
- Some worry about potential misuse (e.g., opaque automated decisions about people) even as others frame decision models as a check against such behavior.