Nvidia Nemotron 3.5 Lightning and NeMo Switchyard
Nvidia’s release of the Nemotron 3.5 Lightning model and the NeMo Switchyard routing library is prompting scrutiny of “smart model routing,” particularly around how it interacts with KV/prompt caching, cost, and reliability in multi-model workflows. Commenters compare Nemotron to Meta’s new 30B Muse Glimmer and Qwen models, generally finding Nvidia’s sparse MoE model fast but weaker for complex coding tasks than similarly sized dense models. A broader thread weighs the future of small, efficient local models amid RAM constraints, with some seeing them as the practical path forward and others arguing that ever-larger, frontier-scale systems will remain dominant, with open models acting as a funnel to Nvidia’s GPU ecosystem.
NeMo Switchyard & Model Routing
- Switchyard is open source but labeled “experimental,” creating confusion about production readiness.
- Docs and README barely mention caching, though the code does, leaving routing–cache strategies unclear.
- Routing strategies sometimes call extra LLMs to choose a model, which adds overhead and seems most suitable for batch tasks (classification, ASR) rather than conversational agents.
- Some see “smart model routing” as over-marketed, given cache and complexity issues; others think it’s viable when well-engineered.
Prompt / KV Caching Across Models
- Several comments clarify that “prompt caching” means KV-cache reuse, not text reuse, and KV caches are strictly model-specific.
- A common pattern: keep separate warm caches per model; when switching models, send only the diff since that model’s last turn, then extend its cache.
- Switching between two cached models slightly lowers hit rate but doesn’t fundamentally break caching.
- Cost breakdown from discussion:
- Input tokens: paid on all models that see the context.
- Output tokens: paid only on the generating model (dominant cost).
- Cached reads: paid per turn; multi-model use doesn’t increase turn count.
- Some argue the main challenge is that a good router must itself be very strong, otherwise routing errors outweigh savings.
MoE vs Dense Models (Nemotron vs Others)
- Nemotron 3.5 Lightning is sparse/MoE; Meta’s Muse Glimmer 30B and various Qwen/Gemma/Laguna dense 27–35B models are discussed as comparisons.
- Benchmarks and anecdotal tests suggest Glimmer and modern Qwen dense models are often higher quality but slower; Lightning and other MoEs are much faster due to fewer active parameters.
- For coding/agentic tasks (e.g., building a collaborative whiteboard), several people report MoE models like Nemotron Lightning and Qwen 35B MoE perform poorly, while ~30B dense models succeed reliably.
- A rule-of-thumb is cited: MoE quality ≈ dense model with parameter count around the geometric mean of total and active params.
Hardware, RAM, and Model Size
- Ongoing “ramapocalypse” is seen as pushing interest in smaller, more efficient, self-hostable models (26–35B, and hypothetical smaller MoEs for 16GB GPUs).
- Others argue history favors scaling big models and then distilling/optimizing, with small-only strategies viewed as risky.
- Discussion covers VRAM usage of KV caches at long context lengths, bandwidth bottlenecks, and practical tokens-per-second on GPUs like the 3090.
Tools, Naming, and Ecosystem Friction
- Confusion over MoE vs dense variants (e.g., Qwen “AxB” names) is common; some tools’ UIs obscure this, leading to user mis-expectations.
- One routing/benchmark site is linked; some see it as helpful, others as quasi-advertising.
Other Tangents
- Brief side threads on minimalist writing as an antidote to AI-driven info overload, and on zero-knowledge-proof identity for social media to mitigate LLM-generated misinformation.