Codestral Mamba

Original Article ↗ Hacker News Discussion ↗

Mistral’s new Codestral Mamba code model, built on the Mamba2 architecture, is prompting interest both for its technical claims—linear-time inference and very long context windows—and for how it stacks up against transformer-based coding assistants like DeepSeek, Copilot, and Claude. Commenters outline the rapid evolution of local LLM tooling (llama.cpp, Ollama, vLLM, ExLlama, GPT4All) and how to run models on consumer hardware, while noting that support for Mamba2 in popular backends is still emerging. There is strong demand for practical integrations such as VS Code extensions and clear API pricing, with many seeing high-quality, possibly local, code assistants as a major commercial opportunity and a way to reduce dependence on proprietary cloud offerings.

Local LLMs and Tooling

  • Many recommend Ollama as the easiest way to run models locally; pairing it with Open WebUI (via Docker) gives a friendly browser UI.
  • Others prefer more feature-rich UIs like text-generation-webui.
  • Alternative entry points: llamafile (single-binary), GPT4All, and direct use of llama.cpp, Exllama, vLLM, or TensorRT-LLM depending on hardware.
  • Hugging Face’s open-llm-leaderboard and community tools like the “gpu_poor” site are cited for model rankings and hardware sizing.

Model Sizes, Hardware, and Quality

  • 7B models run on modest hardware and are seen by some as “very bad” beyond simple tasks, but others argue they’re remarkably capable for summarization and everyday help given their size.
  • 24GB GPUs can run Llama 3 70B in quantized form, though speed and quality claims conflict. Gemma 2 27B is suggested as a strong fit for 24GB VRAM.
  • Apple Silicon’s unified memory makes 7B models feasible but slower than dedicated GPUs.

Open-Source LLM Ecosystem (High-Level History)

  • Thread recaps a short history from early GPT‑2 era to LLaMA, LLaMA 2, Mistral, Mixtral, Llama 3, and Gemma 2, with quantization and CPU/GPU support (llama.cpp, bitsandbytes, Exllama, vLLM, TensorRT‑LLM) driving local adoption.
  • Wrappers like GPT4All and Ollama significantly lowered the barrier to entry.

Codestral Mamba, Mamba Architecture, and Benchmarks

  • Excitement centers on a high-profile Mamba2 code model competing with Transformers while offering linear-time inference and 256k-token context.
  • Some note that DeepSeek models match or beat Codestral Mamba on several benchmarks and that one table mis-highlights results; CodeGeeX4 is said to surpass them “on paper” but isn’t included.
  • Links to primers and explainers on Mamba/state-space models are shared; non-experts find good video and text resources.

IDE and Editor Integration

  • For VS Code/IntelliJ, Continue.dev and Sourcegraph Cody are popular; they can use Ollama or cloud APIs, but Mamba2 isn’t yet supported in llama.cpp, so Codestral Mamba isn’t available via Ollama.
  • Other options: codegpt.co plugins, TabbyML (with older Codestral), and custom editor scripts (e.g., Vim FIM completion via Ollama).

Closed vs Open Code Assistants & UX

  • Open coding models mentioned: CodeLlama, Codestral, DeepSeek-Coder V2, CodeGemma, CodeQwen, WizardCoder, CodeGeeX4; consensus is they still lag GitHub Copilot–class services overall, though some local setups work well.
  • Users report mixed but often strong experiences with Claude 3.5 Sonnet for coding and project-scale help; many feel it outperforms GPT‑4o in practice despite benchmarks.
  • Several dislike Copilot’s perceived decline in quality and explore alternatives like Supermaven, but pricing and token-based limits cause confusion and frustration.

Context Windows and Long-Context Behavior

  • Mamba’s 256k tested context is praised, though some question why it’s lower than Gemini’s claimed million-token range.
  • Participants discuss that newer models handle long context better than older “lost in the middle” behavior, but best practice remains to keep key instructions at the beginning or end.

Miscellaneous

  • Some criticize the product page’s Cleopatra/mamba joke as historically inaccurate and in poor taste.
  • Others think Mistral is missing a revenue opportunity by not shipping an official one-click VS Code extension with a clear paid offering.