Codestral Mamba

Original Article ↗ Hacker News Discussion ↗

Local LLMs and Tooling

  • Many recommend Ollama as the easiest way to run models locally; pairing it with Open WebUI (via Docker) gives a friendly browser UI.
  • Others prefer more feature-rich UIs like text-generation-webui.
  • Alternative entry points: llamafile (single-binary), GPT4All, and direct use of llama.cpp, Exllama, vLLM, or TensorRT-LLM depending on hardware.
  • Hugging Face’s open-llm-leaderboard and community tools like the “gpu_poor” site are cited for model rankings and hardware sizing.

Model Sizes, Hardware, and Quality

  • 7B models run on modest hardware and are seen by some as “very bad” beyond simple tasks, but others argue they’re remarkably capable for summarization and everyday help given their size.
  • 24GB GPUs can run Llama 3 70B in quantized form, though speed and quality claims conflict. Gemma 2 27B is suggested as a strong fit for 24GB VRAM.
  • Apple Silicon’s unified memory makes 7B models feasible but slower than dedicated GPUs.

Open-Source LLM Ecosystem (High-Level History)

  • Thread recaps a short history from early GPT‑2 era to LLaMA, LLaMA 2, Mistral, Mixtral, Llama 3, and Gemma 2, with quantization and CPU/GPU support (llama.cpp, bitsandbytes, Exllama, vLLM, TensorRT‑LLM) driving local adoption.
  • Wrappers like GPT4All and Ollama significantly lowered the barrier to entry.

Codestral Mamba, Mamba Architecture, and Benchmarks

  • Excitement centers on a high-profile Mamba2 code model competing with Transformers while offering linear-time inference and 256k-token context.
  • Some note that DeepSeek models match or beat Codestral Mamba on several benchmarks and that one table mis-highlights results; CodeGeeX4 is said to surpass them “on paper” but isn’t included.
  • Links to primers and explainers on Mamba/state-space models are shared; non-experts find good video and text resources.

IDE and Editor Integration

  • For VS Code/IntelliJ, Continue.dev and Sourcegraph Cody are popular; they can use Ollama or cloud APIs, but Mamba2 isn’t yet supported in llama.cpp, so Codestral Mamba isn’t available via Ollama.
  • Other options: codegpt.co plugins, TabbyML (with older Codestral), and custom editor scripts (e.g., Vim FIM completion via Ollama).

Closed vs Open Code Assistants & UX

  • Open coding models mentioned: CodeLlama, Codestral, DeepSeek-Coder V2, CodeGemma, CodeQwen, WizardCoder, CodeGeeX4; consensus is they still lag GitHub Copilot–class services overall, though some local setups work well.
  • Users report mixed but often strong experiences with Claude 3.5 Sonnet for coding and project-scale help; many feel it outperforms GPT‑4o in practice despite benchmarks.
  • Several dislike Copilot’s perceived decline in quality and explore alternatives like Supermaven, but pricing and token-based limits cause confusion and frustration.

Context Windows and Long-Context Behavior

  • Mamba’s 256k tested context is praised, though some question why it’s lower than Gemini’s claimed million-token range.
  • Participants discuss that newer models handle long context better than older “lost in the middle” behavior, but best practice remains to keep key instructions at the beginning or end.

Miscellaneous

  • Some criticize the product page’s Cleopatra/mamba joke as historically inaccurate and in poor taste.
  • Others think Mistral is missing a revenue opportunity by not shipping an official one-click VS Code extension with a clear paid offering.