Meta AI releases Code Llama 70B

Original Article ↗ Hacker News Discussion ↗

Meta’s release of the 70B-parameter Code Llama model is seen as a major step for open, high-end code generation, drawing comparisons with GPT‑4 and smaller competitors like DeepSeek Coder and Mixtral. Commenters weigh its likely training approach, benchmark performance, and practical usability, especially via quantized variants that can run locally on powerful consumer hardware or cheaply via GPU rental services. A recurring theme is Meta’s strategic motive: by freely releasing strong foundation models, it pressures closed providers like OpenAI, attracts talent, and shifts value toward data, infrastructure, and downstream applications rather than proprietary models alone.

Model capabilities & comparisons

  • Many are excited to see what a 70B Code Llama can do, given strong smaller code models (e.g., Deepseek Coder 6.7B, WaveCoder-Ultra-6.7B) and tools like AlphaCodium (prompt/flow strategy, not a new base model).
  • Some users report Deepseek Coder outperforming same-size Code Llama; a 30–35B Deepseek is said to benchmark similarly to its 7B version.
  • Benchmarks shared (BigCode, EvalPlus, etc.) suggest Code Llama variants are strong but not uniformly best-in-class; claims that some methods beat GPT‑4 on pass@k are referenced but not independently verified in the thread.
  • There is curiosity whether 70B Code Llama can approach GPT‑4 in coding performance and serve as a strong local Copilot alternative.

Hardware, quantization & local usage

  • 70B is considered usable locally only with quantization (typically 4-bit); full-precision inference is seen as wasteful outside research.
  • Reports of successful runs:
    • Mac Studio / MacBook with 64GB+ unified memory: 70B at 4-bit is “usable,” though slower; ~9–10 tok/s on some M1/M2 Ultra setups.
    • RTX 3090 (24GB VRAM): good for ~33B models; 70B requires CPU offload or multi‑GPU, which becomes slow or complex.
    • Multi‑GPU rigs (e.g., multiple P40s, data‑center chassis) can run 70B+ efficiently but require datacenter‑style hardware and power.
  • Debate over whether a 4090 is “enough” for 70B: at realistic 4‑bit quantization it generally is not, unless split across multiple GPUs.
  • Energy cost for local use is estimated to be modest (on the order of cents to low dollars per month for typical interactive workloads).

Tools, workflows & “local Copilot”

  • Popular local stacks: llama.cpp / GGUF, Ollama, text‑generation‑webui, with IDE frontends like Continue (VS Code), Cody, Twinny, JetBrains plugins (CodeGPT), and Emacs modes (e.g., gptel).
  • Strategies discussed:
    • RAG over codebases instead of fine‑tuning, given fast‑changing code.
    • Using hosted APIs (Together, Bedrock, etc.) for large models and local quantized models for day‑to‑day work.
  • Several users already run Deepseek or Code Llama locally as in‑house copilots and report good results; they hope 70B improves further.

Training data & openness

  • Multiple commenters assert 70B Code Llama is very unlikely to be trained on Meta’s internal code due to data extraction risks.
  • Some argue there is actually a shortage of high‑quality open‑source code, and that models already ingest “substantially all” available public code.
  • Strong debate over calling Llama “open source”:
    • Critiques: license restrictions, lack of data and full training stack, and compute barrier mean it isn’t reproducible or truly open.
    • Counterpoints: redistributable weights plus open‑source PyTorch stack are seen as “open enough” and vastly better than pure paper releases.

Meta’s strategy & broader implications

  • Many see Meta’s releases as strategic rather than altruistic:
    • Undercutting OpenAI/Google/Microsoft moats by making capable base models cheap or free.
    • “Commoditize your complement”: make models a commodity so advantage shifts to data, distribution, and applications, where Meta is strong.
    • Attract top AI talent by enabling open publication and community impact.
    • Crowdsource R&D and tooling; next‑gen Meta models benefit from community techniques.
    • Improve content generation, moderation, and metaverse/VR world building internally.
  • Others argue this helps prevent a single closed‑model monopoly and diversifies model suppliers, while still likely leaving only a few giants able to train true frontier models due to GPU scale.
  • Some remain skeptical of Meta’s motives given past scandals; others note that, regardless of intent, the open weight releases are materially beneficial to developers.