Falcon 2

Original Article ↗ Hacker News Discussion ↗

UAE’s Technology Innovation Institute has released Falcon 2, an 11B-parameter open-weight AI model it claims outperforms Meta’s Llama 3 8B and rivals Google’s Gemma 7B, prompting debate over how meaningful those benchmark wins really are given the larger model size and mixed real-world impressions of earlier Falcon versions. Commenters scrutinize the model’s restrictive, changeable license and marketing claims such as being the “only” vision-to-language model, with many arguing this undermines its credibility and practical adoptability. Others note that, even if technically incremental, such sovereign models matter for national control and competition in the rapidly evolving LLM ecosystem.

Benchmarks and Model Quality

  • Falcon 2 11B is claimed (by its creators) to slightly outperform Llama 3 8B and match Gemma 7B on Hugging Face benchmark averages.
  • Several commenters find this odd, saying Llama 3 8B generally outperforms Gemma 7B in their experience and suspect benchmark quirks or contamination.
  • Others note the comparison is among base models; chat-tuned Llama 3 is seen as much better than Gemma chat, which may explain perception gaps.
  • There is skepticism about benchmarking practice: 11B vs 7–8B is not a fair “same class” comparison; automated benchmarks can be misleading; Falcon only narrowly wins on one metric.

Licensing and “Openness” Concerns

  • The Falcon 2 11B license is a modified Apache 2 requiring compliance with an Acceptable Use Policy (AUP) that can change unilaterally.
  • Commenters argue this undermines the claim of being open source and creates ongoing legal risk, since conditions can shift without notice.
  • Debate over enforceability: some think retroactively changing terms for already-distributed weights is likely unenforceable or contradictory; others say the explicit clause may stand but is too risky for serious users.
  • Falcon 1 had earlier license “shenanigans”; Falcon 1 40B is Apache-licensed but seen as obsolete.

Comparisons and Real‑World Use

  • Anecdotes: Llama 3 8B is widely praised as “exceptionally good for its size,” Gemma 7B chat often judged weak; CodeGemma, however, is considered strong for coding.
  • Performance parity of Falcon 2 11B with Llama 3 8B and Mistral 7B despite more parameters is seen as underwhelming.
  • Earlier Falcon models (e.g., 180B) are recalled as heavily hyped but underperforming smaller open models.

Training Setup and Technical Notes

  • Model card: trained on 1024 A100 40GB GPUs for ~2 months with 3D parallelism and FlashAttention 2.
  • Stats cited: ~5T training tokens versus ~15T reported for Llama 3; some doubt extra parameters can offset fewer tokens.

Marketing, Positioning, and Geopolitics

  • Press claims like “only AI model with Vision-to-Language capabilities” are called out as clearly false given GPT‑4V, Claude, Gemini, LLaVA, etc.
  • “Outperforms Llama 3” is viewed as clickbait, especially without addressing Llama 3 70B or LMSYS-style human preference rankings.
  • Some see Falcon as a sovereignty/prestige/media project for the UAE rather than a purely commercial play, with mixed reactions to AI being developed by non-democratic states.