Ask HN: GitHub employees what's going on? Why?

Hacker News Discussion ↗

GitHub’s increasingly frequent outages are being linked to a convergence of factors: a painful migration from its legacy infrastructure to Microsoft Azure, a surge in AI-generated coding activity that reportedly drove commits up 14x in a year, and heavy load from features like Actions and Copilot. Commenters debate whether Microsoft’s ownership and Azure’s reliability are core problems or whether any large platform would struggle to scale a mature, complex system this quickly. Many expect GitHub to respond with stricter limits or new pricing on high-volume and free usage, but worry that chronic instability is already eroding trust in what has become core developer infrastructure.

High-Level Diagnosis

  • Two primary explanations dominate:
    • Massive AI-driven traffic growth (commits, Actions, webhooks) stressing a mature platform.
    • A difficult multi-year migration from GitHub’s legacy stack and own datacenters to Azure, exposing new failure modes.
  • Many believe both are interacting: new load patterns hitting an architecture in transition.

Microsoft Acquisition and Azure Migration

  • Historical uptime graphs show stability pre‑acquisition and more incidents afterward.
  • Some argue this is due to:
    • More features (Actions, Copilot, Codespaces, security, packages) and a much larger surface area to fail.
    • Ongoing migration to Azure, including partial/dual environments, which is described as “painful.”
  • Others see the correlation as evidence Microsoft and/or Azure are the root problem; defenders counter that Azure runs many massive services successfully.

AI‑Driven Growth and “Slop”

  • GitHub leadership is cited claiming ~14× commit growth and rapid user growth, attributed to AI coding and agents.
  • Many suspect huge volumes of low‑value or bot traffic:
    • Repos with thousands of tiny commits.
    • Bots auto‑creating repos, installing apps, and firing webhooks.
    • Scraping attacks and CI/Actions overuse.
  • Debate on whether this is a valid excuse versus a planning failure, since AI‑driven growth has been visible for over a year.

Scaling, Architecture, and Limits

  • Discussion of:
    • GitHub as a shared-state system (repos, PRs, issues) versus mostly stateless LLM APIs, making scaling harder.
    • Rails vs. “modern” stacks; some blame Ruby, others say architecture and design matter more than language.
    • Lack of aggressive limits and rate‑limiting seen as a root cause; calls for per‑user or per‑repo throttles and pricing.
  • One detailed insider-style comment frames this as a once‑in‑a‑generation scaling event on an already huge system, not simple incompetence.

Business, Org, and User Impact

  • Infra seen as a cost center competing with AI initiatives and other priorities.
  • Multiple layoffs are blamed for loss of expertise and slow recovery.
  • Users suggest:
    • Ending or tightening free unlimited private repos.
    • Charging heavy/abusive users or capping daily commits.
  • Others worry that charging more won’t fix quality and that switching costs keep enterprises locked in despite poor uptime.

Recent Incident Postmortem (Aug 17, 2026)

  • Official report (quoted in thread) attributes a major outage to:
    • New traffic peak saturating load balancers.
    • Misconfigured Istio sidecar autoscaling.
    • HAProxy flow exhaustion and retry storms, especially from VS Code/Copilot token logic.
  • Follow‑ups include fixing autoscaling policies, retry behavior, and client amplification, but participants see it as evidence of systemic reliability and operational gaps.