The August 17 outage

Original Article ↗ Hacker News Discussion ↗

A major GitHub outage on August 17, traced to capacity failures, misconfigured load balancing and retry storms during a period of explosive traffic growth, has renewed worries about the platform’s reliability. Commenters link the surge in commits and GitHub Actions runs to AI-generated “slop” code and agentic workflows, arguing that GitHub’s free tier and Copilot usage are overwhelming infrastructure that’s still mid-migration to Azure. Many suggest rate limits, stricter separation of free and paid workloads, or moving to self‑hosted and alternative forges, while others stress that graceful degradation, better load shedding, and resilience patterns matter more than simply adding hardware.

Outage cause and scaling debate

  • Many see the incidents as classic capacity cliffs: systems work fine, then a small load increase (e.g., 2.8B → 2.9B commits/month) triggers hidden bottlenecks and cascading failures.
  • Others argue GitHub should have seen “at-capacity” conditions earlier and designed to avoid such cliffs; just “adding more hardware” (3M CPU cores, 120 PB storage) is seen as insufficient without better architecture and resilience.
  • Some commenters note that this kind of failure is common at hyperscale; others say GitHub’s RCA reads like a team that’s only now building mature large-scale practices.

AI-driven traffic explosion

  • Huge commit growth is widely attributed to AI/agentic coding and automated workflows, not more human developers.
  • Many describe GitHub being flooded with “AI slop”: frequent, low-quality, auto-generated commits, PRs, and issues (Bun is repeatedly cited as an extreme example).
  • There’s disagreement whether this growth is “impressive” (from an infra standpoint) or more like “cancer growth” that degrades the service while adding little value.

Azure, Microsoft, and architecture

  • Strong skepticism that “migrating to Azure” is a solution when Azure is perceived (by some) as a major reliability problem, especially for historically colo-based products like GitHub/LinkedIn.
  • Others push back: any large platform (AWS, Azure, GitHub) will have outages; blaming Azure alone is seen as oversimplified.

Retries, thundering herds, and resilience

  • The retry storms (notably from Copilot/VS Code clients) are viewed as a textbook “thundering herd” / retry amplification problem.
  • Long discussion of best practices: exponential backoff with jitter, circuit breakers, client-side throttling, token buckets, and explicit differentiation between retryable vs non‑retryable errors.
  • Several argue the real root cause is lack of graceful degradation and load shedding, not just insufficient capacity.

Free tier, AI slop, and business model

  • Many suspect GitHub’s economics are strained: massive free hosting (including AI-generated code and Actions) vs finite infra budget.
  • Proposed mitigations: per-user or per-org commit limits, rate limiting, paid minimum tiers (e.g., $1/month), separating free vs paid capacity pools.
  • Counterpoint: Microsoft may willingly subsidize GitHub as a loss leader to drive AI adoption and harvest code as training data, even if that harms reliability.

Alternatives and self‑hosting

  • Strong interest in self-hosted GitLab, Forgejo/Gitea, Codeberg, Sourcehut, and new “AI-native” forges.
  • Tradeoffs discussed:
    • Self-hosting: modest ongoing admin cost but better control, isolation from AI slop, and no central point of failure.
    • GitLab: mature but pricier and “clunky”; fewer outages but not immune.
    • Forgejo/Codeberg: FLOSS, lighter, but weaker CI/app ecosystem; Codeberg’s political stance worries some.
    • Sourcehut: email-driven workflow praised, but paid and niche.

GitHub product experience and workflows

  • Some claim GitHub is “most sophisticated” for integrated issues/PRs/CI; many others say it’s “worst in every individual aspect” but wins via consolidation and network effects.
  • A lot of serious teams now treat GitHub as just a git host and run planning/issue tracking elsewhere (e.g., Linear).
  • Actions is repeatedly cited as unreliable under load and a pain point for enterprises.

Leadership, communication, and trust

  • Noticeable irritation that GitHub’s blog post uses corporate language (“we let you down”) without explicit “sorry” or concrete commitments to refund/compensate paying customers.
  • Some are bothered that the public profile of the GitHub CTO shows no visible coding in the last year; others argue a CTO’s job is leadership, not active coding.
  • Enterprise users in particular are frustrated that outages appear to affect paid and free users alike, and want strict isolation or separate infrastructure for paying customers.

Broader concerns: value, environment, and software quality

  • Several threads question whether the explosive commit growth has yielded noticeably better software or user experiences; many say consumer software is getting worse despite more code.
  • Others counter that AI has empowered non‑coders and enabled hobbyists to build tools they never could before, though these gains are often personal and not widely visible.
  • Environmental impact of AI/data centers sparks debate: some see AI energy use as moderate and offsettable by efficiency gains; others see it as a serious setback for climate goals, especially given scrapped CO₂ targets by major providers.