Lessons from building GitHub code search [video]

Original Article ↗ Hacker News Discussion ↗

GitHub’s revamped code search, showcased in a recent technical talk, prompts both praise for major improvements—like regex support, better scalability, and semantic features powered by custom data structures—and criticism over lost functionality and new constraints. Commenters question the decision to require login for code search, the removal of “sort by recent,” and poor ranking and indexing visibility, arguing these changes erode openness and practical usability. Others highlight broader performance regressions in GitHub’s UI and point to alternative tools such as grep.app and Sourcegraph for more reliable or flexible code search.

Login Requirement for Code Search

  • Many participants ask why public code search now requires login.
  • Explanations proposed:
    • Product/metrics: boosts “active user” numbers and funnels people into GitHub’s ecosystem.
    • Cost/performance and anti-abuse: advanced search is compute-heavy and a major bot target; login enables rate limiting and DDoS protection.
    • Competitive/data strategy: login walls may hinder competitors from mining GitHub data while maximizing GitHub’s ability to mine others.
  • Disagreement on value of anonymous users:
    • Some see them as low-value freeloaders on compute.
    • Others argue they’re future customers and contributors; friction undermines openness and GitHub’s role in the open-source ecosystem.

Openness vs. Walled Garden

  • Several commenters see the login wall as part of a broader trend (like Twitter/Reddit) towards enclosure and monetization of previously open features.
  • Critics argue this erodes trust and pushes projects/users to seek alternatives or local workflows (clone + grep).
  • Defenders emphasize GitHub doesn’t “owe” free compute and must balance mission and sustainability.

Search Features, Ranking, and Missing Results

  • Strong appreciation for the new engine’s capabilities: regex, exact match, fork indexing, better navigation, fewer timeouts, scaling to huge numbers of repos.
  • Frustrations:
    • Removal of “sort by recent” used for tracking new usage patterns and mistakes; some users depended on this.
    • GitHub engineer explains sorting by recency is now technically complex (continuous reindexing, deduplication, Git’s history model) and was heavily abused by scrapers; team prioritized other features.
    • Ranking often surfaces many near-duplicate fork results; users want forks excluded by default.
    • Complaints that search sometimes misses known matches; engineer attributes most cases to repos not yet indexed or hitting documented limits, and agrees visibility into indexing status and exclusions is poor.

Performance and UI Regressions

  • Multiple reports of GitHub’s web UI becoming sluggish or unstable, especially:
    • Syntax-highlighted large files.
    • New React-based interfaces and client-side rendering.
    • Mobile/Android browsers, where pages or system UI can crash.
  • Some users resort to cloning repos or looking into local mirrors/alternative forges due to frontend performance.

Alternatives and Workarounds

  • Frequent mentions of alternatives: grep.app, sourcegraph, Debian code search, local tools (ripgrep, The Silver Searcher), github.dev’s VS Code-like search.
  • Some organizations find commercial search pricing prohibitive and prefer rolling their own code indexing or waiting for future AI tools.

Talk Content and Related Tech

  • Viewers praise the talk and the engineering behind the new system (e.g., custom indexer “Blackbird,” trigram tokenization, deduplication, geometric XOR filters, Tree-sitter-based semantic analysis).
  • Interest in future publication of data structures and implementation details; some are already building custom query languages inspired by GitHub’s syntax.