The Small Website Discoverability Crisis (2021)

Original Article ↗ Hacker News Discussion ↗

Large parts of the independent “small web” are increasingly invisible behind search results dominated by a few big, SEO-optimized platforms. Commenters trade ideas for how to surface personal sites and niche projects—reviving blogrolls, webrings, social bookmarking, curated newsletters, and “small web” search engines—while noting that most users now discover content through walled gardens like TikTok, Instagram, and Reddit. There is broad agreement that hand curation doesn’t scale easily and that Google’s incentives, spam, and link rot all compound the problem, but some argue that small, overlapping communities and human-driven link sharing may be the only sustainable path to discovery.

Is there really a “small web” discoverability crisis?

  • Many commenters feel that interesting small sites are hard or impossible to find via major search engines and social platforms.
  • Others argue the issue reflects user habits: if you live inside walled gardens (YouTube, TikTok, Instagram), you’ll rarely encounter independent sites.
  • Some say that for “average users” the small web effectively doesn’t exist; for power users who browse profiles, repos, RSS, and HN, there’s already more content than time.
  • Several ask for data: traffic trends, index coverage, or other objective measures are largely missing.

Search engines, SEO, and AI

  • Google is widely blamed:
    • Overweighting recency and large, SEO-heavy sites.
    • Under-indexing or de-indexing small and older sites (“discovered – currently not indexed”, “Google jail”).
    • Treating link-heavy pages as spam, discouraging blogrolls/links pages.
  • AI-driven content and AI assistants may worsen the problem by flooding the web with auto‑generated pages and keeping users inside answer boxes, reducing incentive to create original sites.
  • Some see a conflict between ad-funded search and surfacing niche, non-monetized content.

Human curation: blogrolls, webrings, bookmarks

  • Strong nostalgia for blogrolls, links pages, webrings, StumbleUpon, del.icio.us, and early Reddit/Digg as discovery layers.
  • Many endorse the article’s proposal: public bookmark lists, ideally small, annotated, and sometimes shareable as OPML/JSON/RSS.
  • Concerns: link rot, spam/astroturfing, social risk of “endorsing” sites, and the fact that hand curation doesn’t scale.

Existing and proposed tools

  • Mentioned “small web” search engines, random site explorers, bookmarking tools, Neocities-style host+directory hybrids, and RSS-based readers.
  • Ideas floated: federated bookmarking via ActivityPub, browser extensions to publish curated lists, p2p bookmark sharing, “awesome small web” lists, and neural/embedding-based search focused on long-tail content.

Barriers for small-site creators

  • Hard to get traction without buying ads or gaming platforms; self-promotion rules on big communities often punish genuine projects.
  • Tension between building for personal joy vs chasing audience and modern aesthetics.
  • Some question whether mass discoverability is even possible or necessary; others see cultural and preservation value in reaching more people.