ArchiveTeam has finished archiving all goo.gl short links
Archive preservation volunteers have brute‑forced and archived the entire goo.gl URL short link namespace, storing redirects and target content in WARC format on the Internet Archive. Commenters highlight how this complements Google’s plan to deactivate “inactive” links, helps combat link rot for references scattered across the web, and showcases ArchiveTeam’s broader role as an organized, volunteer-powered effort that rescues at‑risk web content beyond what general crawlers like the Wayback Machine or Common Crawl capture. Debates center on the low technical cost of keeping such services running, privacy and access concerns around publishing full datasets, and the need for independent, community-driven archiving in an era of frequent platform shutdowns.
Scope and Method of the goo.gl Archive
- Commenters confirm “all” means exhaustive enumeration of the entire goo.gl keyspace, not just known URLs.
- Volunteers ran a distributed client (“Warrior”) to iterate every possible key, record the HTTP response, and avoid IP bans.
- Since goo.gl no longer issues new links, the namespace is finite and fully searchable.
ArchiveTeam vs Internet Archive
- Several comments clarify the title: ArchiveTeam did the crawling and packaging; Internet Archive is mainly the hosting library.
- ArchiveTeam writes site-specific scripts, coordinates volunteers (via Warrior VMs/Docker), and “grazes” rate limits when sites are shutting down.
- They’re described as the “bucket brigade” rescuing data from dying services; Internet Archive is the storage.
- One anecdote highlights how quickly and efficiently ArchiveTeam infrastructure scaled to archive a video platform.
Google’s Policy Shift and Cost Debate
- People question why Google would deprecate “inactive” links given how simple and cheap a read-only key–value redirector should be.
- Several argue infra costs are negligible for a company like Google; organizational churn and stack churn are speculated as more likely drivers.
- Clarification: “recently clicked” isn’t the criterion; links with activity in late 2024 are kept, others will break.
Dataset Size, Format, and Access
- Confusion over the reported tens–hundreds of TiB leads to explanations: data is stored as WARC files containing full HTTP requests/responses, often including destination content, not just mappings.
- Some wonder why destination pages are archived given they’re no more “at risk” than the rest of the web.
- The WARC sets on archive.org are temporarily access-restricted; explanation relayed is concern over being blocked in the broader “AI scraping wars.”
- This frustrates some volunteers who helped, though others note content is still accessible via the Wayback Machine, just not as bulk dumps.
Privacy and Ethics of Archiving Short URLs
- Debate over whether anyone should have expected privacy: short URLs are easily enumerable, so treating them as secrets is called “silly.”
- Others worry about sensitive materials (private docs, unlisted videos) and compare to earlier incidents where private GPT links were archived.
- Counterpoint: preserving history sometimes requires acting without explicit consent when services are being shuttered.
Wider Web Archiving and Anti–Link-Rot Efforts
- Discussion of similar archives for Reddit (Pushshift, ArcticShift, AcademicTorrents), and speculation about HN datasets.
- A proposal for a blockchain/P2P global web snapshot meets pushback, with some pointing to Common Crawl as a de facto shared corpus, though acknowledged as incomplete.
- Overall, many celebrate the goo.gl effort as a concrete win against link rot, especially for references embedded in old documents and Stack Overflow posts.