Twitter shut off API access; users volunteering their own data for an open API
Lockdowns and steep pricing on APIs at platforms like Twitter/X and Reddit are prompting developers to look for workarounds, including community-run archives and browser extensions that crowdsource data from users’ own browsing. Commenters debate the economics and hosting choices for such projects, the broader shift from an open, syndication-friendly web toward walled gardens optimized for ads and engagement, and the role of AI companies scraping public content. Significant concern centers on privacy and “cognitive security” risks of aggregating identifiable social media histories, alongside questions about whether decentralized alternatives like Mastodon, Bluesky, or Nostr can realistically replace centralized platforms without recreating their problems.
Motivation for a Community Twitter Archive
- Goal: rebuild useful API-like access by letting users donate their own Twitter data exports.
- Seen as a way to escape Twitter’s lockdown, preserve conversation history, and enable new tools (e.g., question-answering over someone’s tweet history).
- Some frame it as helping people migrate off Twitter while keeping their “living wiki” of posts and threads.
Data Storage, Cost, and Infrastructure
- Skepticism about “just put everything in S3” as “cheap.”
- Suggestions to avoid big clouds due to bandwidth costs; proposals for dedicated servers (e.g., unmetered storage boxes) or S3‑compatible object storage / self-hosted systems.
API Lockdowns, Scraping, and the Changing Web
- Widespread frustration with Twitter/Reddit-style API restrictions and high pricing; many foresee a return to heavy web scraping.
- Others argue operators understandably don’t want AI companies and scrapers to extract huge value from their data for free.
- Historical context: early Twitter had RSS, SMS posting/alerts, and a more open API; shutdowns are tied to ad/engagement models.
Privacy, Abuse Risks, and Cognitive Security
- Concern that making identifiable archives queryable enables targeted phishing and other manipulation.
- Calls for explicit consent and clear warnings that data may be permanently mirrored by others.
- Some suggest making datasets private or invite‑only, or smaller community‑scoped collections.
- Worries about data poisoning (fake tweet archives); ideas include requiring multiple independent corroborating uploads and web‑of‑trust mechanisms.
Decentralized Alternatives and Their Trade-offs
- Some advocate moving to Mastodon, Bluesky, or Nostr; others report Mastodon’s culture, admin drama, defederation, and lack of post migration as major drawbacks.
- Debate over whether a federated, server‑oriented design is inherently flawed or simply the only non‑corporate option.
Crowdsourced Scraping via Browser Extensions
- Interest in extensions/userscripts that passively and anonymously upload what users already view (likened to RECAP for PACER or other crowdsourced tools).
- Recognized as TOS‑gray but potentially hard to distinguish from normal browsing; privacy and site‑specific tailoring are unsolved challenges.
Broader Social Media Reflections
- Many describe Twitter as increasingly toxic and engagement‑optimized; a minority say it still works well with careful curation.
- Lock‑ins are attributed to network effects, income from engagement programs, and behavioral inertia despite perceived harm.