Show HN: I scraped 25M Shopify products to build a search engine

Original Article ↗ Hacker News Discussion ↗

A side project that scraped 25 million products from roughly 650,000 Shopify stores to build an independent product search engine drew both technical curiosity and skepticism about its long‑term viability. Commenters dissect how the creator acquired store lists, leveraged public JSON endpoints, and chose an expensive MongoDB‑based stack, while suggesting cheaper, faster alternatives like Elasticsearch, Typesense, Meilisearch, or Postgres for scaling and better relevance. Many also question the product’s differentiation from Shopify’s own Shop app and Amazon, arguing that success will hinge less on crawling scale and more on search quality, curation, fraud filtering, and a clear value proposition for both shoppers and merchants.

Scraping approach & data sources

  • Shopify stores expose standardized JSON endpoints (/products.json, /meta.json), so scraping mostly means downloading these rather than HTML.
  • Several commenters note you can “hammer” /products.json with little rate limiting; traditional anti-bot issues (CAPTCHAs, IP bans) are minimal here.
  • Initial store list came from a paid “BuiltWith” dataset (~2M stores), then filtered by geography and revenue; others mention DNS patterns, products.json presence, or Shopify-specific URL structures as detection methods.
  • Some discuss robots.txt and sitemaps as additional discovery mechanisms, but one person questions using “do-not-scrape” hints as seed data.

Tech stack, cost & performance

  • Current stack: JavaScript crawler, MongoDB Atlas (including Atlas Search), Next.js front-end, AWS infra; cost around $2.2K/month.
  • Many think this is overkill for ~25M products and suggest cheaper, better-suited options: Elasticsearch/OpenSearch, Typesense, Meilisearch, Postgres (with full-text and vectors), or even single bare-metal/Hetzner boxes.
  • Several note that 25M documents is modest for modern search infra and should be fast and inexpensive if architected well.

Search quality & features

  • Users find results for generic queries like “red shoes” weak or noisy; brand matches sometimes outrank obvious intent.
  • Suggestions: vector/semantic search using CLIP or similar, dense image captioning, color detection, and multimodal embeddings to improve relevance.
  • Desired filters: price, “ships to”/“ships from” location, availability, NSFW filtering, delivery country, similarity search, and showing alternative offers for the same product.

Data quality, curation & fraud

  • Multiple comments warn there’s a long tail of low-quality, scammy, or dropshipped Shopify stores.
  • Past projects report heavy manual and automated curation to remove junk, scams, and sensitive/NSFW content; many see curation as critical to differentiation.

Monetization, competition & viability

  • Current model is merchant fees for “verified” and boosted listings; no affiliate revenue yet.
  • Several point to Shopify’s own Shop app and various price comparison engines as direct or indirect competition.
  • Prior similar projects share that technical execution was manageable; user acquisition and monetization were the real bottlenecks.

UX & miscellaneous

  • Feedback highlights slow responses, lack of loading indicators, infinite scroll issues, navigation quirks, Unicode query errors, and desire for easy “open in new tab.”
  • Some question Shopify’s ToS implications; others dismiss this concern or argue it only matters if legal action occurs.