3 ms·

I built a search engine that was able to crawl Amazon and Yelp. The toughest sites were reddit and facebook.

• creato 7y ago
  And google sucks at those too.
• AznHisoka 7y ago
  at scale? millions of pages a week? And now? I wrote a crawler that could crawl Amazon as early as a year ago too, but now it doesn't work.