3 ms·

Check out the serious difficulties the Common Crawl had with crawling 1% of the public internet on donated money and then get back to me with a plan. This is re

Check out the serious difficulties the Common Crawl had with crawling 1% of the public internet on donated money and then get back to me with a plan. This is really, really hard to do for free. Maybe talk to Gates :)

opan 6y ago
TFA specifically mentions not crawling the web, but using a curated list.
- rjurney 6y ago
  I don't follow. A curated list of what? You need to crawl the entire web to have the content of the entire web to index and search.