5 ms·
https://commoncrawl.org/ https://commoncrawl.org/ is a big crawler with all data made public. Probably a good resource for new search engines.
by infogulch 5y ago
https://commoncrawl.org/ https://commoncrawl.org/ is a big crawler with all data made public. Probably a good resource for new search engines.
- boyter 5y agoYes and no. It works for the deep web, but for anything that needs more frequent crawling no. They don't do it often enough. I have mentioned this elsewhere, but id love there to be a single crawl that everyone was able use and build indexes off. It would be especially good for webmasters too as you would have one bot to worry about and optimise for, reducing the load on systems, and allowing for innovation in the search space that cloud flare and other cdn's are stifling because they understandably are blocking crawlers that aren't Google or Bing. For example id love to see something added to robots.txt that allows you to say "crawl at 2am, hit has hard as you want, don't come back for a week" or some such. Having a single crawl could allow for this.