3 ms·
>What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and prov
by RcouF1uZ4gsC 7y ago
>What if you want to build a Google competitor?
Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/
Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
- 4gotunameagain 7y agoWhat if for whatever reason the innovation is on the crawler itself
- kordlessagain 7y agoAn AI crawler.
- AznHisoka 7y agoNot sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.