3 ms·
This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to
by AznHisoka 7y ago
This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
- RcouF1uZ4gsC 7y ago>What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
- 4gotunameagain 7y agoWhat if for whatever reason the innovation is on the crawler itself
- kordlessagain 7y agoAn AI crawler.
- AznHisoka 7y agoNot sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.