4 ms·
I’d like to see them split off crawling from indexing/search, and be forced to provide the complete crawl to anyone who wants it, at cost. Currently, they have
by ericd 2y ago
I’d like to see them split off crawling from indexing/search, and be forced to provide the complete crawl to anyone who wants it, at cost. Currently, they have a privileged position, where almost all websites are willing to let them crawl, while blocking most smaller crawlers pinging them anywhere near as often, and forms a bit of a natural monopoly. Opening that up for all would open up competition in the search interface space a bit, and open things up for more specialized vertical search engines, making it generally a lot easier to start a search site without negatively impacting the sites they're searching with n engines worth of crawler traffic.