2 ms·
That's the result of the crawling, and it preventing competition. Google would much prefer that people complain about the details while ignoring the root cause.
by benatkin 6y ago
That's the result of the crawling, and it preventing competition. Google would much prefer that people complain about the details while ignoring the root cause.
- tyingq 6y agoI don't understand that. The crawling access is mostly the same as it ever was. Google's SERP pages are not. A mutually beneficial search engine that respects it's sources would still crawl the same. Google used to be that. The core problem is incentives: http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html "we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm."
- benatkin 6y agoThat's incorrect. Before the search oligopolies formed, new search engines could start up. There was excite, hotbot, altavista, and more. Now they don't have access. Search these comments for census.gov.
- tyingq 6y agoThere are companies that do pretty well in this space, like ahrefs, for example. They do resort to trickery, like proxy clients that look like home computers or cell phones. But, if a small entity like ahrefs can do it, anyone can do it. In a nutshell, though, I don't see equal access for all crawlers changing anything. Maybe that's the first barrier they hit, but it isn't the biggest or hardest one by far. Bing has good crawler access, but shit market share.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]