3 ms·
Surprised nobody has interpreted search engine spidering bandwidth as a form of public good. There's a limit to the amount of spidering that can happen. It's ok
by awinter-py 9y ago
Surprised nobody has interpreted search engine spidering bandwidth as a form of public good. There's a limit to the amount of spidering that can happen. It's okay if 10 or a 100 companies do it, but if everyone had to do it, the web's access pattern would be exponential and overwhelming.
There's a class of consumer who is technically willing and able to operate their own spider but accepts the lesser convenience of a web search because they recognize it would take down the internet if everyone spidered all the time.
Also surprised the article doesn't mention omnibar search. By defaulting this to G, G is 'abusing' browser dominance to reinforce search dominance.
- xiaoma 9y ago>"the web's access pattern would be exponential and overwhelming." It might well be overwhelming, but it wouldn't be exponential. There are a maximum of N choose 2 connections between N nodes. That number only grows quadratically as N increases, while exponential growth would outstrip quadratic, cubic or any polynomial growth for large values of N.