3 ms·

Can a single centralized engine / business model ever balance all those conflicts? It feels that the time for decentralized search where entities are both contr

Can a single centralized engine / business model ever balance all those conflicts? It feels that the time for decentralized search where entities are both contributing and benefiting from shared search indexes and have the ability to tweak their preferences is, broadly speaking, the space we need to look into?

bborud 5y ago
Centralized vs distributed represents a non-trivial set of technological challenges and differences. Those challenges tended to be poorly understood by anyone who hadn't given the problem of designing a search engine much thought in the past. I don't know if that has improved since I was part of the search engine industry. It probably has in the sense that the absolute number of people who have sufficient understanding of large scale search engines has grown by orders of magnitude in the last 25 years. The challenge is competition. Or rather, the lack thereof. I worked for Google when the last credible challenger threw in the towel and search got pushed closer to being a monopoly / monoculture. At the time Microsoft hadn't gotten their act together. Even as an enthusiastic imbiber of the Google cool-aid, this was deeply worrying. Google distinguished itself because it had to in order to compete. Making suck noticably less and being in tune with the users used to be enough. Because the competition sucked at this. Google no longer has to do that. Jeff Dean doesn't lie awake at night worrying about web search competitors. There was a time when he probably did (if I remember various accounts correctly). I doubt he loses any sleep over it now. Lots of people have tried to do distributed search. Lots and lots. The reason you don't hear about them is that they tend to not produce convincing results. Or in fact any results. In search quality, privacy, fairness. It is much, much harder than you might be tempted to think. And it exists within a field that gets harder every single year since the problem size and complexity keeps growing as the web grows; and grows more complex. If you believe in distributed search I think you should give it a go. Just because everyone has failed before you doesn't necessarily mean it is impossible to do. But it is worth remembering that it is a class of problem that is a lot harder than centralized search. So saying it is the solution is like saying cold fusion is the solution to all our energy needs: it might have been if we knew how to achieve it. But we don't.

bborud 5y ago

Centralized vs distributed represents a non-trivial set of technological challenges and differences. Those challenges tended to be poorly understood by anyone who hadn't given the problem of designing a search engine much thought in the past. I don't know if that has improved since I was part of the search engine industry. It probably has in the sense that the absolute number of people who have sufficient understanding of large scale search engines has grown by orders of magnitude in the last 25 years. The challenge is competition. Or rather, the lack thereof. I worked for Google when the last credible challenger threw in the towel and search got pushed closer to being a monopoly / monoculture. At the time Microsoft hadn't gotten their act together. Even as an enthusiastic imbiber of the Google cool-aid, this was deeply worrying. Google distinguished itself because it had to in order to compete. Making suck noticably less and being in tune with the users used to be enough. Because the competition sucked at this. Google no longer has to do that. Jeff Dean doesn't lie awake at night worrying about web search competitors. There was a time when he probably did (if I remember various accounts correctly). I doubt he loses any sleep over it now. Lots of people have tried to do distributed search. Lots and lots. The reason you don't hear about them is that they tend to not produce convincing results. Or in fact any results. In search quality, privacy, fairness. It is much, much harder than you might be tempted to think. And it exists within a field that gets harder every single year since the problem size and complexity keeps growing as the web grows; and grows more complex. If you believe in distributed search I think you should give it a go. Just because everyone has failed before you doesn't necessarily mean it is impossible to do. But it is worth remembering that it is a class of problem that is a lot harder than centralized search. So saying it is the solution is like saying cold fusion is the solution to all our energy needs: it might have been if we knew how to achieve it. But we don't.