3 ms·

> the index itself should be something operated by the public rather than privately I love this but the act of indexing the web is both complex and ever-evolvi

> the index itself should be something operated by the public rather than privately I love this but the act of indexing the web is both complex and ever-evolving (not to mention expensive), so the market needs to have natural incentives for the public crawler to continue being good at its job. Google is incentivized to have a great index because it feeds into its competitive advantage. If the index is a shared resource, what incentive does the indexing company have to keep improving it? And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? All search companies would have to maintain secondary indices to augment the public one. There's probably some artificial structure needed to make open indexing viable.

cogman10 2y ago
> what incentive does the indexing company have to keep improving it? A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. > And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? Voting/lobbying/etc. Though I'd assume everyone that makes a website will want to be indexed.
- ratorx 2y ago
  NHS is a pretty terrible example. Since it relies on public funding without a direct monetary profit, it is an easy target for defunding when the government needs to cut expenses. And lo and behold that is exactly what has happened. I’m not saying privatisation is better (US healthcare is the textbook counter-example), but just wanted to make the point that socialising something has demonstrably not been a good way to drive innovation and improvement in some cases.
- kajecounterhack 2y ago
  > A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. Sure like there's open street maps for maps, there could be something like that for search indices. But I thought we were discussing the viability of making search indexing an independent, maybe even regulated natural monopoly as a way to democratize search and keep competition improving search itself. And I think the answer is no because search companies will find alpha from maintaining their own secondary indices and use it as a competitive advantage, and the 3rd party index will be (and if not immediately, then over time become) inferior to what the search companies use to differentiate themselves due to natural pressures.

cogman10 2y ago

> what incentive does the indexing company have to keep improving it? A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. > And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? Voting/lobbying/etc. Though I'd assume everyone that makes a website will want to be indexed.

ratorx 2y ago
NHS is a pretty terrible example. Since it relies on public funding without a direct monetary profit, it is an easy target for defunding when the government needs to cut expenses. And lo and behold that is exactly what has happened. I’m not saying privatisation is better (US healthcare is the textbook counter-example), but just wanted to make the point that socialising something has demonstrably not been a good way to drive innovation and improvement in some cases.
kajecounterhack 2y ago
> A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. Sure like there's open street maps for maps, there could be something like that for search indices. But I thought we were discussing the viability of making search indexing an independent, maybe even regulated natural monopoly as a way to democratize search and keep competition improving search itself. And I think the answer is no because search companies will find alpha from maintaining their own secondary indices and use it as a competitive advantage, and the 3rd party index will be (and if not immediately, then over time become) inferior to what the search companies use to differentiate themselves due to natural pressures.