3 ms·

> The index itself is a textbook natural monopoly. I would argue that because of this, the index itself should be something operated by the public rather than

> The index itself is a textbook natural monopoly. I would argue that because of this, the index itself should be something operated by the public rather than privately. This is especially true because it's within the best interests of the public to have an archive of the internet. The wayback machine currently supports that and it's on its last legs. Index the internet with a public search API, then let future search engines hit that an optimize based on internal algorithms to get better results.

kajecounterhack 2y ago
> the index itself should be something operated by the public rather than privately I love this but the act of indexing the web is both complex and ever-evolving (not to mention expensive), so the market needs to have natural incentives for the public crawler to continue being good at its job. Google is incentivized to have a great index because it feeds into its competitive advantage. If the index is a shared resource, what incentive does the indexing company have to keep improving it? And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? All search companies would have to maintain secondary indices to augment the public one. There's probably some artificial structure needed to make open indexing viable.
- cogman10 2y ago
  > what incentive does the indexing company have to keep improving it? A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. > And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? Voting/lobbying/etc. Though I'd assume everyone that makes a website will want to be indexed.
  - ratorx 2y ago
    NHS is a pretty terrible example. Since it relies on public funding without a direct monetary profit, it is an easy target for defunding when the government needs to cut expenses. And lo and behold that is exactly what has happened. I’m not saying privatisation is better (US healthcare is the textbook counter-example), but just wanted to make the point that socialising something has demonstrably not been a good way to drive innovation and improvement in some cases.
  - kajecounterhack 2y ago
    > A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. Sure like there's open street maps for maps, there could be something like that for search indices. But I thought we were discussing the viability of making search indexing an independent, maybe even regulated natural monopoly as a way to democratize search and keep competition improving search itself. And I think the answer is no because search companies will find alpha from maintaining their own secondary indices and use it as a competitive advantage, and the 3rd party index will be (and if not immediately, then over time become) inferior to what the search companies use to differentiate themselves due to natural pressures.
danielmarkbruce 2y ago
The idea that we'd want any type of government entity running search is absurd on several dimensions.
- cogman10 2y ago
  It's only absurd if it's ran as a public-private cooperative. The government directly running something like this can be every bit as efficient as what any private entity could do. It's not as if the gov doesn't already employ in several sectors skilled software developers. It only goes haywire when Reagan/Clinton style "Send it to the private businesses" policies get in the way. When you start talking about a constrained resource like this, it only makes sense to move it to the only entity without profit as its primary motivation.
  - danielmarkbruce 2y ago
    You misunderstand how big the skill gap is between google swes and government swes. Constrained resource doesn't mean what you appear to think it means. You have to take more than econ 101 to understand the real world.
  - kajecounterhack 2y ago
    > It's not as if the gov doesn't already employ in several sectors skilled software developers. Supply and demand -- gov't doesn't pay as well, so top SWEs do not work for gov't. Stronger reward signals attract stronger performers. If you can get paid 2x-10x more to do the same work, who will you go work for? > When you start talking about a constrained resource like this, it only makes sense to move it to the only entity without profit as its primary motivation. If the goal is to give competitors a starting point to mitigate the negative effects of a monopoly, then sure it'd be nice to have an open index that anyone can get. However I'd argue this is an ineffectual strategy because it's likely to be too little to be helpful, and just have low overall quality. Profit is a reward signal that typically signals a customer's demand / need. When organizations like gov't lose that signal, they typically end up optimizing for some far less effective target (someone's whim, political clout) and missing what the underlying need is in reality. Additionally, gov't lacks of a lot of software infrastructure and tend to use outdated tech / must adhere to procurement procedures that don't always yield the right set of tools. So in addition to being unable to understand constituent needs well, and being unable to hire good people, they also have the nasty privilege of working with worse infra when they do get funding. > The government directly running something like this can be every bit as efficient as what any private entity could do. I hope I've illustrated that the answer is no. Also, anecdotally I've worked with gov't on behalf of a big tech company and it was very enlightening to see the structural issues keeping them from succeeding at tasks of even moderate complexity. The US Digital Corps can only do so much :( I'm not anti-gov't though -- gov't _can_ make private entities more efficient though by busting monopolies and enabling maximum competition in the markets. Big tech is certainly due for a shakeup.

kajecounterhack 2y ago

> the index itself should be something operated by the public rather than privately I love this but the act of indexing the web is both complex and ever-evolving (not to mention expensive), so the market needs to have natural incentives for the public crawler to continue being good at its job. Google is incentivized to have a great index because it feeds into its competitive advantage. If the index is a shared resource, what incentive does the indexing company have to keep improving it? And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? All search companies would have to maintain secondary indices to augment the public one. There's probably some artificial structure needed to make open indexing viable.

cogman10 2y ago
> what incentive does the indexing company have to keep improving it? A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. > And what guarantees that the indexing company's strategy doesn't end up being unfair to those affected by their websites getting indexed? Voting/lobbying/etc. Though I'd assume everyone that makes a website will want to be indexed.
- ratorx 2y ago
  NHS is a pretty terrible example. Since it relies on public funding without a direct monetary profit, it is an easy target for defunding when the government needs to cut expenses. And lo and behold that is exactly what has happened. I’m not saying privatisation is better (US healthcare is the textbook counter-example), but just wanted to make the point that socialising something has demonstrably not been a good way to drive innovation and improvement in some cases.
- kajecounterhack 2y ago
  > A publicly owned asset doesn't need to be managed by a private entity. The incentive can be exactly the same incentive that the NHS and NASA operate on. It's a fun problem that you can make a career on making better that is ultimately a public good. Not everything needs to be operated with pure profits in mind. Sure like there's open street maps for maps, there could be something like that for search indices. But I thought we were discussing the viability of making search indexing an independent, maybe even regulated natural monopoly as a way to democratize search and keep competition improving search itself. And I think the answer is no because search companies will find alpha from maintaining their own secondary indices and use it as a competitive advantage, and the 3rd party index will be (and if not immediately, then over time become) inferior to what the search companies use to differentiate themselves due to natural pressures.

danielmarkbruce 2y ago

The idea that we'd want any type of government entity running search is absurd on several dimensions.

cogman10 2y ago
It's only absurd if it's ran as a public-private cooperative. The government directly running something like this can be every bit as efficient as what any private entity could do. It's not as if the gov doesn't already employ in several sectors skilled software developers. It only goes haywire when Reagan/Clinton style "Send it to the private businesses" policies get in the way. When you start talking about a constrained resource like this, it only makes sense to move it to the only entity without profit as its primary motivation.
- danielmarkbruce 2y ago
  You misunderstand how big the skill gap is between google swes and government swes. Constrained resource doesn't mean what you appear to think it means. You have to take more than econ 101 to understand the real world.
- kajecounterhack 2y ago
  > It's not as if the gov doesn't already employ in several sectors skilled software developers. Supply and demand -- gov't doesn't pay as well, so top SWEs do not work for gov't. Stronger reward signals attract stronger performers. If you can get paid 2x-10x more to do the same work, who will you go work for? > When you start talking about a constrained resource like this, it only makes sense to move it to the only entity without profit as its primary motivation. If the goal is to give competitors a starting point to mitigate the negative effects of a monopoly, then sure it'd be nice to have an open index that anyone can get. However I'd argue this is an ineffectual strategy because it's likely to be too little to be helpful, and just have low overall quality. Profit is a reward signal that typically signals a customer's demand / need. When organizations like gov't lose that signal, they typically end up optimizing for some far less effective target (someone's whim, political clout) and missing what the underlying need is in reality. Additionally, gov't lacks of a lot of software infrastructure and tend to use outdated tech / must adhere to procurement procedures that don't always yield the right set of tools. So in addition to being unable to understand constituent needs well, and being unable to hire good people, they also have the nasty privilege of working with worse infra when they do get funding. > The government directly running something like this can be every bit as efficient as what any private entity could do. I hope I've illustrated that the answer is no. Also, anecdotally I've worked with gov't on behalf of a big tech company and it was very enlightening to see the structural issues keeping them from succeeding at tasks of even moderate complexity. The US Digital Corps can only do so much :( I'm not anti-gov't though -- gov't _can_ make private entities more efficient though by busting monopolies and enabling maximum competition in the markets. Big tech is certainly due for a shakeup.