5 ms·

Here's one idea: Make the index open and public, set up a public body to update it, and allow anyone to build a frontend. Access can be charged per query. Other

by wz1000 6y ago

Here's one idea: Make the index open and public, set up a public body to update it, and allow anyone to build a frontend. Access can be charged per query. Other companies/entities/governments will be allowed to mirror the index for essentially the price of data transfer. This is just something I thought up right now, and sure, there might be problems with the approach. But to pretend like there is absolutely nothing that can be done betrays a willful lack of imagination.

jeffbee 6y ago
The way that people underestimate the scale and complexity of Google's indexing always makes me chuckle. I'm especially fond of the idea that other parties will just wantonly copy it, like for research purposes or something. Also I'm sorry to inform you that the way the index shards are produced, arranged in memory/on disk, and the manner in which they are queried and ranked are inextricably linked. You can't make a search-neutral index format that can be queried economically.
- wz1000 6y ago
  > The way that people underestimate the scale and complexity of Google's indexing always makes me chuckle. The same approach that works for Google can also work for a public body supported by public funds. It can also use much of the same systems and maybe even personnel that Google currently employs for this purpose. I'm essentially asking for the search division of Google to be appropriated by the government. > Also I'm sorry to inform you that the way the index shards are produced, arranged in memory/on disk, and the manner in which they are queried and ranked are inextricably linked. You can't make a search-neutral index format that can be queried economically. I'm not asking for a search neutral index - public Google just needs to be (largely) open, transparent and accountable about its ranking algorithm. That's why the stipulation that other entities be allowed to mirror the index - they can optimize the index for their own purposes and rankings on their own hardware. > I'm especially fond of the idea that other parties will just wantonly copy it, like for research purposes or something. Not for research purposes - other entities will be free to copy it for purposes like sovereignty (governments), less/more censorship or to build their own systems on top of it for profit, or any other purpose.
  - jeffbee 6y ago
    There are currently zero governments and only about 4 commercial entities possessing a datacenter large enough to do the job.
    - wz1000 6y ago
      Doesn't the NSA have a bunch of massive datacenters? Regardless of that, the datacenter can be appropriated by the government too.
      - jeffbee 6y ago
        No, the NSA has one facility that would be a small/medium datacenter in the big league, but only if you assume that the NSA is as efficient as Google, which is a bit of a stretch. NSA Utah: 65 MW Google Pryor, Oklahoma: 340 MW
        wz1000 6y ago
        Presumably that Google data center does a lot of compute intensive, non search related stuff - like GCP for one.
        jeffbee 6y ago
        Another thing that people persistently misunderstand is the scale relationship between GCP and the rest of Google.
        wz1000 6y ago
        Even if all you say is true and it is truly impossible for the government to replicate any of what Google does, the point is moot. If the government is going to appropriate Google's index, might as well appropriate the datacenters too. Really, whats Google going to do with them once search is gone? According to you, it is the only thing they have have running there.
- pydry 6y ago
  Are you arguing that it's somehow impossible or technically unreasonable for a startup search engine to piggyback off google's search index in a similar way to how duckduckgo piggybacks off bing's search index? And that the very idea that this could be made possible through law makes you chuckle?
  - joshuamorton 6y ago
    Bing doesn't make its search index public. It has a query api, where you can provide a search query and get Bing's results on a per-query basis. The closest comparison to what this person wants is Common Crawl, which is minuscule compared to the big players, and is a 100TB gzipped download that gets updated monthly.
    - pydry 6y ago
      Bing provides API access to its search index. IIRC the type of queries you can do are a bit more sophisticated than just "what's available via the Bing search engine" (e.g. select a market).
      - jeffbee 6y ago
        That's not index access. You are getting ranked results. Index access would give you the posting list for the term "moose", or the intersection of the posting lists of "moose" and "caribou", or whatever. You can't make a neutral service that returns ranked results, because that's contradictory.
  - post_below 6y ago
    I can't answer for OP but I can say it makes me chuckle too. If the solution was purely to force Google to sell access to their index then yes it seems possible on the surface. But as mentioned index and ranking are inextricably tied together. Even if they weren't, no other organization is going to be able to produce search results comparable to Google using their index. You're underestimating what goes on under the hood. So then the answer (often in these conversations) becomes to open up the ranking algos too. The problems with that are numerous so I'll just point out some of the bigger ones: - Arms race: Search is a constant arms race between providers and 3rd parties trying to game the system. The minute you make the algos public, gamers win the race. Search result quality returns to the way it was in the 90's and stays that way until someone else comes up with proprietary algos that work (but is that even legal at this point in our thought experiment?) - Motivation: If search is open and you therefore can't directly profit from your efforts to improve it (because you automatically give away anything you create to competitors) where is your motivation to keep innovating? - It's harder than you think: Truly, there's so much more going on in modern search indexing and ranking than you likely realize. The chances that some new organization (especially a gov organization) given access to Google's black box as it exists right now would be able to maintain search result quality for any significant length of time is essentially zero. But let's imagine that it's as easy as many people think... Wouldn't the solution then be to build a public alternative rather than effectively killing what we have now?
Isinlor 6y ago
Governments can do that right now too. EU is trying to do that right now with Qwant.com where European Investment Bank invested €25 million. So, I'm curious will you start using Qwant.com from now on? https://www.qwant.com/ https://www.qwant.com/
- wz1000 6y ago
  How are my personal decisions relevant? I'm calling for a government appropriation of the search division of Google.
  - Nasrudith 6y ago
    If you say you want it government run after stealing it (lets not mince words here) and aren't using a government run one right now kind of undermines the sincerity of your arguments.
    - wz1000 6y ago
      > aren't using a government run one right now kind of undermines the sincerity of your arguments. First I never said I'm not. Either way it is pretty irrelevant to the discussion. If, in fact, I am not using a government run one right now, maybe it is precisely because the government hasn't yet stolen Google's? If the government already had something as good as Google, clearly there would be no point in them stealing Google's, would there?

by wz1000 6y ago

jeffbee 6y ago

The way that people underestimate the scale and complexity of Google's indexing always makes me chuckle. I'm especially fond of the idea that other parties will just wantonly copy it, like for research purposes or something. Also I'm sorry to inform you that the way the index shards are produced, arranged in memory/on disk, and the manner in which they are queried and ranked are inextricably linked. You can't make a search-neutral index format that can be queried economically.

wz1000 6y ago
> The way that people underestimate the scale and complexity of Google's indexing always makes me chuckle. The same approach that works for Google can also work for a public body supported by public funds. It can also use much of the same systems and maybe even personnel that Google currently employs for this purpose. I'm essentially asking for the search division of Google to be appropriated by the government. > Also I'm sorry to inform you that the way the index shards are produced, arranged in memory/on disk, and the manner in which they are queried and ranked are inextricably linked. You can't make a search-neutral index format that can be queried economically. I'm not asking for a search neutral index - public Google just needs to be (largely) open, transparent and accountable about its ranking algorithm. That's why the stipulation that other entities be allowed to mirror the index - they can optimize the index for their own purposes and rankings on their own hardware. > I'm especially fond of the idea that other parties will just wantonly copy it, like for research purposes or something. Not for research purposes - other entities will be free to copy it for purposes like sovereignty (governments), less/more censorship or to build their own systems on top of it for profit, or any other purpose.
- jeffbee 6y ago
  There are currently zero governments and only about 4 commercial entities possessing a datacenter large enough to do the job.
  - wz1000 6y ago
    Doesn't the NSA have a bunch of massive datacenters? Regardless of that, the datacenter can be appropriated by the government too.
    - jeffbee 6y ago
      No, the NSA has one facility that would be a small/medium datacenter in the big league, but only if you assume that the NSA is as efficient as Google, which is a bit of a stretch. NSA Utah: 65 MW Google Pryor, Oklahoma: 340 MW
      - wz1000 6y ago
        Presumably that Google data center does a lot of compute intensive, non search related stuff - like GCP for one.
        jeffbee 6y ago
        Another thing that people persistently misunderstand is the scale relationship between GCP and the rest of Google.
        wz1000 6y ago
        Even if all you say is true and it is truly impossible for the government to replicate any of what Google does, the point is moot. If the government is going to appropriate Google's index, might as well appropriate the datacenters too. Really, whats Google going to do with them once search is gone? According to you, it is the only thing they have have running there.
pydry 6y ago
Are you arguing that it's somehow impossible or technically unreasonable for a startup search engine to piggyback off google's search index in a similar way to how duckduckgo piggybacks off bing's search index? And that the very idea that this could be made possible through law makes you chuckle?
- joshuamorton 6y ago
  Bing doesn't make its search index public. It has a query api, where you can provide a search query and get Bing's results on a per-query basis. The closest comparison to what this person wants is Common Crawl, which is minuscule compared to the big players, and is a 100TB gzipped download that gets updated monthly.
  - pydry 6y ago
    Bing provides API access to its search index. IIRC the type of queries you can do are a bit more sophisticated than just "what's available via the Bing search engine" (e.g. select a market).
    - jeffbee 6y ago
      That's not index access. You are getting ranked results. Index access would give you the posting list for the term "moose", or the intersection of the posting lists of "moose" and "caribou", or whatever. You can't make a neutral service that returns ranked results, because that's contradictory.
- post_below 6y ago
  I can't answer for OP but I can say it makes me chuckle too. If the solution was purely to force Google to sell access to their index then yes it seems possible on the surface. But as mentioned index and ranking are inextricably tied together. Even if they weren't, no other organization is going to be able to produce search results comparable to Google using their index. You're underestimating what goes on under the hood. So then the answer (often in these conversations) becomes to open up the ranking algos too. The problems with that are numerous so I'll just point out some of the bigger ones: - Arms race: Search is a constant arms race between providers and 3rd parties trying to game the system. The minute you make the algos public, gamers win the race. Search result quality returns to the way it was in the 90's and stays that way until someone else comes up with proprietary algos that work (but is that even legal at this point in our thought experiment?) - Motivation: If search is open and you therefore can't directly profit from your efforts to improve it (because you automatically give away anything you create to competitors) where is your motivation to keep innovating? - It's harder than you think: Truly, there's so much more going on in modern search indexing and ranking than you likely realize. The chances that some new organization (especially a gov organization) given access to Google's black box as it exists right now would be able to maintain search result quality for any significant length of time is essentially zero. But let's imagine that it's as easy as many people think... Wouldn't the solution then be to build a public alternative rather than effectively killing what we have now?

Isinlor 6y ago

Governments can do that right now too. EU is trying to do that right now with Qwant.com where European Investment Bank invested €25 million. So, I'm curious will you start using Qwant.com from now on? https://www.qwant.com/ https://www.qwant.com/

wz1000 6y ago
How are my personal decisions relevant? I'm calling for a government appropriation of the search division of Google.
- Nasrudith 6y ago
  If you say you want it government run after stealing it (lets not mince words here) and aren't using a government run one right now kind of undermines the sincerity of your arguments.
  - wz1000 6y ago
    > aren't using a government run one right now kind of undermines the sincerity of your arguments. First I never said I'm not. Either way it is pretty irrelevant to the discussion. If, in fact, I am not using a government run one right now, maybe it is precisely because the government hasn't yet stolen Google's? If the government already had something as good as Google, clearly there would be no point in them stealing Google's, would there?