4 ms·

The search index should really be nationalised and treated like a public utility that can't favour one engine over another. At this point only nation states can

The search index should really be nationalised and treated like a public utility that can't favour one engine over another. At this point only nation states can really compete on building a search index - it requires titanic amounts of computing and electrical power. Whereas duck duck go proves that a search engine can operate successfully as a startup.

JoshTriplett 7y ago
"The search index"? Leaving aside any other problems that would cause, that sounds like a recipe for locking in specific formats and technologies forever, and preventing any further innovation. A search index is not simply a function from search string to results. Treating it as such would oversimplify the potential value or future innovation of a search engine. A search index includes the data and metadata of all the pages indexed, plus derived data in various forms ranging from data structures to trained neural networks, plus sufficiently fast access to the data to make it possible to generate new such data structures or train new networks. And all that while continuing to index new pages as fast as possible. A search index used internally by one company can change formats and technologies as long as the products using it have updated accordingly, and can provide many different facets of information. A search index used by multiple entities (likely competing entities) can't easily co-evolve with the search engines using it, can't easily change formats or technologies, can't easily discard older ones (and thus needs far more resources if it innovates), and has a much more difficult time providing previously unanticipated information. Don't get me wrong, I'd love to see a good way to support innovation on the scale of a production search index without requiring billion-dollar infrastructure, but separating the search index from a search engine would substantially increase the complexity and expense of both, and seems quite likely to curtail innovation.
lazyjones 7y ago
> building a search index - it requires titanic amounts of computing and electrical power. That's because it's being done in an extremely wasteful way that isn't really necessary. Proper regulation could enforce a better interface between websites and search engines than stupidly and repeatedly crawling bloated web pages and throwing away almost all their contents. Google has used its computing resources to stay ahead in this game and has no motivation to fix this waste of time and energy. All they've done lately is to suggest using sitemaps for dynamic web sites so their crawlers don't get stuck there forever.
root_axis 7y ago
Can we first nationalize resources that are actually vital to human survival like housing, education, and life-saving medicine?
- pinkfoot 7y ago
  No.
  - root_axis 7y ago
    Care to offer up any reasoning alongside your one-word reply?
    - pinkfoot 7y ago
      No more than the OP’s one line suggestion.
      - root_axis 7y ago
        I didn't bring up the nationalization suggestion, but if nationalization is on the table, why not nationalize resources vital to human survival before relative trivialities like Google's search index?
        pinkfoot 7y ago
        I don’t know, why are you asking me? Make your case.
        root_axis 7y ago
        lol, I posed the question to the person who suggested it, but you decided to inject yourself into the conversation for some inexplicable reason. You don't appear to be engaging in good faith, so I'll end the discussion with you here. Have a great day.

JoshTriplett 7y ago

"The search index"? Leaving aside any other problems that would cause, that sounds like a recipe for locking in specific formats and technologies forever, and preventing any further innovation. A search index is not simply a function from search string to results. Treating it as such would oversimplify the potential value or future innovation of a search engine. A search index includes the data and metadata of all the pages indexed, plus derived data in various forms ranging from data structures to trained neural networks, plus sufficiently fast access to the data to make it possible to generate new such data structures or train new networks. And all that while continuing to index new pages as fast as possible. A search index used internally by one company can change formats and technologies as long as the products using it have updated accordingly, and can provide many different facets of information. A search index used by multiple entities (likely competing entities) can't easily co-evolve with the search engines using it, can't easily change formats or technologies, can't easily discard older ones (and thus needs far more resources if it innovates), and has a much more difficult time providing previously unanticipated information. Don't get me wrong, I'd love to see a good way to support innovation on the scale of a production search index without requiring billion-dollar infrastructure, but separating the search index from a search engine would substantially increase the complexity and expense of both, and seems quite likely to curtail innovation.

lazyjones 7y ago

> building a search index - it requires titanic amounts of computing and electrical power. That's because it's being done in an extremely wasteful way that isn't really necessary. Proper regulation could enforce a better interface between websites and search engines than stupidly and repeatedly crawling bloated web pages and throwing away almost all their contents. Google has used its computing resources to stay ahead in this game and has no motivation to fix this waste of time and energy. All they've done lately is to suggest using sitemaps for dynamic web sites so their crawlers don't get stuck there forever.

root_axis 7y ago

Can we first nationalize resources that are actually vital to human survival like housing, education, and life-saving medicine?

pinkfoot 7y ago
No.
- root_axis 7y ago
  Care to offer up any reasoning alongside your one-word reply?
  - pinkfoot 7y ago
    No more than the OP’s one line suggestion.
    - root_axis 7y ago
      I didn't bring up the nationalization suggestion, but if nationalization is on the table, why not nationalize resources vital to human survival before relative trivialities like Google's search index?
      - pinkfoot 7y ago
        I don’t know, why are you asking me? Make your case.
        root_axis 7y ago
        lol, I posed the question to the person who suggested it, but you decided to inject yourself into the conversation for some inexplicable reason. You don't appear to be engaging in good faith, so I'll end the discussion with you here. Have a great day.