5 ms·

yeah, sure... lets make a system and store all kind of information there, so people can browse it... it would be great to distribute it around the world, maybe

yeah, sure... lets make a system and store all kind of information there, so people can browse it... it would be great to distribute it around the world, maybe on different companies, and sync the data every day so it keeps fresh... I don't know, maybe we can even have every person store their own data on their own private server... but of course, in an open index... </sarcasm>

VladRussian 15y ago
too bad that upvote number for your comment is invisible. It would be interesting to know how many people is able to see it through the same way Edit: looking at the other comments, doesn't seem to be that that many were able to get it. What a disappointing state of minds.
- spiralganglion 15y ago
  The people who "get it" are just going to have a chortle and move on. Commenting to the effect of "I got it, that was funny" doesn't add anything to the conversation.
drivebyacct2 15y ago
I'm not sure I understand your point/perspective.
- DrCatbox 15y ago
  He just told you what the web is today. As it was since 1991.
- nkassis 15y ago
  basically his point is that the internet is the open index. At least that's what I think I understand. Centralizing it all so that one company has to run the index would mean that company would have to foot the entire bandwidth bill which they won't do, so it would end up distributed as the internet already is. It's just a shifting of the traffic to a centralized db which would need to be kept up to date.
  - drivebyacct2 15y ago
    I don't understand why a decentralized, open, free index is a bad thing. It's not the same as it is today and there's no reason one person would have to carry all the load.
    - ctdonath 15y ago
      Every town has a library. Let's just close all the bookstores then.
      - drivebyacct2 15y ago
        Is anyone here capable of having a conversation instead of jabs, quips or snark? Or at least a good reason why a distributed or copied, shared index is a bad thing for anyone?
        ctdonath 15y ago
        Sometimes there's a point to the snark, which is missed by the downvoters. There's more to an index than "hey, let's share an index". That's WHY we have multiple search engines: there will be differing views of what should be in that shared index, and how it should be constructed to facilitate varying needs. There are multiple search engines because each one plays off "hey, our index is better than theirs". My snark makes the point: why not just have a single distributed/copied/shared library system, and do away with pricy competition? Answer: because that single system does not provide everything everyone wants, and people are willing to pay for a different selection & service. The search engine and index are tightly coupled. They're huge, they're complex, they're expensive - and people think they can make a good buck by somehow doing it different. Create a "universal index", and someone will realize they can make money by making their own, leading right back to what we have today. The TFA's key issue isn't really that multiple indexes are crawling his website, it's that they're gathering his data thru the most inefficient means possible - polling every page as often as practical. You don't want universal index, because short of banning competing indexes there will always be competing indexes. You want an agreed-on search interface: a means to serve the indexes what they want at a cost lower than what it costs them to poll-mine your site. Yeah, a universal index is a bad idea. It ignores the fact (you'd think visitors to ycombinator of all sites would get this) that there is money in competition, ergo there will never be a single index. That, and it still doesn't solve the problem: it may reduce the number of crawlers sucking your bandwidth, but it's still polling every page as fast and often as it can.
uxp 15y ago
I'm blinded by what you are trying to imply sarcastically. A rough distributed model could be implemented similar to the way we (hackers/coders) use github as a central repository for a distributed system. People contributing to the index on a private server could do whatever they want but since that instance of the index is not public, no one else will care about what the owner has done to it. Forks can be pushed to a public staging area where others can view it and verify it's accuracy, and then the major players can merge those changes into their forks. The complaint (with github) that it is hard to figure out the canonical repo is also invalid in this model, as one can start with a fork of Google or Yahoo's public repo, and then build their own through merging or hacking directly on it, just like one can fork Linus' Linux kernel, and then merge in other's forks to incorporate other changes. Remember, the index itself, as in the raw data taken in by GoogleBot or Yahoo! Slurp bot, would be the shared information. The analysis of the data, as in pagerank and other factors that Google decides makes one page more relevant to a keyword than the other, would not be shared as that is the bread and butter of each engine.
- DrCatbox 15y ago
  The sarcasm was because the idea precisly explains what we have today, it is called the web.
  - recoiledsnake 15y ago
    So the web wikipedia.org is the same as making the files available at http://dumps.wikimedia.org/enwiki/latest/ http://dumps.wikimedia.org/enwiki/latest/ ?
- ctdonath 15y ago
  His is a variant of "I have a map of the United States... Actual size. It says, 'Scale: 1 mile = 1 mile.' I spent last summer folding it. I hardly ever unroll it. People ask me where I live, and I say, 'E6." - Steven Wright

VladRussian 15y ago

too bad that upvote number for your comment is invisible. It would be interesting to know how many people is able to see it through the same way Edit: looking at the other comments, doesn't seem to be that that many were able to get it. What a disappointing state of minds.

spiralganglion 15y ago
The people who "get it" are just going to have a chortle and move on. Commenting to the effect of "I got it, that was funny" doesn't add anything to the conversation.

drivebyacct2 15y ago

I'm not sure I understand your point/perspective.

DrCatbox 15y ago
He just told you what the web is today. As it was since 1991.
nkassis 15y ago
basically his point is that the internet is the open index. At least that's what I think I understand. Centralizing it all so that one company has to run the index would mean that company would have to foot the entire bandwidth bill which they won't do, so it would end up distributed as the internet already is. It's just a shifting of the traffic to a centralized db which would need to be kept up to date.
- drivebyacct2 15y ago
  I don't understand why a decentralized, open, free index is a bad thing. It's not the same as it is today and there's no reason one person would have to carry all the load.
  - ctdonath 15y ago
    Every town has a library. Let's just close all the bookstores then.
    - drivebyacct2 15y ago
      Is anyone here capable of having a conversation instead of jabs, quips or snark? Or at least a good reason why a distributed or copied, shared index is a bad thing for anyone?
      - ctdonath 15y ago
        Sometimes there's a point to the snark, which is missed by the downvoters. There's more to an index than "hey, let's share an index". That's WHY we have multiple search engines: there will be differing views of what should be in that shared index, and how it should be constructed to facilitate varying needs. There are multiple search engines because each one plays off "hey, our index is better than theirs". My snark makes the point: why not just have a single distributed/copied/shared library system, and do away with pricy competition? Answer: because that single system does not provide everything everyone wants, and people are willing to pay for a different selection & service. The search engine and index are tightly coupled. They're huge, they're complex, they're expensive - and people think they can make a good buck by somehow doing it different. Create a "universal index", and someone will realize they can make money by making their own, leading right back to what we have today. The TFA's key issue isn't really that multiple indexes are crawling his website, it's that they're gathering his data thru the most inefficient means possible - polling every page as often as practical. You don't want universal index, because short of banning competing indexes there will always be competing indexes. You want an agreed-on search interface: a means to serve the indexes what they want at a cost lower than what it costs them to poll-mine your site. Yeah, a universal index is a bad idea. It ignores the fact (you'd think visitors to ycombinator of all sites would get this) that there is money in competition, ergo there will never be a single index. That, and it still doesn't solve the problem: it may reduce the number of crawlers sucking your bandwidth, but it's still polling every page as fast and often as it can.

uxp 15y ago

I'm blinded by what you are trying to imply sarcastically. A rough distributed model could be implemented similar to the way we (hackers/coders) use github as a central repository for a distributed system. People contributing to the index on a private server could do whatever they want but since that instance of the index is not public, no one else will care about what the owner has done to it. Forks can be pushed to a public staging area where others can view it and verify it's accuracy, and then the major players can merge those changes into their forks. The complaint (with github) that it is hard to figure out the canonical repo is also invalid in this model, as one can start with a fork of Google or Yahoo's public repo, and then build their own through merging or hacking directly on it, just like one can fork Linus' Linux kernel, and then merge in other's forks to incorporate other changes. Remember, the index itself, as in the raw data taken in by GoogleBot or Yahoo! Slurp bot, would be the shared information. The analysis of the data, as in pagerank and other factors that Google decides makes one page more relevant to a keyword than the other, would not be shared as that is the bread and butter of each engine.

DrCatbox 15y ago
The sarcasm was because the idea precisly explains what we have today, it is called the web.
- recoiledsnake 15y ago
  So the web wikipedia.org is the same as making the files available at http://dumps.wikimedia.org/enwiki/latest/ http://dumps.wikimedia.org/enwiki/latest/ ?
ctdonath 15y ago
His is a variant of "I have a map of the United States... Actual size. It says, 'Scale: 1 mile = 1 mile.' I spent last summer folding it. I hardly ever unroll it. People ask me where I live, and I say, 'E6." - Steven Wright