4 ms·

It strikes me that the entire article/proposal is based on a faulty premise: "After all, the value is not in the index it is in the analysis of that index." T

It strikes me that the entire article/proposal is based on a faulty premise: "After all, the value is not in the index it is in the analysis of that index." The ability for a given search engine to innovate is based on having control of the index. The line between indexing and analysis isn't quite as clean as what is implied by the article, if only for the simple fact that you can only analyze what is in the index. For example, at it's simplest and index is a list of what words are in what document on the web. But what if I want to give greater weight to words that are in document titles or headings? Then I need to somehow put that into the index. What if I want to use the proximity between words to determine relevance of a result for a particular phrase? Need to get that info into the index, too. In the end, what the author really wants is for someone to maintain a separate copy of the internet for bots. In order for someone to do that, they'd need to charge the bot owners, but the bot owners could just index your content for free, so why would they pay?

mapgrep 15y ago
Three easy reasons search engine owners /might/ pay for a full copy of the web crawl: -Faster. You don't have the latency of millions of HTTP connections, but instead a single download. (Or a few dozen. Or a van full of hard drives.) -Easier. The problem of crawling quickly but politely has been handled for you. The reading of sitemaps has been handled for you. The problem of deciding how deep to crawl, and when to write off a subsite as an endless black hole, has been handled for you. Etc. -Predictable. Figuring out, in advance, how much it is going to cost you to crawl some/all of the web is, to say the least, tricky. Buying a copy with a known price tag provides a measure of certainty. Of course, I am leaving out the potential pitfalls, but the point is there /are/ arguments in favor of buying a copy of the web (and then building your own index).
- geekfactor 15y ago
  All good points. Maintaining a current and high quality index is clearly not free in terms of development time, bandwidth, storage or a host of other factors.
oh_sigh 15y ago
You are setting up a straw man by saying "at it's simplest an index is a list of what words are in what documents". That kind of index(an inverted word index) could be generated from the ideal data format, but it would be stupid store such a rich data set in such a dumb, information-losing format. "The index" would just be a list of key/values. The key would be a URL, and the value would be the content located at that URL. There would also be some kind of metadata attached to the keys to indicate HTTP status code, HTTP header information, last-crawled date, and any other interesting data. From this data set, other, more appropriate indexes could be generated(for example via hadoop) > In the end, what the author really wants is for someone to maintain a separate copy of the internet for bots. Yes, but not for bots. It would be for algorithms. > In order for someone to do that, they'd need to charge the bot owners Probably.. The funding could be like ICANN, whose long-term funding comes from groups that benefit from its services. > but the bot owners could just index your content for free, so why would they pay? How would you create a copy of the internet for free? Are you just going to run your crawler on your home modem while you're at work? Where are you going to store all that data? How are you going to process it? How long is that going to take? Wouldn't it just be easier to(for example) mount a shared EBS volume in EC2 that has the latest, most up to date crawled internet available for your processing?
- munificent 15y ago
  > "The index" would just be a list of key/values. The key would be a URL, and the value would be the content located at that URL. They have that index already. It's called "the internet". If you're storing the entire page content, that's not really much of an "index". > it would be stupid store such a rich data set in such a dumb, information-losing format. Discarding information is the entire purpose of indexing and doing things like map-reduce (emphasis on the reduce!): you discard worthless information so that the stuff you care about is of a manageable size. The reason Google serves up search results quickly is because it carefully controls the data it has to wade through. Adding more data doesn't make it smarter, it makes it slower.
  - oh_sigh 15y ago
    > They have that index already. It's called "the internet". If you're storing the entire page content, that's not really much of an "index". Fine. Let's call it an archive, or cache, or snapshot then. I only used that phrase because it was thrown around in the post I was responding to, and I specifically put that in quotes because I wasn't sure it was the right phrase to use. > Discarding information is the entire purpose of indexing and doing things like map-reduce (emphasis on the reduce!) No. Just no. An index is a data structure that is designed to quickly lookup its elements based on some key. It implies nothing about what your data(neither keys nor elements) look like. Are you serious about "emphasis on reduce!"? "Reduce" is not named reduce because it inherently reduces the amount of data you are working with. It is called "reduce" because 1) an engineer at google liked the sound of it, and 2) it takes a list of intermediary values with all the same key. It is quite easy, and common, to have reduce functions which end up spitting out MORE data than you started with > you discard worthless information so that the stuff you care about is of a manageable size. Sometimes, but that doesn't work in this situation. How do you know what data is worthless before you know what algorithms will be applied to the data? The only acceptable solution is to keep a plaintext copy of the data retrieved from a particular URL.

mapgrep 15y ago

Three easy reasons search engine owners /might/ pay for a full copy of the web crawl: -Faster. You don't have the latency of millions of HTTP connections, but instead a single download. (Or a few dozen. Or a van full of hard drives.) -Easier. The problem of crawling quickly but politely has been handled for you. The reading of sitemaps has been handled for you. The problem of deciding how deep to crawl, and when to write off a subsite as an endless black hole, has been handled for you. Etc. -Predictable. Figuring out, in advance, how much it is going to cost you to crawl some/all of the web is, to say the least, tricky. Buying a copy with a known price tag provides a measure of certainty. Of course, I am leaving out the potential pitfalls, but the point is there /are/ arguments in favor of buying a copy of the web (and then building your own index).

geekfactor 15y ago
All good points. Maintaining a current and high quality index is clearly not free in terms of development time, bandwidth, storage or a host of other factors.

oh_sigh 15y ago

You are setting up a straw man by saying "at it's simplest an index is a list of what words are in what documents". That kind of index(an inverted word index) could be generated from the ideal data format, but it would be stupid store such a rich data set in such a dumb, information-losing format. "The index" would just be a list of key/values. The key would be a URL, and the value would be the content located at that URL. There would also be some kind of metadata attached to the keys to indicate HTTP status code, HTTP header information, last-crawled date, and any other interesting data. From this data set, other, more appropriate indexes could be generated(for example via hadoop) > In the end, what the author really wants is for someone to maintain a separate copy of the internet for bots. Yes, but not for bots. It would be for algorithms. > In order for someone to do that, they'd need to charge the bot owners Probably.. The funding could be like ICANN, whose long-term funding comes from groups that benefit from its services. > but the bot owners could just index your content for free, so why would they pay? How would you create a copy of the internet for free? Are you just going to run your crawler on your home modem while you're at work? Where are you going to store all that data? How are you going to process it? How long is that going to take? Wouldn't it just be easier to(for example) mount a shared EBS volume in EC2 that has the latest, most up to date crawled internet available for your processing?

munificent 15y ago
> "The index" would just be a list of key/values. The key would be a URL, and the value would be the content located at that URL. They have that index already. It's called "the internet". If you're storing the entire page content, that's not really much of an "index". > it would be stupid store such a rich data set in such a dumb, information-losing format. Discarding information is the entire purpose of indexing and doing things like map-reduce (emphasis on the reduce!): you discard worthless information so that the stuff you care about is of a manageable size. The reason Google serves up search results quickly is because it carefully controls the data it has to wade through. Adding more data doesn't make it smarter, it makes it slower.
- oh_sigh 15y ago
  > They have that index already. It's called "the internet". If you're storing the entire page content, that's not really much of an "index". Fine. Let's call it an archive, or cache, or snapshot then. I only used that phrase because it was thrown around in the post I was responding to, and I specifically put that in quotes because I wasn't sure it was the right phrase to use. > Discarding information is the entire purpose of indexing and doing things like map-reduce (emphasis on the reduce!) No. Just no. An index is a data structure that is designed to quickly lookup its elements based on some key. It implies nothing about what your data(neither keys nor elements) look like. Are you serious about "emphasis on reduce!"? "Reduce" is not named reduce because it inherently reduces the amount of data you are working with. It is called "reduce" because 1) an engineer at google liked the sound of it, and 2) it takes a list of intermediary values with all the same key. It is quite easy, and common, to have reduce functions which end up spitting out MORE data than you started with > you discard worthless information so that the stuff you care about is of a manageable size. Sometimes, but that doesn't work in this situation. How do you know what data is worthless before you know what algorithms will be applied to the data? The only acceptable solution is to keep a plaintext copy of the data retrieved from a particular URL.