4 ms·

On the other hand it doesn't seem that much when you consider todays storage density. You can fit around 0.5 PB into one rack nowadays. 200 racks then sounds a

by moe 16y ago

On the other hand it doesn't seem that much when you consider todays storage density. You can fit around 0.5 PB into one rack nowadays. 200 racks then sounds a bit less impressive than 100 Petabytes. However, that ofcourse doesn't account for redundancy, nor for doing anything useful with such a pile of data. Both of which impose some interesting challenges at that scale.

blasdel 16y ago
You think Google's mainline storage of their index is on disk? It is to laugh. I don't think that's been true since around the time they stopped building server shelves out of lego or cardboard. They've been keeping the full text of the web in RAM (for the snippets), with indexes, several times over. With independent live siblings in multiple datacenters.
- moe 16y ago
  Notice the last sentence? I merely tried to put the figure into perspective.
- houseabsolute 16y ago
  Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem cheaper until you consider they'd have to double the number of machines to hold them. Now they mentioned that the size of the index is 100,000,000 GB, which means that they would have spent $2 billion on this RAM alone, not to mention all the other components that are required to house it. Especially considering how much their capital expenditure has fallen lately (http://www.datacenterknowledge.com/archives/2010/01/22/google-capex-edges-higher-in-q4/ http://www.datacenterknowledge.com/archives/2010/01/22/googl...), it doesn't seem very likely that they could afford to spend even $2 billion on this. And the assumptions I made about the price of RAM are pretty generous, considering they'd have had to have been acquiring this for some time, and therefore that the RAM would have been more expensive previously.
  - blasdel 16y ago
    Your retail RAM prices are inflated about 3x, but I think your point is correct. You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds. Perhaps they've done testing and found that absurdly fast results don't matter (or no longer matter) as much as they thought?
    - rxin 16y ago
      Your memory might've been wrong. 0.00025 is just too small a number for a system like this (Multi tier, RPC every where distributed systems).
    - houseabsolute 16y ago
      Sorry, that's just what I saw on Newegg for 8GB sticks. I did not intend to mislead. I am under the impression that most of Google's index is still served out of ram. Certainly I never saw any announcements to the contrary. The other poster also pointed out that sub-millisecond latencies are pretty unlikely. Considering all the diverse sets of data they need to pull from, it would be similarly difficult to believe that tens or hundreds of disk seeks could be carried out in a timely manner for each query to yield a 200 ms total calculation time. If I had to guess they have probably just started taking factors into account that they did not before. For example, perhaps they are measuring total internal latency rather than just the latency of a particular sub-system. Or maybe there are components that go to disk while the posting list lookups are done in RAM.
  - drats 16y ago
    That link has over 7 billion of capex for 2006-2009 there without accounting for their earlier pre-2006 hardware. It's eminently feasible.
    - houseabsolute 16y ago
      Most of that was during a time when RAM was significantly more expensive than my calculations took into account. But in any case, you and I are talking about two different things. You're saying that there is some possible universe where they are using 100 PB of ram on this project alone. I'm claiming that it's very unlikely that this is the same universe as the one we are living in. While I'll gladly admit that you are correct in your claim, I don't think it's useful.
- herdrick 16y ago
  No need to laugh. It was a reasonable assumption.

by moe 16y ago

blasdel 16y ago

You think Google's mainline storage of their index is on disk? It is to laugh. I don't think that's been true since around the time they stopped building server shelves out of lego or cardboard. They've been keeping the full text of the web in RAM (for the snippets), with indexes, several times over. With independent live siblings in multiple datacenters.

moe 16y ago
Notice the last sentence? I merely tried to put the figure into perspective.
houseabsolute 16y ago
Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem cheaper until you consider they'd have to double the number of machines to hold them. Now they mentioned that the size of the index is 100,000,000 GB, which means that they would have spent $2 billion on this RAM alone, not to mention all the other components that are required to house it. Especially considering how much their capital expenditure has fallen lately (http://www.datacenterknowledge.com/archives/2010/01/22/google-capex-edges-higher-in-q4/ http://www.datacenterknowledge.com/archives/2010/01/22/googl...), it doesn't seem very likely that they could afford to spend even $2 billion on this. And the assumptions I made about the price of RAM are pretty generous, considering they'd have had to have been acquiring this for some time, and therefore that the RAM would have been more expensive previously.
- blasdel 16y ago
  Your retail RAM prices are inflated about 3x, but I think your point is correct. You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds. Perhaps they've done testing and found that absurdly fast results don't matter (or no longer matter) as much as they thought?
  - rxin 16y ago
    Your memory might've been wrong. 0.00025 is just too small a number for a system like this (Multi tier, RPC every where distributed systems).
  - houseabsolute 16y ago
    Sorry, that's just what I saw on Newegg for 8GB sticks. I did not intend to mislead. I am under the impression that most of Google's index is still served out of ram. Certainly I never saw any announcements to the contrary. The other poster also pointed out that sub-millisecond latencies are pretty unlikely. Considering all the diverse sets of data they need to pull from, it would be similarly difficult to believe that tens or hundreds of disk seeks could be carried out in a timely manner for each query to yield a 200 ms total calculation time. If I had to guess they have probably just started taking factors into account that they did not before. For example, perhaps they are measuring total internal latency rather than just the latency of a particular sub-system. Or maybe there are components that go to disk while the posting list lookups are done in RAM.
- drats 16y ago
  That link has over 7 billion of capex for 2006-2009 there without accounting for their earlier pre-2006 hardware. It's eminently feasible.
  - houseabsolute 16y ago
    Most of that was during a time when RAM was significantly more expensive than my calculations took into account. But in any case, you and I are talking about two different things. You're saying that there is some possible universe where they are using 100 PB of ram on this project alone. I'm claiming that it's very unlikely that this is the same universe as the one we are living in. While I'll gladly admit that you are correct in your claim, I don't think it's useful.
herdrick 16y ago
No need to laugh. It was a reasonable assumption.