4 ms·

Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for inde

Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem cheaper until you consider they'd have to double the number of machines to hold them. Now they mentioned that the size of the index is 100,000,000 GB, which means that they would have spent $2 billion on this RAM alone, not to mention all the other components that are required to house it. Especially considering how much their capital expenditure has fallen lately (http://www.datacenterknowledge.com/archives/2010/01/22/google-capex-edges-higher-in-q4/ http://www.datacenterknowledge.com/archives/2010/01/22/googl...), it doesn't seem very likely that they could afford to spend even $2 billion on this. And the assumptions I made about the price of RAM are pretty generous, considering they'd have had to have been acquiring this for some time, and therefore that the RAM would have been more expensive previously.

blasdel 16y ago
Your retail RAM prices are inflated about 3x, but I think your point is correct. You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds. Perhaps they've done testing and found that absurdly fast results don't matter (or no longer matter) as much as they thought?
- rxin 16y ago
  Your memory might've been wrong. 0.00025 is just too small a number for a system like this (Multi tier, RPC every where distributed systems).
- houseabsolute 16y ago
  Sorry, that's just what I saw on Newegg for 8GB sticks. I did not intend to mislead. I am under the impression that most of Google's index is still served out of ram. Certainly I never saw any announcements to the contrary. The other poster also pointed out that sub-millisecond latencies are pretty unlikely. Considering all the diverse sets of data they need to pull from, it would be similarly difficult to believe that tens or hundreds of disk seeks could be carried out in a timely manner for each query to yield a 200 ms total calculation time. If I had to guess they have probably just started taking factors into account that they did not before. For example, perhaps they are measuring total internal latency rather than just the latency of a particular sub-system. Or maybe there are components that go to disk while the posting list lookups are done in RAM.
drats 16y ago
That link has over 7 billion of capex for 2006-2009 there without accounting for their earlier pre-2006 hardware. It's eminently feasible.
- houseabsolute 16y ago
  Most of that was during a time when RAM was significantly more expensive than my calculations took into account. But in any case, you and I are talking about two different things. You're saying that there is some possible universe where they are using 100 PB of ram on this project alone. I'm claiming that it's very unlikely that this is the same universe as the one we are living in. While I'll gladly admit that you are correct in your claim, I don't think it's useful.
iamelgringo 16y ago
Why use RAM? You can get a 500GB SSD for $1000. 100 PB would cost you $200 Million, which is peanuts to Google. Or you could use PCIe SSD drives. You can get a TB of PCIe for $3000, and a 5 PCIe x16 mother board at Fry's costs around $300. You could get 100PB of PCIe SSd storage for around $300 M. You could use RAM for very fast cache, and SSD storage for faster than HDD indexing and retrieval.
- houseabsolute 16y ago
  $200M is still a quarter of their capital expenditure in 2009 not including the other parts of the machines. And before 2009 flash was much more expensive and not as fast as it is now. It would surprise me greatly to hear that they had gone this route.
- robk 16y ago
  Ssds aren't reliable enough for Google scale. The rater number of writes before failure for Flash would make replacement a constant effort for them and ruin any economy. You need Ram for the reliability as well as the random access speed.
gojomo 16y ago
The post wording "Caffeine takes up nearly 100 million gigabytes of storage in one database" isn't clear that's all needed for the constantly-consulted index; just that's what the indexing system uses, including perhaps rarely-consulted information. They might also be reporting uncompressed data sizes, or counting all the raw 'storage' used to store info in triplicate (or more). So the live data consulted for almost all web queries might be be much much less than 100 PiB of index data, and thus fit far more economically in RAM.
- houseabsolute 16y ago
  Right. That is why I distinguished between the serving data and the data they discussed in this post.

blasdel 16y ago

Your retail RAM prices are inflated about 3x, but I think your point is correct. You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds. Perhaps they've done testing and found that absurdly fast results don't matter (or no longer matter) as much as they thought?

rxin 16y ago
Your memory might've been wrong. 0.00025 is just too small a number for a system like this (Multi tier, RPC every where distributed systems).
houseabsolute 16y ago
Sorry, that's just what I saw on Newegg for 8GB sticks. I did not intend to mislead. I am under the impression that most of Google's index is still served out of ram. Certainly I never saw any announcements to the contrary. The other poster also pointed out that sub-millisecond latencies are pretty unlikely. Considering all the diverse sets of data they need to pull from, it would be similarly difficult to believe that tens or hundreds of disk seeks could be carried out in a timely manner for each query to yield a 200 ms total calculation time. If I had to guess they have probably just started taking factors into account that they did not before. For example, perhaps they are measuring total internal latency rather than just the latency of a particular sub-system. Or maybe there are components that go to disk while the posting list lookups are done in RAM.

drats 16y ago

That link has over 7 billion of capex for 2006-2009 there without accounting for their earlier pre-2006 hardware. It's eminently feasible.

houseabsolute 16y ago
Most of that was during a time when RAM was significantly more expensive than my calculations took into account. But in any case, you and I are talking about two different things. You're saying that there is some possible universe where they are using 100 PB of ram on this project alone. I'm claiming that it's very unlikely that this is the same universe as the one we are living in. While I'll gladly admit that you are correct in your claim, I don't think it's useful.

iamelgringo 16y ago

Why use RAM? You can get a 500GB SSD for $1000. 100 PB would cost you $200 Million, which is peanuts to Google. Or you could use PCIe SSD drives. You can get a TB of PCIe for $3000, and a 5 PCIe x16 mother board at Fry's costs around $300. You could get 100PB of PCIe SSd storage for around $300 M. You could use RAM for very fast cache, and SSD storage for faster than HDD indexing and retrieval.

houseabsolute 16y ago
$200M is still a quarter of their capital expenditure in 2009 not including the other parts of the machines. And before 2009 flash was much more expensive and not as fast as it is now. It would surprise me greatly to hear that they had gone this route.
robk 16y ago
Ssds aren't reliable enough for Google scale. The rater number of writes before failure for Flash would make replacement a constant effort for them and ruin any economy. You need Ram for the reliability as well as the random access speed.

gojomo 16y ago

The post wording "Caffeine takes up nearly 100 million gigabytes of storage in one database" isn't clear that's all needed for the constantly-consulted index; just that's what the indexing system uses, including perhaps rarely-consulted information. They might also be reporting uncompressed data sizes, or counting all the raw 'storage' used to store info in triplicate (or more). So the live data consulted for almost all web queries might be be much much less than 100 PiB of index data, and thus fit far more economically in RAM.

houseabsolute 16y ago
Right. That is why I distinguished between the serving data and the data they discussed in this post.