6 ms·

RAMCloud - All Your Storage in DRAM

64 points by gjenks 14y ago

DoubleCluster 14y ago
The idea is not that weird. Lots of applications run from RAM most of the time, needing long cache warmups before performing well. If you keep your entire storage in RAM this isn't necessary, and you can do away with the caching layer.
- caw 14y ago
  This looks like the "RAM SAN" products, where there's a bunch of memory and some spinning disks with enough local battery backup to flush to disk. I think the main difference is that they want the data aggregated across multiple machines, which I don't think that previous solution had.
sheraz 14y ago
Somewhat relevant, but on a much smaller scale I suppose: I had to geocode 100million addresses and used PostGIS to get it done. It was painfully slow on disk, so I rented some big-ram servers, created ramdisks, configured postgres to run from those ramdisks, and it was off to the races. What would have taken days of processing was taken down to several hours. The speed improvement was amazing. Even SSD couldn't touch it.
ricardobeat 14y ago
Can anyone shed light on how persistence is achieved? The wiki page doesn't seem to come to any conclusions: https://ramcloud.stanford.edu/wiki/display/ramcloud/Data+persistence https://ramcloud.stanford.edu/wiki/display/ramcloud/Data+per...
- rektide 14y ago
  Hire more server huggers, make extra sure das-blinkenlights stay blinking, buying a second backup generator. And- obviously enough- off site redundancy; multiple data centers. Which they talk about the difficulty of: 10Gbps pipes between DCs is not a lot compared to write throughput of a single computer's main memory, much less a DC's. For some compressible checkpoint logs there might be a way forward. Perhaps we need a new *-cast technology: after any-cast we'll all be needing n-cast, which sends our traffic to the n-nearest DC's? :)
  - s_baby 14y ago
    If it's a distributed network, data can be distributed across nodes using a RAID algorithm.
- rstutsman 14y ago
  Here's the long answer to your question: http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud-recovery.pdf http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud-reco...
rektide 14y ago
From recent coverage of Open Compute on /.: RAM Sled: Facebook wants to replace the leaves and run it on a RAM sled with between 128 GB and 512 GB of memory, and for $500 to $700 per sled. Only a basic CPU would be needed. Total queries would be 450,000 to 1 million key queries per second. http://slashdot.org/topic/datacenter/how-facebook-will-power-graph-search/ http://slashdot.org/topic/datacenter/how-facebook-will-power... Not really any way to know what Facebook is up to here: 512 GB is normally done via 4 sockets, 4 channels, 2 dimms per channel, 16GB dimms; whether FB intends on disrupting any of the factors in this equation for getting mass memory on a single system will be interesting to see. It has been rather shocking to me that FB-DIMMS & other approaches which allow a lot of RAM to be chained to one another- really deep channels- hasn't seen any widescale success. RAM is cheap enough: would that we could plug in a lot of it.
- wmf 14y ago
  If you just care about capacity and cost you can build a system with a $20 ARM SoC for every two DIMMs. The ultimate is BlueGene-style packaging with an SoC on each DIMM, but Facebook's probably not ready for that.
  - duskwuff 14y ago
    Other direction you could go would be to put a bunch of the RAM on a peripheral bus (e.g, PCIe).
- nwh 14y ago
  Similarly, I've been waiting to see an extended modern version of the iRam Box[0], but it doesn't seem to be coming. The 4GB limit is useless now, but imagine the same with hundreds of gigabytes. [0]: https://en.wikipedia.org/wiki/I-RAM https://en.wikipedia.org/wiki/I-RAM
deleted 14y ago
[deleted]
s_baby 14y ago
How is RAMCloud different from something like Redis?
- robotresearcher 14y ago
  It is intended to be durable and available despite host crashes and localized hardware failures. They discuss getting 64GB back into DRAM in 0.6seconds from secondary backup store. The trick is to have many hosts share the burden, so that when one host goes down, 1/100 of its memory is quickly sucked into each of 100 machines from another 100 disk/flash backups. Neat.
  - deleted 14y ago
    [deleted]
  - s_baby 14y ago
    So RAID across network noes.
Nux 14y ago
I often run stuff in ram, the improvements are dramatic. My latest deed was dumping/restoring 20GB of RRDs; doing it in tmpfs was FAST. For running programs stuff can be kept in sync with the likes of anything-sync-daemon. SQL can get trickier, but it can be done.
cbsmith 14y ago
What's odd is they didn't just use nbd + ram disks + RAID. Building a whole key-value store around it makes it a lot less general purpose, and there are perfectly good key-value stores that run off RAM...

64 points by gjenks 14y ago

The idea is not that weird. Lots of applications run from RAM most of the time, needing long cache warmups before performing well. If you keep your entire storage in RAM this isn't necessary, and you can do away with the caching layer.

caw 14y ago
This looks like the "RAM SAN" products, where there's a bunch of memory and some spinning disks with enough local battery backup to flush to disk. I think the main difference is that they want the data aggregated across multiple machines, which I don't think that previous solution had.

sheraz 14y ago

Somewhat relevant, but on a much smaller scale I suppose: I had to geocode 100million addresses and used PostGIS to get it done. It was painfully slow on disk, so I rented some big-ram servers, created ramdisks, configured postgres to run from those ramdisks, and it was off to the races. What would have taken days of processing was taken down to several hours. The speed improvement was amazing. Even SSD couldn't touch it.

ricardobeat 14y ago

Can anyone shed light on how persistence is achieved? The wiki page doesn't seem to come to any conclusions: https://ramcloud.stanford.edu/wiki/display/ramcloud/Data+persistence https://ramcloud.stanford.edu/wiki/display/ramcloud/Data+per...

rektide 14y ago
Hire more server huggers, make extra sure das-blinkenlights stay blinking, buying a second backup generator. And- obviously enough- off site redundancy; multiple data centers. Which they talk about the difficulty of: 10Gbps pipes between DCs is not a lot compared to write throughput of a single computer's main memory, much less a DC's. For some compressible checkpoint logs there might be a way forward. Perhaps we need a new *-cast technology: after any-cast we'll all be needing n-cast, which sends our traffic to the n-nearest DC's? :)
- s_baby 14y ago
  If it's a distributed network, data can be distributed across nodes using a RAID algorithm.
rstutsman 14y ago
Here's the long answer to your question: http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud-recovery.pdf http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud-reco...

rektide 14y ago

From recent coverage of Open Compute on /.: RAM Sled: Facebook wants to replace the leaves and run it on a RAM sled with between 128 GB and 512 GB of memory, and for $500 to $700 per sled. Only a basic CPU would be needed. Total queries would be 450,000 to 1 million key queries per second. http://slashdot.org/topic/datacenter/how-facebook-will-power-graph-search/ http://slashdot.org/topic/datacenter/how-facebook-will-power... Not really any way to know what Facebook is up to here: 512 GB is normally done via 4 sockets, 4 channels, 2 dimms per channel, 16GB dimms; whether FB intends on disrupting any of the factors in this equation for getting mass memory on a single system will be interesting to see. It has been rather shocking to me that FB-DIMMS & other approaches which allow a lot of RAM to be chained to one another- really deep channels- hasn't seen any widescale success. RAM is cheap enough: would that we could plug in a lot of it.

wmf 14y ago
If you just care about capacity and cost you can build a system with a $20 ARM SoC for every two DIMMs. The ultimate is BlueGene-style packaging with an SoC on each DIMM, but Facebook's probably not ready for that.
- duskwuff 14y ago
  Other direction you could go would be to put a bunch of the RAM on a peripheral bus (e.g, PCIe).
nwh 14y ago
Similarly, I've been waiting to see an extended modern version of the iRam Box[0], but it doesn't seem to be coming. The 4GB limit is useless now, but imagine the same with hundreds of gigabytes. [0]: https://en.wikipedia.org/wiki/I-RAM https://en.wikipedia.org/wiki/I-RAM

deleted 14y ago

[deleted]

s_baby 14y ago

How is RAMCloud different from something like Redis?

robotresearcher 14y ago
It is intended to be durable and available despite host crashes and localized hardware failures. They discuss getting 64GB back into DRAM in 0.6seconds from secondary backup store. The trick is to have many hosts share the burden, so that when one host goes down, 1/100 of its memory is quickly sucked into each of 100 machines from another 100 disk/flash backups. Neat.
- deleted 14y ago
  [deleted]
- s_baby 14y ago
  So RAID across network noes.

Nux 14y ago

I often run stuff in ram, the improvements are dramatic. My latest deed was dumping/restoring 20GB of RRDs; doing it in tmpfs was FAST. For running programs stuff can be kept in sync with the likes of anything-sync-daemon. SQL can get trickier, but it can be done.

cbsmith 14y ago

What's odd is they didn't just use nbd + ram disks + RAID. Building a whole key-value store around it makes it a lot less general purpose, and there are perfectly good key-value stores that run off RAM...