6 ms·

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hu

by zaidf 16y ago

Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

suhail 16y ago
It's okay to have hundreds of terrabytes of data but you need to keep your "working set" hot. Even if you have 10TB of data, 1TB of it might actually be hot. Monitoring your working set is tough.
- ericflo 16y ago
  Yea, but that doesn't look like what they're describing here. They seem to be describing having the entire dataset in memory.
- btilly 16y ago
  It is easy to turn swap off. There are many reasons why it can be good to turn swap off in servers. (But only if you've got sufficient monitoring to hear about it before you run out of RAM.)
btilly 16y ago
Depends on what you are doing. If you have high volume and care about latency, you pretty much have to put everything in RAM. If your disk is spinning at 6000 rpm, a given sector spins past 100 times per second, which makes lookup time anywhere from 0 to 1/100th of a second, or 1/200th of a second on average. Disks only let you look for a limited number of things at once, so you can do less than 1000 disk seeks per second. Period. If data is living on disk and you have high query volume, it is really, really easy to blow past that limit. The solution is to shard data on multiple machines in RAM. This gives a fixed cost per unit of RAM. As long as you don't exceed available RAM, it works well. Luckily it isn't hard to monitor available RAM and respond in advance. (They didn't do that in this case.) If you don't care about latency, or have a lower query volume, then you can live with data on disk, and frequently accessed data in RAM. A few well-designed caching layers in front can give you even more headroom. This is much cheaper. But has more complicated failure modes, and they can be tricky to monitor properly.
deleted 16y ago
[deleted]
SriniK 16y ago
In general you want to have memcached implemented for stuff like this. Have most used stuff closer/faster with memcached. I don't understand why they are running the whole nosql db on those monster machines - it just defeats the purpose. And sharding architecture that they are mentioning is quite error prone. I would go with read/write seperate channels for things of their nature - it appears they don't have that either.
- steveklabnik 16y ago
  Your first three sentences are kind of funny; many people actually consider memecached a NoSQL data store by now.
joshstaiger 16y ago
This talk gives a good high level overview of the Facebook infrastructure: http://www.infoq.com/presentations/Scale-at-Facebook http://www.infoq.com/presentations/Scale-at-Facebook In short they use sharded mysql instances (sans joins) as a key-value store with memcache on top of that.
harryh 16y ago
> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a problem so we'll have to switch to another method of calculating how badges are awarded. Several different options here, each with pluses and minuses.
- fleitz 16y ago
  Why not keep the algorithm and process in batches? You don't need to have EVERY users checkin history in RAM at any moment. Throw the checkin on a queue, have a few processes that query a db for the checkin history and you're fine. Heck, if you delay the awards it's also a good excuse to throw the user an alert to come back to the site/app.
  - harryh 16y ago
    That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.
    - carbocation 16y ago
      It also depends on your algorithms. Some algorithms are amenable to "running tallies," so a third possible approach would be to store and update various values based only on the aggregate past data plus the incremental data, instead of looking back through the entire history and recomputing when a new piece of data comes in. This of course depends on whether or not this is even theoretically possible with what you're computing.
      - jorgeortiz85 16y ago
        This is probably feasible for some badges, but would be hard to do for all of them. This approach also increases the complexity of adding new badges, which is undesirable for product and business reasons. It could certainly help in some case though, and it's something we're considering.
jasonwatkinspdx 16y ago
Facebook runs primarily out of ram via memcached. The last numbers I'm aware of were that they had about 200TB in memcache capacity [1]. They use a variety of data stores, but primarily sharded mysql. I don't have recent numbers there, but they were above 1000 master-master pairs as of 2008. While buying that much ram sound costly, that's only looking at capacity. Assuming typical 1u servers and common pricing at the moment, you're paying roughly $1k in capital for each 10GB of ram capacity. However, each of these servers gives you a couple hundred thousand random reads per second. To duplicate that with spinning hard drives would take several hundred spindles at least. SSD's are better, but still, it ends up being a lot of devices to duplicate that io capacity. This is why virtually everyone at stupendous scale (google, facebook, etc) ends up with very ram centric architectures. There's a simple way to decide how much of what storage you need [2]. Look at the distribution of access times. With current technology, in very rough terms, if an item is accessed more than once a day, it's more cost effective to store it on SSD. If it's accessed more than once in an hour, it's more cost effective to store it in ram. [1]: http://perspectives.mvdirona.com/2010/07/01/Velocity2010.aspx http://perspectives.mvdirona.com/2010/07/01/Velocity2010.asp... [2]: http://www.cs.cmu.edu/~damon2007/pdf/graefe07fiveminrule.pdf http://www.cs.cmu.edu/~damon2007/pdf/graefe07fiveminrule.pdf
- rbranson 16y ago
  Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of performance out of spindles, most "old school" DBAs will tell you that to get high throughput for a database you basically need to have RAM = database size.
  - trezor 16y ago
    I have had very acceptable performance for latency critical applications where the db have exceeded the available memory by a factor between 10 and 50. So far I have never heard of any one running a commercial RDBMS reiterate the MySQL-mantra that you need the entire DB in RAM and I find it a very puzzling attitude to efficient database usage.
jbarham 16y ago
> Is it acceptable/preferred to store your entire db in RAM? If you can fit your db into RAM that's the ideal, but of course you also have to have a reliable, persistent backup. The best current compromise is probably PCI based SSD drives. John Ousterhout (of Tcl fame) and his group at Stanford are working on a project called RAMCloud that's exploring the feasibility of storing data in RAM at all times, using disk only as backup. See his projects page http://www.stanford.edu/~ouster/cgi-bin/projects.php http://www.stanford.edu/~ouster/cgi-bin/projects.php for more info. IMO, given the continuous improvements in RAM prices and network latencies, this type of setup for permanent storage will be the norm in data centers in a few years.
- dedward 16y ago
  It doesn't mean the DB files have to be stored in a ramdisk or something like that - you just need enough ram so your working set can fit in ram. These new projects really seem a lot like re-hashing problems that have already been solved over and over again - people just keep forgetting to do proper systems engineering in the first place. There is no magic bullet.
dedward 16y ago
It can be entirely acceptable and preferred. Knowing your working set size and how your memory is being utilized is a critical part of any systems planning... and memory is one heck of a lot faster than disk, hands down, no debate there. Not we're not talking necessarily about a ramdisk here or specialized software (though we could be) - we're talking about proper system design with enough ram and tuning to get the response times you need.

by zaidf 16y ago

suhail 16y ago

It's okay to have hundreds of terrabytes of data but you need to keep your "working set" hot. Even if you have 10TB of data, 1TB of it might actually be hot. Monitoring your working set is tough.

ericflo 16y ago
Yea, but that doesn't look like what they're describing here. They seem to be describing having the entire dataset in memory.
btilly 16y ago
It is easy to turn swap off. There are many reasons why it can be good to turn swap off in servers. (But only if you've got sufficient monitoring to hear about it before you run out of RAM.)

btilly 16y ago

Depends on what you are doing. If you have high volume and care about latency, you pretty much have to put everything in RAM. If your disk is spinning at 6000 rpm, a given sector spins past 100 times per second, which makes lookup time anywhere from 0 to 1/100th of a second, or 1/200th of a second on average. Disks only let you look for a limited number of things at once, so you can do less than 1000 disk seeks per second. Period. If data is living on disk and you have high query volume, it is really, really easy to blow past that limit. The solution is to shard data on multiple machines in RAM. This gives a fixed cost per unit of RAM. As long as you don't exceed available RAM, it works well. Luckily it isn't hard to monitor available RAM and respond in advance. (They didn't do that in this case.) If you don't care about latency, or have a lower query volume, then you can live with data on disk, and frequently accessed data in RAM. A few well-designed caching layers in front can give you even more headroom. This is much cheaper. But has more complicated failure modes, and they can be tricky to monitor properly.

deleted 16y ago

[deleted]

SriniK 16y ago

In general you want to have memcached implemented for stuff like this. Have most used stuff closer/faster with memcached. I don't understand why they are running the whole nosql db on those monster machines - it just defeats the purpose. And sharding architecture that they are mentioning is quite error prone. I would go with read/write seperate channels for things of their nature - it appears they don't have that either.

steveklabnik 16y ago
Your first three sentences are kind of funny; many people actually consider memecached a NoSQL data store by now.

joshstaiger 16y ago

This talk gives a good high level overview of the Facebook infrastructure: http://www.infoq.com/presentations/Scale-at-Facebook http://www.infoq.com/presentations/Scale-at-Facebook In short they use sharded mysql instances (sans joins) as a key-value store with memcache on top of that.

harryh 16y ago

> Is it acceptable/preferred to store your entire db in RAM? This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a problem so we'll have to switch to another method of calculating how badges are awarded. Several different options here, each with pluses and minuses.

fleitz 16y ago
Why not keep the algorithm and process in batches? You don't need to have EVERY users checkin history in RAM at any moment. Throw the checkin on a queue, have a few processes that query a db for the checkin history and you're fine. Heck, if you delay the awards it's also a good excuse to throw the user an alert to come back to the site/app.
- harryh 16y ago
  That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.
  - carbocation 16y ago
    It also depends on your algorithms. Some algorithms are amenable to "running tallies," so a third possible approach would be to store and update various values based only on the aggregate past data plus the incremental data, instead of looking back through the entire history and recomputing when a new piece of data comes in. This of course depends on whether or not this is even theoretically possible with what you're computing.
    - jorgeortiz85 16y ago
      This is probably feasible for some badges, but would be hard to do for all of them. This approach also increases the complexity of adding new badges, which is undesirable for product and business reasons. It could certainly help in some case though, and it's something we're considering.

jasonwatkinspdx 16y ago

Facebook runs primarily out of ram via memcached. The last numbers I'm aware of were that they had about 200TB in memcache capacity [1]. They use a variety of data stores, but primarily sharded mysql. I don't have recent numbers there, but they were above 1000 master-master pairs as of 2008. While buying that much ram sound costly, that's only looking at capacity. Assuming typical 1u servers and common pricing at the moment, you're paying roughly $1k in capital for each 10GB of ram capacity. However, each of these servers gives you a couple hundred thousand random reads per second. To duplicate that with spinning hard drives would take several hundred spindles at least. SSD's are better, but still, it ends up being a lot of devices to duplicate that io capacity. This is why virtually everyone at stupendous scale (google, facebook, etc) ends up with very ram centric architectures. There's a simple way to decide how much of what storage you need [2]. Look at the distribution of access times. With current technology, in very rough terms, if an item is accessed more than once a day, it's more cost effective to store it on SSD. If it's accessed more than once in an hour, it's more cost effective to store it in ram. [1]: http://perspectives.mvdirona.com/2010/07/01/Velocity2010.aspx http://perspectives.mvdirona.com/2010/07/01/Velocity2010.asp... [2]: http://www.cs.cmu.edu/~damon2007/pdf/graefe07fiveminrule.pdf http://www.cs.cmu.edu/~damon2007/pdf/graefe07fiveminrule.pdf

rbranson 16y ago
Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of performance out of spindles, most "old school" DBAs will tell you that to get high throughput for a database you basically need to have RAM = database size.
- trezor 16y ago
  I have had very acceptable performance for latency critical applications where the db have exceeded the available memory by a factor between 10 and 50. So far I have never heard of any one running a commercial RDBMS reiterate the MySQL-mantra that you need the entire DB in RAM and I find it a very puzzling attitude to efficient database usage.

jbarham 16y ago

> Is it acceptable/preferred to store your entire db in RAM? If you can fit your db into RAM that's the ideal, but of course you also have to have a reliable, persistent backup. The best current compromise is probably PCI based SSD drives. John Ousterhout (of Tcl fame) and his group at Stanford are working on a project called RAMCloud that's exploring the feasibility of storing data in RAM at all times, using disk only as backup. See his projects page http://www.stanford.edu/~ouster/cgi-bin/projects.php http://www.stanford.edu/~ouster/cgi-bin/projects.php for more info. IMO, given the continuous improvements in RAM prices and network latencies, this type of setup for permanent storage will be the norm in data centers in a few years.

dedward 16y ago
It doesn't mean the DB files have to be stored in a ramdisk or something like that - you just need enough ram so your working set can fit in ram. These new projects really seem a lot like re-hashing problems that have already been solved over and over again - people just keep forgetting to do proper systems engineering in the first place. There is no magic bullet.

dedward 16y ago

It can be entirely acceptable and preferred. Knowing your working set size and how your memory is being utilized is a critical part of any systems planning... and memory is one heck of a lot faster than disk, hands down, no debate there. Not we're not talking necessarily about a ramdisk here or specialized software (though we could be) - we're talking about proper system design with enough ram and tuning to get the response times you need.