8 ms·

The future of databases is RAM clouds and SSD

38 points by petewarden 17y ago

Andys 17y ago
The problem for RethinkDB is that the firmware on Intel SSDs is advanced enough that it can speed up the normal, free MySQL engine (or postgresql, or oracle, ...) to almost the same sort of speeds. I am yet to see any evidence that RethinkDB's approach is anything but a niche for those who need to squeeze an extra 5-10% that advances in filesystems and flash firmware cannot.
- igrigorik 17y ago
  Huh, interesting. Any resources where I can learn more about these firmware optimizations? Intuitively, that doesn't make much sense: it would seem that dropping a lot of the software overhead associated with concurrency, etc, would yield a significant improvement. Having said that, it's also possible that the actual 'software overhead' is pretty much negligible when compared to disk access times. Would love to learn more on this.
  - wmf 17y ago
    I'm not familiar with RethinkDB, but I have read the H-Store paper and I think some of its optimizations are applicable to RAM but not SSD. SSDs are much faster than disk, but I don't think they're fast enough that you can afford to eliminate caching or locking. In particular, good SSDs are internally parallel and require multiple concurrent I/Os for good performance, so you'll probably have to execute multiple transactions concurrently.
    - igrigorik 17y ago
      Well, if an average random access to disk is ~5ms, and an X25 Intel SSD is ~500 microseconds, then we're already talking a 10x difference and that's for a 'consumer grade' SSD. I'm sure the latency will come down over time as well. If you look at some of the PCI-E SSD solutions, then we're dipping below 100 microseconds, which means we're into 50-60x difference.
    - illumen 17y ago
      Indeed. Even with ram, concurrency is optimal on many server architectures. See NUMA, http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access
- coffeemug 17y ago
  It's not that simple. The FTL addresses two fundamental problems with flash - slow random write performance and wear leveling. It works beautifully on consumer hardware, but performs rather poorly in an enterprise environment. For some workloads you can squeeze a lot more than 5-10% improvement out of the drives. But that's not the interesting part (most of today's workloads are read-intensive anyway). The interesting part comes from being able to execute existing operations in fundamentally different ways. When you're not restricted to sequential reads you can implement a huge chunk of database internals differently - different concurrency models, different implementations of certain joins, etc. This is where the bulk of the performance improvement comes from. Once you do this, you realize that a traditional bottleneck (I/O) is completely blown away. Suddenly, all kinds of things nobody ever bothered to optimize become performance bottlenecks (for example, the query plan is traditionally interpreted, not compiled to native code - not good enough once you get an I/O subsystem that's 250 times faster than what you used to have). Once you get rid of the software bottlenecks, you realize that you're limited by the bus. Since SSDs are highly parallelized devices, you have to change the way you interleave operations for optimal performance. That alone is another order of magnitude. In three years or so, the flash manufacturers will put chips on the SSDs that will allow you to send down programs to them (not unlike the shaders in video cards). That will likely be a bigger revolution than SSDs themselves. Where we are now is just the beginning. You're right that we haven't shown any evidence that our approach is anything but a niche. We will. This stuff just takes a long time to build.
  - neilc 17y ago
    the query plan is traditionally interpreted, not compiled to native code Actually, System R (IBM's pioneering relation database from the mid 1970s) compiled query plans down to machine code.
  - pedrocr 17y ago
    In three years or so, the flash manufacturers will put chips on the SSDs that will allow you to send down programs to them (not unlike the shaders in video cards). That will likely be a bigger revolution than SSDs themselves. Where we are now is just the beginning. This sounds crazy. SSDs have all this firmware and processing power to make them behave like hard drives. If we want to drive the underlying storage directly we might as well rip all that out and let the OS drive the flash directly from the main CPU. Why would we want to be sending programs into the black box? What can the firmware do that the main CPU couldn't if we stopped attaching flash as SATA and instead just created a direct connection that can DMA back and forth to main memory? Some Linux developers have made that point, that they would rather be implementing their own filesystems directly on the flash than dealing with a firmware that is trying to make it appear that the SSD is a hard drive but that then needs TRIM to work around the fact that that is not true. These firmwares are nice for Intel by letting them turn a commodity (flash) into a differentiated (and higher margin) product based on the code in the firmware. Considering Linux already has filesystems that can operate directly over flash and that better ones can be written I'd say we would be better off just getting direct access to the flash.
- Devilboy 17y ago
  We committed to SQL Server a long time ago. It's served us very well and it will take a lot of effort to replace it. To keep up with increasing workloads we've had to purchase reasonably expensive hardware - I think the last server cost us almost $100k. The #1 bottleneck has always been disk IO. To compensate for slow disks you need to increase RAM and make big RAID arrays, both fairly expensive options compared to the Intel SSDs available now. Upgrading our current hardware with only $10k worth of SSDs will add at least a year or two to the life of our SQL Servers.
akamaka 17y ago
I'd like to see a proper quantitative analysis done on this subject. When you take money into account, are SSDs really worth it, or should I just buy more RAM?
- cottsak 17y ago
  For now, i'd say just buy more ram. Basecamp apparently serve 100% of their data from memory.
- sokoloff 17y ago
  If you haven't already maxed the RAM on the server in question, that's generally the first step. When (the active working set) of your DB is much larger than the RAM your server can support, either get a bigger server or put heavy random IO loads on SSDs. You're asking it as an A-xor-B when it's really a case of A-then-B.
- easp 17y ago
  Properly, Flash is another tier in the storage hierarchy. Faster than disk in important measures, but more expensive. Slower than RAM, but cheaper, lower power, and durable. Unfortunately, the infrastrucure for taking easy advantage of Flash isn't very mature yet. SSDs make flash act like a hard disk, but OSs and applications generally don't allow transparent fined-grained data management to take advantage of it. I guess ZFS is best in this regard, since it can migrate hot data to fast devices, but I haven't seen good options on Linux, and I haven't seen hardware RAID controllers that could fill the gap. In your DB, you can place tables on different devices, or use table partitioning to separate recent data from cold, but that isn't as easy as adding more RAM. Ultimately though, cost effectiveness really depends on your workload. MySpace is apparently finding it cost effective to use SSDs for their cache tier. A few months back James Hamilton linked to a study that suggested that for some common workloads, the case for SSDs had to include both price/performance and performance/watt, since it didn't pencil out for each individually. That'll change though, as prices drop and performance improves.
Luyt 17y ago
I put SSDs in all my development machines, and it indeed makes compiling faster (about three times). When prices drop a little, I'll also equip our database server with SSDs. Luckily the database is only 60G.
- lsb 17y ago
  Interesting size; I was talking to guys at a gay dating site, and their worldwide db was around 60 GB also; it could fit in RAM, on the 4XL amazon ec2 instances.
richcollins 17y ago
"RDBMS systems are headed into main memory, which changes the playing field all together" It doesn't change the fact that RDBMSs don't do a good job of returning representations of graphs.
- fauigerzigerk 17y ago
  All the major RDBMS do recursive queries nowadays.
chime 17y ago
One of the most actively-used databases at my company is about 1GB in size and grows at about 100MB/month. It is a Windows-based DB and is very heavy on the reads. I installed SuperVolume http://www.superspeed.com/desktop/supervolume.php http://www.superspeed.com/desktop/supervolume.php on the DB server and created a 4GB disk partition to store the DB files. SuperVolume mirrors the entire 4GB disk partition in RAM and works flawlessly and transparently. I was going to get an SSD for performance enhancement but I don't much see a reason to do that anymore. All the reads happen at the speed of RAM and writes are delayed and non-blocking. Nested queries that used to take 15 minutes now take 45 seconds. Right now the speed issues we have are solely due to unoptimized algorithms and not due to hardware limitations. This is a wonderful spot for a coder to be in because there is so much room for improvement now. If you have Windows systems serving files and data on slow systems, I would highly recommend giving SuperVolume and its related apps a shot before investing in expensive SSDs.
fauigerzigerk 17y ago
That's interesting and very promising. But what I miss in most discussions of relational vs NoSQL is some kind of separation of data model related, performance related and scalability related arguments. These are three rather different issues. There is no way to have a benchmark alone show me what it means to denormalize my data model. Denormalization can help scalability and sometimes performance, but it has huge implications for maintainability, data quality, code complexity and even organisational matters. I do understand the advantages of being able to run any algorithm on data and not be limited by SQL. But abolishing normalized data models altogether and replacing them with application code and custom data structures everywhere makes me very skeptical as someone who had to deal with data quality issues for many years. So that's why I'm glad to see some performance issues of RDBMS improved by new storage architectures and also the innovation going on in terms of recursive queries to deal with hierarchical and graph data structures.
- tom_b 17y ago
  The missing discussion point for me is more fundamental - how and when should I break with my RDBMS data modeling habits? I can't say for sure, but suspect some of the headlong rush into denormalized NoSQL stores is motivated by the ease of the data model. I wonder if once you go down that road, if your "program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of" what you get in a decent RDBMS. (Forgive my hijacking of Greenspun) That said, I'm usually working in denormalized models anyway (star schemas in data warehouses for handy-dandy slice/dice/drill), so I'm completely open to different ways of approaching data processing and analysis problems. I actually feel more free with SQL than limited though. What I would really enjoy would be to have the fully expressive power of SQL embedded in a natural feeling way in a higher level language. I'm interested in Q (of KDB) for this reason, have also been baking my brain in prolog to tear down old data model habits, and hacking some in Ruby and Common Lisp for the same reasons. Of course, I'm probably breaking my career in the process, since I've made a habit of getting paid to work on RDBMS walking the line between hardware/software performance tuning. But because of that, I'm also glad to see new approaches to RDBMS storage and innovation in the whole DB space.
  - 17y ago

38 points by petewarden 17y ago

Andys 17y ago

The problem for RethinkDB is that the firmware on Intel SSDs is advanced enough that it can speed up the normal, free MySQL engine (or postgresql, or oracle, ...) to almost the same sort of speeds. I am yet to see any evidence that RethinkDB's approach is anything but a niche for those who need to squeeze an extra 5-10% that advances in filesystems and flash firmware cannot.

igrigorik 17y ago
Huh, interesting. Any resources where I can learn more about these firmware optimizations? Intuitively, that doesn't make much sense: it would seem that dropping a lot of the software overhead associated with concurrency, etc, would yield a significant improvement. Having said that, it's also possible that the actual 'software overhead' is pretty much negligible when compared to disk access times. Would love to learn more on this.
- wmf 17y ago
  I'm not familiar with RethinkDB, but I have read the H-Store paper and I think some of its optimizations are applicable to RAM but not SSD. SSDs are much faster than disk, but I don't think they're fast enough that you can afford to eliminate caching or locking. In particular, good SSDs are internally parallel and require multiple concurrent I/Os for good performance, so you'll probably have to execute multiple transactions concurrently.
  - igrigorik 17y ago
    Well, if an average random access to disk is ~5ms, and an X25 Intel SSD is ~500 microseconds, then we're already talking a 10x difference and that's for a 'consumer grade' SSD. I'm sure the latency will come down over time as well. If you look at some of the PCI-E SSD solutions, then we're dipping below 100 microseconds, which means we're into 50-60x difference.
  - illumen 17y ago
    Indeed. Even with ram, concurrency is optimal on many server architectures. See NUMA, http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access
coffeemug 17y ago
It's not that simple. The FTL addresses two fundamental problems with flash - slow random write performance and wear leveling. It works beautifully on consumer hardware, but performs rather poorly in an enterprise environment. For some workloads you can squeeze a lot more than 5-10% improvement out of the drives. But that's not the interesting part (most of today's workloads are read-intensive anyway). The interesting part comes from being able to execute existing operations in fundamentally different ways. When you're not restricted to sequential reads you can implement a huge chunk of database internals differently - different concurrency models, different implementations of certain joins, etc. This is where the bulk of the performance improvement comes from. Once you do this, you realize that a traditional bottleneck (I/O) is completely blown away. Suddenly, all kinds of things nobody ever bothered to optimize become performance bottlenecks (for example, the query plan is traditionally interpreted, not compiled to native code - not good enough once you get an I/O subsystem that's 250 times faster than what you used to have). Once you get rid of the software bottlenecks, you realize that you're limited by the bus. Since SSDs are highly parallelized devices, you have to change the way you interleave operations for optimal performance. That alone is another order of magnitude. In three years or so, the flash manufacturers will put chips on the SSDs that will allow you to send down programs to them (not unlike the shaders in video cards). That will likely be a bigger revolution than SSDs themselves. Where we are now is just the beginning. You're right that we haven't shown any evidence that our approach is anything but a niche. We will. This stuff just takes a long time to build.
- neilc 17y ago
  the query plan is traditionally interpreted, not compiled to native code Actually, System R (IBM's pioneering relation database from the mid 1970s) compiled query plans down to machine code.
- pedrocr 17y ago
  In three years or so, the flash manufacturers will put chips on the SSDs that will allow you to send down programs to them (not unlike the shaders in video cards). That will likely be a bigger revolution than SSDs themselves. Where we are now is just the beginning. This sounds crazy. SSDs have all this firmware and processing power to make them behave like hard drives. If we want to drive the underlying storage directly we might as well rip all that out and let the OS drive the flash directly from the main CPU. Why would we want to be sending programs into the black box? What can the firmware do that the main CPU couldn't if we stopped attaching flash as SATA and instead just created a direct connection that can DMA back and forth to main memory? Some Linux developers have made that point, that they would rather be implementing their own filesystems directly on the flash than dealing with a firmware that is trying to make it appear that the SSD is a hard drive but that then needs TRIM to work around the fact that that is not true. These firmwares are nice for Intel by letting them turn a commodity (flash) into a differentiated (and higher margin) product based on the code in the firmware. Considering Linux already has filesystems that can operate directly over flash and that better ones can be written I'd say we would be better off just getting direct access to the flash.
Devilboy 17y ago
We committed to SQL Server a long time ago. It's served us very well and it will take a lot of effort to replace it. To keep up with increasing workloads we've had to purchase reasonably expensive hardware - I think the last server cost us almost $100k. The #1 bottleneck has always been disk IO. To compensate for slow disks you need to increase RAM and make big RAID arrays, both fairly expensive options compared to the Intel SSDs available now. Upgrading our current hardware with only $10k worth of SSDs will add at least a year or two to the life of our SQL Servers.

akamaka 17y ago

I'd like to see a proper quantitative analysis done on this subject. When you take money into account, are SSDs really worth it, or should I just buy more RAM?

cottsak 17y ago
For now, i'd say just buy more ram. Basecamp apparently serve 100% of their data from memory.
sokoloff 17y ago
If you haven't already maxed the RAM on the server in question, that's generally the first step. When (the active working set) of your DB is much larger than the RAM your server can support, either get a bigger server or put heavy random IO loads on SSDs. You're asking it as an A-xor-B when it's really a case of A-then-B.
easp 17y ago
Properly, Flash is another tier in the storage hierarchy. Faster than disk in important measures, but more expensive. Slower than RAM, but cheaper, lower power, and durable. Unfortunately, the infrastrucure for taking easy advantage of Flash isn't very mature yet. SSDs make flash act like a hard disk, but OSs and applications generally don't allow transparent fined-grained data management to take advantage of it. I guess ZFS is best in this regard, since it can migrate hot data to fast devices, but I haven't seen good options on Linux, and I haven't seen hardware RAID controllers that could fill the gap. In your DB, you can place tables on different devices, or use table partitioning to separate recent data from cold, but that isn't as easy as adding more RAM. Ultimately though, cost effectiveness really depends on your workload. MySpace is apparently finding it cost effective to use SSDs for their cache tier. A few months back James Hamilton linked to a study that suggested that for some common workloads, the case for SSDs had to include both price/performance and performance/watt, since it didn't pencil out for each individually. That'll change though, as prices drop and performance improves.

Luyt 17y ago

I put SSDs in all my development machines, and it indeed makes compiling faster (about three times). When prices drop a little, I'll also equip our database server with SSDs. Luckily the database is only 60G.

lsb 17y ago
Interesting size; I was talking to guys at a gay dating site, and their worldwide db was around 60 GB also; it could fit in RAM, on the 4XL amazon ec2 instances.

richcollins 17y ago

"RDBMS systems are headed into main memory, which changes the playing field all together" It doesn't change the fact that RDBMSs don't do a good job of returning representations of graphs.

fauigerzigerk 17y ago
All the major RDBMS do recursive queries nowadays.

chime 17y ago

One of the most actively-used databases at my company is about 1GB in size and grows at about 100MB/month. It is a Windows-based DB and is very heavy on the reads. I installed SuperVolume http://www.superspeed.com/desktop/supervolume.php http://www.superspeed.com/desktop/supervolume.php on the DB server and created a 4GB disk partition to store the DB files. SuperVolume mirrors the entire 4GB disk partition in RAM and works flawlessly and transparently. I was going to get an SSD for performance enhancement but I don't much see a reason to do that anymore. All the reads happen at the speed of RAM and writes are delayed and non-blocking. Nested queries that used to take 15 minutes now take 45 seconds. Right now the speed issues we have are solely due to unoptimized algorithms and not due to hardware limitations. This is a wonderful spot for a coder to be in because there is so much room for improvement now. If you have Windows systems serving files and data on slow systems, I would highly recommend giving SuperVolume and its related apps a shot before investing in expensive SSDs.

fauigerzigerk 17y ago

That's interesting and very promising. But what I miss in most discussions of relational vs NoSQL is some kind of separation of data model related, performance related and scalability related arguments. These are three rather different issues. There is no way to have a benchmark alone show me what it means to denormalize my data model. Denormalization can help scalability and sometimes performance, but it has huge implications for maintainability, data quality, code complexity and even organisational matters. I do understand the advantages of being able to run any algorithm on data and not be limited by SQL. But abolishing normalized data models altogether and replacing them with application code and custom data structures everywhere makes me very skeptical as someone who had to deal with data quality issues for many years. So that's why I'm glad to see some performance issues of RDBMS improved by new storage architectures and also the innovation going on in terms of recursive queries to deal with hierarchical and graph data structures.

tom_b 17y ago
The missing discussion point for me is more fundamental - how and when should I break with my RDBMS data modeling habits? I can't say for sure, but suspect some of the headlong rush into denormalized NoSQL stores is motivated by the ease of the data model. I wonder if once you go down that road, if your "program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of" what you get in a decent RDBMS. (Forgive my hijacking of Greenspun) That said, I'm usually working in denormalized models anyway (star schemas in data warehouses for handy-dandy slice/dice/drill), so I'm completely open to different ways of approaching data processing and analysis problems. I actually feel more free with SQL than limited though. What I would really enjoy would be to have the fully expressive power of SQL embedded in a natural feeling way in a higher level language. I'm interested in Q (of KDB) for this reason, have also been baking my brain in prolog to tear down old data model habits, and hacking some in Ruby and Common Lisp for the same reasons. Of course, I'm probably breaking my career in the process, since I've made a habit of getting paid to work on RDBMS walking the line between hardware/software performance tuning. But because of that, I'm also glad to see new approaches to RDBMS storage and innovation in the whole DB space.
- 17y ago