3 ms·

IMHO more iconoclasm is needed here: "Are you sure you want to use storage in your DBMS?" Obviously some kind of persistent store is needed. But in the moder

by ajross 4y ago

IMHO more iconoclasm is needed here: "Are you sure you want to use storage in your DBMS?" Obviously some kind of persistent store is needed. But in the modern world RAM is so cheap and so performant, and cross-site redundancy/failover techniques are so robust, and sharding paradigms so scalable, that... let's be honest, a deployed database is simply never going to need to restore from a powered-off/pickled representation. Ever. The hard parts of data management are all in RAM now. So... sure, don't mmap() files. But... maybe consider not using a complicated file format at all. Files should be a snapshotted dump, or a log, or some kind of simple combination thereof. If you're worrying about B+ trees and flash latency, you're probably doing it wrong.

marginalia_nu 4y ago
RAM is really fucking expensive though. Even a modest data set of a couple of dozen terabytes will cost an arm and a leg if you're keeping it mostly in RAM (with redundancy).
- ajross 4y ago
  Only for hardware. Once you're at scale and deploying a redundant database, your hardware costs start to vanish anyway. I mean, yes, putting a bunch of drives on a local machine is cheap. But that too is a circumstance where a mmap()-using DBMS is probably inappropriate.
  - loeg 4y ago
    > Once you're at scale and deploying a redundant database, your hardware costs start to vanish anyway. This is really, really false. It would make my life a lot easier if it were true.
    - samhw 4y ago
      I assume they mean "relative to [my idea of how much a company will make when serving that amount of traffic]" - which does depend on the industry they are in, not just in terms of how profitable it is but also its 'user load coefficient', so to speak. (The first job I had was at a 10-person startup in a coworking space, barely making enough revenue to break even, and still consuming a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices. People tend to forget that not every single business is a CRUD web/mobile app whose average user accounts for $20/month revenue against at most a few hundred HTTP requests and a couple megabytes of disk.)
      - Dylan16807 4y ago
        > a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices Even that depends on a lot of factors. Just as an example, if it takes $10 to build and deploy a sensor, and it returns one number per second, then $1 of RAM can hold 2+ years of data before archiving it.
        marginalia_nu 4y ago
        That ignores any type of structure to the data (such as timestamps and which sensor it is). It would also probably need to be indexed somehow.
        Dylan16807 4y ago
        I left spare space for that. Especially if you store a minute of samples at a time.
        samhw 4y ago
        Hmm, how many bits are you allowing for the 'number'? Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Also, is that ordinary NVRAM? How do you protect against bit flips? Google's well-known paper[0] found a 0.22% average incidence of uncorrectable errors per DIMM - that's corruption of more bits than ECC can fix (and I'm not sure that your pricing is even for ECC RAM). Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust (i.e. redundancy against total loss, as well as handling the 'freak errors' in data correctness that become the norm when you're dealing with vast amounts of hardware). I'm a fan of the kind of minimalism that you're advocating, don't get me wrong, but it's from pushing that limit that I've learned what the hard limits are. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c...
        4y ago
mamcx 4y ago
This is only true for the niche subset of "large-ish" datasets and VERY sophisticated customer, where operate a distributed-ram store is "fine". A lot, bit a GIGANTIC margin of the needs for a DB are not close to this, at all. Starting with sqlite, that is likely the most deployed, is impossible that this scenario could be used for it (and that accounting how small most dbs are).
ayende 4y ago
Generally, really bad idea. Assume a 512gb RAM and most of that being in use. With a log file for durability. It sounds fine, but... What is the cost of restarts? You need to read through the log, apply it, etc. That can take a LOT of time, especially on cloud disks
- AtlasBarfed 4y ago
  Your comment is so funny in contrast to the article. Article: "Don't use mmap because modern storage is soooo fast. It will BLOW YOUR MIND!!!" You: "I'm in the cloud!" .........oh. Nevemind.

by ajross 4y ago

marginalia_nu 4y ago

RAM is really fucking expensive though. Even a modest data set of a couple of dozen terabytes will cost an arm and a leg if you're keeping it mostly in RAM (with redundancy).

ajross 4y ago
Only for hardware. Once you're at scale and deploying a redundant database, your hardware costs start to vanish anyway. I mean, yes, putting a bunch of drives on a local machine is cheap. But that too is a circumstance where a mmap()-using DBMS is probably inappropriate.
- loeg 4y ago
  > Once you're at scale and deploying a redundant database, your hardware costs start to vanish anyway. This is really, really false. It would make my life a lot easier if it were true.
  - samhw 4y ago
    I assume they mean "relative to [my idea of how much a company will make when serving that amount of traffic]" - which does depend on the industry they are in, not just in terms of how profitable it is but also its 'user load coefficient', so to speak. (The first job I had was at a 10-person startup in a coworking space, barely making enough revenue to break even, and still consuming a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices. People tend to forget that not every single business is a CRUD web/mobile app whose average user accounts for $20/month revenue against at most a few hundred HTTP requests and a couple megabytes of disk.)
    - Dylan16807 4y ago
      > a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices Even that depends on a lot of factors. Just as an example, if it takes $10 to build and deploy a sensor, and it returns one number per second, then $1 of RAM can hold 2+ years of data before archiving it.
      - marginalia_nu 4y ago
        That ignores any type of structure to the data (such as timestamps and which sensor it is). It would also probably need to be indexed somehow.
        Dylan16807 4y ago
        I left spare space for that. Especially if you store a minute of samples at a time.
      - samhw 4y ago
        Hmm, how many bits are you allowing for the 'number'? Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device. Also, is that ordinary NVRAM? How do you protect against bit flips? Google's well-known paper[0] found a 0.22% average incidence of uncorrectable errors per DIMM - that's corruption of more bits than ECC can fix (and I'm not sure that your pricing is even for ECC RAM). Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust (i.e. redundancy against total loss, as well as handling the 'freak errors' in data correctness that become the norm when you're dealing with vast amounts of hardware). I'm a fan of the kind of minimalism that you're advocating, don't get me wrong, but it's from pushing that limit that I've learned what the hard limits are. [0] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c...
        4y ago

mamcx 4y ago

This is only true for the niche subset of "large-ish" datasets and VERY sophisticated customer, where operate a distributed-ram store is "fine". A lot, bit a GIGANTIC margin of the needs for a DB are not close to this, at all. Starting with sqlite, that is likely the most deployed, is impossible that this scenario could be used for it (and that accounting how small most dbs are).

ayende 4y ago

Generally, really bad idea. Assume a 512gb RAM and most of that being in use. With a log file for durability. It sounds fine, but... What is the cost of restarts? You need to read through the log, apply it, etc. That can take a LOT of time, especially on cloud disks

AtlasBarfed 4y ago
Your comment is so funny in contrast to the article. Article: "Don't use mmap because modern storage is soooo fast. It will BLOW YOUR MIND!!!" You: "I'm in the cloud!" .........oh. Nevemind.