3 ms·

Have you considered disaggregating your storage from your compute and using something like NVMe-oF or a traditional iscsi or fiber channel SAN to mount storage

by psds2 7y ago

Have you considered disaggregating your storage from your compute and using something like NVMe-oF or a traditional iscsi or fiber channel SAN to mount storage to your database server? With a block level storage fabric you can fail over instantly, run read only replicas and even support distributed writes in some cases.

adev_ 7y ago
> With a block level storage fabric you can fail over instantly, run read only replicas and even support distributed writes in some cases. Yes, but it's also a wonderful way to corrupt a DB as soon as anything goes wrong in your storage fabric system. And this type of corruption is generally not the recoverable one.
- p_l 7y ago
  Combine it with a data-safety oriented systems like ZFS and your only issue will be downscaling storage without downtime (though that can also be worked around, if in slightly less safe manner, by putting ZVOLs on thin allocations on SAN)
- pbowyer 7y ago
  > Yes, but it's also a wonderful way to corrupt a DB as soon as anything goes wrong in your storage fabric system. Why is this more likely to corrupt a DB than having it on local disks, when something goes wrong?
  - adev_ 7y ago
    > Why is this more likely to corrupt a DB than having it on local disks, when something goes wrong? Local filesystems (ext4, xfs) have been designed to run over reliable internal bus and over spinning disk. They are (almost) able to stand most of the outage happening in this context ( powercut, corrupted block, missing flush ). Put them over a non-reliable "normal" network, where you can get "savage" cable unplugged, faulty controller, paquet loss, out of order delivery, buggy middle box and you explode the number of scenario that can go wrong and will go wrong. Block level I/O virtualization is amazingly useful, but (in my mind) it should be used with care... I already heard of case in prod where the Master DB, the Slave DB and backup finished all in the same virtualized block-level storage.... try to guess what happened next.
    - pbowyer 7y ago
      Thanks, that's a great explanation.

by psds2 7y ago

adev_ 7y ago

> With a block level storage fabric you can fail over instantly, run read only replicas and even support distributed writes in some cases. Yes, but it's also a wonderful way to corrupt a DB as soon as anything goes wrong in your storage fabric system. And this type of corruption is generally not the recoverable one.

p_l 7y ago
Combine it with a data-safety oriented systems like ZFS and your only issue will be downscaling storage without downtime (though that can also be worked around, if in slightly less safe manner, by putting ZVOLs on thin allocations on SAN)
pbowyer 7y ago
> Yes, but it's also a wonderful way to corrupt a DB as soon as anything goes wrong in your storage fabric system. Why is this more likely to corrupt a DB than having it on local disks, when something goes wrong?
- adev_ 7y ago
  > Why is this more likely to corrupt a DB than having it on local disks, when something goes wrong? Local filesystems (ext4, xfs) have been designed to run over reliable internal bus and over spinning disk. They are (almost) able to stand most of the outage happening in this context ( powercut, corrupted block, missing flush ). Put them over a non-reliable "normal" network, where you can get "savage" cable unplugged, faulty controller, paquet loss, out of order delivery, buggy middle box and you explode the number of scenario that can go wrong and will go wrong. Block level I/O virtualization is amazingly useful, but (in my mind) it should be used with care... I already heard of case in prod where the Master DB, the Slave DB and backup finished all in the same virtualized block-level storage.... try to guess what happened next.
  - pbowyer 7y ago
    Thanks, that's a great explanation.