4 ms·

"You may corrupt the data in case of container crash where database didn’t shutdown correctly. And lose the most important part of your service." This is the p

by bborud 10y ago

"You may corrupt the data in case of container crash where database didn’t shutdown correctly. And lose the most important part of your service." This is the point in the article where you know you don't need to read the rest. If your data integrity hinges on your database being able to shut down correctly you will be disappointed. The author believes in fairytales.

takeda 10y ago
It's kind of interesting how knowledge works with some subjects. If you don't know anything, you will agree with the statement. When you know a bit more, you will disagree, but when you learn more than that you will once agree. It's true that real databases need to guarantee data won't disappear even at power loss, so you would think that container crash should be comparable with power loss, if not more trivial. The thing is that the database can provide such guarantees (write things in correct order, write to disk when database says so etc), but only if the underlying system provides specific guarantees to the database. The storage drivers are quite buggy, so reliability of your data is still in hands of these drivers.
- evgen 10y ago
  And then when you know even more you come to realize that filesystems can lie to you, the OS can lie to you, and the physical disks can lie to you. If you are depending on anything in that path to actually assure you that it has written the data via any path other than flushing as many caches as you can and reading the data back out then you will eventually be disappointed.
- bborud 10y ago
  You stopped one iteration short. You assume you can, and will, know that your OS and your disks do not lie to you. Yes, persisting data in a consistent and durable manner is hard. It is damn hard. It was hard 20 years ago when systems required to store obscene amounts of data started to become more common and it is hard today. (This reminds me of a discussion a couple of years ago on how to kill processes. There are two schools of thought. One is that you should go through the SIGTERM - wait - SIGKILL dance, because that's "being nice". The other is that you always send SIGKILL immediately and instead engineer systems that can deal with it)
  - takeda 10y ago
    Yes it does lie especially with fsync() because that call's purpose was to flush all caches to disk, which is an expensive operation. Since then NCQ/TCQ were added to disks and also systems like Linux implemented write barriers to enable more control[1]. [1] https://monolight.cc/2011/06/barriers-caches-filesystems/ https://monolight.cc/2011/06/barriers-caches-filesystems/

by bborud 10y ago

takeda 10y ago

It's kind of interesting how knowledge works with some subjects. If you don't know anything, you will agree with the statement. When you know a bit more, you will disagree, but when you learn more than that you will once agree. It's true that real databases need to guarantee data won't disappear even at power loss, so you would think that container crash should be comparable with power loss, if not more trivial. The thing is that the database can provide such guarantees (write things in correct order, write to disk when database says so etc), but only if the underlying system provides specific guarantees to the database. The storage drivers are quite buggy, so reliability of your data is still in hands of these drivers.

evgen 10y ago
And then when you know even more you come to realize that filesystems can lie to you, the OS can lie to you, and the physical disks can lie to you. If you are depending on anything in that path to actually assure you that it has written the data via any path other than flushing as many caches as you can and reading the data back out then you will eventually be disappointed.
bborud 10y ago
You stopped one iteration short. You assume you can, and will, know that your OS and your disks do not lie to you. Yes, persisting data in a consistent and durable manner is hard. It is damn hard. It was hard 20 years ago when systems required to store obscene amounts of data started to become more common and it is hard today. (This reminds me of a discussion a couple of years ago on how to kill processes. There are two schools of thought. One is that you should go through the SIGTERM - wait - SIGKILL dance, because that's "being nice". The other is that you always send SIGKILL immediately and instead engineer systems that can deal with it)
- takeda 10y ago
  Yes it does lie especially with fsync() because that call's purpose was to flush all caches to disk, which is an expensive operation. Since then NCQ/TCQ were added to disks and also systems like Linux implemented write barriers to enable more control[1]. [1] https://monolight.cc/2011/06/barriers-caches-filesystems/ https://monolight.cc/2011/06/barriers-caches-filesystems/