3 ms·

There's different kinds of data loss. There's data loss because you lose the whole drive; because you lost a whole write; because a write was only partially wri

There's different kinds of data loss. There's data loss because you lose the whole drive; because you lost a whole write; because a write was only partially written. Half the problem with NIH solutions is, what happens when you try to read from your bespoke binary format, and the result is corrupted in some way? So much of the value of battle-tested, multi-decade-old databases is that those are solved problems that you, the engineer building on top of the database, do not need to worry about. Of course data loss is alright when you're talking about a few records within a billion. It is categorically unacceptable when AWS loses your drive, you try to restore from backup, the application crashes when trying to use the restored backup because of "corruption", the executives are pissed because downtime is reaching into the hours/days while you frantically try to FedEx a laptop to the one engineer who knows your bespoke binary format and can maybe heal the backup by hand except he's on vacation and didn't bring his laptop with him.

pas 3y ago
> Half the problem with NIH solutions is, what happens when you try to read from your bespoke binary format, and the result is corrupted in some way? restoring an EBS snapshot seems pretty similar to restoring Aurora/RDS, binary format or not. if you know you have problems. (they don't mention checksums in the blog post. or any kind of error handling, just that they can buffer some writes.) usually the problem with NIH solutions is that evolving/extending them is hard. (of course multi-decade projects are also not magically free of ossified parts, we just euphemistically think of them as trade-offs.)
- kikimora 3y ago
  How EBS snapshots going to be consistent? I mean AWS takes them at random time, a half of a write may be captured. Another, less common scenario is silent data corruption you never notice until you need to restore.
  - pas 3y ago
    I assume they do the snapshotting from some script. (Stop writes, flush, snapshot, restart writes.) If not, then they probably have some way of detecting partial writes. It seems that they do fsync() every second. > Another, less common scenario is silent data corruption you never notice until you need to restore. I tried to find what kind of guarantees EBS offers, but they only talk about durability and availability ( https://aws.amazon.com/ebs/sla/ https://aws.amazon.com/ebs/sla/ ), nothing about data corruption, individual sector failure. (So this could mean that they either won't even notice ... or you might get back a full zero page - assuming they detect some underlying error, but due to encryption at rest the internal checksum fails, assuming there is one.) It's nice that AWS recommends you to just ignore read errors ... https://repost.aws/questions/QUGqXF7BcOQKCopHJcnnDrRA/ebs-volume-i-o-error https://repost.aws/questions/QUGqXF7BcOQKCopHJcnnDrRA/ebs-vo... :D :D :D

pas 3y ago

> Half the problem with NIH solutions is, what happens when you try to read from your bespoke binary format, and the result is corrupted in some way? restoring an EBS snapshot seems pretty similar to restoring Aurora/RDS, binary format or not. if you know you have problems. (they don't mention checksums in the blog post. or any kind of error handling, just that they can buffer some writes.) usually the problem with NIH solutions is that evolving/extending them is hard. (of course multi-decade projects are also not magically free of ossified parts, we just euphemistically think of them as trade-offs.)

kikimora 3y ago
How EBS snapshots going to be consistent? I mean AWS takes them at random time, a half of a write may be captured. Another, less common scenario is silent data corruption you never notice until you need to restore.
- pas 3y ago
  I assume they do the snapshotting from some script. (Stop writes, flush, snapshot, restart writes.) If not, then they probably have some way of detecting partial writes. It seems that they do fsync() every second. > Another, less common scenario is silent data corruption you never notice until you need to restore. I tried to find what kind of guarantees EBS offers, but they only talk about durability and availability ( https://aws.amazon.com/ebs/sla/ https://aws.amazon.com/ebs/sla/ ), nothing about data corruption, individual sector failure. (So this could mean that they either won't even notice ... or you might get back a full zero page - assuming they detect some underlying error, but due to encryption at rest the internal checksum fails, assuming there is one.) It's nice that AWS recommends you to just ignore read errors ... https://repost.aws/questions/QUGqXF7BcOQKCopHJcnnDrRA/ebs-volume-i-o-error https://repost.aws/questions/QUGqXF7BcOQKCopHJcnnDrRA/ebs-vo... :D :D :D