3 ms·
For a high-valued production data, if you think your data is safe after it's (finally) written to disk, then you're doing it wrong. Your data is not safe until
by ypcx 15y ago
For a high-valued production data, if you think your data is safe after it's (finally) written to disk, then you're doing it wrong. Your data is not safe until a) it left the physical database machine, b) it left the physical datacenter to be stored into multiple datacenters around the world (e.g. S3).
Thus ideally, the (horribly slow) disk doesn't even come into play, especially for in-memory DBs. You buffer the data in memory before they are sent out of the machine/datacenter, but you make sure to mirror this buffer at multiple separate physical machines (which your database cluster should support), in case one goes down or over. Once the data are committed into a replicated store, you can clear that buffer. Fast and reliable.
This is not to say that there are zillion cases where harddrive is still the ideal persistence device. After all, it's very hard to destroy a harddrive in a way to make the data unrecoverable (of course, I'm talking about cases where RAID failed or wasn't present). However in reality, data from broken harddrives are seldom attempted to be recovered, mainly I guess due to the price and relatively long service waiting times.
- antirez 15y agoIf you read the article carefully there are multiple mentions about this, specifically I wrote that RDB persistence is just perfect for this: a single-file compact representation of data to send far away :)
- ypcx 15y agoHmm, I was thinking more of "sending chunks of AOF" out to a separate, distributed storage (and the RDB snapshot file occasionally). Loading data from network could then be slower than from a disk, but also faster than from a disk, depending on network speed and number of machines to read from.