3 ms·
so gow donyou test it for such cases as discussed here? Like a power outage while writing to disk? Maybe you were lucky and didn't hit thst in production, yet.
by johannes1234321 5y ago
so gow donyou test it for such cases as discussed here? Like a power outage while writing to disk? Maybe you were lucky and didn't hit thst in production, yet.
- andai 5y agoThis could be simulated with a VM, or perhaps (maybe the underlying OS abstractions would alter the result) by simply killing the process?
- bayindirh 5y agoWe pull the plugs out of the systems while they're running (sometimes knowingly, sometimes unknowingly). "kill -9" is almost the same thing in most cases.
- mst 5y agoSounds like you're doing something similar to FoundationDB's Sinkhole thing. I rarely get to work on projects where we have the budget to destruction test things that hard, but I am absolutely in love with the concept.
- johannes1234321 5y agoNope. The actual harddrive then still has power and will finish its work.
- bayindirh 5y agoWe operate with a cattle and pet model [0], and we use a lot of services on that cattle servers. Their power can go out, disks can go bad, hardware can fry. We've seen a lot of things. As an HPC center, we absolutely hammer what we have. Even manufacturers baffled how we can fry our servers. We're eating memory controllers in one generation, and on board NICs were being cooked on others for example. XFS and EXT4 can handle a lot of abuse, incl. power loss without any problems during a heavy write. Regardless of the services we run, we didn't loss any data during a power outage, and for us, power outage means "power outage during full load". Making sure that your scientific computation can continue from the point you left it is a big business. Nobody wants to lose three weeks or a month just because a server gave out its magic smoke. [0]: https://www.engineyard.com/blog/pets-vs-cattle/ https://www.engineyard.com/blog/pets-vs-cattle/