3 ms·

The testing I did back when I did this sort of thing was to put on a filesystem stress test automatically on boot, and then plug the whole system into a lamp ti

by cwillu 3y ago

The testing I did back when I did this sort of thing was to put on a filesystem stress test automatically on boot, and then plug the whole system into a lamp timer set to hard cut the power every 15 minutes. I went through _dozens_ of cards, until I finally bit the bullet and went with the “100x more expensive than consumer grade”-grade from digikey. Every other option eventually failed, in the system-no-longer-detects-any-storage-in-the-slot sense of failure.

jqpabc123 3y ago
It sounds like the abrupt power cut test eventually produced a voltage spike that physically damaged components on the card. Ever consider that your system's power or card interface design might be to blame as much as the card? For a lot of applications, spike suppression hardware components or a simple UPS might be a more cost effective investment that could also provide other ancillary benefits. Perhaps what we have here is some enterprising company that has incorporated a few cheap spike suppression components costing pennies each inside the card itself --- and now sells the cards as "industrial grade" for 100X as much. I would bet money that they haven't developed their own "industrial" flash memory chips.
- eternityforest 3y ago
  They don't publish specs on consumer cards, so it's all speculation without heavy reverse engineering. There seems to be people saying that it's not just voltage spikes, it's power failure corrupting data at the FTL layer, that the card wasn't designed to recover from. If it were up to me, SD based devices like the RasPi would have a tiny capacitor, diode, and supervisor circuit, so that the SD could never lose power until at least a few ms after the CPU did, but I'm not the one in charge of such things, all I can do is spec a UPS or an industrial card. Or more likely, just keep the logs in RAM as much as possible and use fatrace to hunt down various card destroying daemons.
- cwillu 3y ago
  https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf “The flash memory devices we studied in this work demonstrated unexpected behavior when power failure occurs. The error rates do not always decrease as the operation proceeds, and power failure can corrupt the data from operations that completed successfully. We also found that relying on blocks that have been programmed or erased during a power failure is unreliable, even if the data appears to be intact.” Yes, the hardware it's plugged into could also solve (or at least alleviate) the issues. It's not really useful when you're provided the hardware and told to make it work. “Get the manufacturer to acknowledge the issue, release a new revision, and then start replacing installed hardware in the field in 6 months when you start receiving it” is not preferable to “spend an extra 100 dollars on a 25 thousand dollar electrical/instrumentation installation, on an sd-card that has a demonstrated ability to deal with the hardware we have today”. I've literally spent more money on a simple push button, of which I needed dozens on such an installation (for example: https://www.digikey.ca/en/products/detail/schneider-electric/XB4BW3165/16613561 https://www.digikey.ca/en/products/detail/schneider-electric... ). :)

by cwillu 3y ago

jqpabc123 3y ago

It sounds like the abrupt power cut test eventually produced a voltage spike that physically damaged components on the card. Ever consider that your system's power or card interface design might be to blame as much as the card? For a lot of applications, spike suppression hardware components or a simple UPS might be a more cost effective investment that could also provide other ancillary benefits. Perhaps what we have here is some enterprising company that has incorporated a few cheap spike suppression components costing pennies each inside the card itself --- and now sells the cards as "industrial grade" for 100X as much. I would bet money that they haven't developed their own "industrial" flash memory chips.

eternityforest 3y ago
They don't publish specs on consumer cards, so it's all speculation without heavy reverse engineering. There seems to be people saying that it's not just voltage spikes, it's power failure corrupting data at the FTL layer, that the card wasn't designed to recover from. If it were up to me, SD based devices like the RasPi would have a tiny capacitor, diode, and supervisor circuit, so that the SD could never lose power until at least a few ms after the CPU did, but I'm not the one in charge of such things, all I can do is spec a UPS or an industrial card. Or more likely, just keep the logs in RAM as much as possible and use fatrace to hunt down various card destroying daemons.
cwillu 3y ago
https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf https://cseweb.ucsd.edu/~swanson/papers/DAC2011PowerCut.pdf “The flash memory devices we studied in this work demonstrated unexpected behavior when power failure occurs. The error rates do not always decrease as the operation proceeds, and power failure can corrupt the data from operations that completed successfully. We also found that relying on blocks that have been programmed or erased during a power failure is unreliable, even if the data appears to be intact.” Yes, the hardware it's plugged into could also solve (or at least alleviate) the issues. It's not really useful when you're provided the hardware and told to make it work. “Get the manufacturer to acknowledge the issue, release a new revision, and then start replacing installed hardware in the field in 6 months when you start receiving it” is not preferable to “spend an extra 100 dollars on a 25 thousand dollar electrical/instrumentation installation, on an sd-card that has a demonstrated ability to deal with the hardware we have today”. I've literally spent more money on a simple push button, of which I needed dozens on such an installation (for example: https://www.digikey.ca/en/products/detail/schneider-electric/XB4BW3165/16613561 https://www.digikey.ca/en/products/detail/schneider-electric... ). :)