3 ms·

This comes up every time someone mentions the corruption. We don't power off our many pis, but they are on UPSs (we use them for a security system.) However, t

by chisleu 12y ago

This comes up every time someone mentions the corruption. We don't power off our many pis, but they are on UPSs (we use them for a security system.) However, the corruption happens so much that we keep disk images and have a spare on hand when we have to reboot them for any reason because 1/4 of the time they will not boot up. Even if it were corrupting the FILES being written when hard booted, it shouldn't corrupt the FILESYSTEM being written. TLDR, nuh uh.

tdicola 12y ago
> Even if it were corrupting the FILES being written when hard booted, it shouldn't corrupt the FILESYSTEM being written. That's not how SD cards work unfortunately. When an SD card erases data, like if it's updating a file allocation table, it has to erase an entire block of flash and not just the bytes you care about. The card will throw the block into a cache in memory, erase the block, and then rewrite the parts of the block that weren't erased. However if you pull out power during that operation you'll completely kill the block and destroy the filesystem. There's a great video here from Linuxconf Auckland about using SD cards for embedded systems and these issues: https://www.youtube.com/watch?v=K3zb6p0thQU https://www.youtube.com/watch?v=K3zb6p0thQU
- mark-r 12y ago
  I wish we had a filesystem that was tuned to the peculiarities of flash memory. It should be aware of the large erase blocks and optimize for that.
- zurn 12y ago
  There's no inherent reason SD cards have to be like this, more sophisticated SSDs are designed to stay readable after power loss. But most SD cards are cheap and nasty so they don't.
- chisleu 12y ago
  I get that, but I don't get how that applies to the corruption we are seeing. We aren't having power fail issues.
pja 12y ago
The metadata logging on modern Linux filesystems is designed with the consistency guarantees offered by spinning magnetic storage in mind. SD cards (and SSDs for that matter) are completely different - pull the plug when the card is busy updating its internal block indexes and you’ll corrupt a big chunk of the card & no filesystem level metadata logging is going to prevent that. If you could use a flash filesystem at the OS level & talk directly to the SDcard flash blocks then you might be able to pull the plug & not risk corrupting an SDcard, but as things currently stand I don’t think that’s possible.

by chisleu 12y ago

tdicola 12y ago

> Even if it were corrupting the FILES being written when hard booted, it shouldn't corrupt the FILESYSTEM being written. That's not how SD cards work unfortunately. When an SD card erases data, like if it's updating a file allocation table, it has to erase an entire block of flash and not just the bytes you care about. The card will throw the block into a cache in memory, erase the block, and then rewrite the parts of the block that weren't erased. However if you pull out power during that operation you'll completely kill the block and destroy the filesystem. There's a great video here from Linuxconf Auckland about using SD cards for embedded systems and these issues: https://www.youtube.com/watch?v=K3zb6p0thQU https://www.youtube.com/watch?v=K3zb6p0thQU

mark-r 12y ago
I wish we had a filesystem that was tuned to the peculiarities of flash memory. It should be aware of the large erase blocks and optimize for that.
zurn 12y ago
There's no inherent reason SD cards have to be like this, more sophisticated SSDs are designed to stay readable after power loss. But most SD cards are cheap and nasty so they don't.
chisleu 12y ago
I get that, but I don't get how that applies to the corruption we are seeing. We aren't having power fail issues.

pja 12y ago

The metadata logging on modern Linux filesystems is designed with the consistency guarantees offered by spinning magnetic storage in mind. SD cards (and SSDs for that matter) are completely different - pull the plug when the card is busy updating its internal block indexes and you’ll corrupt a big chunk of the card & no filesystem level metadata logging is going to prevent that. If you could use a flash filesystem at the OS level & talk directly to the SDcard flash blocks then you might be able to pull the plug & not risk corrupting an SDcard, but as things currently stand I don’t think that’s possible.