5 ms·

Wondering, if there is flash memory friendly filesystem, that is actually not overwriting existing blocks until card is full, but write any changes to new memor

Wondering, if there is flash memory friendly filesystem, that is actually not overwriting existing blocks until card is full, but write any changes to new memory cells, rather than overwriting existing ones. This is not a problem for devices like cameras, since they only create new files, so wear is distributed among all cells evenly (assuming you remove all files when card is full, before taking new photos), but it's real problem for devices like raspberry pi and rapsbian, doing lot of updates (log files, ...). And, yes I understand root partition, could be set to read-only and log stored on tmpfs, but I'm still curious. PS: As I still struggle to understand, why I'm getting downvotes. So please be so kind and write why, so I can eventually delete this comment.

kalleboo 7y ago
There are a few https://en.wikipedia.org/wiki/Flash_file_system https://en.wikipedia.org/wiki/Flash_file_system
- deleted 7y ago
  [deleted]
rocqua 7y ago
I thought most flash devices did their own block mapping to implement this. Or is that just SSDs?
- krylon 7y ago
  AFAIU, you are correct, but apparently, "most flash devices" does not include MicroSD cards. At least that is how I remember it.
  - pjc50 7y ago
    No, they've definitely got a Flash translation layer in there, and a microcontroller to run it. It's pretty much unavoidable for making a working flash device with reasonable performance on traditional filesystems.
    - zzzcpan 7y ago
      But SD cards, CF cards, eMMCs, USB sticks - unless enterprise graded and claim otherwise, all have either very primitive wear leveling in their FTL or none at all, it's only SSDs that have full blown log structured algorithms.
- loeg 7y ago
  Only tiny tiny embedded devices don't use a wear-leveling controller of some kind.
mrob 7y ago
It's called a log-structured file system. LWN has a good overview: https://lwn.net/Articles/353411/ https://lwn.net/Articles/353411/ SSDs implement them internally, and some Android devices use the F2FS filesystem: https://en.wikipedia.org/wiki/F2FS https://en.wikipedia.org/wiki/F2FS
- deleted 7y ago
  [deleted]
- toolslive 7y ago
  Log-structured is conceptually robust, but SSDs/NVMes have failure modes that can do things like "in this extent of 64MB, all bytes have their 6th bit erased". So it's an illusion to think that in case of a crash, the things you did not touch remained unharmed.
  - toolslive 7y ago
    here's a paper that will shatter some illusions: https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf https://www.usenix.org/system/files/conference/fast13/fast13...
    - londons_explore 7y ago
      Considering how the hardware doesn't provide the serializability guarantees that it claims, why do we pay the high software performance and complexity hit to try to get the same? Lockfiles, O_SYNC, flush(), etc. all become unnecessary if we just assume that all data is at risk in case of improper poweroff. libeatmydata does this, and dramatically increases performance for some workloads.
      - blattimwind 7y ago
        I/O has always been a happy-path-only adventure in mainstream software and hardware. Attempting consistent I/O (kernel and hardware will try their best to thwart any attempt) etc. may help in some cases, but ultimately there are hardly any guarantees when it comes to power loss, and your data may be gone or corrupted no matter what you did.

kalleboo 7y ago

There are a few https://en.wikipedia.org/wiki/Flash_file_system https://en.wikipedia.org/wiki/Flash_file_system

deleted 7y ago
[deleted]

rocqua 7y ago

I thought most flash devices did their own block mapping to implement this. Or is that just SSDs?

krylon 7y ago
AFAIU, you are correct, but apparently, "most flash devices" does not include MicroSD cards. At least that is how I remember it.
- pjc50 7y ago
  No, they've definitely got a Flash translation layer in there, and a microcontroller to run it. It's pretty much unavoidable for making a working flash device with reasonable performance on traditional filesystems.
  - zzzcpan 7y ago
    But SD cards, CF cards, eMMCs, USB sticks - unless enterprise graded and claim otherwise, all have either very primitive wear leveling in their FTL or none at all, it's only SSDs that have full blown log structured algorithms.
loeg 7y ago
Only tiny tiny embedded devices don't use a wear-leveling controller of some kind.

mrob 7y ago

It's called a log-structured file system. LWN has a good overview: https://lwn.net/Articles/353411/ https://lwn.net/Articles/353411/ SSDs implement them internally, and some Android devices use the F2FS filesystem: https://en.wikipedia.org/wiki/F2FS https://en.wikipedia.org/wiki/F2FS

deleted 7y ago
[deleted]
toolslive 7y ago
Log-structured is conceptually robust, but SSDs/NVMes have failure modes that can do things like "in this extent of 64MB, all bytes have their 6th bit erased". So it's an illusion to think that in case of a crash, the things you did not touch remained unharmed.
- toolslive 7y ago
  here's a paper that will shatter some illusions: https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf https://www.usenix.org/system/files/conference/fast13/fast13...
  - londons_explore 7y ago
    Considering how the hardware doesn't provide the serializability guarantees that it claims, why do we pay the high software performance and complexity hit to try to get the same? Lockfiles, O_SYNC, flush(), etc. all become unnecessary if we just assume that all data is at risk in case of improper poweroff. libeatmydata does this, and dramatically increases performance for some workloads.
    - blattimwind 7y ago
      I/O has always been a happy-path-only adventure in mainstream software and hardware. Attempting consistent I/O (kernel and hardware will try their best to thwart any attempt) etc. may help in some cases, but ultimately there are hardly any guarantees when it comes to power loss, and your data may be gone or corrupted no matter what you did.