4 ms·

No, this issue isn't something the filesystem can work around. The issue is that random writes leave you with all your erase units partially overwritten, and n

No, this issue isn't something the filesystem can work around. The issue is that random writes leave you with all your erase units partially overwritten, and none of them fully overwritten. If the writes really are random then there's no data layout that will avoid this issue - this is a fundamental issue when write size < erase unit size.

markhahn 9y ago
I really don't think you read the paper...
Dylan16807 9y ago
That problem is actually simple to solve. You append new versions until you hit some threshold, at which point you actually compact/GC. You don't have a 1:1 mapping. Getting the best performance requires careful tuning, but reducing erase count by a large factor is quite straightforward.
- wtallis 9y ago
  FYI, the guy you're replying to is the author of bcache and bcachefs. He's pretty familiar with the tradeoffs of the "simple" solution you refer to. Fragmenting when you have a partial block invalidation lets you avoid an immediate and costly read-modify-write cycle (both in terms of performance and write endurance). But storing metadata to keep track of those fragments does provide some write amplification of its own, and the eventual garbage collection is a pretty big source of further write amplification. And this requires a non-trivial spare area to keep write amplification under control and performance high. And doing all this on top of an existing flash translation layer that's already doing the same thing makes it even more complicated, and it means you might not be saving anywhere near as many media writes as a naive analysis would suggest. New SSDs are starting to allow for application-layer hints about data lifetime, in order to allow for the SSD's flash translation layer to do a sort of generational garbage collection. This actually does provide a big reduction to write amplification (and improve performance) beyond the baseline that you get with a traditional SSD with or without a filesystem that tries to be flash-friendly.
  - Dylan16807 9y ago
    > FYI, the guy you're replying to is the author of bcache and bcachefs. He's pretty familiar with the tradeoffs of the "simple" solution you refer to. Then I don't know why he oversimplified so much. It's not a silver bullet, but it's absolutely a workaround the filesystem can use to mitigate the issue. > And doing all this on top of an existing flash translation layer that's already doing the same thing makes it even more complicated, and it means you might not be saving anywhere near as many media writes as a naive analysis would suggest. Sure, but "some drives already have that mitigation" is a far cry from "filesystems can't apply that mitigation". And the end result is still a reduction in write amplification, no matter who does it.
    - wtallis 9y ago
      > Sure, but "some drives already have that mitigation" is a far cry from "filesystems can't apply that mitigation". ALL flash-based SSDs have FTLs that work more or less as described above. They wouldn't survive even short-term light usage if they did a full erase block R/M/W cycle on every 512B sector modification. If you don't even understand that most basic principle of what SSDs are doing under the hood, you're in no position to judge what a filesystem might be able to do to help. It's as naive as suggesting zipping a file twice.
      - Dylan16807 9y ago
        But it's equally naive to state that it's a 'fundamental issue' that writing a bunch of text to disk requires a lot of I/O, and that compression doesn't help. He didn't just say filesystem smarts were redundant with controller smarts, he said no data layout could avoid the problem. That's a big misstatement of the problem.
        wtallis 9y ago
        > But it's equally naive to state that it's a 'fundamental issue' that writing a bunch of text to disk requires a lot of I/O, and that compression doesn't help. That wasn't stated anywhere in this thread. > He didn't just say filesystem smarts were redundant with controller smarts, he said no data layout could avoid the problem. That's a big misstatement of the problem. You're imagining the conversation to be about a different problem than the one that was explicitly referred to. The fundamental problem under discussion is that random writes will thrash any data structure. You might be able to convert a stream of random writes to sequential writes on the first pass of filling the disk, but once you have a data set on disk and are updating it randomly (which is what almost all real-world random writes are), nothing can prevent the fragmentation and need for garbage collection. Doing it in the filesystem layer won't magically be far more effective than what's already being done at the drive level.
- baruch 9y ago
  If you only append you do not get the effect that you want, you actually need to send the writes as a large consecutive write for the SSD to do what they want in this paper. If you send many smaller writes the SSD will for sure split them to different nand dies to get the maximum write parallelization. They will be grouped for the maximum write efficiency which will not necessarily be the same as erase efficiency.
baruch 9y ago
The one thing they are saying in that paper is that they make the fs work to have its writes at about the size of the erase group which would in fact help. A problem I can see with using this on large SSDs is that their erase group size is now fairly large (at one point I was told by a vendor the erase size is 1GB in an 800GB 3WPD SSD).
caf 9y ago
The linked research paper describes a log-structured filesystem called SFS, which they built from NILFS2 in order to implement and test the random-write mitigations they propose. Samsung then submitted a clean sheet implementation of a flash-friendly log-structured filesystem called F2FS ( https://lkml.org/lkml/2012/10/5/205 https://lkml.org/lkml/2012/10/5/205 ).

markhahn 9y ago

I really don't think you read the paper...

Dylan16807 9y ago

That problem is actually simple to solve. You append new versions until you hit some threshold, at which point you actually compact/GC. You don't have a 1:1 mapping. Getting the best performance requires careful tuning, but reducing erase count by a large factor is quite straightforward.

wtallis 9y ago
FYI, the guy you're replying to is the author of bcache and bcachefs. He's pretty familiar with the tradeoffs of the "simple" solution you refer to. Fragmenting when you have a partial block invalidation lets you avoid an immediate and costly read-modify-write cycle (both in terms of performance and write endurance). But storing metadata to keep track of those fragments does provide some write amplification of its own, and the eventual garbage collection is a pretty big source of further write amplification. And this requires a non-trivial spare area to keep write amplification under control and performance high. And doing all this on top of an existing flash translation layer that's already doing the same thing makes it even more complicated, and it means you might not be saving anywhere near as many media writes as a naive analysis would suggest. New SSDs are starting to allow for application-layer hints about data lifetime, in order to allow for the SSD's flash translation layer to do a sort of generational garbage collection. This actually does provide a big reduction to write amplification (and improve performance) beyond the baseline that you get with a traditional SSD with or without a filesystem that tries to be flash-friendly.
- Dylan16807 9y ago
  > FYI, the guy you're replying to is the author of bcache and bcachefs. He's pretty familiar with the tradeoffs of the "simple" solution you refer to. Then I don't know why he oversimplified so much. It's not a silver bullet, but it's absolutely a workaround the filesystem can use to mitigate the issue. > And doing all this on top of an existing flash translation layer that's already doing the same thing makes it even more complicated, and it means you might not be saving anywhere near as many media writes as a naive analysis would suggest. Sure, but "some drives already have that mitigation" is a far cry from "filesystems can't apply that mitigation". And the end result is still a reduction in write amplification, no matter who does it.
  - wtallis 9y ago
    > Sure, but "some drives already have that mitigation" is a far cry from "filesystems can't apply that mitigation". ALL flash-based SSDs have FTLs that work more or less as described above. They wouldn't survive even short-term light usage if they did a full erase block R/M/W cycle on every 512B sector modification. If you don't even understand that most basic principle of what SSDs are doing under the hood, you're in no position to judge what a filesystem might be able to do to help. It's as naive as suggesting zipping a file twice.
    - Dylan16807 9y ago
      But it's equally naive to state that it's a 'fundamental issue' that writing a bunch of text to disk requires a lot of I/O, and that compression doesn't help. He didn't just say filesystem smarts were redundant with controller smarts, he said no data layout could avoid the problem. That's a big misstatement of the problem.
      - wtallis 9y ago
        > But it's equally naive to state that it's a 'fundamental issue' that writing a bunch of text to disk requires a lot of I/O, and that compression doesn't help. That wasn't stated anywhere in this thread. > He didn't just say filesystem smarts were redundant with controller smarts, he said no data layout could avoid the problem. That's a big misstatement of the problem. You're imagining the conversation to be about a different problem than the one that was explicitly referred to. The fundamental problem under discussion is that random writes will thrash any data structure. You might be able to convert a stream of random writes to sequential writes on the first pass of filling the disk, but once you have a data set on disk and are updating it randomly (which is what almost all real-world random writes are), nothing can prevent the fragmentation and need for garbage collection. Doing it in the filesystem layer won't magically be far more effective than what's already being done at the drive level.
baruch 9y ago
If you only append you do not get the effect that you want, you actually need to send the writes as a large consecutive write for the SSD to do what they want in this paper. If you send many smaller writes the SSD will for sure split them to different nand dies to get the maximum write parallelization. They will be grouped for the maximum write efficiency which will not necessarily be the same as erase efficiency.

baruch 9y ago

The one thing they are saying in that paper is that they make the fs work to have its writes at about the size of the erase group which would in fact help. A problem I can see with using this on large SSDs is that their erase group size is now fairly large (at one point I was told by a vendor the erase size is 1GB in an 800GB 3WPD SSD).

caf 9y ago

The linked research paper describes a log-structured filesystem called SFS, which they built from NILFS2 in order to implement and test the random-write mitigations they propose. Samsung then submitted a clean sheet implementation of a flash-friendly log-structured filesystem called F2FS ( https://lkml.org/lkml/2012/10/5/205 https://lkml.org/lkml/2012/10/5/205 ).