4 ms·

Rather silly title, but that's true of many papers reusing this meme. The abstract presents an alternative to wear leveling that they refer to as "capacity var

Rather silly title, but that's true of many papers reusing this meme. The abstract presents an alternative to wear leveling that they refer to as "capacity variance", also known as breaking compatibility with all previous filesystems, drivers, and bootloaders. Wear leveling has been a necessary evil, because without it flash storage simply would not be usable at all as hard drive replacements/successors. Nowadays some of the pieces are in place to allow software to interact with SSDs in a more natural fashion, but it still isn't possible to completely forego the FTL that emulates hard drive behavior.

cma 4y ago
FTL == flash translation layer
shagie 4y ago
> Rather silly title, but that's true of many papers reusing this meme. "Considered Harmful" Essays Considered Harmful - https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html // https://news.ycombinator.com/item?id=9744916 https://news.ycombinator.com/item?id=9744916 (from 2015, other posts didn't gain much traction)
remram 4y ago
The best I can say if I get an article titled "Wear leveling considered harmful" is say "no, it's not". "Considered harmful" is appropriate for a PSA that the whole community considers something to be harmful. It's terrible for an essay trying to argue that it should be avoided even though that is not the community standard (yet). The title should be "Why wear leveling is harmful", "A case against wear leveling", or something similar. I don't understand why people keep using those stupid titles.
- ranger_danger 4y ago
  You don't understand why clickbait is clickbait?
- bitwize 4y ago
  "Considered harmful" is shorthand for "Why I consider this harmful". Has been since the first one, "GO TO Considered Harmful" by Dijkstra, which came out in a time when everybody used GOTO or its equivalent.
  - afiori 4y ago
    Dijkstra's title was a shorthand for "I believe you and everybody else should consider this harmful"; appropriate for the strong claim that GOTO was bad and was making programming as a whole worse. Using "X considered harmful" in place of "Y is better than X" is just capitalizing on the meme.
  - remram 4y ago
    Even that has internal logic. "GOTO is considered harmful in the projects that I run, so we don't use it there, and this is why". If an SSD manufacturer explained that they consider the feature harmful, and they are no longer including it in products, that's something. Here it's a consumer expressing that they think they don't like the feature and think they would like someone to make products a different way. Completely different.
- eternityforest 4y ago
  If they are proposing something else that does what wear levelling does, it should be "Future replacements for SSD wear levelling" or "Next generation wear levelling for solid state storage", if they are in fact just suggesting we move the leveling to the host. Obviously we need some leveling somewhere or there's gonna be problems when someone uses an overwritable file....
ChrisLomont 4y ago
>also known as breaking compatibility with all previous filesystems.... Pretty much every filesystem in use has the ability to mark sections of a drive as bad and unusable. Pretty much every SSD in use has more blocks than it shows to the filesystem. It would not be hard to leverage both of these and likely implement this paper with zero change to the filesystem or drivers, and the SSD can remap bad sectors to the end of the space on the drive, allowing bootloaders and the like to work exactly as they do now.
- wtallis 4y ago
  Remapping bad sectors to the end of the drive doesn't solve much. For starters, GPT actually expects the end of the drive to be usable, so we'd need to introduce a new partition table format or make the drive an active participant in updating the partition table. But more importantly, you cannot simply hand-wave away the challenges of having NAND erase blocks that are huge compared to both NAND page sizes and the allocation units used by host software. If you make a drive that is at first glance "compatible" with existing software and filesystems but behind the scenes has to do eg. 24MB read-modify-write cycles for any size of host write, the drive may be much simpler than current SSDs but will be dead within a week or two at most (and painfully slow until then).
  - ChrisLomont 4y ago
    SSDs already do all you just listed behind the scenes. The FTL is already mapping such things behind the scenes so filesystems don't have to implement it. As the paper mentions, changes to these cycles lengthens the lifetime. The filesystem at the os level won't change.
kragen 4y ago
Maybe: ① when we really need wear leveling because of previous filesystems, we should implement it as a kernel module instead of in the SSD firmware? Then we can debug the failures it causes, tune its performance to our use case, and tune the software running on top of it to perform well with its strengths and weaknesses. Compatibility with previous bootloaders isn't important because writing a bootloader for a given piece of hardware is a one-afternoon task. ② The nearly 3× increase in firesystem capacity the authors report might be worth a significant amount of software work? ③ Hard disks are so different from SSDs that we're probably paying much bigger penalties than that 3× for forcing SSDs to pretend to be hard disks? To be concrete, hard disks are about eighty thousand times slower for random access than modern SDRAM: 8000 μs versus 0.1 μs. Their sequential transfer rate is maybe 250 megabytes per second, which means that the "bandwidth-delay product" is about 4 megabytes; if you're accessing data in chunks of less than 4 megabytes, you're wasting most of your disk's time seeking instead of transferring data. 25 years ago this number was about 256K. A Samsung Evo 970 1TB SSD over NVMe PCIe3 can peak at 100k iops and 6.5 GB/s (personal communication) which works out to 64 kilobytes. I've heard that other SSDs commonly can sustain full read bandwidth down to 4KiB per I/O operation. But writes are an order of magnitude slower (100 μs instead of 10 μs) and erases are an order of magnitude slower than that (1000 μs instead of 100 μs). So, aside from the shitty endurances down in the mere hundreds of writes, even when you aren't breaking it, the SSD's performance characteristics are very, very different from a disk's. If you drive it with systems software designed for a disk, you're probably wasting most of its power. For example: ⓐ because the absolute bandwidth to memory is on the order of 10% of memcpy bandwidth, and 30× the bandwidth from a disk, lots of architectural choices that prefer caching stuff in RAM pay off much more poorly with an SSD (possibly worsening performance rather than improving it), and architectural choices that put one or more memcpys on the data path cost proportionally much more. ⓑ because the "bandwidth-delay product" is 60–1000 times smaller, many design tradeoffs made to reduce random seeks on disks are probably a bad deal. At the very least you want to reduce the size of your B-tree nodes. ⓒ because writes are so much more expensive than reads, design tradeoffs that increase writes in order to reduce reads (for example, to sort nearby records together in a database, or defrag your disk) are very often a bad deal on SSDs. ⓓ because erases (which affect large contiguous areas in order to get acceptable performance) are so much more expensive than reads or writes, sequential writes are enormously cheaper than random writes, but in a different way than on disks. SMR has a similar kind of issue. ⓔ we have the ridiculous situation described in the paper where the SSD wastes 65% of its storage capacity in order to perpetuate the illusion that it's just a really fast disk. 65%! (If the simulator they used is correct.)
cryptonector 4y ago
First of all the filesystems out there, ZFS is the best placed to deal with that break and do its own, intelligent wear leveling. So I wouldn't mind ;) More seriously, the OS can add its own layer between raw, unleveled SSD, and the filesystem, and do its own wear leveling. But, really, the best solution to the need for wear leveling is to switch to CoW filesystems. The biggest problem is the need for many uberblocks, but I guess each transaction's root blocks can pre-allocate the next one's, and then the real uberblocks can be overwritten every N transactions, thus limiting the number of writes to uberblocks. Plus, of course, there should be many uberblocks, not all written every N transactions.
- mgerdts 4y ago
  > ZFS is the best placed to deal with that break and do its own, intelligent wear leveling Part of wear leveling is to rewrite old static data to blocks that are more worn so that the relatively fresh blocks that have been long squatted upon can take their turn at getting wear. ZFS has a long history about not being great at moving data to a different location, even though some of the brightest minds in this space have been highly motivated to solve this. Google "BP rewrite".
  - cryptonector 4y ago
    I'm quite aware of the BP rewrite problem. I should explain that what I had in mind is that nearly worn out blocks should be marked no-longer-to-be-written-to. I left that part out due to writing too quickly. The point is there's no need to move blocks, just stop writing to worn-out blocks, and then the CoW nature of the filesystem will take care of the rest. Eventually you'll run out of non-worn-out blocks and the filesystem will become effectively read-only. (Naturally one would want to keep a count of the number of worn-out blocks in the volume, and one would need an internal file in which to keep track of worn-out blocks, probably.) As for BP rewrite, the fundamental design fault in ZFS that makes BP rewrite hard is that block pointers include locations in them and so the checksums of blocks that have block pointers must change when any of those pointer-to blocks are relocated. What should have happened instead is that physical locations should have been separated so as to avoid block checksums binding physical block addresses: a) block pointers should have had no physical location in them, b) every block that contain block pointers should have been followed immediately by a "cache" array of physical pointers corresponding to the logical block pointers in that block, c) the "cache" of physical block addresses would then be easily excluded from block checksum computations, d) block caches would have been easy to overwrite. Once block checksums in block pointers do not bind physical addresses, you can then traverse the tree and relocate blocks by copying them to new locations then re-writing the physical block addresses in the caches (that are now not bound into those checksums). This is a very simple process. You do need to avoid creating multiple copies of blocks that are reachable via multiple paths (because of hardlinks). This proposal tends to obtain replies to the effect of that design increasing the risk of checksum collisions. That's fallacious for cryptographic checksums. I continue to be shocked that the BP rewrite problem isn't properly understood. However, it is probably too late to fix ZFS to make BP rewrite easy. The reason is two-fold: 1) the lack of BP rewrite has been worked around for things like vdev evacuation, 2) moving physical locations out of the block pointer would essentially yield a completely different on-disk layout for znodes and indirect nodes and so on, and moving to the new on-disk format would result in special-casing a lot of code, and would be quite a process. It would almost be easier to start over.
pclmulqdq 4y ago
I am not a storage person, but I have been storage-adjacent for a while. That said, it looks like storage abstractions need a huge overhaul, and we are just sitting here pretending things are fine. SSDs are just very different devices than HDDs. Zoned NVMe, which gives an append-only log abstraction, is a big step in the right direction. By the way, it turns out that hard drives today also include a fair amount of software (not as much as SSDs, but a lot) to emulate a hard drive of yesteryear. The time of the "block device" may be over.
- AnIdiotOnTheNet 4y ago
  Every computing abstraction needs a huge overhaul. We're working on a mountain made of layered abstractions wrapped around concepts from the 70s.

cma 4y ago

FTL == flash translation layer

shagie 4y ago

> Rather silly title, but that's true of many papers reusing this meme. "Considered Harmful" Essays Considered Harmful - https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html // https://news.ycombinator.com/item?id=9744916 https://news.ycombinator.com/item?id=9744916 (from 2015, other posts didn't gain much traction)

remram 4y ago

The best I can say if I get an article titled "Wear leveling considered harmful" is say "no, it's not". "Considered harmful" is appropriate for a PSA that the whole community considers something to be harmful. It's terrible for an essay trying to argue that it should be avoided even though that is not the community standard (yet). The title should be "Why wear leveling is harmful", "A case against wear leveling", or something similar. I don't understand why people keep using those stupid titles.

ranger_danger 4y ago
You don't understand why clickbait is clickbait?
bitwize 4y ago
"Considered harmful" is shorthand for "Why I consider this harmful". Has been since the first one, "GO TO Considered Harmful" by Dijkstra, which came out in a time when everybody used GOTO or its equivalent.
- afiori 4y ago
  Dijkstra's title was a shorthand for "I believe you and everybody else should consider this harmful"; appropriate for the strong claim that GOTO was bad and was making programming as a whole worse. Using "X considered harmful" in place of "Y is better than X" is just capitalizing on the meme.
- remram 4y ago
  Even that has internal logic. "GOTO is considered harmful in the projects that I run, so we don't use it there, and this is why". If an SSD manufacturer explained that they consider the feature harmful, and they are no longer including it in products, that's something. Here it's a consumer expressing that they think they don't like the feature and think they would like someone to make products a different way. Completely different.
eternityforest 4y ago
If they are proposing something else that does what wear levelling does, it should be "Future replacements for SSD wear levelling" or "Next generation wear levelling for solid state storage", if they are in fact just suggesting we move the leveling to the host. Obviously we need some leveling somewhere or there's gonna be problems when someone uses an overwritable file....

ChrisLomont 4y ago

>also known as breaking compatibility with all previous filesystems.... Pretty much every filesystem in use has the ability to mark sections of a drive as bad and unusable. Pretty much every SSD in use has more blocks than it shows to the filesystem. It would not be hard to leverage both of these and likely implement this paper with zero change to the filesystem or drivers, and the SSD can remap bad sectors to the end of the space on the drive, allowing bootloaders and the like to work exactly as they do now.

wtallis 4y ago
Remapping bad sectors to the end of the drive doesn't solve much. For starters, GPT actually expects the end of the drive to be usable, so we'd need to introduce a new partition table format or make the drive an active participant in updating the partition table. But more importantly, you cannot simply hand-wave away the challenges of having NAND erase blocks that are huge compared to both NAND page sizes and the allocation units used by host software. If you make a drive that is at first glance "compatible" with existing software and filesystems but behind the scenes has to do eg. 24MB read-modify-write cycles for any size of host write, the drive may be much simpler than current SSDs but will be dead within a week or two at most (and painfully slow until then).
- ChrisLomont 4y ago
  SSDs already do all you just listed behind the scenes. The FTL is already mapping such things behind the scenes so filesystems don't have to implement it. As the paper mentions, changes to these cycles lengthens the lifetime. The filesystem at the os level won't change.

kragen 4y ago

Maybe: ① when we really need wear leveling because of previous filesystems, we should implement it as a kernel module instead of in the SSD firmware? Then we can debug the failures it causes, tune its performance to our use case, and tune the software running on top of it to perform well with its strengths and weaknesses. Compatibility with previous bootloaders isn't important because writing a bootloader for a given piece of hardware is a one-afternoon task. ② The nearly 3× increase in firesystem capacity the authors report might be worth a significant amount of software work? ③ Hard disks are so different from SSDs that we're probably paying much bigger penalties than that 3× for forcing SSDs to pretend to be hard disks? To be concrete, hard disks are about eighty thousand times slower for random access than modern SDRAM: 8000 μs versus 0.1 μs. Their sequential transfer rate is maybe 250 megabytes per second, which means that the "bandwidth-delay product" is about 4 megabytes; if you're accessing data in chunks of less than 4 megabytes, you're wasting most of your disk's time seeking instead of transferring data. 25 years ago this number was about 256K. A Samsung Evo 970 1TB SSD over NVMe PCIe3 can peak at 100k iops and 6.5 GB/s (personal communication) which works out to 64 kilobytes. I've heard that other SSDs commonly can sustain full read bandwidth down to 4KiB per I/O operation. But writes are an order of magnitude slower (100 μs instead of 10 μs) and erases are an order of magnitude slower than that (1000 μs instead of 100 μs). So, aside from the shitty endurances down in the mere hundreds of writes, even when you aren't breaking it, the SSD's performance characteristics are very, very different from a disk's. If you drive it with systems software designed for a disk, you're probably wasting most of its power. For example: ⓐ because the absolute bandwidth to memory is on the order of 10% of memcpy bandwidth, and 30× the bandwidth from a disk, lots of architectural choices that prefer caching stuff in RAM pay off much more poorly with an SSD (possibly worsening performance rather than improving it), and architectural choices that put one or more memcpys on the data path cost proportionally much more. ⓑ because the "bandwidth-delay product" is 60–1000 times smaller, many design tradeoffs made to reduce random seeks on disks are probably a bad deal. At the very least you want to reduce the size of your B-tree nodes. ⓒ because writes are so much more expensive than reads, design tradeoffs that increase writes in order to reduce reads (for example, to sort nearby records together in a database, or defrag your disk) are very often a bad deal on SSDs. ⓓ because erases (which affect large contiguous areas in order to get acceptable performance) are so much more expensive than reads or writes, sequential writes are enormously cheaper than random writes, but in a different way than on disks. SMR has a similar kind of issue. ⓔ we have the ridiculous situation described in the paper where the SSD wastes 65% of its storage capacity in order to perpetuate the illusion that it's just a really fast disk. 65%! (If the simulator they used is correct.)

cryptonector 4y ago

First of all the filesystems out there, ZFS is the best placed to deal with that break and do its own, intelligent wear leveling. So I wouldn't mind ;) More seriously, the OS can add its own layer between raw, unleveled SSD, and the filesystem, and do its own wear leveling. But, really, the best solution to the need for wear leveling is to switch to CoW filesystems. The biggest problem is the need for many uberblocks, but I guess each transaction's root blocks can pre-allocate the next one's, and then the real uberblocks can be overwritten every N transactions, thus limiting the number of writes to uberblocks. Plus, of course, there should be many uberblocks, not all written every N transactions.

mgerdts 4y ago
> ZFS is the best placed to deal with that break and do its own, intelligent wear leveling Part of wear leveling is to rewrite old static data to blocks that are more worn so that the relatively fresh blocks that have been long squatted upon can take their turn at getting wear. ZFS has a long history about not being great at moving data to a different location, even though some of the brightest minds in this space have been highly motivated to solve this. Google "BP rewrite".
- cryptonector 4y ago
  I'm quite aware of the BP rewrite problem. I should explain that what I had in mind is that nearly worn out blocks should be marked no-longer-to-be-written-to. I left that part out due to writing too quickly. The point is there's no need to move blocks, just stop writing to worn-out blocks, and then the CoW nature of the filesystem will take care of the rest. Eventually you'll run out of non-worn-out blocks and the filesystem will become effectively read-only. (Naturally one would want to keep a count of the number of worn-out blocks in the volume, and one would need an internal file in which to keep track of worn-out blocks, probably.) As for BP rewrite, the fundamental design fault in ZFS that makes BP rewrite hard is that block pointers include locations in them and so the checksums of blocks that have block pointers must change when any of those pointer-to blocks are relocated. What should have happened instead is that physical locations should have been separated so as to avoid block checksums binding physical block addresses: a) block pointers should have had no physical location in them, b) every block that contain block pointers should have been followed immediately by a "cache" array of physical pointers corresponding to the logical block pointers in that block, c) the "cache" of physical block addresses would then be easily excluded from block checksum computations, d) block caches would have been easy to overwrite. Once block checksums in block pointers do not bind physical addresses, you can then traverse the tree and relocate blocks by copying them to new locations then re-writing the physical block addresses in the caches (that are now not bound into those checksums). This is a very simple process. You do need to avoid creating multiple copies of blocks that are reachable via multiple paths (because of hardlinks). This proposal tends to obtain replies to the effect of that design increasing the risk of checksum collisions. That's fallacious for cryptographic checksums. I continue to be shocked that the BP rewrite problem isn't properly understood. However, it is probably too late to fix ZFS to make BP rewrite easy. The reason is two-fold: 1) the lack of BP rewrite has been worked around for things like vdev evacuation, 2) moving physical locations out of the block pointer would essentially yield a completely different on-disk layout for znodes and indirect nodes and so on, and moving to the new on-disk format would result in special-casing a lot of code, and would be quite a process. It would almost be easier to start over.

pclmulqdq 4y ago

I am not a storage person, but I have been storage-adjacent for a while. That said, it looks like storage abstractions need a huge overhaul, and we are just sitting here pretending things are fine. SSDs are just very different devices than HDDs. Zoned NVMe, which gives an append-only log abstraction, is a big step in the right direction. By the way, it turns out that hard drives today also include a fair amount of software (not as much as SSDs, but a lot) to emulate a hard drive of yesteryear. The time of the "block device" may be over.

AnIdiotOnTheNet 4y ago
Every computing abstraction needs a huge overhaul. We're working on a mountain made of layered abstractions wrapped around concepts from the 70s.