7 ms·

The part about filesystems is slightly incorrect. > The way the file system handles this is incompatible with the workings of NAND flash. That's true of most

The part about filesystems is slightly incorrect. > The way the file system handles this is incompatible with the workings of NAND flash. That's true of most conventional filesystems, but log-structured filesystems are much more flash-friendly. That's why there has been a resurgence of interest in them, and also why a typical flash translation layer bears a striking resemblance to a log-structured FS. There are also flash-specific filesystems. > to an HDD all sectors are the same. This is not true because of bad blocks. Every disk has a reserve of blocks that can be remapped in place of a detected bad block, transparently, much like flash blocks are remapped. Beyond that, it's also useful for a disk to know which blocks are not in use so it can return all zeroes without actually hitting the media. There are special flags to force media access and commands to physically zero a block, for the cases where those are needed, but often they're not. Trim/discard actually gets pretty complicated, especially when things like RAID and virtual block devices are involved.

rckoepke 7y ago
> to an HDD all sectors are the same. Also I believe some humans, and (filesystems?) intentionally stored certain data towards the inside/outside of the HDD because the simple cylinder geometry allowed faster reads in those regions. However, I'm not seeing conclusive proof that modern HDDs show performance variation with respect to radius.
- monocasa 7y ago
  Io schedulers would also understand the geometry and reorder commands to minimize seek distance (and therefore seek time).
  - notacoward 7y ago
    I used to do some of that a long time ago, and it was actually kind of fun. There's a whole different set of optimizations to do on flash, both for performance and (more importantly in many use cases) power efficiency, and I think that would be fun too.
  - tzs 7y ago
    At some point, I think around the mid '90s or early 2000s but maybe earlier, though, seek time was fast enough on widely available drives that on average for random access you spent about as much time on the right cylinder waiting for the sector you wanted to rotate to under the head as you did on seeking to the right cylinder. You could get some decent gains then if you made your scheduler take rotation into account. A long seek that arrived just before the target sector came under the head could be faster than a short seek that would arrive just after the sector passed the head. On the other hand, taking rotation into account could make the scheduler quite a bit more complex. You needed a model that could predict seek time well, and you needed to know the angular position of each sector in its cylinder. I don't think that there were any drives that would tell you this. SCSI drives wouldn't even tell you the geometry. IDE drives would tell you a geometry, but it didn't necessarily have anything to do with the actual geometry of the drive. At the time I worked at a company that was working on disk performance enhancement software (e.g., drivers with better scheduling, utilities that would log disk accesses and then rearrange data on the disk so that the I/O patterns in the logs would be faster [1], and that sort of thing). We had a program that could get the real disk geometry. It did so by doing a lot of random I/O and looking at the timing of when the results came back. If there were no disk cache, this would be fairly easy. (Well, it didn't necessarily get the real geometry, but rather a purported geometry and seek and rotational characteristics that could predict I/O time well). For instance, read some random sector T, then read another random sector, then read T again. Look at the time difference between when you started getting data back on the two reads of T. This should be a multiple of the rotation time. If the disk has caching that can still work but you need to read a lot of random sectors between the two reads of T to try to get the first read out of the cache. Anyway, we had to give up on that approach because the program to analyze the disk took a few days of constant I/O to finish. Management decided that most consumers would not put up with such a long setup procedure. [1] Yes, that could mean that it would purposefully make files more fragmented. A fairly common pattern was for a program to open a bunch of data files and read a header from each. E.g., some big GUI programs would do that for a large number of font files. Arranging that program and those data files on disk so that you have the program code that gets loaded before the header reads, then the headers of all the files, and then the rest of the file data, could give you a nice speed boost. The flaw in this method is that, to use the above example, if another big GUI program also uses those same font files, the layout that makes the first program go fast might suck for the second program. If you've got a computer that you mostly only use for one task, though, it can be a viable approach.
- wtallis 7y ago
  > However, I'm not seeing conclusive proof that modern HDDs show performance variation with respect to radius. They definitely still do; it's fundamental to drives that run at fixed RPM but maintain high areal density across the entire platter. One of the 1TB drives I have lying around does about 183MB/s at the beginning of the disk, 148MB/s in the middle, and 97MB/s at the end.
  - ascar 7y ago
    I'm really interested in how you benchmarked that? Is there a "simple" way to specify the physical location on a HD data should be written to?
    - wtallis 7y ago
      Hard drives generally use a fairly simple linear mapping between LBAs and physical location. Low LBAs are on the outer edge of the platter where transfer speeds are highest, and high LBAs are on the inner edge where speeds are lowest. Unlike SSDs, hard drives don't need wear leveling, so there's no reason to break from that pattern except in the relatively rare instance of damaged sectors. At smaller scales, the layout of individual tracks can vary quite a bit, but that doesn't have as much impact on overall sequential transfer speed. See http://blog.stuffedcow.net/2019/09/hard-disk-geometry-microbenchmarking/ http://blog.stuffedcow.net/2019/09/hard-disk-geometry-microb...
      - zozbot234 7y ago
        Your link was discussed @ https://news.ycombinator.com/item?id=21492345 https://news.ycombinator.com/item?id=21492345
        wtallis 7y ago
        Yep. I found it interesting enough to bookmark, and now I tend to pull it up when somebody wants to know how hard drives really work.
      - mehrdadn 7y ago
- kabdib 7y ago
  This is still very much true, to the point that game consoles will reserve areas of rotating media for different purposes, depending on speed (e.g., disk caches on the Xbox are on the outer tracks). For optical media the layout of assets can be critical to the user experience. You probably want your startup assets located on the outer, faster tracks.
- marcosdumay 7y ago
  People reduce the number of cylinders to keep seek time under control. It just happens that if you are going to choose what cylinders to use, you better choose the inner ones that are slightly faster.
  - Hackbraten 7y ago
    Wouldn’t the outer ones be slightly faster in terms of both transfer rate and head travel time?
- CrazyStat 7y ago
  I remember reading a blog post ca. 2005, before SSDs were common, in which the author explained how he had identified all the files his system needed to boot and the order in which they were read from disk, and then contrived to put them all in the correct order in a contiguous block at the start of the drive to eliminate seeks during boot entirely. This apparently shaved a couple seconds off his boot time.
  - dspillett 7y ago
    There are automated solutions to this. Windows has had similar tricks built in (at various levels of cleverness, depending on the Windows variant you run) for a while now: https://en.wikipedia.org/wiki/Prefetcher https://en.wikipedia.org/wiki/Prefetcher No doubt there are similar options for Linux and other OSs. A similar trick I've seen is using a small SSD as a manual cache: copy in the files that need to be fast and mount it with the larger filesystem (on slower drives) as a unioned filesystem. Though just using block device based automatic cacheing may be easier and safer than rolling your own mess. There are a few options for Linux though some are not currently maintained (see https://serverfault.com/questions/969302/linux-ssd-as-hdd-cache https://serverfault.com/questions/969302/linux-ssd-as-hdd-ca... amongst other places), and some IO controllers support it directly (even some motherboards have this built in) without needing to bother your OS with the details at all.
guenthert 7y ago
> but log-structured filesystems are much more flash-friendly. That's why there has been a resurgence of interest in them I could swear I read such in 2010 -- did that actually happen in the last ten years?
- osmarks 7y ago
  f2fs (https://en.wikipedia.org/wiki/F2FS https://en.wikipedia.org/wiki/F2FS) is now popular for Android-based mobile devices.
  - guenthert 7y ago
    That counts, thanks!
rodgerd 7y ago
He's also inaccurate in characterizing all cells as "1 or 0"; digital electronics have always been a cut-off of an underlying analogue value (such as voltage).
- outworlder 7y ago
  They explain this in more detail later on. See "SSD Reads" and "interpreting the results".
namibj 7y ago
I briefly searched for a source of (mostly) FTL-less microSD cards (I'd be fine with them doing wear leveling-related remapping of whole blocks, as long as it's power-loss-safe, but I don't want them to expend any effort on emulating 512/4k random write capability). I didn't find anything but bunnie's blog entry[0] on hacking/re-flashing the firmware on the card's controller. Due to how important it is to have a good FTL, and the pretty much native suitability of log-structured data formats, like those based on RocksDB or InfluxDB (both not unlikely for data handled and stored on RPi-like SBCs), it'd be much better for reliability and performance reasons to let these LSM engines deal with NAND flash's block erase/sector write/sector read behavior. [0]: https://www.bunniestudios.com/blog/?p=3554 https://www.bunniestudios.com/blog/?p=3554

rckoepke 7y ago

> to an HDD all sectors are the same. Also I believe some humans, and (filesystems?) intentionally stored certain data towards the inside/outside of the HDD because the simple cylinder geometry allowed faster reads in those regions. However, I'm not seeing conclusive proof that modern HDDs show performance variation with respect to radius.

monocasa 7y ago
Io schedulers would also understand the geometry and reorder commands to minimize seek distance (and therefore seek time).
- notacoward 7y ago
  I used to do some of that a long time ago, and it was actually kind of fun. There's a whole different set of optimizations to do on flash, both for performance and (more importantly in many use cases) power efficiency, and I think that would be fun too.
- tzs 7y ago
  At some point, I think around the mid '90s or early 2000s but maybe earlier, though, seek time was fast enough on widely available drives that on average for random access you spent about as much time on the right cylinder waiting for the sector you wanted to rotate to under the head as you did on seeking to the right cylinder. You could get some decent gains then if you made your scheduler take rotation into account. A long seek that arrived just before the target sector came under the head could be faster than a short seek that would arrive just after the sector passed the head. On the other hand, taking rotation into account could make the scheduler quite a bit more complex. You needed a model that could predict seek time well, and you needed to know the angular position of each sector in its cylinder. I don't think that there were any drives that would tell you this. SCSI drives wouldn't even tell you the geometry. IDE drives would tell you a geometry, but it didn't necessarily have anything to do with the actual geometry of the drive. At the time I worked at a company that was working on disk performance enhancement software (e.g., drivers with better scheduling, utilities that would log disk accesses and then rearrange data on the disk so that the I/O patterns in the logs would be faster [1], and that sort of thing). We had a program that could get the real disk geometry. It did so by doing a lot of random I/O and looking at the timing of when the results came back. If there were no disk cache, this would be fairly easy. (Well, it didn't necessarily get the real geometry, but rather a purported geometry and seek and rotational characteristics that could predict I/O time well). For instance, read some random sector T, then read another random sector, then read T again. Look at the time difference between when you started getting data back on the two reads of T. This should be a multiple of the rotation time. If the disk has caching that can still work but you need to read a lot of random sectors between the two reads of T to try to get the first read out of the cache. Anyway, we had to give up on that approach because the program to analyze the disk took a few days of constant I/O to finish. Management decided that most consumers would not put up with such a long setup procedure. [1] Yes, that could mean that it would purposefully make files more fragmented. A fairly common pattern was for a program to open a bunch of data files and read a header from each. E.g., some big GUI programs would do that for a large number of font files. Arranging that program and those data files on disk so that you have the program code that gets loaded before the header reads, then the headers of all the files, and then the rest of the file data, could give you a nice speed boost. The flaw in this method is that, to use the above example, if another big GUI program also uses those same font files, the layout that makes the first program go fast might suck for the second program. If you've got a computer that you mostly only use for one task, though, it can be a viable approach.
wtallis 7y ago
> However, I'm not seeing conclusive proof that modern HDDs show performance variation with respect to radius. They definitely still do; it's fundamental to drives that run at fixed RPM but maintain high areal density across the entire platter. One of the 1TB drives I have lying around does about 183MB/s at the beginning of the disk, 148MB/s in the middle, and 97MB/s at the end.
- ascar 7y ago
  I'm really interested in how you benchmarked that? Is there a "simple" way to specify the physical location on a HD data should be written to?
  - wtallis 7y ago
    Hard drives generally use a fairly simple linear mapping between LBAs and physical location. Low LBAs are on the outer edge of the platter where transfer speeds are highest, and high LBAs are on the inner edge where speeds are lowest. Unlike SSDs, hard drives don't need wear leveling, so there's no reason to break from that pattern except in the relatively rare instance of damaged sectors. At smaller scales, the layout of individual tracks can vary quite a bit, but that doesn't have as much impact on overall sequential transfer speed. See http://blog.stuffedcow.net/2019/09/hard-disk-geometry-microbenchmarking/ http://blog.stuffedcow.net/2019/09/hard-disk-geometry-microb...
    - zozbot234 7y ago
      Your link was discussed @ https://news.ycombinator.com/item?id=21492345 https://news.ycombinator.com/item?id=21492345
      - wtallis 7y ago
        Yep. I found it interesting enough to bookmark, and now I tend to pull it up when somebody wants to know how hard drives really work.
    - mehrdadn 7y ago
kabdib 7y ago
This is still very much true, to the point that game consoles will reserve areas of rotating media for different purposes, depending on speed (e.g., disk caches on the Xbox are on the outer tracks). For optical media the layout of assets can be critical to the user experience. You probably want your startup assets located on the outer, faster tracks.
marcosdumay 7y ago
People reduce the number of cylinders to keep seek time under control. It just happens that if you are going to choose what cylinders to use, you better choose the inner ones that are slightly faster.
- Hackbraten 7y ago
  Wouldn’t the outer ones be slightly faster in terms of both transfer rate and head travel time?
CrazyStat 7y ago
I remember reading a blog post ca. 2005, before SSDs were common, in which the author explained how he had identified all the files his system needed to boot and the order in which they were read from disk, and then contrived to put them all in the correct order in a contiguous block at the start of the drive to eliminate seeks during boot entirely. This apparently shaved a couple seconds off his boot time.
- dspillett 7y ago
  There are automated solutions to this. Windows has had similar tricks built in (at various levels of cleverness, depending on the Windows variant you run) for a while now: https://en.wikipedia.org/wiki/Prefetcher https://en.wikipedia.org/wiki/Prefetcher No doubt there are similar options for Linux and other OSs. A similar trick I've seen is using a small SSD as a manual cache: copy in the files that need to be fast and mount it with the larger filesystem (on slower drives) as a unioned filesystem. Though just using block device based automatic cacheing may be easier and safer than rolling your own mess. There are a few options for Linux though some are not currently maintained (see https://serverfault.com/questions/969302/linux-ssd-as-hdd-cache https://serverfault.com/questions/969302/linux-ssd-as-hdd-ca... amongst other places), and some IO controllers support it directly (even some motherboards have this built in) without needing to bother your OS with the details at all.

guenthert 7y ago

> but log-structured filesystems are much more flash-friendly. That's why there has been a resurgence of interest in them I could swear I read such in 2010 -- did that actually happen in the last ten years?

osmarks 7y ago
f2fs (https://en.wikipedia.org/wiki/F2FS https://en.wikipedia.org/wiki/F2FS) is now popular for Android-based mobile devices.
- guenthert 7y ago
  That counts, thanks!

rodgerd 7y ago

He's also inaccurate in characterizing all cells as "1 or 0"; digital electronics have always been a cut-off of an underlying analogue value (such as voltage).

outworlder 7y ago
They explain this in more detail later on. See "SSD Reads" and "interpreting the results".

namibj 7y ago

I briefly searched for a source of (mostly) FTL-less microSD cards (I'd be fine with them doing wear leveling-related remapping of whole blocks, as long as it's power-loss-safe, but I don't want them to expend any effort on emulating 512/4k random write capability). I didn't find anything but bunnie's blog entry[0] on hacking/re-flashing the firmware on the card's controller. Due to how important it is to have a good FTL, and the pretty much native suitability of log-structured data formats, like those based on RocksDB or InfluxDB (both not unlikely for data handled and stored on RPi-like SBCs), it'd be much better for reliability and performance reasons to let these LSM engines deal with NAND flash's block erase/sector write/sector read behavior. [0]: https://www.bunniestudios.com/blog/?p=3554 https://www.bunniestudios.com/blog/?p=3554