7 ms·

A File System All Its Own – specialized for SSDs

100 points by cnahr 14y ago

joe_bleau 14y ago
The MLC and SLC NAND trends in figure 1 are confusing me. Historically, wasn't SLC first? Yet the graph shows pricing for MLC back to 2001, and SLC back to only 2007-ish. It correctly shows that MLC is less expensive than SLC. Maybe he didn't have old price data for SLC?
- mbjorling 14y ago
  It's hard for people to get data on flash chips / prices without signing an NDA. He probably found the data from various places and stitched it together from that.
  - joe_bleau 14y ago
    Hmm. I read the chart as price for the entire drive, not the 'raw materials' of the drive.
- adamleventhal 14y ago
  The data is from the good folks at Objective Analysis.
Sami_Lehtinen 14y ago
Layering stuff for legacy reasons isn't anything new. It was a smart idea to connect flat digital displays using ADC and VGA cable and display adapter with DAC. Still many did it, and some people even doing it today. It doesn't still make any sense what so ever.
- vy8vWJlco 14y ago
  It makes perfect sense when, say, that's the only cable you have on hand and when "mostly working for cheap" beats "not working, but technically correct and optimal." And then there are those to whom "lossy but non-DRM analog" (VGA) typically beats "lossless digital with DRM capability that will sneak up on you when you least want it" (HDMI).
  - petsos 14y ago
    What the hell am I going to do with a VGA display when the only output I have is a serial cable? And then why should I pay for a VGA cable? I can connect my vt100 anywhere with a cable I can make myself.
    - vy8vWJlco 14y ago
      I'll bite. I didn't say VGA was cutting egde, or the one true cable, or that real programmers use butterflies - only that it remains practical for a very large number of uses. I believe the longevity of VGA has a lot to do with its compatibility and explicit lack of DRM features. I have often had a choice and gone with "good enough" VGA when digital was an option, simply because I am aware that by buying HDMI I am not only financially supporting and licensing DRM, but am committing to a technology that can be used against me. I may not be the norm, but I am far from a Luddite.
      - shrughes 14y ago
        There are other digital options besides HDMI...
        vy8vWJlco 14y ago
        Yes, DVI was popular there for a while, and there are an assortment of others, but HDMI seems to have outpaced DVI in my encounters (and the others are quite niche: some Mac, Intel, etc...). DVI is still attractive, but things are shipping without it in favor of HDMI (in my experiences), and that only further helps extend VGA's lifespan. People already have VGA cables, and their only "upgrade" path to digital is often HDMI, so VGA remains the lesser of evils based on price and personal lifestyle/ethics.
        LukeShu 14y ago
        My experience is similar, but I'll add that DVI->HDMI cables are cheap, and mean that you can use DVI even when your displays are HDMI.
hobbes78 14y ago
Is exFAT no good?
- mbell 14y ago
  In short - No its not a very good file system. Even ignoring the licensing / patent issues with it, its non-journaled and only has a single FAT in most implementations; Its easily corrupted and difficult to repair. It also lacks a number of useful features like pre-allocation, robust meta-data, etc.
jpalomaki 14y ago
Wouldn't it make sense to use an object storage style interface to SSDs? Instead of managing sectors and cylinders the SSD would provide interface for managing objects, pretty much like cloud storage services like S3.
- mbjorling 14y ago
  It's one way to look at it. However, having an object interface toward the SSD does not solve the problem of variability that the author mentions. The variability is caused by the "incompatible" NAND flash interface (read, write and erase), while the IO interface to the host system is read/write (and occasional trim to let the device know of unused pages). Therefore, another interface, other than a simple read/write is the holy grail. This interface might be one that give various guarantees for the user, e.g. atomic operations, etc. It doesn't need to only be an object / page store.
- baruch 14y ago
  For some uses this may be a better option, it will still require handling of the flash chips. Assuming SAS connectivity as the interface this is perfectly possible with the OSD standard from T10 (SCSI committee). SATA is not going to work since it is a block interface only and not easily extensible in a sane way.
baruch 14y ago
The problem is that most users (home & enterprise) just want things to work, they don't really care much how to get there and to have the best efficiency. It won't be too hard to have a good filesystem that works over raw NAND flash but it will not work on older OSes, it will not work in the enterprise storage market and so there will be less buyers and thus it will cost more so no one will buy it and it will not be made. Even the enterprise storage folks just want the damn flash devices to just work without the storage folks doing anything with them. It's taken to extremes sometimes and the flash vendors just do whatever they are told since there is a lot of market in whatever the software-defined engineers want. Except the engineers mostly want to deal with high level algorithms and to brag how fast their algorithm is without really thinking about the hardware. Hardware is hard. Besides they can do something with the hardware that is already on the market rather than envision something better. TL;DR unless someone will hold the stick at both ends (software and hardware) no one will make a reduced layer solution.
- baruch 14y ago
  Case in point for the cost: I looked at buying an OpenSSD (referenced in the article) to write my own firmware to a lowly 64GB device. Was quoted $2000. There is nothing else getting close to this openness on the market.
  - adamleventhal 14y ago
    ... and the prognosis for OpenSSD is terminal.
    - baruch 14y ago
      It seems to be geared only towards research insititutes and a specific one at that, most of the work using OpenSSD is concentrated on that one university.
- dman 14y ago
  Hardware makers make this hard for themselves. Having expensive devkits and requiring NDA's for doing the most trivial things creates a dynamic where the only players able to jump through the monetary and bureaucratic hoops are large vendors. Those vendors are unable to deliver true innovation because they have so much vested in the status quo.
stcredzero 14y ago
The memory hierarchy needs to be revised to take into account the different performance characteristics of Flash RAM vs. hard drives. There is no disputing that NAND Flash SSD are very different from Dynamic RAM, static RAM, and HD.
frozenport 14y ago
As a side note, Lustre seems to work fine on SSDs.
- adamleventhal 14y ago
  Everything works "fine" on SSDs! They were designed to drop right into place.
deleted 14y ago
[deleted]
vy8vWJlco 14y ago
"Layering the file system translation on top of the flash translation is inefficient and impedes performance." "For many years SSDs were almost exclusively built to seamlessly replace hard drives; they not only supported the same block-device interface" The point of storage is to be able to put anything you want on it. That contract is the block interface, and includes the ability to change the filesystem. A file with internal structures is also a filesystem. The interfaces are fine. Change for change's sake should be avoided. (Providing a bypass, SSD-optimized interface is fine, but, ahem: "put down the crack pipes"... https://news.ycombinator.com/item?id=5541063 https://news.ycombinator.com/item?id=5541063 )
- petsos 14y ago
  > That contract is the block interface, and includes the ability to change the filesystem. The article argues that we should change this contract. > The interfaces are fine. The article argues that they are not. > Change for change's sake should be avoided. The article argues that we should change them for performance's sake.
  - vy8vWJlco 14y ago
    >> Change for change's sake should be avoided. > The article argues that we should change them for performance's sake. I also said providing a bypass was reasonable, but the article gives the impression that the block-level interface is yesterday's jam, and something less than the starting point. It is the starting point and will continue to be because block-level storage is the major use-case. Block-level access accounts for basically all bulk storage in /dev. Extra performance and features (ioctl calls, or a management interface) are gravy, but without the block-level interface, it is not accessible to 99.9% of all software and will not serve for general storage, including pre-existing filesystems, from FAT12 to Btrfs. There are applications for which those are interfaces. There's no reason to make it difficult to apply those layers. Looking forward, if it can't store current and unforseen filesystems out of the box (the block interface), even if that means a 50% reduction in speed, it doesn't deserve the name "mass storage." Noone wants a key-value store even if it is 100% faster. They may be faster, but it doesn't look like storage. And there's no need for it to look different either, because people expect to be able to use it like a block device. Take WD Green drives with their larger allocation sizes. There happens to be an optimal cluster size, but the interface is still that of block IO. If flash works best for a certain allocation size or other tweaks, or even a specific high-level formatting, fine, but it's still going to need to provide the block IO interface as a starting point if it's going to be used as a drive. It's nice to see at least one punitive downvote though.
mav3r1ck 14y ago
Not quite sure what this article is talking about: https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_optimized_for_flash_memory.2C_solid_state_media https://en.wikipedia.org/wiki/List_of_file_systems#File_syst... I personally believe log-based file systems are a perfect match due to never saving the same file repeatedly to the same location (so provides built in wear-leveling) and one can optimize writes by always clearing the head of the log for the next write.
- adamleventhal 14y ago
  That list is specious at best. Take ZFS. I designed the flash integration for ZFS; it's used as a caching tier. ZFS is definitely not optimized for use with flash as its primary backing store. The same is true for some of the other filesystems in the list; offhand: CASL and WAFL. Most of the rest are designed for embedded use cases, are research toys, or are embedded research toys.
kalleboo 14y ago
> Again, this approach today requires a vendor that can assert broad control over the whole system—from the file system to the interface, controller, and flash media. Apple would be well-positioned here if they still cared about their Macs. HFS is due for a replacement anyway after 30 years. (it could be done on iOS devices too, but flash I/O performance doesn't seem to be a the major bottleneck for those uses)
- baruch 14y ago
  They actually have all the components under their arm, they bought Anobit which made fabulous SSDs and are snatching engineers around here. Anobit SSDs were the best I've seen so far in terms of consistent performance.
  - adamleventhal 14y ago
    Agreed that Apple is very well-positioned for this -- the question is whether they care about the problem. A purpose-built filesystem would improve performance and longevity at a lower cost. Is it worth it for Apple to invest in a brand new file system and data path?
- DanBC 14y ago
  I would love Apple if they created an excellent file systems specifically for SSD use, and if they also opened that file system to other users. HFS+ has some nice features. I don't know anyone not running OS X using it.

100 points by cnahr 14y ago

The MLC and SLC NAND trends in figure 1 are confusing me. Historically, wasn't SLC first? Yet the graph shows pricing for MLC back to 2001, and SLC back to only 2007-ish. It correctly shows that MLC is less expensive than SLC. Maybe he didn't have old price data for SLC?

mbjorling 14y ago
It's hard for people to get data on flash chips / prices without signing an NDA. He probably found the data from various places and stitched it together from that.
- joe_bleau 14y ago
  Hmm. I read the chart as price for the entire drive, not the 'raw materials' of the drive.
adamleventhal 14y ago
The data is from the good folks at Objective Analysis.

Sami_Lehtinen 14y ago

Layering stuff for legacy reasons isn't anything new. It was a smart idea to connect flat digital displays using ADC and VGA cable and display adapter with DAC. Still many did it, and some people even doing it today. It doesn't still make any sense what so ever.

vy8vWJlco 14y ago
It makes perfect sense when, say, that's the only cable you have on hand and when "mostly working for cheap" beats "not working, but technically correct and optimal." And then there are those to whom "lossy but non-DRM analog" (VGA) typically beats "lossless digital with DRM capability that will sneak up on you when you least want it" (HDMI).
- petsos 14y ago
  What the hell am I going to do with a VGA display when the only output I have is a serial cable? And then why should I pay for a VGA cable? I can connect my vt100 anywhere with a cable I can make myself.
  - vy8vWJlco 14y ago
    I'll bite. I didn't say VGA was cutting egde, or the one true cable, or that real programmers use butterflies - only that it remains practical for a very large number of uses. I believe the longevity of VGA has a lot to do with its compatibility and explicit lack of DRM features. I have often had a choice and gone with "good enough" VGA when digital was an option, simply because I am aware that by buying HDMI I am not only financially supporting and licensing DRM, but am committing to a technology that can be used against me. I may not be the norm, but I am far from a Luddite.
    - shrughes 14y ago
      There are other digital options besides HDMI...
      - vy8vWJlco 14y ago
        Yes, DVI was popular there for a while, and there are an assortment of others, but HDMI seems to have outpaced DVI in my encounters (and the others are quite niche: some Mac, Intel, etc...). DVI is still attractive, but things are shipping without it in favor of HDMI (in my experiences), and that only further helps extend VGA's lifespan. People already have VGA cables, and their only "upgrade" path to digital is often HDMI, so VGA remains the lesser of evils based on price and personal lifestyle/ethics.
        LukeShu 14y ago
        My experience is similar, but I'll add that DVI->HDMI cables are cheap, and mean that you can use DVI even when your displays are HDMI.

hobbes78 14y ago

Is exFAT no good?

mbell 14y ago
In short - No its not a very good file system. Even ignoring the licensing / patent issues with it, its non-journaled and only has a single FAT in most implementations; Its easily corrupted and difficult to repair. It also lacks a number of useful features like pre-allocation, robust meta-data, etc.

jpalomaki 14y ago

Wouldn't it make sense to use an object storage style interface to SSDs? Instead of managing sectors and cylinders the SSD would provide interface for managing objects, pretty much like cloud storage services like S3.

mbjorling 14y ago
It's one way to look at it. However, having an object interface toward the SSD does not solve the problem of variability that the author mentions. The variability is caused by the "incompatible" NAND flash interface (read, write and erase), while the IO interface to the host system is read/write (and occasional trim to let the device know of unused pages). Therefore, another interface, other than a simple read/write is the holy grail. This interface might be one that give various guarantees for the user, e.g. atomic operations, etc. It doesn't need to only be an object / page store.
baruch 14y ago
For some uses this may be a better option, it will still require handling of the flash chips. Assuming SAS connectivity as the interface this is perfectly possible with the OSD standard from T10 (SCSI committee). SATA is not going to work since it is a block interface only and not easily extensible in a sane way.

baruch 14y ago

The problem is that most users (home & enterprise) just want things to work, they don't really care much how to get there and to have the best efficiency. It won't be too hard to have a good filesystem that works over raw NAND flash but it will not work on older OSes, it will not work in the enterprise storage market and so there will be less buyers and thus it will cost more so no one will buy it and it will not be made. Even the enterprise storage folks just want the damn flash devices to just work without the storage folks doing anything with them. It's taken to extremes sometimes and the flash vendors just do whatever they are told since there is a lot of market in whatever the software-defined engineers want. Except the engineers mostly want to deal with high level algorithms and to brag how fast their algorithm is without really thinking about the hardware. Hardware is hard. Besides they can do something with the hardware that is already on the market rather than envision something better. TL;DR unless someone will hold the stick at both ends (software and hardware) no one will make a reduced layer solution.

baruch 14y ago
Case in point for the cost: I looked at buying an OpenSSD (referenced in the article) to write my own firmware to a lowly 64GB device. Was quoted $2000. There is nothing else getting close to this openness on the market.
- adamleventhal 14y ago
  ... and the prognosis for OpenSSD is terminal.
  - baruch 14y ago
    It seems to be geared only towards research insititutes and a specific one at that, most of the work using OpenSSD is concentrated on that one university.
dman 14y ago
Hardware makers make this hard for themselves. Having expensive devkits and requiring NDA's for doing the most trivial things creates a dynamic where the only players able to jump through the monetary and bureaucratic hoops are large vendors. Those vendors are unable to deliver true innovation because they have so much vested in the status quo.

stcredzero 14y ago

The memory hierarchy needs to be revised to take into account the different performance characteristics of Flash RAM vs. hard drives. There is no disputing that NAND Flash SSD are very different from Dynamic RAM, static RAM, and HD.

frozenport 14y ago

As a side note, Lustre seems to work fine on SSDs.

adamleventhal 14y ago
Everything works "fine" on SSDs! They were designed to drop right into place.

deleted 14y ago

[deleted]

vy8vWJlco 14y ago

"Layering the file system translation on top of the flash translation is inefficient and impedes performance." "For many years SSDs were almost exclusively built to seamlessly replace hard drives; they not only supported the same block-device interface" The point of storage is to be able to put anything you want on it. That contract is the block interface, and includes the ability to change the filesystem. A file with internal structures is also a filesystem. The interfaces are fine. Change for change's sake should be avoided. (Providing a bypass, SSD-optimized interface is fine, but, ahem: "put down the crack pipes"... https://news.ycombinator.com/item?id=5541063 https://news.ycombinator.com/item?id=5541063 )

petsos 14y ago
> That contract is the block interface, and includes the ability to change the filesystem. The article argues that we should change this contract. > The interfaces are fine. The article argues that they are not. > Change for change's sake should be avoided. The article argues that we should change them for performance's sake.
- vy8vWJlco 14y ago
  >> Change for change's sake should be avoided. > The article argues that we should change them for performance's sake. I also said providing a bypass was reasonable, but the article gives the impression that the block-level interface is yesterday's jam, and something less than the starting point. It is the starting point and will continue to be because block-level storage is the major use-case. Block-level access accounts for basically all bulk storage in /dev. Extra performance and features (ioctl calls, or a management interface) are gravy, but without the block-level interface, it is not accessible to 99.9% of all software and will not serve for general storage, including pre-existing filesystems, from FAT12 to Btrfs. There are applications for which those are interfaces. There's no reason to make it difficult to apply those layers. Looking forward, if it can't store current and unforseen filesystems out of the box (the block interface), even if that means a 50% reduction in speed, it doesn't deserve the name "mass storage." Noone wants a key-value store even if it is 100% faster. They may be faster, but it doesn't look like storage. And there's no need for it to look different either, because people expect to be able to use it like a block device. Take WD Green drives with their larger allocation sizes. There happens to be an optimal cluster size, but the interface is still that of block IO. If flash works best for a certain allocation size or other tweaks, or even a specific high-level formatting, fine, but it's still going to need to provide the block IO interface as a starting point if it's going to be used as a drive. It's nice to see at least one punitive downvote though.

mav3r1ck 14y ago

Not quite sure what this article is talking about: https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_optimized_for_flash_memory.2C_solid_state_media https://en.wikipedia.org/wiki/List_of_file_systems#File_syst... I personally believe log-based file systems are a perfect match due to never saving the same file repeatedly to the same location (so provides built in wear-leveling) and one can optimize writes by always clearing the head of the log for the next write.

adamleventhal 14y ago
That list is specious at best. Take ZFS. I designed the flash integration for ZFS; it's used as a caching tier. ZFS is definitely not optimized for use with flash as its primary backing store. The same is true for some of the other filesystems in the list; offhand: CASL and WAFL. Most of the rest are designed for embedded use cases, are research toys, or are embedded research toys.

kalleboo 14y ago

> Again, this approach today requires a vendor that can assert broad control over the whole system—from the file system to the interface, controller, and flash media. Apple would be well-positioned here if they still cared about their Macs. HFS is due for a replacement anyway after 30 years. (it could be done on iOS devices too, but flash I/O performance doesn't seem to be a the major bottleneck for those uses)

baruch 14y ago
They actually have all the components under their arm, they bought Anobit which made fabulous SSDs and are snatching engineers around here. Anobit SSDs were the best I've seen so far in terms of consistent performance.
- adamleventhal 14y ago
  Agreed that Apple is very well-positioned for this -- the question is whether they care about the problem. A purpose-built filesystem would improve performance and longevity at a lower cost. Is it worth it for Apple to invest in a brand new file system and data path?
DanBC 14y ago
I would love Apple if they created an excellent file systems specifically for SSD use, and if they also opened that file system to other users. HFS+ has some nice features. I don't know anyone not running OS X using it.