3 ms·

The more I think about it, the more I'm thinking that allowing SSDs to present a higher level API than block storage would be an incredible win. Consider: - Cu

The more I think about it, the more I'm thinking that allowing SSDs to present a higher level API than block storage would be an incredible win. Consider: - Currently, SSD controller firmware goes to incredible lengths to present something that is not a contiguous block device (due to the way flash works) as a contiguous block device to the OS. Likely this includes a huge amount of logic for storing and updating the mapping between logical blocks and actual hardware storage location. Since the control APIs currently are very low level, the controller basically has to buffer everything, transform that into what the flash needs to do, and then execute, not unlike how a CPU decodes instructions and converts to uops or reorders. Presumably this is quite difficult as early controllers had all sorts of problems such as performance tanking when the disk got close to full. - The filesystem then pretends that the very non-uniform storage provided by the SSD is actually a uniform block and introduces a layer of metadata in a bespoke format to determine the actual disk locations of files, directories, and their associated metadata. Along with that, it can include lots of very nice extra features such as encryption, journaling, hashing (for integrity and de-duplication), CoW, symlinks, hardlinks, directory hardlinks, etc. - It seems likely then that were the storage API to be raised to a higher level (files and metadata, rather than block storage), the SSD controller can implement every one of those features with superior reliability, performance, and power usage. We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance. What would happen if a file's extents mapping was simply the physical location mapping the SSD controller is already doing? If if could transparently perform hashing on read/write or keep a proper journal of file operations? Handle CoW transparently? And all of that implemented in a way that those just become standard storage features rather than requiring a certain OS to run.

theamk 7y ago
I'd really like to go other direction, and have SSDs expose non-uniform storage. Right now, if my filesystem dies, I have a ton of tools and manuals for recovery. This is the format is well known, and diagnostics tools are easily available. If my SSD dies, it's just gone. In two cases I had, the drive just was not allowing any file reads at all -- I had zero chance to recover the data. The drive internals are totally opaque. So no, I would prefer my SSDs to be super dumb -- I do not trust the manufacturers to get recovery tools right.
- jabl 7y ago
  +1 I'd love to see something like Open Channel SSD's ( http://lightnvm.io/ http://lightnvm.io/ , https://openchannelssd.readthedocs.io/en/latest/ https://openchannelssd.readthedocs.io/en/latest/ ) becoming mainstream. Spec at http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf > This specification defines a class of SSDs named Open-Channel SSDs. They are different from traditional SSDs in that they expose the internal parallelism of an SSD and allow the host to manage them. Maintaining the parallelism on the host enables benefits such as I/O isolation, predictable latencies, and software-defined non-volatile memory management, that scales to hundreds or thousands of workers per terabyte.
imtringued 7y ago
>We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance Yeah but are you really sure that hardware encryption is safer? Are you sure that it's not secretly saving the key somewhere where it can be extracted? Are you sure that the hardware encryption works at all and that it's not just writing the data unencrypted?
- cyphar 7y ago
  In fact there are so many examples of hardware encryption being objectively broken[1]. I wouldn't trust it at all, and treat it as plain-text storage that probably has more firmware bugs that will lead to data loss. [1]: https://news.ycombinator.com/item?id=18382975 https://news.ycombinator.com/item?id=18382975
unionpivo 7y ago
You do realize its all software at one point right ? It just that instead of running on main cpu, it would run on SSD controller's microchip. And will probably be binary only, that people wont be able to inspect or fix. Samsung has good hardware, but not that great software. I'd rather have the opposite. SSD hardware exposing their internals, to os developers.

theamk 7y ago

I'd really like to go other direction, and have SSDs expose non-uniform storage. Right now, if my filesystem dies, I have a ton of tools and manuals for recovery. This is the format is well known, and diagnostics tools are easily available. If my SSD dies, it's just gone. In two cases I had, the drive just was not allowing any file reads at all -- I had zero chance to recover the data. The drive internals are totally opaque. So no, I would prefer my SSDs to be super dumb -- I do not trust the manufacturers to get recovery tools right.

jabl 7y ago
+1 I'd love to see something like Open Channel SSD's ( http://lightnvm.io/ http://lightnvm.io/ , https://openchannelssd.readthedocs.io/en/latest/ https://openchannelssd.readthedocs.io/en/latest/ ) becoming mainstream. Spec at http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf > This specification defines a class of SSDs named Open-Channel SSDs. They are different from traditional SSDs in that they expose the internal parallelism of an SSD and allow the host to manage them. Maintaining the parallelism on the host enables benefits such as I/O isolation, predictable latencies, and software-defined non-volatile memory management, that scales to hundreds or thousands of workers per terabyte.

imtringued 7y ago

>We've already seen where software encryption went to hardware encryption -- the result is that the encryption is mostly transparent to performance Yeah but are you really sure that hardware encryption is safer? Are you sure that it's not secretly saving the key somewhere where it can be extracted? Are you sure that the hardware encryption works at all and that it's not just writing the data unencrypted?

cyphar 7y ago
In fact there are so many examples of hardware encryption being objectively broken[1]. I wouldn't trust it at all, and treat it as plain-text storage that probably has more firmware bugs that will lead to data loss. [1]: https://news.ycombinator.com/item?id=18382975 https://news.ycombinator.com/item?id=18382975

unionpivo 7y ago

You do realize its all software at one point right ? It just that instead of running on main cpu, it would run on SSD controller's microchip. And will probably be binary only, that people wont be able to inspect or fix. Samsung has good hardware, but not that great software. I'd rather have the opposite. SSD hardware exposing their internals, to os developers.