5 ms·

Can someone tell me what the use case for this would be?

by chadash 7y ago

ricardobeat 7y ago
You can see for yourself in the third paragraph. Streamlines key-value storage and might replace database backends.
- mruts 7y ago
  Why is this worth paying a premium for, though? Maybe a SQL SSD could get some adoption, but a simple KV store? How would that ever be a bottleneck (they are relatively easy to make blazing fast, unlike SQL) in your application? How many people/corporations actually want something like this?
  - tantalor 7y ago
    > How would that ever be a bottleneck Disk I/O is always the bottleneck.
    - rosser 7y ago
      Disk IO is far from the only bottleneck, or necessarily even the most common one. Source: My day job has been as a DBA for ~15 years.
    - ralusek 7y ago
      Usually* In web apps
paggle 7y ago
The hash table for mapping keys to values will be implemented in hardware on the device, rather than needing to come off the SSD to the CPU.
fyfy18 7y ago
Now your SSD can be web scale too!
Nican 7y ago
Quite a few databases lately are using RocksDB as their backend. Really, relational databases are just a key-value store with a well-optimized query planner on top of it. See: -CockroachDB (a spinoff of Google's Spanner) -MySQL's MyRocks (Facebook) -YugaByte (Postgres compatible sharded SQL) -Ceph (Opensource alternative to Amazon S3) I am guessing this is looking into improving the performance even further.
stickac 7y ago
Any application using LevelDB/RocksDB as their backend, e.g. Bitcoin.
rodgerd 7y ago
It's removing a layer of complexity from your storage: just as the article explains (does anyone read the articles? Anyone?) there's currently a lot of overhead in SSDs mapping their underlying model into pretending to be hard drives with 512 byte or 4K sectors, just like 40 year old spinning platter. Your filesystem is then layering a bunch of work on top of that to map the things you care about - files - into a bunch of fragments and metadata into those 4K chunks. You would gain the ability to do things like: 1. Throw away all the spinning disk emulation code in the SSD. 2. Align the FS level primitives with the storage: if you've set your RAID chunks to 64K per array member (for example), store a 64K object in one write, not break it into 16 x 4 K blocks. If your ZFS filesystem is set to 1 MB records, write 1 MB objects to disk, not many 4 K chunks. 3. Variable sized objects mean your filesystem could simply dispatch whole files as objects: if the FS knows your photo is a 20 MB file and your source code file is 1K, it no longer has to break the photo into many blocks, or waste a whole 4K block on a 1K file, it writes a 20 MB object and a 1K object. 4. Applications could access the storage even more directly where it makes sense: Postgres, for example, stores large records via the toast mechanism, where very large column in a row will be stored as a separately to the rest of the table (so as not to blow out the table files). You could extend that special case to simply address the storage directly, and not bother with filesystem overhead at all.
bob1029 7y ago
I have been working on a hobby database that uses a key-value store as the backend. Currently, it is structured as a binary tree such that an append-only log is possible (ideal for handling updates on a traditional SSD). This is a very complex and time-consuming part of the implementation. With this KV drive, I could potentially 'hardware accelerate' my database backend and all of the complexity of log-structured merge trees falls away. It would just be a handful of calls into a KV SSD library that handles the magic of getting a KV pair to disk. All I would need to worry about are the higher-order primitives I can build on top of a KV store. Using hashing schemes and clever metadata structures, you can encode virtually unlimited amounts of information into a 256 bit key. One simple flat store of KV items can contain all of your indexes, objects, scripts, jobs, settings, etc. Additionally, using hashed metadata keys lends itself well to sharing the store across unlimited drives and nodes. 256 bits comprises an unimaginably large key space, and SHA512 is actually even faster to execute on most x86 64-bit hardware if you can eat the 2x overhead on keys (which can be safely assumed to be inexhaustible through the heat death of the universe at this point).

Can someone tell me what the use case for this would be?

ricardobeat 7y ago

You can see for yourself in the third paragraph. Streamlines key-value storage and might replace database backends.

mruts 7y ago
Why is this worth paying a premium for, though? Maybe a SQL SSD could get some adoption, but a simple KV store? How would that ever be a bottleneck (they are relatively easy to make blazing fast, unlike SQL) in your application? How many people/corporations actually want something like this?
- tantalor 7y ago
  > How would that ever be a bottleneck Disk I/O is always the bottleneck.
  - rosser 7y ago
    Disk IO is far from the only bottleneck, or necessarily even the most common one. Source: My day job has been as a DBA for ~15 years.
  - ralusek 7y ago
    Usually* In web apps

paggle 7y ago

The hash table for mapping keys to values will be implemented in hardware on the device, rather than needing to come off the SSD to the CPU.

fyfy18 7y ago

Now your SSD can be web scale too!

Nican 7y ago

Quite a few databases lately are using RocksDB as their backend. Really, relational databases are just a key-value store with a well-optimized query planner on top of it. See: -CockroachDB (a spinoff of Google's Spanner) -MySQL's MyRocks (Facebook) -YugaByte (Postgres compatible sharded SQL) -Ceph (Opensource alternative to Amazon S3) I am guessing this is looking into improving the performance even further.

stickac 7y ago

Any application using LevelDB/RocksDB as their backend, e.g. Bitcoin.

rodgerd 7y ago

It's removing a layer of complexity from your storage: just as the article explains (does anyone read the articles? Anyone?) there's currently a lot of overhead in SSDs mapping their underlying model into pretending to be hard drives with 512 byte or 4K sectors, just like 40 year old spinning platter. Your filesystem is then layering a bunch of work on top of that to map the things you care about - files - into a bunch of fragments and metadata into those 4K chunks. You would gain the ability to do things like: 1. Throw away all the spinning disk emulation code in the SSD. 2. Align the FS level primitives with the storage: if you've set your RAID chunks to 64K per array member (for example), store a 64K object in one write, not break it into 16 x 4 K blocks. If your ZFS filesystem is set to 1 MB records, write 1 MB objects to disk, not many 4 K chunks. 3. Variable sized objects mean your filesystem could simply dispatch whole files as objects: if the FS knows your photo is a 20 MB file and your source code file is 1K, it no longer has to break the photo into many blocks, or waste a whole 4K block on a 1K file, it writes a 20 MB object and a 1K object. 4. Applications could access the storage even more directly where it makes sense: Postgres, for example, stores large records via the toast mechanism, where very large column in a row will be stored as a separately to the rest of the table (so as not to blow out the table files). You could extend that special case to simply address the storage directly, and not bother with filesystem overhead at all.

bob1029 7y ago

I have been working on a hobby database that uses a key-value store as the backend. Currently, it is structured as a binary tree such that an append-only log is possible (ideal for handling updates on a traditional SSD). This is a very complex and time-consuming part of the implementation. With this KV drive, I could potentially 'hardware accelerate' my database backend and all of the complexity of log-structured merge trees falls away. It would just be a handful of calls into a KV SSD library that handles the magic of getting a KV pair to disk. All I would need to worry about are the higher-order primitives I can build on top of a KV store. Using hashing schemes and clever metadata structures, you can encode virtually unlimited amounts of information into a 256 bit key. One simple flat store of KV items can contain all of your indexes, objects, scripts, jobs, settings, etc. Additionally, using hashed metadata keys lends itself well to sharing the store across unlimited drives and nodes. 256 bits comprises an unimaginably large key space, and SHA512 is actually even faster to execute on most x86 64-bit hardware if you can eat the 2x overhead on keys (which can be safely assumed to be inexhaustible through the heat death of the universe at this point).