4 ms·

I don’t understand the comments saying that this obsoletes databases, or that this is a good substrate for relational databases. The interface is for random ac

I don’t understand the comments saying that this obsoletes databases, or that this is a good substrate for relational databases. The interface is for random access. Quite a few database optimizations depend on sequential access, I.e. accessing records in key order following a random access. This is why B-trees are so important. Sequential access in key order does not appear to be a possibility with this technology.

the_duke 7y ago
> The interface is for random access. I'm not sure where you are getting this idea. How keys are organized would be up to the device, and devices could support multiple schemes. Clustering keys in a sorted order and fast in-order iteration seems like an essential requirement. Just scanning over the specification [1], I see an iterator interface for key groups (6.4) and a setting for ordering (5.4.3) Edit: just to clarify, I agree that this would in no way obsolete databases. But software DBs could utilize key/value disks internally. [1] https://www.snia.org/sites/default/files/technical_work/KVSAPI/Key%20Value%20Storage%20API%20v1.0.pdf https://www.snia.org/sites/default/files/technical_work/KVSA...
- zokier 7y ago
  See slide 43 here https://www.snia.org/sites/default/files/SDC/2017/presentations/Object_ObjectDriveStorage/Ki_Yang_Seok_Key_Value_SSD_Explained_Concept_Device_System_and_Standard.pdf https://www.snia.org/sites/default/files/SDC/2017/presentati...
  - the_duke 7y ago
    Very interesting presentation. Slide 43 is for write performance though. RocksDB does a lot of clever things with writes, like delayed and batched flushing with a WAL for recovery. Would be interesting how that benchmark was written. Other slides with read benchmarks show very significant performance improvements. But comparing directly to a software DB seems inappropriate to me anyway. We can't expect (and definitely wouldn't want) a hard disk to offer something as complex as RocksDB/LLDB/etc. I would rather imagine that the software DBs start taking advantage of key/value disks internally.
- aledalgrande 7y ago
  Is that basically like having B-trees on the keys? I guess you would need also an API to retrieve multiple values at once, like `MGET` in Redis.
teraflop 7y ago
In addition to what the other commenter said: sequential access is considerably less critical for SSDs than it is for magnetic hard drives. It's still somewhat faster, but only by maybe 2x or so, compared to a factor of several hundred for a magnetic drive.
- mlyle 7y ago
  Yes, but... You might have many, many sequentially related records in one 4k block of an index retrieved from a SSD. Maybe 200. Then in turn you can retrieve those index blocks sequentially and get a performance improvement. In turn, when you're doing a merge or bitmap scan over the index, this can make a really big difference.
blaisio 7y ago
I agree. If all you can do is ask for a key, then you can't optimize the order in which you fetch them. That does matter, because if you fetch keys in the same block you can load the block once. Maybe the new optimizations the SSD can do with this level of abstraction will make up for the difference in performance. Or maybe they can try to organize keys by sorting or something like that.
endorphone 7y ago
Aggregations and the like require sequential access, but random access is very valuable for RDBMS where in most cases the vast majority of operations are random (write a record, pull a record, update an index, etc). Many of the systems in play have an extraordinary number of optimizations for sequential access -- assuming terrible random access -- but would still hugely benefit from something like this. However those who say it obsoletes anything beyond the most trivial of KV stores are way off the mark. It's a possible optimization for all sorts of database systems, certainly including RDBMS systems (many of which are layered over a KV store or sorts). There are disk systems for Oracle DBs that can run a subset of SQL right at the storage later, filtering by predicates, doing index searches, etc.

the_duke 7y ago

> The interface is for random access. I'm not sure where you are getting this idea. How keys are organized would be up to the device, and devices could support multiple schemes. Clustering keys in a sorted order and fast in-order iteration seems like an essential requirement. Just scanning over the specification [1], I see an iterator interface for key groups (6.4) and a setting for ordering (5.4.3) Edit: just to clarify, I agree that this would in no way obsolete databases. But software DBs could utilize key/value disks internally. [1] https://www.snia.org/sites/default/files/technical_work/KVSAPI/Key%20Value%20Storage%20API%20v1.0.pdf https://www.snia.org/sites/default/files/technical_work/KVSA...

zokier 7y ago
See slide 43 here https://www.snia.org/sites/default/files/SDC/2017/presentations/Object_ObjectDriveStorage/Ki_Yang_Seok_Key_Value_SSD_Explained_Concept_Device_System_and_Standard.pdf https://www.snia.org/sites/default/files/SDC/2017/presentati...
- the_duke 7y ago
  Very interesting presentation. Slide 43 is for write performance though. RocksDB does a lot of clever things with writes, like delayed and batched flushing with a WAL for recovery. Would be interesting how that benchmark was written. Other slides with read benchmarks show very significant performance improvements. But comparing directly to a software DB seems inappropriate to me anyway. We can't expect (and definitely wouldn't want) a hard disk to offer something as complex as RocksDB/LLDB/etc. I would rather imagine that the software DBs start taking advantage of key/value disks internally.
aledalgrande 7y ago
Is that basically like having B-trees on the keys? I guess you would need also an API to retrieve multiple values at once, like `MGET` in Redis.

teraflop 7y ago

In addition to what the other commenter said: sequential access is considerably less critical for SSDs than it is for magnetic hard drives. It's still somewhat faster, but only by maybe 2x or so, compared to a factor of several hundred for a magnetic drive.

mlyle 7y ago
Yes, but... You might have many, many sequentially related records in one 4k block of an index retrieved from a SSD. Maybe 200. Then in turn you can retrieve those index blocks sequentially and get a performance improvement. In turn, when you're doing a merge or bitmap scan over the index, this can make a really big difference.

blaisio 7y ago

I agree. If all you can do is ask for a key, then you can't optimize the order in which you fetch them. That does matter, because if you fetch keys in the same block you can load the block once. Maybe the new optimizations the SSD can do with this level of abstraction will make up for the difference in performance. Or maybe they can try to organize keys by sorting or something like that.

endorphone 7y ago

Aggregations and the like require sequential access, but random access is very valuable for RDBMS where in most cases the vast majority of operations are random (write a record, pull a record, update an index, etc). Many of the systems in play have an extraordinary number of optimizations for sequential access -- assuming terrible random access -- but would still hugely benefit from something like this. However those who say it obsoletes anything beyond the most trivial of KV stores are way off the mark. It's a possible optimization for all sorts of database systems, certainly including RDBMS systems (many of which are layered over a KV store or sorts). There are disk systems for Oracle DBs that can run a subset of SQL right at the storage later, filtering by predicates, doing index searches, etc.