6 ms·

I've only ever used SQL and relational databases. What are the use cases of Key-value stores? What's the canonical example of where they are a clearly the right

by dcl 3y ago

I've only ever used SQL and relational databases. What are the use cases of Key-value stores? What's the canonical example of where they are a clearly the right solution?

huntertwo 3y ago
No schema validation so you get faster writes, generally allows for easier scaling
ok_dad 3y ago
Caching
jerf 3y ago
In a nutshell, you trade in the various guarantees relational databases provide, like transactions, relational integrity, and high levels of queryability for higher performance on what a key/value store can do, simplicity of API, and some things that can be easier for the system to provide precisely because it is offering fewer guarantees. Particularly distribution... SQL is hard to distribute precisely because of the guarantees it offers. When to use it? You must be sure you have a case where the additional guarantees relationship databases provide are not necessary, and you either need the simplicity of usage or deployment. Or in rare cases, speed, but whereas most people seem to act as if this the main reason, I consider it a relatively poor reason. A relational DB with its guarantees turned off (i.e., no relations, no transactions, tables with just a key and a value) perform fairly closely to a key/value store for a wide range of scaling needs. There is a top end where this matters, but fewer programmers need this than think they need this. Still, it is a valid concern, and if you don't need relational guarantees it may let you scale down the instance size. Most of the cases I see where key/value stores are a good choice relate more to the simplicity than performance. I love me my Postgres, but nothing compares to the simplicity of just tossing up a Memcache somewhere and solving my problem, if that's what my problem calls for. No schema, no migrations, an API so simple my local programming language may well simply integrate it with my native associative array syntax... if you are careful to use it only where you aren't going to need relational functionality the bang for the buck can be very nice.
- jmartrican 3y ago
  I'm a big fan of relational DBs. But I do still have a Redis instance running in my environment. I know that my relational DB can easily do what Redis is doing but there are two big reasons that I will disclose below for why I still use Redis. 1) The system that is reading the data from Redis is a front-end system. I have concerns with security. As a front-end system, it can be accessed from the Internet and potentially hacked. As such, I make sure this system does not have access to the DB. 2) Preserve DB connections, size (affects costs of storing backups and time to hydrate), CPU, and memory. I would really like to keep my DB as trim and spry as possible. I am using traditional relational DB and not distributed. So in an effort to keep my complexity in DB management down, I offload some work to Redis. I could have created a second relational DB that has low security requirements and no relational object mapping, but I didnt think of that till now (I will explore this further in the next few days). In my case the data that is stored in Redis is accessed a lot and consists of data from multiple tables, so it was a good pick for moving to Redis. Performance wise, I suspect relational DB could perform jus as fast, but again I want to offload traffic off of the DB to not have to grow it or go distributed. If I was a better DB admin, I could probably created views or other relational DB features. But I felt it was easy to just let my API backend code (which has access to data that is also not in DB or might need to be formatted or calculated) construct the final data object then store a copy in Redis, whenever the data is updated.
- zinodaur 3y ago
  Great writeup. I use a KV store for work, but only because the alternative for our data size (>1PB, >1 trillion keys), sharded sql, is totally awful. Distributed KV stores like foundationDB, dynamoDB actually do offer transactions , which is a huge win and I think their main selling point. I guess I'm like you - I really don't understand why someone who had another choice would opt in to a KV store (barring something like memcache, or something really high performance using an embedded KV store).
somethingAlex 3y ago
A concrete example would be a users shopping cart, as they build it. You don’t need the niceties of a fully ACID compliant DB, you need write performance, and extremely high availability. That was at least a chief use case spotlighted in the original Dynamo paper by Amazon that what the precursor to AWS’ DynamoDB paper. Not to say that couldn’t be done with Postgres but of course they were dealing with insane scale on Amazon Day.
- AdieuToLogic 3y ago
  > A concrete example would be a users shopping cart, as they build it. You don’t need the niceties of a fully ACID compliant DB, you need write performance, and extremely high availability. Using a key-value store for shopping carts can work for awhile, especially for the use-case you describe, but fails when system functionality grows beyond retrieving only by a cart ID. And when using a persistent store which does not provide ACID capabilities, the system will ultimately have to enforce at least atomicity and consistency via server logic.
  - makeitdouble 3y ago
    > when system functionality grows beyond retrieving only by a cart ID. Either you manage your system so it never ever uses any other key than a cart ID (services have been running for decades keeping the same unique ID, that's not some unreasonable thing). Or you migrate your data to match the completely wild new requirement, and taking costly steps to deal with a funfamental business change would be seen as reasonable in most orgs.
- ledauphin 3y ago
  it's a common misconception that modern non-relational stores (such as DynamoDB) aren't ACID compliant. DynamoDB offers ACID transactions, even across tables, as of several years ago. not that you're saying they don't, but some people might interpret your comment that way.
  - xwowsersx 3y ago
    That's right. FoundationDB and ArangoDB, among others, are also ACID-compliant, I believe.
sakopov 3y ago
At my old job we used DynamoDB in a microservice architecture and I always thought it was a perfect fit. Not sure about JunoDB, but DynamoDB is notoriously difficult to index for various querying patterns. This seemed like less of an issue for microservices because the models are generally very compact and easy to query.
AdieuToLogic 3y ago
> What are the use cases of Key-value stores? The ideal use-case is when there is one, and only one, property used for retrieval which is guaranteed unique by the system. Less ideal, but often very performant, is when retrieval always uses one property which may not be unique. Once retrieval requires anything other than a single predefined property, querying key-value persistent stores degrade into linear searches.
- BulgarianIdiot 3y ago
  Relational dbs are mostly keyval stores inside. An index is a keyval store projection of another keyval store (but say, keying it for another subset of the value). B-tree indices and hashmaps are ways to represent keyval stores for quick look-up (b-trees are convenient as they allow range look-ups, & are automatically sorted, while hashmaps aren't, but have lower overhead for lookup and storage). In essence everything is keyval. A sparse array is an ordered keyval store with integer keys (also technically everything is ordered, too, but some orders are stable, and useful, while others aren't). A dense array is an adjacency-optimized version of a sparse array where the key is implicit based on computable offset within a larger dense array, your address space. RAM address space is also variations on that theme. Raw disk storage. And file systems. Everything is. Maybe I spend too much time messing with storage, but I can't see it any other way at this point.
twic 3y ago
Don't think of key-value stores as an alternative to relational databases. Think of them as really big hashtables.
throwaway892238 3y ago
When you don't need a relational data model and you just need to store random key=value entries
NavinF 3y ago
Caching. It'd be pretty silly to use a relational database for something that trivially shards across servers, often doesn't have any consistency requirements, and only needs 3 columns (request, response, expiry time). I don't think it's even possible to use postgres as a traditional cache. Eg this trigger looks extremely slow and I would be unsurprised if a human holding ctrl+shift+r could single-handedly DoS a cache like this: https://stackoverflow.com/questions/26046816/is-there-a-way-to-set-an-expiry-time-after-which-a-data-entry-is-automaticall/26063344#26063344 https://stackoverflow.com/questions/26046816/is-there-a-way-...
- jmartrican 3y ago
  You can use Postgres for caching, and many other things. https://www.amazingcto.com/postgres-for-everything/ https://www.amazingcto.com/postgres-for-everything/
  - ilyt 3y ago
    Missing the point entirely
    - jmartrican 3y ago
      I was responding to this comment. > I don't think it's even possible to use postgres as a traditional cache. The link I posted shows that you can use Postgres as a traditional cache. Specifically, the SO link showing that Postgres cannot do caching (expiring old data) is explicitly addressed in the link I posted. One would have to got out of their way to miss the point. And miss it entirely.
      - ilyt 3y ago
        theoretically possible is far away from practical. That's my point. It would take quite a bit of work to do even subset of what most other solutions come out of the box, at most likely much lower performance (the reason to use cache in first place). You can put a fucking text files on the disk as cache and `find -atime X -delete` to clear it, doesn't mean you should use it.
firecall 3y ago
They have a list of Common Use Cases in the article! :-)
dikei 3y ago
Key-Value stores are the OG of datastore: most SQL and relational databases are built upon Key-Value stores

by dcl 3y ago

I've only ever used SQL and relational databases. What are the use cases of Key-value stores? What's the canonical example of where they are a clearly the right solution?

huntertwo 3y ago

No schema validation so you get faster writes, generally allows for easier scaling

ok_dad 3y ago

Caching

jerf 3y ago

In a nutshell, you trade in the various guarantees relational databases provide, like transactions, relational integrity, and high levels of queryability for higher performance on what a key/value store can do, simplicity of API, and some things that can be easier for the system to provide precisely because it is offering fewer guarantees. Particularly distribution... SQL is hard to distribute precisely because of the guarantees it offers. When to use it? You must be sure you have a case where the additional guarantees relationship databases provide are not necessary, and you either need the simplicity of usage or deployment. Or in rare cases, speed, but whereas most people seem to act as if this the main reason, I consider it a relatively poor reason. A relational DB with its guarantees turned off (i.e., no relations, no transactions, tables with just a key and a value) perform fairly closely to a key/value store for a wide range of scaling needs. There is a top end where this matters, but fewer programmers need this than think they need this. Still, it is a valid concern, and if you don't need relational guarantees it may let you scale down the instance size. Most of the cases I see where key/value stores are a good choice relate more to the simplicity than performance. I love me my Postgres, but nothing compares to the simplicity of just tossing up a Memcache somewhere and solving my problem, if that's what my problem calls for. No schema, no migrations, an API so simple my local programming language may well simply integrate it with my native associative array syntax... if you are careful to use it only where you aren't going to need relational functionality the bang for the buck can be very nice.

jmartrican 3y ago
I'm a big fan of relational DBs. But I do still have a Redis instance running in my environment. I know that my relational DB can easily do what Redis is doing but there are two big reasons that I will disclose below for why I still use Redis. 1) The system that is reading the data from Redis is a front-end system. I have concerns with security. As a front-end system, it can be accessed from the Internet and potentially hacked. As such, I make sure this system does not have access to the DB. 2) Preserve DB connections, size (affects costs of storing backups and time to hydrate), CPU, and memory. I would really like to keep my DB as trim and spry as possible. I am using traditional relational DB and not distributed. So in an effort to keep my complexity in DB management down, I offload some work to Redis. I could have created a second relational DB that has low security requirements and no relational object mapping, but I didnt think of that till now (I will explore this further in the next few days). In my case the data that is stored in Redis is accessed a lot and consists of data from multiple tables, so it was a good pick for moving to Redis. Performance wise, I suspect relational DB could perform jus as fast, but again I want to offload traffic off of the DB to not have to grow it or go distributed. If I was a better DB admin, I could probably created views or other relational DB features. But I felt it was easy to just let my API backend code (which has access to data that is also not in DB or might need to be formatted or calculated) construct the final data object then store a copy in Redis, whenever the data is updated.
zinodaur 3y ago
Great writeup. I use a KV store for work, but only because the alternative for our data size (>1PB, >1 trillion keys), sharded sql, is totally awful. Distributed KV stores like foundationDB, dynamoDB actually do offer transactions , which is a huge win and I think their main selling point. I guess I'm like you - I really don't understand why someone who had another choice would opt in to a KV store (barring something like memcache, or something really high performance using an embedded KV store).

somethingAlex 3y ago

A concrete example would be a users shopping cart, as they build it. You don’t need the niceties of a fully ACID compliant DB, you need write performance, and extremely high availability. That was at least a chief use case spotlighted in the original Dynamo paper by Amazon that what the precursor to AWS’ DynamoDB paper. Not to say that couldn’t be done with Postgres but of course they were dealing with insane scale on Amazon Day.

AdieuToLogic 3y ago
> A concrete example would be a users shopping cart, as they build it. You don’t need the niceties of a fully ACID compliant DB, you need write performance, and extremely high availability. Using a key-value store for shopping carts can work for awhile, especially for the use-case you describe, but fails when system functionality grows beyond retrieving only by a cart ID. And when using a persistent store which does not provide ACID capabilities, the system will ultimately have to enforce at least atomicity and consistency via server logic.
- makeitdouble 3y ago
  > when system functionality grows beyond retrieving only by a cart ID. Either you manage your system so it never ever uses any other key than a cart ID (services have been running for decades keeping the same unique ID, that's not some unreasonable thing). Or you migrate your data to match the completely wild new requirement, and taking costly steps to deal with a funfamental business change would be seen as reasonable in most orgs.
ledauphin 3y ago
it's a common misconception that modern non-relational stores (such as DynamoDB) aren't ACID compliant. DynamoDB offers ACID transactions, even across tables, as of several years ago. not that you're saying they don't, but some people might interpret your comment that way.
- xwowsersx 3y ago
  That's right. FoundationDB and ArangoDB, among others, are also ACID-compliant, I believe.

sakopov 3y ago

At my old job we used DynamoDB in a microservice architecture and I always thought it was a perfect fit. Not sure about JunoDB, but DynamoDB is notoriously difficult to index for various querying patterns. This seemed like less of an issue for microservices because the models are generally very compact and easy to query.

AdieuToLogic 3y ago

> What are the use cases of Key-value stores? The ideal use-case is when there is one, and only one, property used for retrieval which is guaranteed unique by the system. Less ideal, but often very performant, is when retrieval always uses one property which may not be unique. Once retrieval requires anything other than a single predefined property, querying key-value persistent stores degrade into linear searches.

BulgarianIdiot 3y ago
Relational dbs are mostly keyval stores inside. An index is a keyval store projection of another keyval store (but say, keying it for another subset of the value). B-tree indices and hashmaps are ways to represent keyval stores for quick look-up (b-trees are convenient as they allow range look-ups, & are automatically sorted, while hashmaps aren't, but have lower overhead for lookup and storage). In essence everything is keyval. A sparse array is an ordered keyval store with integer keys (also technically everything is ordered, too, but some orders are stable, and useful, while others aren't). A dense array is an adjacency-optimized version of a sparse array where the key is implicit based on computable offset within a larger dense array, your address space. RAM address space is also variations on that theme. Raw disk storage. And file systems. Everything is. Maybe I spend too much time messing with storage, but I can't see it any other way at this point.

twic 3y ago

Don't think of key-value stores as an alternative to relational databases. Think of them as really big hashtables.

throwaway892238 3y ago

When you don't need a relational data model and you just need to store random key=value entries

NavinF 3y ago

Caching. It'd be pretty silly to use a relational database for something that trivially shards across servers, often doesn't have any consistency requirements, and only needs 3 columns (request, response, expiry time). I don't think it's even possible to use postgres as a traditional cache. Eg this trigger looks extremely slow and I would be unsurprised if a human holding ctrl+shift+r could single-handedly DoS a cache like this: https://stackoverflow.com/questions/26046816/is-there-a-way-to-set-an-expiry-time-after-which-a-data-entry-is-automaticall/26063344#26063344 https://stackoverflow.com/questions/26046816/is-there-a-way-...

jmartrican 3y ago
You can use Postgres for caching, and many other things. https://www.amazingcto.com/postgres-for-everything/ https://www.amazingcto.com/postgres-for-everything/
- ilyt 3y ago
  Missing the point entirely
  - jmartrican 3y ago
    I was responding to this comment. > I don't think it's even possible to use postgres as a traditional cache. The link I posted shows that you can use Postgres as a traditional cache. Specifically, the SO link showing that Postgres cannot do caching (expiring old data) is explicitly addressed in the link I posted. One would have to got out of their way to miss the point. And miss it entirely.
    - ilyt 3y ago
      theoretically possible is far away from practical. That's my point. It would take quite a bit of work to do even subset of what most other solutions come out of the box, at most likely much lower performance (the reason to use cache in first place). You can put a fucking text files on the disk as cache and `find -atime X -delete` to clear it, doesn't mean you should use it.

firecall 3y ago

They have a list of Common Use Cases in the article! :-)

dikei 3y ago

Key-Value stores are the OG of datastore: most SQL and relational databases are built upon Key-Value stores