6 ms·

Meanwhile, the rest of the CRUD world is still using local RDBMS in place of local key-value stores. I understand the point the abstract is trying to make, but

Meanwhile, the rest of the CRUD world is still using local RDBMS in place of local key-value stores. I understand the point the abstract is trying to make, but the title is sensationalized at best.

anonu 7y ago
I remember a headline on HN from a month or two ago. Something like "You're not Google". Turns out that a run of the mill RDBMS will fit 99% of the problems you're trying to solve. Edit: https://news.ycombinator.com/item?id=19576092 https://news.ycombinator.com/item?id=19576092
- pwodhouse 7y ago
  The authors advocate for run of the mill rdbms plus their sharing system.
- lmm 7y ago
  What makes an RDBMS "run of the mill" and a key-value store not? If anything key-value stores tend to be much simpler, both conceptually and in terms of implementation.
  - acdha 7y ago
    An RDBMS can handle considerably more tasks due to the power of the relational tools, and there’s a huge existing library of tools built around that model. That means that there are far more projects which can be implemented using only an RDBMS than only a KV store, and in the modern era both are easy cloud options with SSDs pushing caching past the level that most projects will ever need.
    - Justsignedup 7y ago
      There's even databases that are focused on increasing the capabilities of rdbms. Point is that sqlite could take people way further than they think. Postgres incredibly far. By the time you need a caching layer where key value stores shine best you have many options you can do and resources to do them.
      - snaky 7y ago
        > Postgres incredibly far Up to the millions of TPS actually. https://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/ https://akorotkov.github.io/blog/2016/05/09/scalability-towa...
  - jmull 7y ago
    I think you're misunderstanding what "run of the mill" means. It isn't an expression of simplicity. Anyway, look it up, and you'll have the answer to your question.
    - naasking 7y ago
      "Run of the mill" is not about simplicity, but refers to something common, average, commotidized.
  - nkozyra 7y ago
    I think the point is that RDBMS are considered stodgy and uncool despite the fact that they can be extremely sophisticated and powerful. Cheap, "faster," less-reliable stores became very big around the Mongo explosion - and don't get me wrong they can have a place, particularly Memcached and Redis which are great for ephemeral data - but you can do a lot of that with a RDBMS.
    - wlesieutre 7y ago
      I'm reminded of this https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
    - vidarh 7y ago
      There was also a point where "internet scale" was impossible to handle on regular RDBMS's without massive amounts of pain. But disk IO performance has gone up dramatically since, and capacities likewise. So a lot of caching that many of us had to do to scale systems a decade or two ago is now not necessary until/unless you get several magnitudes up in scale, and while the number of internet users has gone up, so has competition, and the proportion of services that gets successful to need scale up to several magnitudes beyond the old chokepoints at the time is tiny.
  - ben509 7y ago
    A SQL DBMS (run of the mill) gives you a well thought out data model (the SQL model, derived from the relational model) that implements your business logic in terms of a well designed, well understood data model. Conceptually, the relational model has relations and atoms. The algebra is built on the primitives conjoin, disjoin, project and rename. You can't get much simpler while being as expressive. The SQL model adds a good deal of complexity, but at least it's standardized and you can generally ignore the complexity you don't need. A KV store is conceptually simple if you stick to using it as a KV store. Once you try to implement any kind of schema, you have to build that from scratch. As you add business logic to it, you wind up inventing an ad hoc data model, which is likely to be conceptually more complex than either the SQL or relational model. You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you.
    - robodale 7y ago
      Somebody please hard-code this comment to the top, because it is spot on.
    - stcredzero 7y ago
      You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you. I think this idea needs to be expressed more clearly in CompSci and programming. What is the exact nature of the complexity explosion you are talking about here? Is there something analogous to the increase in complexity going from regular expressions to stack machines? What math are we talking about here? It seems like everyone just explains the relational algebra, then leaves it right there.
      - ben509 7y ago
        > It seems like everyone just explains the relational algebra, then leaves it right there. Fair point. > What is the exact nature of the complexity explosion you are talking about here? I'm thinking in terms of the entities of Occam's razor: "entities should not be multiplied unnecessarily." "Entities" is pretty abstract, we can't identify what complexity is directly, but what we can do is imagine, "what if we tried to build a mathematical model that captures a real life application?" If we did that, and had a mathematical description of a thing we wrote, we can formalize it by trying to reduce it to some minimal set of axioms. And then, your more complex rules are derived from those axioms, and if you get your math right those complex rules will be consistent. If you're very clever, you can make it reasonably intuitive. If you have something that's very complex, what you'd observe after modelling it is you have mostly axioms and very few rules are able to be derived from those axioms. That is, the rules are just the rules and there's no broader reason for them to be so, or deeper consistent patterns. And, maybe some of those rules wind up being contradictory, and they may lack orthogonality. The relational algebra, being an algebra, is a set of operatations that are closed over the universe of relations, so it's very nicely orthogonal and reduces to a small set of primitives. As relations can be visualized as "tables" they're relatively intuitive, and using techniques such as normalization you can also structure around potential anomalies that add unwanted complexity. You can also control where your complexity goes. An integrity constraint can enforce some rules about what may be in a relational variable, and if that's enforced, all code that wants to use data from that relation can simply assume that the data maintains that structure. Thus the complexity can be centralized in the system. Now, that's in the ideal world with a True Relational DBMS, we have to deal with SQL and vendor-specific SQL at that, so we have a rather complex underlying model. But it's typically Pretty Good. When we build such a model ad hoc, we go by our intuition and are constrained by the market. Our intutition leads us to use more complex structures, that is, structures that if they were expressed mathematically would use a larger set of axioms to describe them. Then, as we want those structures to interoperate, we are unwittingly merging two sets of axioms. Worse, because we're typicaly expressing it in application libraries, we wind up repeating code, so the complexity is spread all over the application and it especially multiplies if "conventions" are copy-pasta'd. Hopefully that explains the nature of the explosion of entities / complexity; let me know if I can expand on anything.
- draw_down 7y ago
  Yeah, well, some people are google. For example, the authors of this paper.
- tootie 7y ago
  Lots of dev opt for DBs like Redis or Mongo specifically because they're simpler and easier to develop for. Not for scaling. I think that can easily be a trap though because they are very easy for simple use cases, but get hairy really quickly when you start needing complex relationships.
  - beardbandit 7y ago
    You can use RDBMS for anything that requires complex relationships and map them to unique identifiers in the Mongo documents. In most cases though, Mongo fits use cases, we never try to use it as a wholesale RDBMS replacement.
    - tracker1 7y ago
      Agreed... Mongo also works well for denormalizing data structures into something that renders faster to the browser. A single key/document->json->get->parse-> render is much faster than a query across many joins and/or sub-queries. As an aside, if you're doing it anyway, it's also a good point to dump to .json.gz in S3 (or similar) as a secondary backup system.
drawkbox 7y ago
Fast key-value stores are really great for caching layers, as the main source of truth on data not as much. Whether these are local in-memory or external like Redis/memcached/mongo etc. Any good large scale system will have a caching layer for in memory or external cache store, even small/medium apps may use them for performance and ultimately cost when it comes to cloud, putting more on cpu/local than external databases. It is really what works for the system the best as with anything in engineering, sometimes needed, sometimes not, it depends. Memcached really started out as a caching layer, it is right there in the name. Once NoSQL started the caching initial focus was lost for a bit, but I think fast key-value stores are returning to mostly being cache layers, which work great with any system, including RDBMS backed of any size. Cloud key-value stores or local in-memory are almost required to compete on speed/performance even for marketing (search results) and even cloud costs of going to the database as well as taking weight off the database which is usually the hardest hit. Caching layers are useful in all systems/platforms, usually local in-memory first, then shared/external if costs/budget/performance needed. You can even do local caching on the instance, and a remote key-value store, both in front of the database for backup on performance/speed, though cache busting should be implemented from the start when data changes to prevent stale data.
liveoneggs 7y ago
the real irony is that RDBMS are all built on fast k:v stores :)
- HelloNurse 7y ago
  The real point is that you don't want to reimplement a poor, ad-hoc RDBMS inside your application.
- posix_me_less 7y ago
  What do you mean, a filesystem?
  - bjornagoransson 7y ago
    I think it's a reference to RocksDB, and maybe also InnoDB
  - vidarh 7y ago
    Pretty much all RDBMSs can be considered a set of key-value collections, and their storage backends often conceptually are pretty much that. Assuming a row-oriented storage engine, a simple approach is: * A main key/value collection consisting of primary key (key) mapping to a serialization of the row (value) * One key/value collection per index consisting of each index function applied to the row (key) mapping to the primary key (or some internal object id; the value) It's a bit of a simplification as an RDBMS needs to maintain a sort-order and allow e.g. range queries, and need interfaces that allow reading/writing lots of rows with low per-row overhead, and so it's not that they'd be a good match for just any key-value store.
- bottled_poe 7y ago
  I’m not sure what you mean? Aren’t most databases built on assumptions of continuous disk space mapped by indexed arrays? How does k:v fit into the picture?
  - marcosdumay 7y ago
    The most common high-level use of a DBMS is the pair "given this key, get me the record that is its value" and "store this record under a new key". It's not exactly equal to the usual k:v usage (because it uses meaningless keys), but it's very similar. That said, I don't think this is the usage scenario most DBMS are optimized for, because despite being common, it's a very fast case, so there is relatively little to gain. At the same time, this is the only usage of k:v storage, so you bet it's optimized for that.

Meanwhile, the rest of the CRUD world is still using local RDBMS in place of local key-value stores. I understand the point the abstract is trying to make, but the title is sensationalized at best.

anonu 7y ago

I remember a headline on HN from a month or two ago. Something like "You're not Google". Turns out that a run of the mill RDBMS will fit 99% of the problems you're trying to solve. Edit: https://news.ycombinator.com/item?id=19576092 https://news.ycombinator.com/item?id=19576092

pwodhouse 7y ago
The authors advocate for run of the mill rdbms plus their sharing system.
lmm 7y ago
What makes an RDBMS "run of the mill" and a key-value store not? If anything key-value stores tend to be much simpler, both conceptually and in terms of implementation.
- acdha 7y ago
  An RDBMS can handle considerably more tasks due to the power of the relational tools, and there’s a huge existing library of tools built around that model. That means that there are far more projects which can be implemented using only an RDBMS than only a KV store, and in the modern era both are easy cloud options with SSDs pushing caching past the level that most projects will ever need.
  - Justsignedup 7y ago
    There's even databases that are focused on increasing the capabilities of rdbms. Point is that sqlite could take people way further than they think. Postgres incredibly far. By the time you need a caching layer where key value stores shine best you have many options you can do and resources to do them.
    - snaky 7y ago
      > Postgres incredibly far Up to the millions of TPS actually. https://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/ https://akorotkov.github.io/blog/2016/05/09/scalability-towa...
- jmull 7y ago
  I think you're misunderstanding what "run of the mill" means. It isn't an expression of simplicity. Anyway, look it up, and you'll have the answer to your question.
  - naasking 7y ago
    "Run of the mill" is not about simplicity, but refers to something common, average, commotidized.
- nkozyra 7y ago
  I think the point is that RDBMS are considered stodgy and uncool despite the fact that they can be extremely sophisticated and powerful. Cheap, "faster," less-reliable stores became very big around the Mongo explosion - and don't get me wrong they can have a place, particularly Memcached and Redis which are great for ephemeral data - but you can do a lot of that with a RDBMS.
  - wlesieutre 7y ago
    I'm reminded of this https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
  - vidarh 7y ago
    There was also a point where "internet scale" was impossible to handle on regular RDBMS's without massive amounts of pain. But disk IO performance has gone up dramatically since, and capacities likewise. So a lot of caching that many of us had to do to scale systems a decade or two ago is now not necessary until/unless you get several magnitudes up in scale, and while the number of internet users has gone up, so has competition, and the proportion of services that gets successful to need scale up to several magnitudes beyond the old chokepoints at the time is tiny.
- ben509 7y ago
  A SQL DBMS (run of the mill) gives you a well thought out data model (the SQL model, derived from the relational model) that implements your business logic in terms of a well designed, well understood data model. Conceptually, the relational model has relations and atoms. The algebra is built on the primitives conjoin, disjoin, project and rename. You can't get much simpler while being as expressive. The SQL model adds a good deal of complexity, but at least it's standardized and you can generally ignore the complexity you don't need. A KV store is conceptually simple if you stick to using it as a KV store. Once you try to implement any kind of schema, you have to build that from scratch. As you add business logic to it, you wind up inventing an ad hoc data model, which is likely to be conceptually more complex than either the SQL or relational model. You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you.
  - robodale 7y ago
    Somebody please hard-code this comment to the top, because it is spot on.
  - stcredzero 7y ago
    You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you. I think this idea needs to be expressed more clearly in CompSci and programming. What is the exact nature of the complexity explosion you are talking about here? Is there something analogous to the increase in complexity going from regular expressions to stack machines? What math are we talking about here? It seems like everyone just explains the relational algebra, then leaves it right there.
    - ben509 7y ago
      > It seems like everyone just explains the relational algebra, then leaves it right there. Fair point. > What is the exact nature of the complexity explosion you are talking about here? I'm thinking in terms of the entities of Occam's razor: "entities should not be multiplied unnecessarily." "Entities" is pretty abstract, we can't identify what complexity is directly, but what we can do is imagine, "what if we tried to build a mathematical model that captures a real life application?" If we did that, and had a mathematical description of a thing we wrote, we can formalize it by trying to reduce it to some minimal set of axioms. And then, your more complex rules are derived from those axioms, and if you get your math right those complex rules will be consistent. If you're very clever, you can make it reasonably intuitive. If you have something that's very complex, what you'd observe after modelling it is you have mostly axioms and very few rules are able to be derived from those axioms. That is, the rules are just the rules and there's no broader reason for them to be so, or deeper consistent patterns. And, maybe some of those rules wind up being contradictory, and they may lack orthogonality. The relational algebra, being an algebra, is a set of operatations that are closed over the universe of relations, so it's very nicely orthogonal and reduces to a small set of primitives. As relations can be visualized as "tables" they're relatively intuitive, and using techniques such as normalization you can also structure around potential anomalies that add unwanted complexity. You can also control where your complexity goes. An integrity constraint can enforce some rules about what may be in a relational variable, and if that's enforced, all code that wants to use data from that relation can simply assume that the data maintains that structure. Thus the complexity can be centralized in the system. Now, that's in the ideal world with a True Relational DBMS, we have to deal with SQL and vendor-specific SQL at that, so we have a rather complex underlying model. But it's typically Pretty Good. When we build such a model ad hoc, we go by our intuition and are constrained by the market. Our intutition leads us to use more complex structures, that is, structures that if they were expressed mathematically would use a larger set of axioms to describe them. Then, as we want those structures to interoperate, we are unwittingly merging two sets of axioms. Worse, because we're typicaly expressing it in application libraries, we wind up repeating code, so the complexity is spread all over the application and it especially multiplies if "conventions" are copy-pasta'd. Hopefully that explains the nature of the explosion of entities / complexity; let me know if I can expand on anything.
draw_down 7y ago
Yeah, well, some people are google. For example, the authors of this paper.
tootie 7y ago
Lots of dev opt for DBs like Redis or Mongo specifically because they're simpler and easier to develop for. Not for scaling. I think that can easily be a trap though because they are very easy for simple use cases, but get hairy really quickly when you start needing complex relationships.
- beardbandit 7y ago
  You can use RDBMS for anything that requires complex relationships and map them to unique identifiers in the Mongo documents. In most cases though, Mongo fits use cases, we never try to use it as a wholesale RDBMS replacement.
  - tracker1 7y ago
    Agreed... Mongo also works well for denormalizing data structures into something that renders faster to the browser. A single key/document->json->get->parse-> render is much faster than a query across many joins and/or sub-queries. As an aside, if you're doing it anyway, it's also a good point to dump to .json.gz in S3 (or similar) as a secondary backup system.

drawkbox 7y ago

Fast key-value stores are really great for caching layers, as the main source of truth on data not as much. Whether these are local in-memory or external like Redis/memcached/mongo etc. Any good large scale system will have a caching layer for in memory or external cache store, even small/medium apps may use them for performance and ultimately cost when it comes to cloud, putting more on cpu/local than external databases. It is really what works for the system the best as with anything in engineering, sometimes needed, sometimes not, it depends. Memcached really started out as a caching layer, it is right there in the name. Once NoSQL started the caching initial focus was lost for a bit, but I think fast key-value stores are returning to mostly being cache layers, which work great with any system, including RDBMS backed of any size. Cloud key-value stores or local in-memory are almost required to compete on speed/performance even for marketing (search results) and even cloud costs of going to the database as well as taking weight off the database which is usually the hardest hit. Caching layers are useful in all systems/platforms, usually local in-memory first, then shared/external if costs/budget/performance needed. You can even do local caching on the instance, and a remote key-value store, both in front of the database for backup on performance/speed, though cache busting should be implemented from the start when data changes to prevent stale data.

liveoneggs 7y ago

the real irony is that RDBMS are all built on fast k:v stores :)

HelloNurse 7y ago
The real point is that you don't want to reimplement a poor, ad-hoc RDBMS inside your application.
posix_me_less 7y ago
What do you mean, a filesystem?
- bjornagoransson 7y ago
  I think it's a reference to RocksDB, and maybe also InnoDB
- vidarh 7y ago
  Pretty much all RDBMSs can be considered a set of key-value collections, and their storage backends often conceptually are pretty much that. Assuming a row-oriented storage engine, a simple approach is: * A main key/value collection consisting of primary key (key) mapping to a serialization of the row (value) * One key/value collection per index consisting of each index function applied to the row (key) mapping to the primary key (or some internal object id; the value) It's a bit of a simplification as an RDBMS needs to maintain a sort-order and allow e.g. range queries, and need interfaces that allow reading/writing lots of rows with low per-row overhead, and so it's not that they'd be a good match for just any key-value store.
bottled_poe 7y ago
I’m not sure what you mean? Aren’t most databases built on assumptions of continuous disk space mapped by indexed arrays? How does k:v fit into the picture?
- marcosdumay 7y ago
  The most common high-level use of a DBMS is the pair "given this key, get me the record that is its value" and "store this record under a new key". It's not exactly equal to the usual k:v usage (because it uses meaningless keys), but it's very similar. That said, I don't think this is the usage scenario most DBMS are optimized for, because despite being common, it's a very fast case, so there is relatively little to gain. At the same time, this is the only usage of k:v storage, so you bet it's optimized for that.