37 ms·

I remember a headline on HN from a month or two ago. Something like "You're not Google". Turns out that a run of the mill RDBMS will fit 99% of the problems yo

by anonu 7y ago

I remember a headline on HN from a month or two ago. Something like "You're not Google". Turns out that a run of the mill RDBMS will fit 99% of the problems you're trying to solve. Edit: https://news.ycombinator.com/item?id=19576092 https://news.ycombinator.com/item?id=19576092

pwodhouse 7y ago
The authors advocate for run of the mill rdbms plus their sharing system.
lmm 7y ago
What makes an RDBMS "run of the mill" and a key-value store not? If anything key-value stores tend to be much simpler, both conceptually and in terms of implementation.
- acdha 7y ago
  An RDBMS can handle considerably more tasks due to the power of the relational tools, and there’s a huge existing library of tools built around that model. That means that there are far more projects which can be implemented using only an RDBMS than only a KV store, and in the modern era both are easy cloud options with SSDs pushing caching past the level that most projects will ever need.
  - Justsignedup 7y ago
    There's even databases that are focused on increasing the capabilities of rdbms. Point is that sqlite could take people way further than they think. Postgres incredibly far. By the time you need a caching layer where key value stores shine best you have many options you can do and resources to do them.
    - snaky 7y ago
      > Postgres incredibly far Up to the millions of TPS actually. https://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/ https://akorotkov.github.io/blog/2016/05/09/scalability-towa...
- jmull 7y ago
  I think you're misunderstanding what "run of the mill" means. It isn't an expression of simplicity. Anyway, look it up, and you'll have the answer to your question.
  - naasking 7y ago
    "Run of the mill" is not about simplicity, but refers to something common, average, commotidized.
- nkozyra 7y ago
  I think the point is that RDBMS are considered stodgy and uncool despite the fact that they can be extremely sophisticated and powerful. Cheap, "faster," less-reliable stores became very big around the Mongo explosion - and don't get me wrong they can have a place, particularly Memcached and Redis which are great for ephemeral data - but you can do a lot of that with a RDBMS.
  - wlesieutre 7y ago
    I'm reminded of this https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
  - vidarh 7y ago
    There was also a point where "internet scale" was impossible to handle on regular RDBMS's without massive amounts of pain. But disk IO performance has gone up dramatically since, and capacities likewise. So a lot of caching that many of us had to do to scale systems a decade or two ago is now not necessary until/unless you get several magnitudes up in scale, and while the number of internet users has gone up, so has competition, and the proportion of services that gets successful to need scale up to several magnitudes beyond the old chokepoints at the time is tiny.
- ben509 7y ago
  A SQL DBMS (run of the mill) gives you a well thought out data model (the SQL model, derived from the relational model) that implements your business logic in terms of a well designed, well understood data model. Conceptually, the relational model has relations and atoms. The algebra is built on the primitives conjoin, disjoin, project and rename. You can't get much simpler while being as expressive. The SQL model adds a good deal of complexity, but at least it's standardized and you can generally ignore the complexity you don't need. A KV store is conceptually simple if you stick to using it as a KV store. Once you try to implement any kind of schema, you have to build that from scratch. As you add business logic to it, you wind up inventing an ad hoc data model, which is likely to be conceptually more complex than either the SQL or relational model. You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you.
  - robodale 7y ago
    Somebody please hard-code this comment to the top, because it is spot on.
  - stcredzero 7y ago
    You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you. I think this idea needs to be expressed more clearly in CompSci and programming. What is the exact nature of the complexity explosion you are talking about here? Is there something analogous to the increase in complexity going from regular expressions to stack machines? What math are we talking about here? It seems like everyone just explains the relational algebra, then leaves it right there.
    - ben509 7y ago
      > It seems like everyone just explains the relational algebra, then leaves it right there. Fair point. > What is the exact nature of the complexity explosion you are talking about here? I'm thinking in terms of the entities of Occam's razor: "entities should not be multiplied unnecessarily." "Entities" is pretty abstract, we can't identify what complexity is directly, but what we can do is imagine, "what if we tried to build a mathematical model that captures a real life application?" If we did that, and had a mathematical description of a thing we wrote, we can formalize it by trying to reduce it to some minimal set of axioms. And then, your more complex rules are derived from those axioms, and if you get your math right those complex rules will be consistent. If you're very clever, you can make it reasonably intuitive. If you have something that's very complex, what you'd observe after modelling it is you have mostly axioms and very few rules are able to be derived from those axioms. That is, the rules are just the rules and there's no broader reason for them to be so, or deeper consistent patterns. And, maybe some of those rules wind up being contradictory, and they may lack orthogonality. The relational algebra, being an algebra, is a set of operatations that are closed over the universe of relations, so it's very nicely orthogonal and reduces to a small set of primitives. As relations can be visualized as "tables" they're relatively intuitive, and using techniques such as normalization you can also structure around potential anomalies that add unwanted complexity. You can also control where your complexity goes. An integrity constraint can enforce some rules about what may be in a relational variable, and if that's enforced, all code that wants to use data from that relation can simply assume that the data maintains that structure. Thus the complexity can be centralized in the system. Now, that's in the ideal world with a True Relational DBMS, we have to deal with SQL and vendor-specific SQL at that, so we have a rather complex underlying model. But it's typically Pretty Good. When we build such a model ad hoc, we go by our intuition and are constrained by the market. Our intutition leads us to use more complex structures, that is, structures that if they were expressed mathematically would use a larger set of axioms to describe them. Then, as we want those structures to interoperate, we are unwittingly merging two sets of axioms. Worse, because we're typicaly expressing it in application libraries, we wind up repeating code, so the complexity is spread all over the application and it especially multiplies if "conventions" are copy-pasta'd. Hopefully that explains the nature of the explosion of entities / complexity; let me know if I can expand on anything.
draw_down 7y ago
Yeah, well, some people are google. For example, the authors of this paper.
tootie 7y ago
Lots of dev opt for DBs like Redis or Mongo specifically because they're simpler and easier to develop for. Not for scaling. I think that can easily be a trap though because they are very easy for simple use cases, but get hairy really quickly when you start needing complex relationships.
- beardbandit 7y ago
  You can use RDBMS for anything that requires complex relationships and map them to unique identifiers in the Mongo documents. In most cases though, Mongo fits use cases, we never try to use it as a wholesale RDBMS replacement.
  - tracker1 7y ago
    Agreed... Mongo also works well for denormalizing data structures into something that renders faster to the browser. A single key/document->json->get->parse-> render is much faster than a query across many joins and/or sub-queries. As an aside, if you're doing it anyway, it's also a good point to dump to .json.gz in S3 (or similar) as a secondary backup system.

by anonu 7y ago

pwodhouse 7y ago

The authors advocate for run of the mill rdbms plus their sharing system.

lmm 7y ago

What makes an RDBMS "run of the mill" and a key-value store not? If anything key-value stores tend to be much simpler, both conceptually and in terms of implementation.

acdha 7y ago
An RDBMS can handle considerably more tasks due to the power of the relational tools, and there’s a huge existing library of tools built around that model. That means that there are far more projects which can be implemented using only an RDBMS than only a KV store, and in the modern era both are easy cloud options with SSDs pushing caching past the level that most projects will ever need.
- Justsignedup 7y ago
  There's even databases that are focused on increasing the capabilities of rdbms. Point is that sqlite could take people way further than they think. Postgres incredibly far. By the time you need a caching layer where key value stores shine best you have many options you can do and resources to do them.
  - snaky 7y ago
    > Postgres incredibly far Up to the millions of TPS actually. https://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/ https://akorotkov.github.io/blog/2016/05/09/scalability-towa...
jmull 7y ago
I think you're misunderstanding what "run of the mill" means. It isn't an expression of simplicity. Anyway, look it up, and you'll have the answer to your question.
- naasking 7y ago
  "Run of the mill" is not about simplicity, but refers to something common, average, commotidized.
nkozyra 7y ago
I think the point is that RDBMS are considered stodgy and uncool despite the fact that they can be extremely sophisticated and powerful. Cheap, "faster," less-reliable stores became very big around the Mongo explosion - and don't get me wrong they can have a place, particularly Memcached and Redis which are great for ephemeral data - but you can do a lot of that with a RDBMS.
- wlesieutre 7y ago
  I'm reminded of this https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- vidarh 7y ago
  There was also a point where "internet scale" was impossible to handle on regular RDBMS's without massive amounts of pain. But disk IO performance has gone up dramatically since, and capacities likewise. So a lot of caching that many of us had to do to scale systems a decade or two ago is now not necessary until/unless you get several magnitudes up in scale, and while the number of internet users has gone up, so has competition, and the proportion of services that gets successful to need scale up to several magnitudes beyond the old chokepoints at the time is tiny.
ben509 7y ago
A SQL DBMS (run of the mill) gives you a well thought out data model (the SQL model, derived from the relational model) that implements your business logic in terms of a well designed, well understood data model. Conceptually, the relational model has relations and atoms. The algebra is built on the primitives conjoin, disjoin, project and rename. You can't get much simpler while being as expressive. The SQL model adds a good deal of complexity, but at least it's standardized and you can generally ignore the complexity you don't need. A KV store is conceptually simple if you stick to using it as a KV store. Once you try to implement any kind of schema, you have to build that from scratch. As you add business logic to it, you wind up inventing an ad hoc data model, which is likely to be conceptually more complex than either the SQL or relational model. You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you.
- robodale 7y ago
  Somebody please hard-code this comment to the top, because it is spot on.
- stcredzero 7y ago
  You may not be aware of the complexity of your ad hoc model, but math will inevitably remind you. I think this idea needs to be expressed more clearly in CompSci and programming. What is the exact nature of the complexity explosion you are talking about here? Is there something analogous to the increase in complexity going from regular expressions to stack machines? What math are we talking about here? It seems like everyone just explains the relational algebra, then leaves it right there.
  - ben509 7y ago
    > It seems like everyone just explains the relational algebra, then leaves it right there. Fair point. > What is the exact nature of the complexity explosion you are talking about here? I'm thinking in terms of the entities of Occam's razor: "entities should not be multiplied unnecessarily." "Entities" is pretty abstract, we can't identify what complexity is directly, but what we can do is imagine, "what if we tried to build a mathematical model that captures a real life application?" If we did that, and had a mathematical description of a thing we wrote, we can formalize it by trying to reduce it to some minimal set of axioms. And then, your more complex rules are derived from those axioms, and if you get your math right those complex rules will be consistent. If you're very clever, you can make it reasonably intuitive. If you have something that's very complex, what you'd observe after modelling it is you have mostly axioms and very few rules are able to be derived from those axioms. That is, the rules are just the rules and there's no broader reason for them to be so, or deeper consistent patterns. And, maybe some of those rules wind up being contradictory, and they may lack orthogonality. The relational algebra, being an algebra, is a set of operatations that are closed over the universe of relations, so it's very nicely orthogonal and reduces to a small set of primitives. As relations can be visualized as "tables" they're relatively intuitive, and using techniques such as normalization you can also structure around potential anomalies that add unwanted complexity. You can also control where your complexity goes. An integrity constraint can enforce some rules about what may be in a relational variable, and if that's enforced, all code that wants to use data from that relation can simply assume that the data maintains that structure. Thus the complexity can be centralized in the system. Now, that's in the ideal world with a True Relational DBMS, we have to deal with SQL and vendor-specific SQL at that, so we have a rather complex underlying model. But it's typically Pretty Good. When we build such a model ad hoc, we go by our intuition and are constrained by the market. Our intutition leads us to use more complex structures, that is, structures that if they were expressed mathematically would use a larger set of axioms to describe them. Then, as we want those structures to interoperate, we are unwittingly merging two sets of axioms. Worse, because we're typicaly expressing it in application libraries, we wind up repeating code, so the complexity is spread all over the application and it especially multiplies if "conventions" are copy-pasta'd. Hopefully that explains the nature of the explosion of entities / complexity; let me know if I can expand on anything.

draw_down 7y ago

Yeah, well, some people are google. For example, the authors of this paper.

tootie 7y ago

Lots of dev opt for DBs like Redis or Mongo specifically because they're simpler and easier to develop for. Not for scaling. I think that can easily be a trap though because they are very easy for simple use cases, but get hairy really quickly when you start needing complex relationships.

beardbandit 7y ago
You can use RDBMS for anything that requires complex relationships and map them to unique identifiers in the Mongo documents. In most cases though, Mongo fits use cases, we never try to use it as a wholesale RDBMS replacement.
- tracker1 7y ago
  Agreed... Mongo also works well for denormalizing data structures into something that renders faster to the browser. A single key/document->json->get->parse-> render is much faster than a query across many joins and/or sub-queries. As an aside, if you're doing it anyway, it's also a good point to dump to .json.gz in S3 (or similar) as a secondary backup system.