7 ms·

Document DBs are like blockchain projects - overhyped and worse than existing solutions for nearly every use case. Why do ostensibly smart engineers keep fallin

Document DBs are like blockchain projects - overhyped and worse than existing solutions for nearly every use case. Why do ostensibly smart engineers keep falling for this stuff?

wongarsu 8y ago
> Why do ostensibly smart engineers keep falling for this stuff? Review driven design is likely a large contributer. If you know you will be looking for a new job in 2 years, you better get some experience in $latestTrend instead of $provenTechnology.
wwweston 8y ago
For one thing, a good chunk of what happens in the industry isn't particularly well described by the term "engineering," which in my mind describes someone who is well-trained and experience-versed in relevant scientific models for the application domain and the dynamics of relevant "materials" and their interaction that can be composed into a solution. It might be the term "craft" better describes a lot of software development activity. For another, it may be everyone is susceptible to the influence of trends. Even engineers. And the more complex the details of a subdomain of software engineering are, the bigger the tradeoff between becoming someone who can make a true engineering assessment in that niche and developing expertise elsewhere...
- ownagefool 8y ago
  It's also largely a fact that we're paid better for following hypes, at least in the UK. Thus a smart "engineer" who also cares about their bank balance pretty much has to use everything new to stay relevant. Further, most of the use cases are fairly shallow, so it doesn't actually matter that much, and you'll be paid more to paper over the cracks. We need to change the market, but it doesn't like the strong and stable engineers are getting lead roles to pay more strong and stable engineers to build boring tech, despite that actually being better for most businesses.
zzzcpan 8y ago
They are not overhyped, definitely not as much as old school RDBMS stuff. It's just most engineers can't make good database choices no matter what database they choose. Or more generally they can't make good infrastructure choices, as those are out of their competence and are mostly about things like operations and distributed systems, that take a long time and a lot of experience to get to a level of good decisions.
- michelpp 8y ago
  You cannot be old-school and hype. That's the reason why it is old school. And the people who can't make good database design choices are exactly the kind of people who should be using SQL. Postgres knows how to optimize and plan queries efficiently based on the actual distributions of values in your dataset. These poor choosers should be doing that... by hand? https://www.postgresql.org/docs/11/planner-optimizer.html https://www.postgresql.org/docs/11/planner-optimizer.html
  - zzzcpan 8y ago
    This is what I'm talking about. You think that some feature that makes performance unpredictable is that important. While it's the last thing you should care about.
    - sparkie 8y ago
      What people should care about is the integrity and consistency of the data they're storing. RDBMS do an extremely good job of generalising this problem with reasonable performance, which suits the majority of real world problems. The ideas behind it are understood by most of the industry. It should be the default choice (or plain filesystem storage), unless you have a specific requirement for something different. In the latter case, this should be people who are well informed about the database choices they're making.
    - astine 8y ago
      The only way for query performance to be truly predictable is for your data to be static. If that's not your situation then a query optimizer is incredibly helpful because it means that you don't have to rewrite your queries every time you add an index.
hnbroseph 8y ago
perhaps they don't agree with your thesis.
bdcravens 8y ago
Probably for the same reason people have a build process to load in 100s of KBs of Javascript in order to render an <h1> tag, and the compile it all down to HTML and declare static sites are the wave of the future.
jsd1982 8y ago
Sometimes the choice isn't in the hands of the person most qualified to make that decision.
sandstrom 8y ago
Well, we're kind of comparing MongoDB of ~2011 (when The Guardian started using them) with Postgres of today. One major change is that in 2010 you couldn't always run your whole DB in RAM, so there were some real performance benefits with MongoDB. Another difference is that MongoDB was early with great JSON-support. Something that Postgres has since gained. I think there are pros and cons with both. If I'd chose one today, for most tasks I'd probably chose Postgres. That said, to understand why people made those choices, you need to 'teleport back' to that time and compare them in that year (given the tradeoffs at that time).
- sgt101 8y ago
  Oh! From The Guardian write up I had the impression they were using it until mid 2018..
  - Drdrdrq 8y ago
    It is not easy to switch, so until the pain is too big, one doesn't do it.
- purple_ducks 8y ago
  If somebody is teleporting back then, the fact Mongo had a _global_ Mongo instance level lock for writes should be more than enough for people to run screaming. I worked with databases that only had table level locks(not row level) and there were more than enough occasions I cursed the creators. Instance level(& indeed DB level) is insanity unless your DB is a read only DB.
  - underwater 8y ago
    Not saying you're wrong, but a news site and CMS has different database requirements than many other products. They have very few write actions; in the Guardian's case hundreds of authors publishing a few articles a day, versus millions of users reading data. Writes would be sporadic and batched. News products also changes more rapidly than you might expect. A modern news team may be writing and shipping code to better cover a breaking news event. I can imagine why a document store would be appealing to them.
deleted 8y ago
[deleted]
frakkingcylons 8y ago
You're going to need some supporting arguments if you're actually trying to say that every document database is not worth using.

Document DBs are like blockchain projects - overhyped and worse than existing solutions for nearly every use case. Why do ostensibly smart engineers keep falling for this stuff?

wongarsu 8y ago

> Why do ostensibly smart engineers keep falling for this stuff? Review driven design is likely a large contributer. If you know you will be looking for a new job in 2 years, you better get some experience in $latestTrend instead of $provenTechnology.

wwweston 8y ago

For one thing, a good chunk of what happens in the industry isn't particularly well described by the term "engineering," which in my mind describes someone who is well-trained and experience-versed in relevant scientific models for the application domain and the dynamics of relevant "materials" and their interaction that can be composed into a solution. It might be the term "craft" better describes a lot of software development activity. For another, it may be everyone is susceptible to the influence of trends. Even engineers. And the more complex the details of a subdomain of software engineering are, the bigger the tradeoff between becoming someone who can make a true engineering assessment in that niche and developing expertise elsewhere...

ownagefool 8y ago
It's also largely a fact that we're paid better for following hypes, at least in the UK. Thus a smart "engineer" who also cares about their bank balance pretty much has to use everything new to stay relevant. Further, most of the use cases are fairly shallow, so it doesn't actually matter that much, and you'll be paid more to paper over the cracks. We need to change the market, but it doesn't like the strong and stable engineers are getting lead roles to pay more strong and stable engineers to build boring tech, despite that actually being better for most businesses.

zzzcpan 8y ago

They are not overhyped, definitely not as much as old school RDBMS stuff. It's just most engineers can't make good database choices no matter what database they choose. Or more generally they can't make good infrastructure choices, as those are out of their competence and are mostly about things like operations and distributed systems, that take a long time and a lot of experience to get to a level of good decisions.

michelpp 8y ago
You cannot be old-school and hype. That's the reason why it is old school. And the people who can't make good database design choices are exactly the kind of people who should be using SQL. Postgres knows how to optimize and plan queries efficiently based on the actual distributions of values in your dataset. These poor choosers should be doing that... by hand? https://www.postgresql.org/docs/11/planner-optimizer.html https://www.postgresql.org/docs/11/planner-optimizer.html
- zzzcpan 8y ago
  This is what I'm talking about. You think that some feature that makes performance unpredictable is that important. While it's the last thing you should care about.
  - sparkie 8y ago
    What people should care about is the integrity and consistency of the data they're storing. RDBMS do an extremely good job of generalising this problem with reasonable performance, which suits the majority of real world problems. The ideas behind it are understood by most of the industry. It should be the default choice (or plain filesystem storage), unless you have a specific requirement for something different. In the latter case, this should be people who are well informed about the database choices they're making.
  - astine 8y ago
    The only way for query performance to be truly predictable is for your data to be static. If that's not your situation then a query optimizer is incredibly helpful because it means that you don't have to rewrite your queries every time you add an index.

hnbroseph 8y ago

perhaps they don't agree with your thesis.

bdcravens 8y ago

Probably for the same reason people have a build process to load in 100s of KBs of Javascript in order to render an <h1> tag, and the compile it all down to HTML and declare static sites are the wave of the future.

jsd1982 8y ago

Sometimes the choice isn't in the hands of the person most qualified to make that decision.

sandstrom 8y ago

Well, we're kind of comparing MongoDB of ~2011 (when The Guardian started using them) with Postgres of today. One major change is that in 2010 you couldn't always run your whole DB in RAM, so there were some real performance benefits with MongoDB. Another difference is that MongoDB was early with great JSON-support. Something that Postgres has since gained. I think there are pros and cons with both. If I'd chose one today, for most tasks I'd probably chose Postgres. That said, to understand why people made those choices, you need to 'teleport back' to that time and compare them in that year (given the tradeoffs at that time).

sgt101 8y ago
Oh! From The Guardian write up I had the impression they were using it until mid 2018..
- Drdrdrq 8y ago
  It is not easy to switch, so until the pain is too big, one doesn't do it.
purple_ducks 8y ago
If somebody is teleporting back then, the fact Mongo had a _global_ Mongo instance level lock for writes should be more than enough for people to run screaming. I worked with databases that only had table level locks(not row level) and there were more than enough occasions I cursed the creators. Instance level(& indeed DB level) is insanity unless your DB is a read only DB.
- underwater 8y ago
  Not saying you're wrong, but a news site and CMS has different database requirements than many other products. They have very few write actions; in the Guardian's case hundreds of authors publishing a few articles a day, versus millions of users reading data. Writes would be sporadic and batched. News products also changes more rapidly than you might expect. A modern news team may be writing and shipping code to better cover a breaking news event. I can imagine why a document store would be appealing to them.

deleted 8y ago

[deleted]

frakkingcylons 8y ago

You're going to need some supporting arguments if you're actually trying to say that every document database is not worth using.