4 ms·

I submitted this article because of a single paragraph: > When it came to the different types of NoSQL databases, it was pretty clear that a document store bes

by orf 9y ago

I submitted this article because of a single paragraph: > When it came to the different types of NoSQL databases, it was pretty clear that a document store best suited our needs. We needed to be able to query based on the data so key/value was out, there was no need to have the ability to make all of the complex connections between data of a graph db, and nobody actually knows what a column db is (jk but our data is not the right use case for it). So document it was. We would have to change our data layout a bit to really work well with most document databases, our current data structure was similar to what you would expect for a relational db with tables etc since dynamo also uses a table structure. I wanted something that was ACID, none of that eventual consistency nonsense. That took a lot of them out of the race right there. I don't get it. He needed arbitrary querying, relations, ACID, tables. And yet... his first thought was to go to a document-based no-sql database and is only later hit with the sudden realization that his relational data is bet suited in a relational database. What has gone so wrong here that an otherwise smart and educated engineer makes these kinds of fundamental mistakes? Is it the success of no-sql marketing departments, and the apparent un-sexiness of traditional relational databases? Is it education? The job market? A fear of SQL (that you never really have to write anyway...)? Is it the quick-fix non-relational databases give you? As far as I can see there are very very very few nails that a PostgreSQL hammer cannot hit. I guess that's part of the point of the article, but why does it need an article to explain this? How can we stop these kinds of articles from needing to be written? > The change drastically increased the speed of our API, some requests are up to 6x faster than they were with Dynamo. The account details request, our previously slowest call, took around 12 seconds with Dynamo and was reduced to 2 seconds with Postgres! sigh Edit: Not sure why this has been moved to the bottom of the thread. I wasn't trying to single this particular developer out, I was merely commenting on this troubling trend. Apologies if this came across in the wrong way.

misterbowfinger 9y ago
When I hear "brogrammer", I'm usually a little put-off because it's kind of a troll term. But this article, wow. You can call me on it - brogrammers are ruining the industry.
- dccoolgai 9y ago
  Easy there, Hoss. This guy is just giving an honest account of the decisions he made and how it turned out. Doesn't say anything offensive or exclusive at all. Certain attitudes are harmful to the industry and people wanting to learn and share their experiences...
  - collyw 9y ago
    "I’ve been writing node for 4 years and like any other true node fanboy, if I need to use a database, I go straight to NoSQL." Yes, thats the way an engineer should be making decisions about arguably the most important part of the application.
    - dccoolgai 9y ago
      So everyone that makes a bad decision and openly takes their time to write about it so others can learn from their mistakes is a "brogrammer"? Great engineers become great engineers by making mistakes and learning from them.
      - collyw 9y ago
        My problem is with the reasoning behind the decision (or rather the complete lack of it). (And it wasn't me that used "brogramer").
justicezyx 9y ago
He likely has no formal training in database and therefore are not exposed to things alike. Side topic: AWS did agressive move to throw it's resources to University, and pays off hansomely. Those new blood coming to our profession now has the concept of infrastructure ~= AWS (mileage varies between people, but that's the idea).
virmundi 9y ago
Keep in mind that the Postgres hammer is a non-standard hammer. Sure it is pretty SQL compliant (except there are no dirty reads thanks to MVCC), but if you use JSON or the arrays or the many other wonderful features of Postgres, you are now tied to Postgres. You are also slowly inching your way into Document Stores.
- tootie 9y ago
  The biggest pro to Dynamo of Postgres in an AWS world is just price. It's very attractive to try to shove a square peg in a round hole to save money. Of course, you end up costing yourself far more in the long run if it's such a bad fit.
- orf 9y ago
  The same can be said about any database though. No one is like another. SQL as a standard is roughly followed, and PostgreSQL is very compliant. That's more than can be said about any non-relational database - each has it's own query language. I use JSON fields and arrays quite a bit, but that doesn't mean I need a document store. It means I have small portions of data that is non-relational, or is better suited to use arrays/json than strict tables, but I still want everything else consistent, enforced, and also have all the other bells and whistles Postgres gives you (FTS, partial indexes, rich types everywhere, ACID, the list goes on and on). Being tied to Postgres is often better than being tied to a document store. At least you can dip your toes into non-rel as little or much as you like (where it makes sense), rather than diving in at the deep end and realizing you can't swim.
  - virmundi 9y ago
    What about ArangoDB? You can perform joins, even across the cluster. It's a document store. You don't have transactions across the cluster (yet, I'm pretty sure I'm right). You get sharding and distribution automatically. Again, at this point you're out of the SQL world, why not go fully out?
- eropple 9y ago
  I can't think of a datastore whose purely SQL-compliant feature set is worth using; to that end, using any RDBMS with a reasonable feature set is going to be a product lock-in of some kind. And I've been around and babysat a lot of people's Mongo stuff; I'll die on the hill that being tied to Postgres is a hell of a lot better for the 10th, the 50th, and the 98th percentile of developers (and companies) than a MongoDB or the like.
toomuchtodo 9y ago
"I’m not a database guy, I’m a node guy." This school of thought that a SWE can know the entire stack, infra to app, is coming home to roost. You are the technical debt. Learn your fundementals, and if a relational database isn't part of your foundational knowledge, learn it now or get out.
- orf 9y ago
  I get what you mean, but to me that's the same as saying "I'm not a HTTP guy, I'm a web guy". I mean, sure, he may be a "node guy", but he uses node to make web applications. A large part of that is persistence, i.e databases. For a lot of applications it's one of the most important part of your stack, if not the most important part. It's your treasure trove, it's your crown jewels. Your app is just a thing that shuffles state around in the database. He knew a fair bit about NoSQL databases, but why did he not even consider a relational one? I'm not trying to single this specific guy out, merely asking what people think is causing the trend of engineers over-using non-rel databases for relational data. Edit: didn't see your edit. you're spot on.
  - eropple 9y ago
    I spend a lot of time, as a consultant, talking to people who've already made technical decisions that are biting them in the ass and it seems to mostly boil down to good press and an impressive of the five-minute demo. The mental effort required to learn new stuff is scary and always looks worse than it really is; easy demos hook novice and many-years-a-novice developers to the point where, well, of course it seems obvious not to use RDBMSes, they're hard and you might have to spend time on learning them and you might even look silly or dumb (gasp!) when you're doing it. From the outside, Postgres looks hard. It isn't hard, but it looks it, because it was not designed around a five-minute demo. I use it for everything until I have a very compelling reason not to. But I have an established library of scripts and systems to make it easy, because I am both an infrastructure guy and a software developer so the idea of "I'm going to write a Vagrantfile in two minutes flat and drop in this provisioner" is something that's top-of-mind for me. (Or I just steal the Vagrantfile from another project, whatever.) But I've made that investment and it's paying off--if one hasn't made that investment, it's scary And then doing it in production? Whether it's a hosted service like RDS or doing it yourself, you have to answer a lot of capital-Q Questions along the way and that's effort that seems unrelated to actually getting your software done. It's not just "why didn't this person think about a RDBMS?". It's that all of these easy-answer-but-not-really systems (fill in your own) are capturing the minds of developers. The quest for abstractions and ways to make ourselves better able to do our jobs is a good one. The uncritical adoption of tools that might not actually do that is not, but that's a cultural thing not limited to datastores.
StreamBright 9y ago
NodeJS and MongoDB is frequently used together and I think some fraction of the NodeJS developers thinks that NoSQL is just the better, newer, faster solution to data storage. I regurarly run into customers who hire us to make their data model work and some non-trivial cases end up being moving them to PostgreSQL. Data engineering actually is a more complex topic than some people think and it really bites you when you are actually using the solution for a while.
venture_lol 9y ago
It's way too easy for anyone to pick up Node and CRUD a few pieces of data into a NOSQL store and voila, I am an architect :) Selecting the right framework, platform, model for current needs with reasonable forward looking support takes experience and strong formal fundamentals.
- collyw 9y ago
  Unfortunately job ads imply than many places prefer the former.
jopsen 9y ago
Often people don't want to do database admin stuff, dynamodb has none of that. On the bright side, relational is coming back with RDS, cloud-sql, spanner and cockroachdb. With any luck NoSQL will be forgotten :)

by orf 9y ago

misterbowfinger 9y ago

When I hear "brogrammer", I'm usually a little put-off because it's kind of a troll term. But this article, wow. You can call me on it - brogrammers are ruining the industry.

dccoolgai 9y ago
Easy there, Hoss. This guy is just giving an honest account of the decisions he made and how it turned out. Doesn't say anything offensive or exclusive at all. Certain attitudes are harmful to the industry and people wanting to learn and share their experiences...
- collyw 9y ago
  "I’ve been writing node for 4 years and like any other true node fanboy, if I need to use a database, I go straight to NoSQL." Yes, thats the way an engineer should be making decisions about arguably the most important part of the application.
  - dccoolgai 9y ago
    So everyone that makes a bad decision and openly takes their time to write about it so others can learn from their mistakes is a "brogrammer"? Great engineers become great engineers by making mistakes and learning from them.
    - collyw 9y ago
      My problem is with the reasoning behind the decision (or rather the complete lack of it). (And it wasn't me that used "brogramer").

justicezyx 9y ago

He likely has no formal training in database and therefore are not exposed to things alike. Side topic: AWS did agressive move to throw it's resources to University, and pays off hansomely. Those new blood coming to our profession now has the concept of infrastructure ~= AWS (mileage varies between people, but that's the idea).

virmundi 9y ago

Keep in mind that the Postgres hammer is a non-standard hammer. Sure it is pretty SQL compliant (except there are no dirty reads thanks to MVCC), but if you use JSON or the arrays or the many other wonderful features of Postgres, you are now tied to Postgres. You are also slowly inching your way into Document Stores.

tootie 9y ago
The biggest pro to Dynamo of Postgres in an AWS world is just price. It's very attractive to try to shove a square peg in a round hole to save money. Of course, you end up costing yourself far more in the long run if it's such a bad fit.
orf 9y ago
The same can be said about any database though. No one is like another. SQL as a standard is roughly followed, and PostgreSQL is very compliant. That's more than can be said about any non-relational database - each has it's own query language. I use JSON fields and arrays quite a bit, but that doesn't mean I need a document store. It means I have small portions of data that is non-relational, or is better suited to use arrays/json than strict tables, but I still want everything else consistent, enforced, and also have all the other bells and whistles Postgres gives you (FTS, partial indexes, rich types everywhere, ACID, the list goes on and on). Being tied to Postgres is often better than being tied to a document store. At least you can dip your toes into non-rel as little or much as you like (where it makes sense), rather than diving in at the deep end and realizing you can't swim.
- virmundi 9y ago
  What about ArangoDB? You can perform joins, even across the cluster. It's a document store. You don't have transactions across the cluster (yet, I'm pretty sure I'm right). You get sharding and distribution automatically. Again, at this point you're out of the SQL world, why not go fully out?
eropple 9y ago
I can't think of a datastore whose purely SQL-compliant feature set is worth using; to that end, using any RDBMS with a reasonable feature set is going to be a product lock-in of some kind. And I've been around and babysat a lot of people's Mongo stuff; I'll die on the hill that being tied to Postgres is a hell of a lot better for the 10th, the 50th, and the 98th percentile of developers (and companies) than a MongoDB or the like.

toomuchtodo 9y ago

"I’m not a database guy, I’m a node guy." This school of thought that a SWE can know the entire stack, infra to app, is coming home to roost. You are the technical debt. Learn your fundementals, and if a relational database isn't part of your foundational knowledge, learn it now or get out.

orf 9y ago
I get what you mean, but to me that's the same as saying "I'm not a HTTP guy, I'm a web guy". I mean, sure, he may be a "node guy", but he uses node to make web applications. A large part of that is persistence, i.e databases. For a lot of applications it's one of the most important part of your stack, if not the most important part. It's your treasure trove, it's your crown jewels. Your app is just a thing that shuffles state around in the database. He knew a fair bit about NoSQL databases, but why did he not even consider a relational one? I'm not trying to single this specific guy out, merely asking what people think is causing the trend of engineers over-using non-rel databases for relational data. Edit: didn't see your edit. you're spot on.
- eropple 9y ago
  I spend a lot of time, as a consultant, talking to people who've already made technical decisions that are biting them in the ass and it seems to mostly boil down to good press and an impressive of the five-minute demo. The mental effort required to learn new stuff is scary and always looks worse than it really is; easy demos hook novice and many-years-a-novice developers to the point where, well, of course it seems obvious not to use RDBMSes, they're hard and you might have to spend time on learning them and you might even look silly or dumb (gasp!) when you're doing it. From the outside, Postgres looks hard. It isn't hard, but it looks it, because it was not designed around a five-minute demo. I use it for everything until I have a very compelling reason not to. But I have an established library of scripts and systems to make it easy, because I am both an infrastructure guy and a software developer so the idea of "I'm going to write a Vagrantfile in two minutes flat and drop in this provisioner" is something that's top-of-mind for me. (Or I just steal the Vagrantfile from another project, whatever.) But I've made that investment and it's paying off--if one hasn't made that investment, it's scary And then doing it in production? Whether it's a hosted service like RDS or doing it yourself, you have to answer a lot of capital-Q Questions along the way and that's effort that seems unrelated to actually getting your software done. It's not just "why didn't this person think about a RDBMS?". It's that all of these easy-answer-but-not-really systems (fill in your own) are capturing the minds of developers. The quest for abstractions and ways to make ourselves better able to do our jobs is a good one. The uncritical adoption of tools that might not actually do that is not, but that's a cultural thing not limited to datastores.

StreamBright 9y ago

NodeJS and MongoDB is frequently used together and I think some fraction of the NodeJS developers thinks that NoSQL is just the better, newer, faster solution to data storage. I regurarly run into customers who hire us to make their data model work and some non-trivial cases end up being moving them to PostgreSQL. Data engineering actually is a more complex topic than some people think and it really bites you when you are actually using the solution for a while.

venture_lol 9y ago

It's way too easy for anyone to pick up Node and CRUD a few pieces of data into a NOSQL store and voila, I am an architect :) Selecting the right framework, platform, model for current needs with reasonable forward looking support takes experience and strong formal fundamentals.

collyw 9y ago
Unfortunately job ads imply than many places prefer the former.

jopsen 9y ago

Often people don't want to do database admin stuff, dynamodb has none of that. On the bright side, relational is coming back with RDS, cloud-sql, spanner and cockroachdb. With any luck NoSQL will be forgotten :)