6 ms·

Why did NoSQL become popular? Was it the huge datasets required by the internet? Before SQL, there was already no SQL. I don't mean that as a semantic joke, bu

Why did NoSQL become popular? Was it the huge datasets required by the internet? Before SQL, there was already no SQL. I don't mean that as a semantic joke, but that databases existed before relational databases that were faster. Relational DB were too slow to even be usable, until B-trees made them barely feasible in performance (and still much slower than previous DB). The advantage was flexibility: you could change the database organization without having to rewrite application. Similarly, if your application needed data in a different form, you could make it seem that the database was already in that form. So SQL was like a glue between systems that could transform the structure of the data - much like high school algebra can put an equation into a different form, that is equivalent but more convenient. I can imagine, that back in 1970's, computing power was growing much faster year-by-year, than typical database sizes were. So, although "slow", they became "fast enough" for more and more use-cases. But in 2010's, internet datasets were growing much faster - and computing power wasn't. So Relational DBs weren't "fast enough" for these cases... hence "NoSQL". Is that about right?

overcast 6y ago
NoSQL rocketed in popularity, because it required zero working knowledge of how databases work. You could get up and running on a project, without having to worry about what tables, columns, and their relationships to one another meant. You could throw ANY data in, and generally get it back out.
- sharadov 6y ago
  Exactly! Developers did not want to learn messy databases. I think a lot of folks without experience entered the industry ( mostly from boot camps and such and ruined everything)
- dodobirdlord 6y ago
  I think that's not the reason. NoSQL rocketed in popularity on the back of adoption by a few large companies with scale problems that had to abandon relational databases due to scale issues. If the requirement is to serve very high low-latency throughput to back something like shopping on Amazon, then relational databases and SQL in particular aren't very helpful. You know your data access patterns up front, and can optimize your database to support exactly your API's access patterns. Ad hoc queries on the production database are prohibited, data analysis work gets done with some kind of ETL pipeline, and the choice to trade off any part of ACID for more throughput and lower latency is a no-brainer.
  - overcast 6y ago
    A few large companies helped to get it onto the radar of a lot of inexperienced developers, who found how easy it was just to plug away on it. All of that performance nonsense was second fiddle by a long shot for the vast majority of users.
- claytongulick 6y ago
  I agree with you partially. I also think that it became popular because a lot of data for many cases is naturally semi-structured documents / complex graphs. Strictly modeling this in many cases is either impossible or prohibitively complex. Like most tech fads, NoSQL is best used as just another tool in the toolbox. It is entirely appropriate for a huge class of problems, and entirely inappropriate for others.
  - f6v 6y ago
    I can almost guarantee that for most of the SaaS startups which still go with the React, Node, Mongo stack the data is structured. They have users, orders and whatnot. It just takes some experience to foresee the upcoming product changes. But as someone said here, the nosql stack has been incredibly popular among recent bootcamp graduates.
    - claytongulick 6y ago
      We're going to have to disagree there. Loosely structured graph data is a thing, and there are many uses cases for it. This is why pretty much every rdbms these days has a json datatype.
barrkel 6y ago
NoSQL didn't support joins (it's much easier to get predictable performance from key lookups), was trivial to shard (because no joins), and supported the common B2C scaling pattern of small amounts of data on millions of users. With SQL it's easy to write a badly performing query which does lots of inefficient joins in the DB. NoSQL doesn't give you as many tools to offload your computation so it lives in your app instead, where it's easier to scale out (i.e. throw money at the problem). I'm not getting into schemas vs schemaless (e.g. the cost of migrations when you have billions of rows) or denormalization (e.g. stuffing joined entites into both ends of an association, ugh), etc., there are other pros and cons. But IMO the lack of compute offload is a positive feature of most NoSQL.
- jolux 6y ago
  But you’re also offloading data integrity to your application, which can introduce significant latency if you don’t do it right.
aidenn0 6y ago
> But in 2010's, internet datasets were growing much faster - and computing power wasn't. So Relational DBs weren't "fast enough" for these cases... hence "NoSQL". I joke that the only thing a NoSQL database can do faster than an RDBMS is give you the wrong answer to a query. There are two different features that people typically think of when they think of NoSQL as compared to a traditional RDBMS: 1. Document database (i.e. unstructured, or at least weakly structured data) 2. Eventual consistency You could have an ACID document database, or an eventually consistent relational database, so the two are actually orthogonal. It's definitely easier to get something up and running quickly if you aren't going to implement all of SQL though. If you make money serving up ads on lists of documents, then getting any answer quickly is going to be better than getting the "right" answer slowly, so for google at least, this made sense, as google's databases actually were that big. IMO the rest of it was people cargo-culting, not understanding the hard-won knowledge of the 70s, and also not realizing that, if your working set was 100GB, you could just buy a server with 256GB of ram and have zero performance issues. Prototypes went up fast, benchmarks were great, and then somewhere down the line, someone wanted to run a query on the data, and discovered that fast writes come at a significant cost.
- claytongulick 6y ago
  I don't know. After hating it a lot for a long time, I've come to really appreciate MongoDB's aggregation pipelines. They're very explicit and easy to control / optimize / understand. Yes, it sort of feels like you're writing an EXPLAIN plan, but I'm starting to actually enjoy that now that I've gotten deeper into the guts of it.
- fctorial 6y ago
  With m.2 nvme, you don't even need a ton of ram for that. They're fast enough.
- strken 6y ago
  If your database keeps growing, you'll eventually exceed the write capacity of a single server, and have to write to more than one place. You're correct that most of the NoSQL hype was cargo-culting, but I don't think it came solely from Google trying to serve ads quicker, I think it also came from companies having too much data to scale vertically.
deleted 6y ago
[deleted]

overcast 6y ago

NoSQL rocketed in popularity, because it required zero working knowledge of how databases work. You could get up and running on a project, without having to worry about what tables, columns, and their relationships to one another meant. You could throw ANY data in, and generally get it back out.

sharadov 6y ago
Exactly! Developers did not want to learn messy databases. I think a lot of folks without experience entered the industry ( mostly from boot camps and such and ruined everything)
dodobirdlord 6y ago
I think that's not the reason. NoSQL rocketed in popularity on the back of adoption by a few large companies with scale problems that had to abandon relational databases due to scale issues. If the requirement is to serve very high low-latency throughput to back something like shopping on Amazon, then relational databases and SQL in particular aren't very helpful. You know your data access patterns up front, and can optimize your database to support exactly your API's access patterns. Ad hoc queries on the production database are prohibited, data analysis work gets done with some kind of ETL pipeline, and the choice to trade off any part of ACID for more throughput and lower latency is a no-brainer.
- overcast 6y ago
  A few large companies helped to get it onto the radar of a lot of inexperienced developers, who found how easy it was just to plug away on it. All of that performance nonsense was second fiddle by a long shot for the vast majority of users.
claytongulick 6y ago
I agree with you partially. I also think that it became popular because a lot of data for many cases is naturally semi-structured documents / complex graphs. Strictly modeling this in many cases is either impossible or prohibitively complex. Like most tech fads, NoSQL is best used as just another tool in the toolbox. It is entirely appropriate for a huge class of problems, and entirely inappropriate for others.
- f6v 6y ago
  I can almost guarantee that for most of the SaaS startups which still go with the React, Node, Mongo stack the data is structured. They have users, orders and whatnot. It just takes some experience to foresee the upcoming product changes. But as someone said here, the nosql stack has been incredibly popular among recent bootcamp graduates.
  - claytongulick 6y ago
    We're going to have to disagree there. Loosely structured graph data is a thing, and there are many uses cases for it. This is why pretty much every rdbms these days has a json datatype.

barrkel 6y ago

NoSQL didn't support joins (it's much easier to get predictable performance from key lookups), was trivial to shard (because no joins), and supported the common B2C scaling pattern of small amounts of data on millions of users. With SQL it's easy to write a badly performing query which does lots of inefficient joins in the DB. NoSQL doesn't give you as many tools to offload your computation so it lives in your app instead, where it's easier to scale out (i.e. throw money at the problem). I'm not getting into schemas vs schemaless (e.g. the cost of migrations when you have billions of rows) or denormalization (e.g. stuffing joined entites into both ends of an association, ugh), etc., there are other pros and cons. But IMO the lack of compute offload is a positive feature of most NoSQL.

jolux 6y ago
But you’re also offloading data integrity to your application, which can introduce significant latency if you don’t do it right.

aidenn0 6y ago

> But in 2010's, internet datasets were growing much faster - and computing power wasn't. So Relational DBs weren't "fast enough" for these cases... hence "NoSQL". I joke that the only thing a NoSQL database can do faster than an RDBMS is give you the wrong answer to a query. There are two different features that people typically think of when they think of NoSQL as compared to a traditional RDBMS: 1. Document database (i.e. unstructured, or at least weakly structured data) 2. Eventual consistency You could have an ACID document database, or an eventually consistent relational database, so the two are actually orthogonal. It's definitely easier to get something up and running quickly if you aren't going to implement all of SQL though. If you make money serving up ads on lists of documents, then getting any answer quickly is going to be better than getting the "right" answer slowly, so for google at least, this made sense, as google's databases actually were that big. IMO the rest of it was people cargo-culting, not understanding the hard-won knowledge of the 70s, and also not realizing that, if your working set was 100GB, you could just buy a server with 256GB of ram and have zero performance issues. Prototypes went up fast, benchmarks were great, and then somewhere down the line, someone wanted to run a query on the data, and discovered that fast writes come at a significant cost.

claytongulick 6y ago
I don't know. After hating it a lot for a long time, I've come to really appreciate MongoDB's aggregation pipelines. They're very explicit and easy to control / optimize / understand. Yes, it sort of feels like you're writing an EXPLAIN plan, but I'm starting to actually enjoy that now that I've gotten deeper into the guts of it.
fctorial 6y ago
With m.2 nvme, you don't even need a ton of ram for that. They're fast enough.
strken 6y ago
If your database keeps growing, you'll eventually exceed the write capacity of a single server, and have to write to more than one place. You're correct that most of the NoSQL hype was cargo-culting, but I don't think it came solely from Google trying to serve ads quicker, I think it also came from companies having too much data to scale vertically.

deleted 6y ago

[deleted]