7 ms·

The NoSQL movement

37 points by tux1968 15y ago

jaylevitt 15y ago
I can't even count how many mistakes this article makes. A random sampling: - Relational databases were designed for a world where availability is unimportant, like transaction processing. Um, no. OLTP has four letters, not two, and the first are as important as the last. Tandem was providing five-9's systems in the '80s and '90s for ATM networks, lottery systems, airline reservations, credit cards, etc. Tandem is a fault-tolerant relational database in hardware. - Up-front schema design is a poor fit in a world where data requirements are fluid: I don't think "up front" means what you think it means. You can change schemas on the fly nowadays; your schema design is no more up-front than your coding is. - You can't have millions of columns in a relational database: True, and you wouldn't; you'd normalize that. This is an important difference, but not a disadvantage of relational databases, any more than saying "In a relational database, you'd join URLs with IP addresses, and maybe five other tables; this design isn't even conceivable in a NoSQL database." - To optimize relational performance, you "do away with joins wherever possible": 1995 called, and it wants its MyISAM back. - Two-phase commit is so obsolete, even banks don't use it: Of course they do. You still need a two-phase commit to make sure the other end got your data; whether "got your data" happens in the customer path or during reconciliation is a design decision. How, exactly, do you think they discover that you and your spouse both got the money? - "relational databases were developed when distributed systems were rare and exotic at best." That's nothing; when Von Neumann machines were developed, we didn't even have transistors. Some legacy architectures keep on working. - "absolute consistency isn't a hard requirement for banks": see above. Yes it is. - "So the CAP theorem is historically irrelevant to relational databases: they're good at providing consistency, and they have been adapted to provide high availability with some success, but they are hard to partition without extreme effort or extreme cost." ... Wh... Bu... That's not even wrong. - "consistency requirements of many social applications are very soft." I like the Facebook example from a recent article on causal consistency: I de-friend my boss and then post that I'm quitting. Certain kinds of consistency are in fact critical to social applications. There are many good reasons to design around a NoSQL database instead of a relational one. This article provides fewer than zero of them.
- mjb 15y ago
  I agree with you - this article does a very poor job of communicating the advantages of 'NoSQL' databases over traditional solutions. It spends too many words setting up a straw man (SQL == not partitioned, not highly available, not redundant, etc) and too few actually making it's case. Lines like this are, at best, a gross oversimplification of relational database performance tuning: "But when you need to optimize performance, you look at the queries you actually perform, then merge tables to create longer rows, and do away with joins wherever possible." And this: "We require sub-second responses to queries." Really? Single big-box OLTP systems have many issues, but if you are getting query times greater than a second on a typical online database then you are either doing something wrong or have specific requirements. Neither of those things will be fixed by blindly picking 'NoSQL' over 'SQL'. "any significant database needs to be distributed." I could easily list, off the top of my head, 100 significant databases that aren't distributed. Maybe some of them should be, but saying that "any significant database needs to be distributed" is a real stretch. > There are many good reasons to design around a NoSQL database instead of a relational one. This article provides fewer than zero of them. Yeah, that's the thing. An article of a quarter of the length with more research, less strawman bashing and less breathlessness could have made the case for NoSQL much more effectively.
- opendomain 15y ago
  - availability means access to the data, not just uptime - I have never seen changing schemas on the fly in a RDBMS programmatically (except admin functions). I have seen overloading of column types however. - "super column" data stores have a deistinct design advantage that you can not get from relational DBS. - everyone that is using relational does the same thing when they get big data: denormalize. From materilzed views al the way to sharding and replication. - banks use transactions, but not as a two phase commit. it is a single atomic record - not partial updates. - we still have some species from the age of the dinosaurs, but the land is rules by those that have evolved. - I agree that banks should be consistent. Not sure what he author was saying here - CAP is relevant to ALL datastores. Consistancy, Availability, Partition tolerance : pick any two. - if your post does not get committed to Facebook, it may be important to YOU, but the application is designed more for availability
ryutin 15y ago
The survey of nosql technologies sounds more like a rationale for another O'Reilly bookshelf! I mean, there are definitely arguments that can be made for alternatives to relational databases, but without those special cases, in such an early stage of maturation and without standards, it's worthy for pursuing by the hardy cowboy or blissful novice. As the nosql technologies do mature and standards emerge, I think it should be expected that they will be subsumed into existing database products as new features.
- einhverfr 15y ago
  There's already some work on this in PostgreSQL with hstore and Javascript as a stored procedure languae.
einhverfr 15y ago
The article is, IMO, total crap. There are advantages to NoSQL in some cases. I am even happy to write about them. However, they are not really the advantages listed here. The first is that SQL databases can in fact be distributed in some cases. Look at Postgres-XC for an example of write-scalable distributed shards while maintaining a consistent, relational model. The second is the focus on analytics. NoSQL analytics is very much of a problematic area. Doing the analysis typically requires very intensive searches of the entire data, and NoSQL databases are not optimized for this. Consequently analytical data tends to be slow to build and then maintained on data input, rather than generated ad hoc from existing data (using the entered data as a single point of truth). This leads to a lack of flexibility even though prepared reports load quickly. It isn't clear to me how different this would be from summary tables maintained with triggers. Now NoSQL has some advantages: 1) Where you don't need ad hoc analytics, where you have highly defined functional requirements, and where you have well defined network protocols for interop, development is often faster, and performance is better. 2) I think it could be very interesting as a network transparent back-plane if you will for various kinds of network services.
cageface 15y ago
The debate over NoSQL vs SQL datastores seems to carry many of the same overtones as the debate over static vs dynamic typing in programming languages, with similar arguments being made on both sides.
troymc 15y ago
Having the two categories SQL and NoSQL is a bit like having the two categories "books" and "non-books." The latter category includes giraffes, planets, feelings, and windmills. Okay, maybe it's not that bad, but NoSQL databases still make a huge non-homogeneous set of things.
jacques_chester 15y ago
Here's my theory. NoSQL came about because smart people were exposed to relational technology in this order: 1. A university course or book that mostly focused on SQL and then normalised design followed by 2. Using MySQL in production. What's missing from this picture is learning the other halves: 1. Normalisation matters to transaction processing. Fast queries is another matter entirely and usually only gets airily waved at in many books and university courses. I went through an entire semester without seeing "OLAP". Techniques I learned on the job were kept to the "Advanced Databases" course which was only taught sporadically. The idea that OLAP is some high mystery is just silly. It's join-beating, denormalising stuff, like NoSQL, just with decades of literature and code to back it up. 2. It also matters that MySQL is not the benchmark of relational technology performance or features. My day job is working with Oracle databases. The price, the odd absence of useful features because It's Never Been Done That Way (I'm looking at you, primary key triggers and booleans-stored-as-char(1)) ... sometimes it's amazing that people pay so much for it. Then you see the manuals, the supporting tools[1] and the performance a half-decent DBA can massage and you get that this stuff isn't as bad as the sticker price says. For my own work, postgresql is where it's at. But for a big site I'd look to DB2, Oracle RAC, Teradata, NonStop, Greenplum and on and on and on before betting on 5-year old technology reinventing a 50-year old paradigm that didn't work real well the first time around. [1] Except SQL Developer. What a dog.
- einhverfr 15y ago
  I don't really buy that. I have built applications using BDB on the back-end. It works really well for some things. It just has other shortcomings development-wise that usually makes up for the benefits and then some. The thing is that object-oriented and relational design are fundamentally different. It is very rare to find people who are very good (or even equally good) at both. They are entirely different design disciplines aimed at very different problems. What non-relational db's do is they free the developer up from this second, very different discipine using tools that usually perform very well for standard OO operations. That's a powerful thing. Of course it's also powerful to have a math engine which can take your stored data, digest it, and spit out a report based on criteria you hadn't thought of until 30 seconds ago. And iterating through a collection of objects? Not a good way to do that.
DiabloD3 15y ago
There is no such thing as the NoSQL movement.... its the "NoMySQL and MySQL is the only SQL implementation in the world because we're all PHP users and have never heard of PostgreSQL" movement. And it doesn't help a lot of the NoSQL dbs out there have SQL-like query languages.
mbailey 15y ago
I love the ad for MS SQL server in the middle of the page.
wyuenho 15y ago
Every time someone says you can't do schema-less design as easily in RDBMS', I cite this EAV article on Wikipedia. http://en.wikipedia.org/wiki/Entity–attribute–value_model http://en.wikipedia.org/wiki/Entity–attribute–value_model
- olalonde 15y ago
  Or you can use an ORM that generates the schema on the fly. (Technically, you still have a schema in both cases.)
olalonde 15y ago
After reading the article and comments here, I'm left even more confused. Anyone care to explain some use cases for NoSQL databases?

37 points by tux1968 15y ago

I can't even count how many mistakes this article makes. A random sampling: - Relational databases were designed for a world where availability is unimportant, like transaction processing. Um, no. OLTP has four letters, not two, and the first are as important as the last. Tandem was providing five-9's systems in the '80s and '90s for ATM networks, lottery systems, airline reservations, credit cards, etc. Tandem is a fault-tolerant relational database in hardware. - Up-front schema design is a poor fit in a world where data requirements are fluid: I don't think "up front" means what you think it means. You can change schemas on the fly nowadays; your schema design is no more up-front than your coding is. - You can't have millions of columns in a relational database: True, and you wouldn't; you'd normalize that. This is an important difference, but not a disadvantage of relational databases, any more than saying "In a relational database, you'd join URLs with IP addresses, and maybe five other tables; this design isn't even conceivable in a NoSQL database." - To optimize relational performance, you "do away with joins wherever possible": 1995 called, and it wants its MyISAM back. - Two-phase commit is so obsolete, even banks don't use it: Of course they do. You still need a two-phase commit to make sure the other end got your data; whether "got your data" happens in the customer path or during reconciliation is a design decision. How, exactly, do you think they discover that you and your spouse both got the money? - "relational databases were developed when distributed systems were rare and exotic at best." That's nothing; when Von Neumann machines were developed, we didn't even have transistors. Some legacy architectures keep on working. - "absolute consistency isn't a hard requirement for banks": see above. Yes it is. - "So the CAP theorem is historically irrelevant to relational databases: they're good at providing consistency, and they have been adapted to provide high availability with some success, but they are hard to partition without extreme effort or extreme cost." ... Wh... Bu... That's not even wrong. - "consistency requirements of many social applications are very soft." I like the Facebook example from a recent article on causal consistency: I de-friend my boss and then post that I'm quitting. Certain kinds of consistency are in fact critical to social applications. There are many good reasons to design around a NoSQL database instead of a relational one. This article provides fewer than zero of them.

mjb 15y ago
I agree with you - this article does a very poor job of communicating the advantages of 'NoSQL' databases over traditional solutions. It spends too many words setting up a straw man (SQL == not partitioned, not highly available, not redundant, etc) and too few actually making it's case. Lines like this are, at best, a gross oversimplification of relational database performance tuning: "But when you need to optimize performance, you look at the queries you actually perform, then merge tables to create longer rows, and do away with joins wherever possible." And this: "We require sub-second responses to queries." Really? Single big-box OLTP systems have many issues, but if you are getting query times greater than a second on a typical online database then you are either doing something wrong or have specific requirements. Neither of those things will be fixed by blindly picking 'NoSQL' over 'SQL'. "any significant database needs to be distributed." I could easily list, off the top of my head, 100 significant databases that aren't distributed. Maybe some of them should be, but saying that "any significant database needs to be distributed" is a real stretch. > There are many good reasons to design around a NoSQL database instead of a relational one. This article provides fewer than zero of them. Yeah, that's the thing. An article of a quarter of the length with more research, less strawman bashing and less breathlessness could have made the case for NoSQL much more effectively.
opendomain 15y ago
- availability means access to the data, not just uptime - I have never seen changing schemas on the fly in a RDBMS programmatically (except admin functions). I have seen overloading of column types however. - "super column" data stores have a deistinct design advantage that you can not get from relational DBS. - everyone that is using relational does the same thing when they get big data: denormalize. From materilzed views al the way to sharding and replication. - banks use transactions, but not as a two phase commit. it is a single atomic record - not partial updates. - we still have some species from the age of the dinosaurs, but the land is rules by those that have evolved. - I agree that banks should be consistent. Not sure what he author was saying here - CAP is relevant to ALL datastores. Consistancy, Availability, Partition tolerance : pick any two. - if your post does not get committed to Facebook, it may be important to YOU, but the application is designed more for availability

ryutin 15y ago

The survey of nosql technologies sounds more like a rationale for another O'Reilly bookshelf! I mean, there are definitely arguments that can be made for alternatives to relational databases, but without those special cases, in such an early stage of maturation and without standards, it's worthy for pursuing by the hardy cowboy or blissful novice. As the nosql technologies do mature and standards emerge, I think it should be expected that they will be subsumed into existing database products as new features.

einhverfr 15y ago
There's already some work on this in PostgreSQL with hstore and Javascript as a stored procedure languae.

einhverfr 15y ago

The article is, IMO, total crap. There are advantages to NoSQL in some cases. I am even happy to write about them. However, they are not really the advantages listed here. The first is that SQL databases can in fact be distributed in some cases. Look at Postgres-XC for an example of write-scalable distributed shards while maintaining a consistent, relational model. The second is the focus on analytics. NoSQL analytics is very much of a problematic area. Doing the analysis typically requires very intensive searches of the entire data, and NoSQL databases are not optimized for this. Consequently analytical data tends to be slow to build and then maintained on data input, rather than generated ad hoc from existing data (using the entered data as a single point of truth). This leads to a lack of flexibility even though prepared reports load quickly. It isn't clear to me how different this would be from summary tables maintained with triggers. Now NoSQL has some advantages: 1) Where you don't need ad hoc analytics, where you have highly defined functional requirements, and where you have well defined network protocols for interop, development is often faster, and performance is better. 2) I think it could be very interesting as a network transparent back-plane if you will for various kinds of network services.

cageface 15y ago

The debate over NoSQL vs SQL datastores seems to carry many of the same overtones as the debate over static vs dynamic typing in programming languages, with similar arguments being made on both sides.

troymc 15y ago

Having the two categories SQL and NoSQL is a bit like having the two categories "books" and "non-books." The latter category includes giraffes, planets, feelings, and windmills. Okay, maybe it's not that bad, but NoSQL databases still make a huge non-homogeneous set of things.

jacques_chester 15y ago

Here's my theory. NoSQL came about because smart people were exposed to relational technology in this order: 1. A university course or book that mostly focused on SQL and then normalised design followed by 2. Using MySQL in production. What's missing from this picture is learning the other halves: 1. Normalisation matters to transaction processing. Fast queries is another matter entirely and usually only gets airily waved at in many books and university courses. I went through an entire semester without seeing "OLAP". Techniques I learned on the job were kept to the "Advanced Databases" course which was only taught sporadically. The idea that OLAP is some high mystery is just silly. It's join-beating, denormalising stuff, like NoSQL, just with decades of literature and code to back it up. 2. It also matters that MySQL is not the benchmark of relational technology performance or features. My day job is working with Oracle databases. The price, the odd absence of useful features because It's Never Been Done That Way (I'm looking at you, primary key triggers and booleans-stored-as-char(1)) ... sometimes it's amazing that people pay so much for it. Then you see the manuals, the supporting tools[1] and the performance a half-decent DBA can massage and you get that this stuff isn't as bad as the sticker price says. For my own work, postgresql is where it's at. But for a big site I'd look to DB2, Oracle RAC, Teradata, NonStop, Greenplum and on and on and on before betting on 5-year old technology reinventing a 50-year old paradigm that didn't work real well the first time around. [1] Except SQL Developer. What a dog.

einhverfr 15y ago
I don't really buy that. I have built applications using BDB on the back-end. It works really well for some things. It just has other shortcomings development-wise that usually makes up for the benefits and then some. The thing is that object-oriented and relational design are fundamentally different. It is very rare to find people who are very good (or even equally good) at both. They are entirely different design disciplines aimed at very different problems. What non-relational db's do is they free the developer up from this second, very different discipine using tools that usually perform very well for standard OO operations. That's a powerful thing. Of course it's also powerful to have a math engine which can take your stored data, digest it, and spit out a report based on criteria you hadn't thought of until 30 seconds ago. And iterating through a collection of objects? Not a good way to do that.

DiabloD3 15y ago

There is no such thing as the NoSQL movement.... its the "NoMySQL and MySQL is the only SQL implementation in the world because we're all PHP users and have never heard of PostgreSQL" movement. And it doesn't help a lot of the NoSQL dbs out there have SQL-like query languages.

mbailey 15y ago

I love the ad for MS SQL server in the middle of the page.

wyuenho 15y ago

Every time someone says you can't do schema-less design as easily in RDBMS', I cite this EAV article on Wikipedia. http://en.wikipedia.org/wiki/Entity–attribute–value_model http://en.wikipedia.org/wiki/Entity–attribute–value_model

olalonde 15y ago
Or you can use an ORM that generates the schema on the fly. (Technically, you still have a schema in both cases.)

olalonde 15y ago

After reading the article and comments here, I'm left even more confused. Anyone care to explain some use cases for NoSQL databases?