3 ms·

> The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. Also inc

by jeswin 11y ago

> The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. Also incredibly expensive. Take this case, Google took off because they were able to scale-out with off-the-shelf hardware, compared to the millions banks were pouring in for scale-up configurations which handled much less load. Scale-up can quickly hit hard limits, before it becomes exponentially expensive to continue on the path. This is true even today. NoSQL movement, if you wanna call it that, took off because most apps (including Google, including Financial Services, even Health Care) don't need some types of consistency offered by relational databases. Many large apps are significantly de-normalized and have many foreign-key less tables, often filled up by scheduled jobs. That's fine for most apps; NoSQL architectures recognize that and users consider that in design. For the majority of use-cases out there, NoSQL databases offer enough consistency. For the remaining use-cases, there are tools available in NoSQL databases to make them work, though it requires a bit of work.

andyidsinga 11y ago
yep. see also : shared nothing architecture
AlisdairO 11y ago
> For the majority of use-cases out there, NoSQL databases offer enough consistency. For applications with a nontrivial data model, ensuring that each logical operation only does a single document update (or that multiple nontransactional updates cause no conflict in maintaining consistency in the presence of other logical operations) is actually really challenging - and it adds a substantial design overhead to every new feature added. I think you're being extremely optimistic in your assertion that NoSQL systems are that widely safely applicable. My experience has been that NoSQL-based systems stay mostly-consistent because they happen to experience very little concurrent activity, not through understanding/design on the users' part. This is not to make light of the situations where NoSQL systems shine, but the idea that higher levels of consistency are rarely useful does not match my experience at all. > NoSQL movement, if you wanna call it that, took off because most apps (including Google, including Financial Services, even Health Care) don't need some types of consistency offered by relational databases. I'd say that Mongo (for example) took off because: - They really nailed the setup experience for new users (which RDBMSs historically sucked at). - The data model is much easier for simple apps. - They had some fairly creative techniques for making their system look good - unacknowledged writes, and claims of support for replication which didn't really fully work. - Most programmers don't really understand the ins-and-outs of maintaining consistency.
marktangotango 11y ago
>> Many large apps are significantly de-normalized and have many foreign-key less tables, often filled up by scheduled jobs. That's fine for most apps; NoSQL architectures recognize that and users consider that in design. There's a boat load of assumed knowledge in this quote, how likely is it that someone not familiar with rdbms would know what a foreign key is, for example? Not very likely I think. I posit you give developers too much credit.
nickpsecurity 11y ago
In startup mode, I expect companies to do whatever works. Google's initial decisions were correct. Later, they dedicated expensive teams to inventing brand new solutions for about every aspect of their operation. That's when my critique apples. For instance, they might have looked at expensive NonStop, identified what they could copy with COTS tech, and made a cheap knockoff. They could've done the same with prior MPP or cluster tech which already scaled to thousands of nodes with open source management tools and microsecond latency. Much more efficient than web stacks, too. The one example I saw of them copying and improving past methods was Google File System. As I predicted, that good choice laid foundation for awesome stuff such as BigTable, Spanner, and F1. NoSQL movement's origins are highly debatable. Here's what I saw in its beginning: an explosion of articles on the subject after a few success stories about big companies doing massive scale on cheap servers. Zealots argued that mature, strong-consistency solutions had problems. So, instead of fixing those, we should just ditch them and strong consistency for The New, Great Thing. The only time I even saw a real analysis of cost-benefits was a few articles which focused on a narrow class of applications where data consistency didn't really matter. So, my conclusion was that the movement was two things: 95% a social phenomenon that happens with each IT fad; 5% a result of weaknesses of relational model and tools. That last part makes sense and is why I've always opposed RDBMS's. On your last point, do you have a citation that shows weak consistency databases offer enough integrity for "majority of use-cases?" I thought the majority of use cases for databases were protecting data that's important to a business: orders, billing, HR, inventory, customer information, and so on. I usually just tell them to use PostgresSQL and that covers it with high reliability. If you really can back your claim, though, I'll be glad to start transitioning my R&D toward designs that trade against integrity of data.

by jeswin 11y ago

andyidsinga 11y ago

yep. see also : shared nothing architecture

AlisdairO 11y ago

> For the majority of use-cases out there, NoSQL databases offer enough consistency. For applications with a nontrivial data model, ensuring that each logical operation only does a single document update (or that multiple nontransactional updates cause no conflict in maintaining consistency in the presence of other logical operations) is actually really challenging - and it adds a substantial design overhead to every new feature added. I think you're being extremely optimistic in your assertion that NoSQL systems are that widely safely applicable. My experience has been that NoSQL-based systems stay mostly-consistent because they happen to experience very little concurrent activity, not through understanding/design on the users' part. This is not to make light of the situations where NoSQL systems shine, but the idea that higher levels of consistency are rarely useful does not match my experience at all. > NoSQL movement, if you wanna call it that, took off because most apps (including Google, including Financial Services, even Health Care) don't need some types of consistency offered by relational databases. I'd say that Mongo (for example) took off because: - They really nailed the setup experience for new users (which RDBMSs historically sucked at). - The data model is much easier for simple apps. - They had some fairly creative techniques for making their system look good - unacknowledged writes, and claims of support for replication which didn't really fully work. - Most programmers don't really understand the ins-and-outs of maintaining consistency.

marktangotango 11y ago

>> Many large apps are significantly de-normalized and have many foreign-key less tables, often filled up by scheduled jobs. That's fine for most apps; NoSQL architectures recognize that and users consider that in design. There's a boat load of assumed knowledge in this quote, how likely is it that someone not familiar with rdbms would know what a foreign key is, for example? Not very likely I think. I posit you give developers too much credit.

nickpsecurity 11y ago

In startup mode, I expect companies to do whatever works. Google's initial decisions were correct. Later, they dedicated expensive teams to inventing brand new solutions for about every aspect of their operation. That's when my critique apples. For instance, they might have looked at expensive NonStop, identified what they could copy with COTS tech, and made a cheap knockoff. They could've done the same with prior MPP or cluster tech which already scaled to thousands of nodes with open source management tools and microsecond latency. Much more efficient than web stacks, too. The one example I saw of them copying and improving past methods was Google File System. As I predicted, that good choice laid foundation for awesome stuff such as BigTable, Spanner, and F1. NoSQL movement's origins are highly debatable. Here's what I saw in its beginning: an explosion of articles on the subject after a few success stories about big companies doing massive scale on cheap servers. Zealots argued that mature, strong-consistency solutions had problems. So, instead of fixing those, we should just ditch them and strong consistency for The New, Great Thing. The only time I even saw a real analysis of cost-benefits was a few articles which focused on a narrow class of applications where data consistency didn't really matter. So, my conclusion was that the movement was two things: 95% a social phenomenon that happens with each IT fad; 5% a result of weaknesses of relational model and tools. That last part makes sense and is why I've always opposed RDBMS's. On your last point, do you have a citation that shows weak consistency databases offer enough integrity for "majority of use-cases?" I thought the majority of use cases for databases were protecting data that's important to a business: orders, billing, HR, inventory, customer information, and so on. I usually just tell them to use PostgresSQL and that covers it with high reliability. If you really can back your claim, though, I'll be glad to start transitioning my R&D toward designs that trade against integrity of data.