8 ms·

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by develope

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql. There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.

exclusiv 8y ago
It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch. ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer. I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.
- logophobia 8y ago
  I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that: * Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..) * "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema. * It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.
  - exclusiv 8y ago
    Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves. Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course. Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO. Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.
threeseed 8y ago
Not sure what you are talking about here. There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL). And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.
- logophobia 8y ago
  Cassandra is AP (available and partition tolerant), with tunable consistency. HBase is CP (consistent and partition tolerant, but not always available in case of network partitions). Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new. I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications. Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.
  - qaq 8y ago
    Spanner is CP I think the A part comes from reliability of google network infrastructure which is not part of DB system's design per se.
    - logophobia 8y ago
      I think I misunderstood that yes. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-.... TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available). I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.
189723954 8y ago
>don't want to take the time to properly learn and optimize sql. This sounds like an old-wives tail at this point.
- logophobia 8y ago
  I've met developers like that. Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive. Not all developers are formally schooled, a lot of people don't know what a relational database model is. I've seen people: * Iterate over entire tables to get to a small number of records * Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries) * Denormalize everything, and then wonder why things are inconsistent and hard to query
  - sqlcook 8y ago
    It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards. From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.
    - matwood 8y ago
      > It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.

exclusiv 8y ago

It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch. ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer. I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.

logophobia 8y ago
I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that: * Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..) * "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema. * It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.
- exclusiv 8y ago
  Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves. Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course. Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO. Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.

threeseed 8y ago

Not sure what you are talking about here. There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL). And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.

logophobia 8y ago
Cassandra is AP (available and partition tolerant), with tunable consistency. HBase is CP (consistent and partition tolerant, but not always available in case of network partitions). Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new. I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications. Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.
- qaq 8y ago
  Spanner is CP I think the A part comes from reliability of google network infrastructure which is not part of DB system's design per se.
  - logophobia 8y ago
    I think I misunderstood that yes. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-.... TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available). I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.

189723954 8y ago

>don't want to take the time to properly learn and optimize sql. This sounds like an old-wives tail at this point.

logophobia 8y ago
I've met developers like that. Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive. Not all developers are formally schooled, a lot of people don't know what a relational database model is. I've seen people: * Iterate over entire tables to get to a small number of records * Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries) * Denormalize everything, and then wonder why things are inconsistent and hard to query
- sqlcook 8y ago
  It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards. From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.
  - matwood 8y ago
    > It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.