7 ms·

> NoSQL came into existence because the databases at the time couldn’t handle the scale required. Arguably the first "NoSQL" database was a DIT, generally acce

> NoSQL came into existence because the databases at the time couldn’t handle the scale required. Arguably the first "NoSQL" database was a DIT, generally accessed via the LDAP protocol. OpenLDAP is one of the better known open source instances, but Novell eDirectory was there in the early part of this century, as was MS AD. In the case of Directory Services, it's a completely different approach to both SQL and the common "Document Store" approach of modern "NoSQL" options. Unless you go cowboy-mode, you're still adhering to schemas (albeit with the option to combine schemas), you still use indexed attributes to conduct searches, but you have a literal tree of objects with a more flexible layout (e.g. multi-value attributes without needing to use a JSON column) than with a regular SQL RDBMS. The rise in popularity of 'modern' NoSQL options (most commonly document stores) is IMO driven by a combination of lazy developers and too much Kool-aid consumption.

logophobia 8y ago
I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql. There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.
- exclusiv 8y ago
  It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch. ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer. I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.
  - logophobia 8y ago
    I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that: * Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..) * "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema. * It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.
    - exclusiv 8y ago
      Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves. Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course. Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO. Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.
- threeseed 8y ago
  Not sure what you are talking about here. There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL). And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.
  - logophobia 8y ago
    Cassandra is AP (available and partition tolerant), with tunable consistency. HBase is CP (consistent and partition tolerant, but not always available in case of network partitions). Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new. I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications. Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.
    - qaq 8y ago
      Spanner is CP I think the A part comes from reliability of google network infrastructure which is not part of DB system's design per se.
      - logophobia 8y ago
        I think I misunderstood that yes. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-.... TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available). I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.
- 189723954 8y ago
  >don't want to take the time to properly learn and optimize sql. This sounds like an old-wives tail at this point.
  - logophobia 8y ago
    I've met developers like that. Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive. Not all developers are formally schooled, a lot of people don't know what a relational database model is. I've seen people: * Iterate over entire tables to get to a small number of records * Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries) * Denormalize everything, and then wonder why things are inconsistent and hard to query
    - sqlcook 8y ago
      It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards. From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.
      - matwood 8y ago
        > It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.
mikekchar 8y ago
There were NoSQL databases well before LDAP was a thing. We called them "Object databases". The idea was that doing object relational mapping was a PITA and wouldn't it be better to store your domain objects directly in the database? (As an aside, in case you are curious, the answer is: No. Generally it's not a good idea) Of course if your data is not relational, you can't do SQL. :-)
- pjmlp 8y ago
  Memories of POET and Gemstone spring to mind...
- dragonwriter 8y ago
  > Of course if your data is not relational, you can't do SQL. The relational model is, as I understand it, fully general, so this literally cannot be the case. There is a problem of “if you do not know the shape of the data in advance” and “if the RDBMSs available to you do not efficiently handle data of the particular shape and access pattern you are using”, but “relational” isn't an inherent property of data, it's a property of a data model, and any data that can be modelled can be modelled relationally. That doesn't mean it is efficiently handled by every, or any, existing RDBMS, though.
  - threeseed 8y ago
    There are plenty of models that don't work well modelled relationally. Few examples here: (1) Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ? You can't. Since every SQL database has a tiny column limit. In Cassandra that is a trivial O(1) fetch. (2) Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records. (3) Star schema with a fact table and thousands of dimensions. In a SQL database that is thousand joins. In a JSON document store those dimensions can be embedded as well as made available as relations and again that is an O(1) fetch.
    - davidgould 8y ago
threeseed 8y ago
So the developers at Google (BigTable), Facebook (Cassandra), Yahoo (HBase), Apple (FoundationDB), Redis etc are "lazy" and have drunk "too much Kool-aid". Because these technologies every day power the most demanding applications on the planet.
- mmt 8y ago
  I think the accusation was against the developers who caused the rise in popularity (i.e. the users), not the developers who created the technologies (i.e. the authors). In that light, it might be best to exclude BigTable, since it's only available publically as a service.
  - threeseed 8y ago
    Facebook, Yahoo, Google etc built these databases to use not as coding exercises. So are they lazy and drinking the kool aid for not relying on SQL databases instead of building their own ? Or maybe SQL isn't perfect for every use case and other developers should stop arrogantly belittling those that choose to architect their own systems a particular way.
    - stephenr 8y ago
      I'd say that 99.9% of organisations using any database technology, don't have the scaling problems of Google, Yahoo, Facebook or Apple. They objectively can't be "drinking the koolaid" or cargoculting something, because they've specifically created their own solutions, to solve their own problems. I specifically didn't say SQL is perfect - I said that they're very popular for the wrong reasons, and often are used in situations that a relational database would be a better solution.
      - threeseed 8y ago
        So it's okay for developers to build their own database for their specific use case. But it's not okay for developers to use a database for their specific use case. Makes sense.
189723954 8y ago
>lazy developers and too much Kool-aid consumption. When your argument is based on personal attacks of the people using that thing, it sounds more like you have an emotional investment in the alternative rather than a rational argument.
- qaq 8y ago
  It is a very rational argument people want to copy what Big boys are doing often fairly rationally as they want that stuff on the resume. I've seen countless projects using "BIG data" tools for dealing with 100GB or less of data that has very modest grows rate.

logophobia 8y ago

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql. There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.

exclusiv 8y ago
It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch. ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer. I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.
- logophobia 8y ago
  I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that: * Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..) * "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema. * It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.
  - exclusiv 8y ago
    Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves. Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course. Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO. Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.
threeseed 8y ago
Not sure what you are talking about here. There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL). And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.
- logophobia 8y ago
  Cassandra is AP (available and partition tolerant), with tunable consistency. HBase is CP (consistent and partition tolerant, but not always available in case of network partitions). Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new. I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications. Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.
  - qaq 8y ago
    Spanner is CP I think the A part comes from reliability of google network infrastructure which is not part of DB system's design per se.
    - logophobia 8y ago
      I think I misunderstood that yes. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-.... TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available). I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.
189723954 8y ago
>don't want to take the time to properly learn and optimize sql. This sounds like an old-wives tail at this point.
- logophobia 8y ago
  I've met developers like that. Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive. Not all developers are formally schooled, a lot of people don't know what a relational database model is. I've seen people: * Iterate over entire tables to get to a small number of records * Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries) * Denormalize everything, and then wonder why things are inconsistent and hard to query
  - sqlcook 8y ago
    It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards. From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.
    - matwood 8y ago
      > It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.

mikekchar 8y ago

There were NoSQL databases well before LDAP was a thing. We called them "Object databases". The idea was that doing object relational mapping was a PITA and wouldn't it be better to store your domain objects directly in the database? (As an aside, in case you are curious, the answer is: No. Generally it's not a good idea) Of course if your data is not relational, you can't do SQL. :-)

pjmlp 8y ago
Memories of POET and Gemstone spring to mind...
dragonwriter 8y ago
> Of course if your data is not relational, you can't do SQL. The relational model is, as I understand it, fully general, so this literally cannot be the case. There is a problem of “if you do not know the shape of the data in advance” and “if the RDBMSs available to you do not efficiently handle data of the particular shape and access pattern you are using”, but “relational” isn't an inherent property of data, it's a property of a data model, and any data that can be modelled can be modelled relationally. That doesn't mean it is efficiently handled by every, or any, existing RDBMS, though.
- threeseed 8y ago
  There are plenty of models that don't work well modelled relationally. Few examples here: (1) Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ? You can't. Since every SQL database has a tiny column limit. In Cassandra that is a trivial O(1) fetch. (2) Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records. (3) Star schema with a fact table and thousands of dimensions. In a SQL database that is thousand joins. In a JSON document store those dimensions can be embedded as well as made available as relations and again that is an O(1) fetch.
  - davidgould 8y ago

threeseed 8y ago

So the developers at Google (BigTable), Facebook (Cassandra), Yahoo (HBase), Apple (FoundationDB), Redis etc are "lazy" and have drunk "too much Kool-aid". Because these technologies every day power the most demanding applications on the planet.

mmt 8y ago
I think the accusation was against the developers who caused the rise in popularity (i.e. the users), not the developers who created the technologies (i.e. the authors). In that light, it might be best to exclude BigTable, since it's only available publically as a service.
- threeseed 8y ago
  Facebook, Yahoo, Google etc built these databases to use not as coding exercises. So are they lazy and drinking the kool aid for not relying on SQL databases instead of building their own ? Or maybe SQL isn't perfect for every use case and other developers should stop arrogantly belittling those that choose to architect their own systems a particular way.
  - stephenr 8y ago
    I'd say that 99.9% of organisations using any database technology, don't have the scaling problems of Google, Yahoo, Facebook or Apple. They objectively can't be "drinking the koolaid" or cargoculting something, because they've specifically created their own solutions, to solve their own problems. I specifically didn't say SQL is perfect - I said that they're very popular for the wrong reasons, and often are used in situations that a relational database would be a better solution.
    - threeseed 8y ago
      So it's okay for developers to build their own database for their specific use case. But it's not okay for developers to use a database for their specific use case. Makes sense.

189723954 8y ago

>lazy developers and too much Kool-aid consumption. When your argument is based on personal attacks of the people using that thing, it sounds more like you have an emotional investment in the alternative rather than a rational argument.

qaq 8y ago
It is a very rational argument people want to copy what Big boys are doing often fairly rationally as they want that stuff on the resume. I've seen countless projects using "BIG data" tools for dealing with 100GB or less of data that has very modest grows rate.