7 ms·

You could also argue that the 20th century relational database is a terrible way to model information as it is so different than the way we humans actually do.

You could also argue that the 20th century relational database is a terrible way to model information as it is so different than the way we humans actually do. Hierarchies are a key to information modeling for us, but a disaster with relational. We do not store information in countless tables with increasingly akward joins between them, we think in terms of relations between things and their hierarchies of relations. One thing is not in 20 different places referenced via a key id, its just in one place mentally linked in relation to many others. For this reason, the noSql databases, like Apache Solr etc (which are open source at no cost), are far superior to actually capturing information in a more human way, and doing it at far bigger scale and faster. So I would say the last thing you would want to start with on any modern knowledge/information system would be SQL.

eska 4y ago
Yes, humans think in hierarchies, but they think in multiple hierarchies. You’re in one hierarchy at work, another in your family, another at your Karate dojo. This cannot be represented in a single hierarchy. In non-relational you therefore end up duplicating data and suffer the associated ACID and performance issues. The only reason people are fine with it is because they don’t care about data integrity and won’t fix these problems in hindsight, but that’s what the non-relational database developers tell them to do since they declare it’s not their responsibility.
simonw 4y ago
What you're saying sounds good on paper, but I don't think real-world experience backs it up. I've not seen a non-relational system really have sticking power for human knowledge representation yet, despite many people making many attempts at it. Meanwhile, most SQL databases include excellent support for JSON column types now, if you want to mix in some data that doesn't fit neatly into rows and columns.
- gibsonf1 4y ago
  We are using it for our conceptual AI with space-time digital twin, and so are many others: https://solr.apache.org/community.html#powered-by https://solr.apache.org/community.html#powered-by
  - simonw 4y ago
    I've used Solr and Elasticsearch to build all sorts of things over my career, but I've rarely used them as the central point of truth for my data. Instead, I keep my data in a relational database and denormalize aspects of it out to them in order to answer queries that aren't a good fit for my database: https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/ https://2017.djangocon.us/talks/the-denormalized-query-engin...
    - gibsonf1 4y ago
      And why is that, what do you get out of relational in that case that Solr doesn't give you? The Solr boolean query interface is very impressive and very fast. The caveat is you need a good essentialized schema.
      - simonw 4y ago
        Transactions and joins.
        gibsonf1 4y ago
        I would call joins a bug of relational rather than a feature. If you have all the data in one collection, you don't need or want joins. For transactions, we do that server side (in our own system) as we have what we call temporary awareness created for each request that only writes when all operations have completed successfully before importing the state changes into primary awareness for that user.
        simonw 4y ago
        Every large system I've ever encountered that's built on a NoSQL system such as MongoDB or Elasticsearch has inevitably ended up reinventing joins, poorly.
        gibsonf1 4y ago
woooooo 4y ago
We all did this from like 2008-2015. You wind up defining relationships in code instead, across dozens of API calls without a central spec and it's super easy to get wrong. If it were as easy as you say we'd all be using MongoDB now.
- gibsonf1 4y ago
  That would be a different approach than us as our entire schema consists of 4 doc types with 14 keys/functons which models the universe, and then we use specifications in the data to control inference and system operations etc. So it's becoming a no-code system powered by the specs captured in the data.
  - woooooo 4y ago
    If your business domain is document-centric without a lot of strong foreign key relationships between them, then you do you. Just saying other domains can run into "denormalization hell" and "super complicated API traversal".
    - gibsonf1 4y ago
      We do both documents, the internals of documents and a full model of space-time with our digital twin technology, but we've spent decades arriving at a extremely essentialized lower ontology schema with which we can model everything - thats really the secret to it.
- tracker1 4y ago
  I tend to think of a classifieds site as pretty much the ideal case for a typical nosql approach... especially Mongo being a great fit. Today, I might use PostgreSQL/Cockroach with some fixed and a JSON column for extra bits based on type of classified. Not everything fits into this mold, it's a matter of mix/match today, and there are no clear rules for how you get into scaling as needed...
indymike 4y ago
The whole no-sql era was caused by various performance issues that made the SQL databases of that era unusable, or difficult to work with for certain kinds of workloads and data needs. As SQL databases evolved to support these workloads with features like JSONfields, the need for highly specialized database servers is less than it was, but there are certainly places where noSQLs shine. Cache and data structure servers, schemaless for persistence where everything is defined in frontend code, huge scale, etc... > the noSql databases, like Apache Solr SOLR (and Elastic Search) is a search engine, and is not a database, and should never, not be confused as one. > We do not store information in countless tables with increasingly akward joins between them, The whole "no joins" thing was really silly as the first thing that you do when working with a noSQL is write code like this parent=someThing.get(someObject); children=someObject.get(childrenMatchingCriteria); Which is a join. What Mongo and other noSQLs did was ship a reasonably performant and more importantly, easy for the developer way to store JSON objects. I spent a good number of years building on CouchDB, Firebase, DynamoDB and Mongo because it was easy to go fast, especially where the back end was just storing settings objects... Ironically, every project eventually needed two things that noSQL was supposed to not need: schemas and joins... so we implemented them in code (often in a React, Angular or Vue frontend).
- gibsonf1 4y ago
  The primary issue is if you are trying to model information the way humans use it and need it, relational tables are just not even close, hierarchies are a disaster, etc. We humans don't have/use tables mentally.
  - simonw 4y ago
    I just don't think that's true. Humans think in terms of categories of thing. The reason things live in the same category is that they share the same attributes. The relational model works really well for this. I'm ready to be convinced otherwise, but I'm going to need some really good examples of "human data" that fits NoSQL systems but can't be reasonably represented relationally.
    - 4y ago
crazygringo 4y ago
Funny, to me relational databases seem to fit human thinking much better. Hierarchies aren't a disaster with relational, they're trivial. A table maps product attributes to products, products to orders, orders to customers. But relational allows you to escape the limitations of strict hierarchy. Because another table also maps products to suppliers, and orders to shipments, and orders to customer satisfaction surveys. People think with hierarchies yes, but those hierarchies are built out of relations. I'd actually say we think in relations, and that some of those relations can be expressed as hierarchies, but others (sometimes most) can't at all. If you're using "countless tables" then you might have a problem with your modeling. And if you're using "increasingly awkward joins" you might be writing your joins wrong, or again modeling the data in a confusing way. Or you just have a prejudice against joins -- they're no more "awkward" than pointers in C. Database normal forms exist for a reason, because they ensure you're storing data in the way relational databases are designed to work with it. I'd say the first thing you should start with on any modern knowledge/information system is SQL, and only migrate away from that if you're absolutely sure it's necessary.

eska 4y ago

Yes, humans think in hierarchies, but they think in multiple hierarchies. You’re in one hierarchy at work, another in your family, another at your Karate dojo. This cannot be represented in a single hierarchy. In non-relational you therefore end up duplicating data and suffer the associated ACID and performance issues. The only reason people are fine with it is because they don’t care about data integrity and won’t fix these problems in hindsight, but that’s what the non-relational database developers tell them to do since they declare it’s not their responsibility.

simonw 4y ago

What you're saying sounds good on paper, but I don't think real-world experience backs it up. I've not seen a non-relational system really have sticking power for human knowledge representation yet, despite many people making many attempts at it. Meanwhile, most SQL databases include excellent support for JSON column types now, if you want to mix in some data that doesn't fit neatly into rows and columns.

gibsonf1 4y ago
We are using it for our conceptual AI with space-time digital twin, and so are many others: https://solr.apache.org/community.html#powered-by https://solr.apache.org/community.html#powered-by
- simonw 4y ago
  I've used Solr and Elasticsearch to build all sorts of things over my career, but I've rarely used them as the central point of truth for my data. Instead, I keep my data in a relational database and denormalize aspects of it out to them in order to answer queries that aren't a good fit for my database: https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/ https://2017.djangocon.us/talks/the-denormalized-query-engin...
  - gibsonf1 4y ago
    And why is that, what do you get out of relational in that case that Solr doesn't give you? The Solr boolean query interface is very impressive and very fast. The caveat is you need a good essentialized schema.
    - simonw 4y ago
      Transactions and joins.
      - gibsonf1 4y ago
        I would call joins a bug of relational rather than a feature. If you have all the data in one collection, you don't need or want joins. For transactions, we do that server side (in our own system) as we have what we call temporary awareness created for each request that only writes when all operations have completed successfully before importing the state changes into primary awareness for that user.
        simonw 4y ago
        Every large system I've ever encountered that's built on a NoSQL system such as MongoDB or Elasticsearch has inevitably ended up reinventing joins, poorly.
        gibsonf1 4y ago

woooooo 4y ago

We all did this from like 2008-2015. You wind up defining relationships in code instead, across dozens of API calls without a central spec and it's super easy to get wrong. If it were as easy as you say we'd all be using MongoDB now.

gibsonf1 4y ago
That would be a different approach than us as our entire schema consists of 4 doc types with 14 keys/functons which models the universe, and then we use specifications in the data to control inference and system operations etc. So it's becoming a no-code system powered by the specs captured in the data.
- woooooo 4y ago
  If your business domain is document-centric without a lot of strong foreign key relationships between them, then you do you. Just saying other domains can run into "denormalization hell" and "super complicated API traversal".
  - gibsonf1 4y ago
    We do both documents, the internals of documents and a full model of space-time with our digital twin technology, but we've spent decades arriving at a extremely essentialized lower ontology schema with which we can model everything - thats really the secret to it.
tracker1 4y ago
I tend to think of a classifieds site as pretty much the ideal case for a typical nosql approach... especially Mongo being a great fit. Today, I might use PostgreSQL/Cockroach with some fixed and a JSON column for extra bits based on type of classified. Not everything fits into this mold, it's a matter of mix/match today, and there are no clear rules for how you get into scaling as needed...

indymike 4y ago

The whole no-sql era was caused by various performance issues that made the SQL databases of that era unusable, or difficult to work with for certain kinds of workloads and data needs. As SQL databases evolved to support these workloads with features like JSONfields, the need for highly specialized database servers is less than it was, but there are certainly places where noSQLs shine. Cache and data structure servers, schemaless for persistence where everything is defined in frontend code, huge scale, etc... > the noSql databases, like Apache Solr SOLR (and Elastic Search) is a search engine, and is not a database, and should never, not be confused as one. > We do not store information in countless tables with increasingly akward joins between them, The whole "no joins" thing was really silly as the first thing that you do when working with a noSQL is write code like this parent=someThing.get(someObject); children=someObject.get(childrenMatchingCriteria); Which is a join. What Mongo and other noSQLs did was ship a reasonably performant and more importantly, easy for the developer way to store JSON objects. I spent a good number of years building on CouchDB, Firebase, DynamoDB and Mongo because it was easy to go fast, especially where the back end was just storing settings objects... Ironically, every project eventually needed two things that noSQL was supposed to not need: schemas and joins... so we implemented them in code (often in a React, Angular or Vue frontend).

gibsonf1 4y ago
The primary issue is if you are trying to model information the way humans use it and need it, relational tables are just not even close, hierarchies are a disaster, etc. We humans don't have/use tables mentally.
- simonw 4y ago
  I just don't think that's true. Humans think in terms of categories of thing. The reason things live in the same category is that they share the same attributes. The relational model works really well for this. I'm ready to be convinced otherwise, but I'm going to need some really good examples of "human data" that fits NoSQL systems but can't be reasonably represented relationally.
  - 4y ago

crazygringo 4y ago

Funny, to me relational databases seem to fit human thinking much better. Hierarchies aren't a disaster with relational, they're trivial. A table maps product attributes to products, products to orders, orders to customers. But relational allows you to escape the limitations of strict hierarchy. Because another table also maps products to suppliers, and orders to shipments, and orders to customer satisfaction surveys. People think with hierarchies yes, but those hierarchies are built out of relations. I'd actually say we think in relations, and that some of those relations can be expressed as hierarchies, but others (sometimes most) can't at all. If you're using "countless tables" then you might have a problem with your modeling. And if you're using "increasingly awkward joins" you might be writing your joins wrong, or again modeling the data in a confusing way. Or you just have a prejudice against joins -- they're no more "awkward" than pointers in C. Database normal forms exist for a reason, because they ensure you're storing data in the way relational databases are designed to work with it. I'd say the first thing you should start with on any modern knowledge/information system is SQL, and only migrate away from that if you're absolutely sure it's necessary.