6 ms·

I think the issue isn't SQL, but the table paradigm for storing data. Humans do not store data in separate tables that need joining, they store data in a fully

I think the issue isn't SQL, but the table paradigm for storing data. Humans do not store data in separate tables that need joining, they store data in a fully connected graph (hyper-graph). Its about relationships and hierarchies - the graph allows incredibly fast hierarchical reasoning, as most things involve hierarchical reasoning. The relational table, and Sql by correlation, are terrible at the human approach to working with data especially hierarchies.

warrenm 4y ago
I work extensively with Splunk which is dominantly based on noSQL underneath (MongoDB, among other, proprietary technologies) I've also recently been [re]introduced to graph databases (which are highly similar to the pre-relational network database paradigm) You can simulate graph relationships with an RDBMS or noSQL - but you shouldn't You can simulate an RDBMS with a graph db or noSQL - but you shouldn't You can simulate noSQL with graph and RDBMS tools - but, again, you shouldn't They all have their place - and ca even work quite well together, if you use them as they're intended
- gibsonf1 4y ago
  But thats just it, having many silos for connected information is just not the way we humans do it, nor necessary. A single conceptually indexed space/time hypergraph (as we humans do it) for any part of the world is all you need, scalable up to many billions of edges for that one part of the world.
  - tremon 4y ago
    having many silos for connected information is just not the way we humans do it What do you mean? I can think of many silos containing connected information: municipal residence records, marriage registries, birth records, police reports, tax records, medical records (for each hospital) are all silos connected by a citizen's identity.
    - gibsonf1 4y ago
      You missed the "humans" part of my statement. We humans mentally have a fully linked graph (hypergraph) of what we know through space-time and conceptually indexed. We do not have a whole bunch of disconnected data silos in our minds. Modelling the very successful human approach to data and reality modeling may make more sense then using these 50 year old tables and disconnected silos?
- johannes1234321 4y ago
  > You can simulate noSQL with graph and RDBMS tools - but, again, you shouldn't What is the feature which makes the noSQL which you shouldn't do in a relational database? To me noSQL always looks like a subset of relational database. The only thing, maybe, is that you can truly put everything in, but with modern JSON features and all the other things I don't see a downside in using relational. (Except a little learning curve, while that can be hidden behind some ORM or something if you really want for the beginning)
  - warrenm 4y ago
    The "schema on the fly" is the draw for noSQL - nothing has to be rigidly predefined like an RDBMS mandates
    - johannes1234321 4y ago
      CREATE TABLE foo (doc JSON); done. If that's really what you want.
      - warrenm 4y ago
        For every bit of JSON that comes in? No And then having to figure out what all the fields are after the fact? No That's why noSQL exists
        johannes1234321 4y ago
        What is the thin noSQL brings to the table which a relational system doesn't have? (Except that writing a depth search in a JSON document in SQL is a bit more cumbersome, but if that is a concern that can be wrapped in a data access library to generate the SQL)
        warrenm 4y ago
        I think I've explained that above - it's not having to be rigidly tied to a schema :)
gfody 4y ago
sql doesn't specify how data is stored, it's only a description of the data you want. how your data is connected or not at rest is up to the engine and how you leverage its engine-specific features.
- xpe 4y ago
  Sort of. "Does not specify" does not mean there is no connection between logical and physical layers. The relational model has strong implications for what a storage engine should do well.
  - gfody 4y ago
    > The relational model has strong implications for what a storage engine should do well. agreed, the implication is that the engine should do everything well. rdbms is a product of being pulled in every direction, there's generally an index strategy for everything including fully connected hyper graphs, and that's how they ought to be because we can write wildly complex recursive sql queries and the goal is an efficient ideally optimal execution plan
    - xpe 4y ago
      > agreed, the implication is that the engine should do everything well. I don't think you mean "everything, equally well". But if you mean that database vendors feel pressure to have their "RDBMS" be able to "be competitive" across a wide variety of use cases and metrics, I tend to agree.
Shorel 4y ago
It's the other way around. We use tables because they help us to massively simplify the processing of data. Do you want to deal with graphs? The algorithms involved are harder than what your intuition is telling you. In fact, the first databases, before relational databases were in vogue, were hierarchical/network databases. https://en.wikipedia.org/wiki/CODASYL https://en.wikipedia.org/wiki/CODASYL They were obsoleted and replaced when RDMS appeared.
- xpe 4y ago
  You are talking past the other person... too much certainty, too few questions. CODASYL is probably not remotely close to what the GP is talking about w.r.t. how humans think in graphs.
xpe 4y ago
There are more recent aspects of SQL that do hierarchical processing well. Graph processing is often decent depending on the particular DBMS. The syntax and conceptualization of these queries is not as well socialized though.
xpe 4y ago
Do you think computers should try to store data like humans? Why? Perhaps you don't mean storage but rather a conceptual model?
xpe 4y ago
"Most things involve hierarchical reasoning" begs the question: what is the universe of processing you assume? Many would counter by saying basic index lookups comprise a vast majority of DB processing time.

warrenm 4y ago

I work extensively with Splunk which is dominantly based on noSQL underneath (MongoDB, among other, proprietary technologies) I've also recently been [re]introduced to graph databases (which are highly similar to the pre-relational network database paradigm) You can simulate graph relationships with an RDBMS or noSQL - but you shouldn't You can simulate an RDBMS with a graph db or noSQL - but you shouldn't You can simulate noSQL with graph and RDBMS tools - but, again, you shouldn't They all have their place - and ca even work quite well together, if you use them as they're intended

gibsonf1 4y ago
But thats just it, having many silos for connected information is just not the way we humans do it, nor necessary. A single conceptually indexed space/time hypergraph (as we humans do it) for any part of the world is all you need, scalable up to many billions of edges for that one part of the world.
- tremon 4y ago
  having many silos for connected information is just not the way we humans do it What do you mean? I can think of many silos containing connected information: municipal residence records, marriage registries, birth records, police reports, tax records, medical records (for each hospital) are all silos connected by a citizen's identity.
  - gibsonf1 4y ago
    You missed the "humans" part of my statement. We humans mentally have a fully linked graph (hypergraph) of what we know through space-time and conceptually indexed. We do not have a whole bunch of disconnected data silos in our minds. Modelling the very successful human approach to data and reality modeling may make more sense then using these 50 year old tables and disconnected silos?
johannes1234321 4y ago
> You can simulate noSQL with graph and RDBMS tools - but, again, you shouldn't What is the feature which makes the noSQL which you shouldn't do in a relational database? To me noSQL always looks like a subset of relational database. The only thing, maybe, is that you can truly put everything in, but with modern JSON features and all the other things I don't see a downside in using relational. (Except a little learning curve, while that can be hidden behind some ORM or something if you really want for the beginning)
- warrenm 4y ago
  The "schema on the fly" is the draw for noSQL - nothing has to be rigidly predefined like an RDBMS mandates
  - johannes1234321 4y ago
    CREATE TABLE foo (doc JSON); done. If that's really what you want.
    - warrenm 4y ago
      For every bit of JSON that comes in? No And then having to figure out what all the fields are after the fact? No That's why noSQL exists
      - johannes1234321 4y ago
        What is the thin noSQL brings to the table which a relational system doesn't have? (Except that writing a depth search in a JSON document in SQL is a bit more cumbersome, but if that is a concern that can be wrapped in a data access library to generate the SQL)
        warrenm 4y ago
        I think I've explained that above - it's not having to be rigidly tied to a schema :)

gfody 4y ago

sql doesn't specify how data is stored, it's only a description of the data you want. how your data is connected or not at rest is up to the engine and how you leverage its engine-specific features.

xpe 4y ago
Sort of. "Does not specify" does not mean there is no connection between logical and physical layers. The relational model has strong implications for what a storage engine should do well.
- gfody 4y ago
  > The relational model has strong implications for what a storage engine should do well. agreed, the implication is that the engine should do everything well. rdbms is a product of being pulled in every direction, there's generally an index strategy for everything including fully connected hyper graphs, and that's how they ought to be because we can write wildly complex recursive sql queries and the goal is an efficient ideally optimal execution plan
  - xpe 4y ago
    > agreed, the implication is that the engine should do everything well. I don't think you mean "everything, equally well". But if you mean that database vendors feel pressure to have their "RDBMS" be able to "be competitive" across a wide variety of use cases and metrics, I tend to agree.

Shorel 4y ago

It's the other way around. We use tables because they help us to massively simplify the processing of data. Do you want to deal with graphs? The algorithms involved are harder than what your intuition is telling you. In fact, the first databases, before relational databases were in vogue, were hierarchical/network databases. https://en.wikipedia.org/wiki/CODASYL https://en.wikipedia.org/wiki/CODASYL They were obsoleted and replaced when RDMS appeared.

xpe 4y ago
You are talking past the other person... too much certainty, too few questions. CODASYL is probably not remotely close to what the GP is talking about w.r.t. how humans think in graphs.

xpe 4y ago

There are more recent aspects of SQL that do hierarchical processing well. Graph processing is often decent depending on the particular DBMS. The syntax and conceptualization of these queries is not as well socialized though.

xpe 4y ago

Do you think computers should try to store data like humans? Why? Perhaps you don't mean storage but rather a conceptual model?

xpe 4y ago

"Most things involve hierarchical reasoning" begs the question: what is the universe of processing you assume? Many would counter by saying basic index lookups comprise a vast majority of DB processing time.