6 ms·

NoSQL Data Modeling Techniques (2012)

127 points by agmm 6y ago

SeriousM 6y ago
The article is from 2012 but still very relevant today!
elitan 6y ago
I thought this was going to be one of those oneliner posts: Use SQL Like this: https://www.amazon.com/Why-Socialism-Works-Harrison-Lievesley/dp/1521531218 https://www.amazon.com/Why-Socialism-Works-Harrison-Lievesle...
LukeEF 6y ago
To my mind, the point of databases is to provide as accurate a representation of some aspect of external reality as possible. The advantage of graph databases is that they model the world as things that have properties and relationships with other things. This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases. Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins. In this respect, even the simplest graph database such as Neo4j — which models the world as a bunch of JSON documents, some of which may contain pointers to other JSON documents, is much better than even the fanciest RDBMS. Granted there is no taxonomy or schema, and support for temporality is basic, but it’s easy to produce a much more naturalistic model of the world than will ever be possible if you have to break the things up into relations.
- alex_anglin 6y ago
  Agreed that graph databases are super useful for a variety of purposes. However, it's usually worth consideration as to whether a whole separate data store is needed for that. Postgres, for instance, has uses that would allow for this work within a RDBMS: https://postgresconf.org/conferences/2020/program/proposals/postgres-as-a-graph-database https://postgresconf.org/conferences/2020/program/proposals/...
  - dumb1224 6y ago
    Agreed. Actually even the openlink virtuoso database project is kind of using the RDBMS approach in its underlying implementation. https://en.wikipedia.org/wiki/Virtuoso_Universal_Server https://en.wikipedia.org/wiki/Virtuoso_Universal_Server However part of the power of the graph database that utilising a linked dataset such as RDFs is that you will be able to use SPARQL to query the graphs, which is more suitable for sets / graphs.
- asavinov 6y ago
  > This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases. One approach to modeling data based on mappings (mathematical functions) is the concept-oriented model [1] implemented in [2]. Its main feature is that it gets rid of joins, groupby and map-reduce by manipulating data using operations with functions (mappings). > Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins. One old related general idea is to assume the existence of universal relation. Such an approach is referred to as the universal relation model (URM) [3, 4]. [1] A. Savinov, Concept-oriented model: Modeling and processing data using functions, Eprint: arXiv:1911.07225 [cs.DB], 2019 https://www.researchgate.net/publication/337336089_Concept-oriented_model_Modeling_and_processing_data_using_functions https://www.researchgate.net/publication/337336089_Concept-o... [2] https://github.com/asavinov/prosto https://github.com/asavinov/prosto Prosto Data Processing Toolkit: No join-groupby, No map-reduce [3] https://en.wikipedia.org/wiki/Universal_relation_assumption https://en.wikipedia.org/wiki/Universal_relation_assumption [4] R. Fagin, A.O. Mendelzon and J.D. Ullman, A Simplified Universal Relation Assumption and Its Properties. ACM Trans. Database Syst., 7(3), 343-360 (1982).
- nivertech 6y ago
  The most important problem that DBMSes are solving is how not to lose data in case of a power outage. If your computer using non-volatile memory (NVRAM) then database is just a data structure. The secondary problem is how to organize and manage the data, but that is highly dependent on the workloads in question.
  - smilliken 6y ago
    Even with NVRAM, you still want to use an append-only log instead of mutable data structures in order to support streaming replication, incremental backup, concurrent writes, and triggers.
    - nivertech 6y ago
      > Even with NVRAM, you still want to use an append-only log instead of mutable data structures Nothing preventing you from implementing AOL in NVRAM > to support streaming replication, incremental backup, concurrent writes, and triggers All distributed systems stuff equally applies to both in-memory and persistent data structures
  - zffr 6y ago
    Wouldn't NVRAM only give you durability (D in ACID)? I think it might be too reductionist to say a database on NVRAM is just a data structure. Even with NVRAM you still need to maintain atomicity, consistency and isolation.
    - nivertech 6y ago
      Agree, but not every DBMS supports transactions, and even when they're supported the application code may avoid using them for various reasons.
michaelmior 6y ago
There has been a fair bit of work since this time on automated data modeling for NoSQL databases given a conceptual model and information on the workload. Some of this work is my own[0], but there has been a lot of other related activity. One thing that I've been happy to see explored more is cost modeling[1] in NoSQL databases as this makes any sort of automated reasoning much easier. I doubt we'll ever get to the point where data modeling by an expert is never needed, but I think there's a lot we can do to make things easier and more performant for novices. [0] http://ieeexplore.ieee.org/abstract/document/7967690/ http://ieeexplore.ieee.org/abstract/document/7967690/ [1] https://www.researchgate.net/publication/350366905_A_cost_model_for_random_access_queries_in_document_stores https://www.researchgate.net/publication/350366905_A_cost_mo...
kuharich 5y ago
Past comments: https://news.ycombinator.com/item?id=12733425 https://news.ycombinator.com/item?id=12733425

127 points by agmm 6y ago

The article is from 2012 but still very relevant today!

I thought this was going to be one of those oneliner posts: Use SQL Like this: https://www.amazon.com/Why-Socialism-Works-Harrison-Lievesley/dp/1521531218 https://www.amazon.com/Why-Socialism-Works-Harrison-Lievesle...

LukeEF 6y ago

To my mind, the point of databases is to provide as accurate a representation of some aspect of external reality as possible. The advantage of graph databases is that they model the world as things that have properties and relationships with other things. This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases. Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins. In this respect, even the simplest graph database such as Neo4j — which models the world as a bunch of JSON documents, some of which may contain pointers to other JSON documents, is much better than even the fanciest RDBMS. Granted there is no taxonomy or schema, and support for temporality is basic, but it’s easy to produce a much more naturalistic model of the world than will ever be possible if you have to break the things up into relations.

alex_anglin 6y ago
Agreed that graph databases are super useful for a variety of purposes. However, it's usually worth consideration as to whether a whole separate data store is needed for that. Postgres, for instance, has uses that would allow for this work within a RDBMS: https://postgresconf.org/conferences/2020/program/proposals/postgres-as-a-graph-database https://postgresconf.org/conferences/2020/program/proposals/...
- dumb1224 6y ago
  Agreed. Actually even the openlink virtuoso database project is kind of using the RDBMS approach in its underlying implementation. https://en.wikipedia.org/wiki/Virtuoso_Universal_Server https://en.wikipedia.org/wiki/Virtuoso_Universal_Server However part of the power of the graph database that utilising a linked dataset such as RDFs is that you will be able to use SPARQL to query the graphs, which is more suitable for sets / graphs.
asavinov 6y ago
> This is closer to the way that humans perceive the world — mapping between whatever aspect of external reality you are interested in and the data model is an order of magnitude easier than with relational databases. One approach to modeling data based on mappings (mathematical functions) is the concept-oriented model [1] implemented in [2]. Its main feature is that it gets rid of joins, groupby and map-reduce by manipulating data using operations with functions (mappings). > Everything is pre-joined — you don’t have to disassemble objects into normalised tables and reassemble them with joins. One old related general idea is to assume the existence of universal relation. Such an approach is referred to as the universal relation model (URM) [3, 4]. [1] A. Savinov, Concept-oriented model: Modeling and processing data using functions, Eprint: arXiv:1911.07225 [cs.DB], 2019 https://www.researchgate.net/publication/337336089_Concept-oriented_model_Modeling_and_processing_data_using_functions https://www.researchgate.net/publication/337336089_Concept-o... [2] https://github.com/asavinov/prosto https://github.com/asavinov/prosto Prosto Data Processing Toolkit: No join-groupby, No map-reduce [3] https://en.wikipedia.org/wiki/Universal_relation_assumption https://en.wikipedia.org/wiki/Universal_relation_assumption [4] R. Fagin, A.O. Mendelzon and J.D. Ullman, A Simplified Universal Relation Assumption and Its Properties. ACM Trans. Database Syst., 7(3), 343-360 (1982).
nivertech 6y ago
The most important problem that DBMSes are solving is how not to lose data in case of a power outage. If your computer using non-volatile memory (NVRAM) then database is just a data structure. The secondary problem is how to organize and manage the data, but that is highly dependent on the workloads in question.
- smilliken 6y ago
  Even with NVRAM, you still want to use an append-only log instead of mutable data structures in order to support streaming replication, incremental backup, concurrent writes, and triggers.
  - nivertech 6y ago
    > Even with NVRAM, you still want to use an append-only log instead of mutable data structures Nothing preventing you from implementing AOL in NVRAM > to support streaming replication, incremental backup, concurrent writes, and triggers All distributed systems stuff equally applies to both in-memory and persistent data structures
- zffr 6y ago
  Wouldn't NVRAM only give you durability (D in ACID)? I think it might be too reductionist to say a database on NVRAM is just a data structure. Even with NVRAM you still need to maintain atomicity, consistency and isolation.
  - nivertech 6y ago
    Agree, but not every DBMS supports transactions, and even when they're supported the application code may avoid using them for various reasons.

michaelmior 6y ago

There has been a fair bit of work since this time on automated data modeling for NoSQL databases given a conceptual model and information on the workload. Some of this work is my own[0], but there has been a lot of other related activity. One thing that I've been happy to see explored more is cost modeling[1] in NoSQL databases as this makes any sort of automated reasoning much easier. I doubt we'll ever get to the point where data modeling by an expert is never needed, but I think there's a lot we can do to make things easier and more performant for novices. [0] http://ieeexplore.ieee.org/abstract/document/7967690/ http://ieeexplore.ieee.org/abstract/document/7967690/ [1] https://www.researchgate.net/publication/350366905_A_cost_model_for_random_access_queries_in_document_stores https://www.researchgate.net/publication/350366905_A_cost_mo...

kuharich 5y ago

Past comments: https://news.ycombinator.com/item?id=12733425 https://news.ycombinator.com/item?id=12733425