4 ms·

So you have to use sharding because your databases have limited capacity. But you can just join in the app, because the app have unlimited memory?

by olavk 8y ago

EGreg 8y ago
The app doesn’t hold the entire database at a time. The app simply does the following: 1) Get the root record(s) from id(s) 2) See what related records it needs, combine them into a list of ids, partition list by shard 3) Ask each shard for the corresponding records 4) Repeat from 2 if necessary 5) Return this whole tree / graph to the user Graph databases can do this in O(1) instead of O(log N) lookups. Relational joins are just one way to achieve this, which can be made atomic in the ACID sense. However, as you scale up your website, eg with 100,000,000 users, it would be silly to do massive joins. Google even says this in their docs now, for BigQuery. Instead, design your systems from the beginnig to be as parallel as possible, if you think they will scale. Look at the problems with Ethereum for example. Or Twitter fail whales of the past.
- olavk 8y ago
  A graph database basically caches related entities with the root entity, so lookups are fast when the query follows the paths of the graph. The price is that any other query is extremely costly. But if it works for your use case, more power to you. But not really relevant in a discussion about ORMs - Object Relational Mappers. The relational model was designed to address the limitations of the network/graph database, especially to allow arbitrary (ad-hoc) querying and to decouple the physical storage from the logical model. But if you don't need all that, a graph database may be fine.
  - EGreg 8y ago
    Most applications don’t need that. But if they do, just have indexes!
- dmitriid 8y ago
  Acolyer again: https://blog.acolyer.org/2017/07/07/do-we-need-specialized-graph-databases-benchmarking-real-time-social-networking-applications/ https://blog.acolyer.org/2017/07/07/do-we-need-specialized-g... For the vast majority of use cases regular SQL databases blow graph databases out of the water. Unless you go for graph-specific algorithms like Shortest Path, and even then...

So you have to use sharding because your databases have limited capacity. But you can just join in the app, because the app have unlimited memory?

by olavk 8y ago

So you have to use sharding because your databases have limited capacity. But you can just join in the app, because the app have unlimited memory?

EGreg 8y ago

The app doesn’t hold the entire database at a time. The app simply does the following: 1) Get the root record(s) from id(s) 2) See what related records it needs, combine them into a list of ids, partition list by shard 3) Ask each shard for the corresponding records 4) Repeat from 2 if necessary 5) Return this whole tree / graph to the user Graph databases can do this in O(1) instead of O(log N) lookups. Relational joins are just one way to achieve this, which can be made atomic in the ACID sense. However, as you scale up your website, eg with 100,000,000 users, it would be silly to do massive joins. Google even says this in their docs now, for BigQuery. Instead, design your systems from the beginnig to be as parallel as possible, if you think they will scale. Look at the problems with Ethereum for example. Or Twitter fail whales of the past.

olavk 8y ago
A graph database basically caches related entities with the root entity, so lookups are fast when the query follows the paths of the graph. The price is that any other query is extremely costly. But if it works for your use case, more power to you. But not really relevant in a discussion about ORMs - Object Relational Mappers. The relational model was designed to address the limitations of the network/graph database, especially to allow arbitrary (ad-hoc) querying and to decouple the physical storage from the logical model. But if you don't need all that, a graph database may be fine.
- EGreg 8y ago
  Most applications don’t need that. But if they do, just have indexes!
dmitriid 8y ago
Acolyer again: https://blog.acolyer.org/2017/07/07/do-we-need-specialized-graph-databases-benchmarking-real-time-social-networking-applications/ https://blog.acolyer.org/2017/07/07/do-we-need-specialized-g... For the vast majority of use cases regular SQL databases blow graph databases out of the water. Unless you go for graph-specific algorithms like Shortest Path, and even then...