6 ms·

I implemented pretty much exactly this setup once. For what it's worth, there's a lot of footguns with this approach. You need to be careful about infinite cyc

I implemented pretty much exactly this setup once. For what it's worth, there's a lot of footguns with this approach. You need to be careful about infinite cycles and things like that. You also just do not get the ergonomics of using a query language that is meant for graph data. Once you get it all setup and experience the issues you will no doubt be able to build whatever you want but there's a learning curve. In the end we switched to using Neo4j and were able to build new features a lot more quickly than we would've been able to on Postgres. It's also worth mentioning that there are many ways to store graph data, and using an "adjacency list" like is being done here may or may not be the best way for your use case. Neo4j I think uses a different approach where nodes store info about edges they are connected to directly, so you don't even need to hit an index to follow an edge.

mistermann 4y ago
Piggybacking on your Neo4J reference....can anyone suggest any resources that would be good for someone from the SQL world who is very uninformed on graph databases to quickly get their head around the key ideas, but even more importantly, how to choose a platform among the offerings (licensing being I think a key issue, I've heard Neo4J can get expensive)...assume a large, "social media" scale system, to be safe.
- viksit 4y ago
  you can use datomic as a good example. https://hashrocket.com/blog/posts/using-datomic-as-a-graph-database https://hashrocket.com/blog/posts/using-datomic-as-a-graph-d... another good example using predicates - https://github.com/threatgrid/asami/wiki/2.-Introduction https://github.com/threatgrid/asami/wiki/2.-Introduction
- joshspankit 4y ago
  As someone who hated being in the SQL world and couldn’t figure out why until years later when I found out about graph databases, here’s one big shift: Graph database store true relationships. The R in RDBMS and the concept of relationships by primary keys are lies (imo). They lead people away from what true relationships can be. Databases like Neo4j are not about doing any sort of PK or join or merge-before-query. Looking up by connection is the fastest way to find information. If your RDBMS had one table for phone numbers, one for email addresses, one for first names, one for last names, one for street, and so-on it would take huge amounts of time to query to find all the detail for a single person. With a graph database it takes a small amount of time to find the first bit of info (let’s say phone number since it’s easily sorted), and then every other bit of linked data is essentially free.
  - eurasiantiger 4y ago
    Effectively, moving from RDB to a graph database just shifts relationship resolution from query time to insert/update time. Sometimes this can be an advantage. I’d even wager usually.
    - deleted 4y ago
      [deleted]
    - bavell 4y ago
      Well said and agreed!
    - mistermann 4y ago
      This is very good news for my purposes, thank you!
    - lolive 4y ago
      It also switches you from schema-based data model to schemaless data model.
      - deleted 4y ago
        [deleted]
  - tejtm 4y ago
    Apologies in advance; the R for relational is with respect to the tuple, that is amongst the unordered items which are serialized within a row in a table and not the keys which link different tables, the lie you find does not originate in the definition. As someone who has wanted to love the graph dbs, keep with narrow focus top down queries where the index free adjacency lets you avoid touching the bulk of your data, and stay away from bulk queries that may need to visit everywhere multiple times, which is unfortunately where things get interesting.
  - WolfOliver 4y ago
    The R in RDBMS stands for the mathematical term of relations which is defined as a subset of a cartesian product.
- henryfjordan 4y ago
  Neo4j has a self-hosted community edition that is free, though you need to update to Enterprise to expand your DB beyond one node. It's worth noting though that scaling from a single node to a cluster is going to have some pretty huge performance issues whether you are using SQL or Neo4j to store your graph. The power of graph databases is that it should be very inexpensive to follow an edge in the graph but when that leads to a network hop you lose a lot of performance. If you are comfortable fitting your data onto one box, then development speed is probably more important than other factors and I would just try a few databases out and especially pay attention to how good the libraries are in your programming language of choice. Neo4j for instance had high quality libraries in Java but the experience in Python was not as good. If you have a lot of data or need next-level performance, I would start by doing some research on the various ways to store graph data and then pick a DB that supports the approach you want to take.
  - j45 4y ago
    Vaticles typed bid intriguing
- taubek 4y ago
  For some basics of graph modeling take a look at https://memgraph.com/docs/memgraph/tutorials/graph-modeling https://memgraph.com/docs/memgraph/tutorials/graph-modeling. Some differences between SQL and graph databases are covered at https://memgraph.com/blog/graph-database-vs-relational-database https://memgraph.com/blog/graph-database-vs-relational-datab....
- pottertheotter 4y ago
  I was looking at graph databases/Neo4j a lot early last year and found Neo4j has a lot of good resources. You can get this book for free through their website. [1] This have some good documentation[2], including this page "Transition from relational to graph database".[3] [1] https://neo4j.com/graph-databases-book/ https://neo4j.com/graph-databases-book/ [2] https://neo4j.com/docs/getting-started/current/ https://neo4j.com/docs/getting-started/current/ [3] https://neo4j.com/docs/getting-started/current/appendix/graphdb-concepts/graphdb-vs-rdbms/ https://neo4j.com/docs/getting-started/current/appendix/grap...
- bavell 4y ago
  I've been using EdgeDB in a greenfield project and loving it. Some rough edges (forgive the pun) but fantastic for prototyping / smaller projects and could be a good candidate for large projecs. I like it better than when I used neo4j a few years back for a client project. Much better license and I really like that it's built on top on postgres.
- deleted 4y ago
  [deleted]
- xpe 4y ago
  > how to choose a platform among the offerings (licensing being I think a key issue, I've heard Neo4J can get expensive) Safest bet (knowing nothing else about your needs): Build at least three prototypes, preferably rapidly and in parallel. Only after doing so will you be able to ask the right questions that speak accurately to your specific needs and therefore be able to map actual requirements to the different graph databases. I’d be curious what other experienced graph database people would say about their first bigger projects. Looking back, my first graph projects involved significant iteration and several restarts. And these were still necessary after having significant background knowledge about the algorithms, offerings, and trade offs. There are many people in roles/orgs that don’t have the humility or organizational cover to admit that these early projects are largely experimental learning experiences. > ...assume a large, "social media" scale system, to be safe. This broad assumption is going to cost you. Instead, I’d encourage you to reframe this assumption in more specific terms such as read, write patterns; need for consistency, availability; support expectations; your talent pool; and lots more. Estimates can be on a log10 scale: 1, 10, 100, 1000 and so on,
cryptonector 4y ago
> You need to be careful about infinite cycles and things like that. Not really, just never forget this: always use `UNION`, and never `UNION ALL`, in recursive queries!
- twic 4y ago
  There are still ways to go wrong. Try writing a query which explores a graph from a starting node, building up a path from the start to each node as a string. Even using UNION you will get an infinite loop, because every visit to a node in the loop has a different path, so it makes a unique row.
  - cryptonector 4y ago
    The aggregation needs to be done after you've computed the transitive closure, not while you're computing it.
    - blowski 4y ago
      To me, that all sounds like you need to be careful.
      - cryptonector 4y ago
        I mean, yes? This is easy to learn the hard way, but it's also easy to learn. Once learned you don't forget.
        misnome 4y ago
        You started the thread by disagreeing with “you need to be careful”, and the OP is pointing out that your thread conclusion is “you need to be careful”.
        xpe 4y ago
        Maybe, but both shared useful points along the way. Elaboration IOW. All good.

mistermann 4y ago

Piggybacking on your Neo4J reference....can anyone suggest any resources that would be good for someone from the SQL world who is very uninformed on graph databases to quickly get their head around the key ideas, but even more importantly, how to choose a platform among the offerings (licensing being I think a key issue, I've heard Neo4J can get expensive)...assume a large, "social media" scale system, to be safe.

viksit 4y ago
you can use datomic as a good example. https://hashrocket.com/blog/posts/using-datomic-as-a-graph-database https://hashrocket.com/blog/posts/using-datomic-as-a-graph-d... another good example using predicates - https://github.com/threatgrid/asami/wiki/2.-Introduction https://github.com/threatgrid/asami/wiki/2.-Introduction
joshspankit 4y ago
As someone who hated being in the SQL world and couldn’t figure out why until years later when I found out about graph databases, here’s one big shift: Graph database store true relationships. The R in RDBMS and the concept of relationships by primary keys are lies (imo). They lead people away from what true relationships can be. Databases like Neo4j are not about doing any sort of PK or join or merge-before-query. Looking up by connection is the fastest way to find information. If your RDBMS had one table for phone numbers, one for email addresses, one for first names, one for last names, one for street, and so-on it would take huge amounts of time to query to find all the detail for a single person. With a graph database it takes a small amount of time to find the first bit of info (let’s say phone number since it’s easily sorted), and then every other bit of linked data is essentially free.
- eurasiantiger 4y ago
  Effectively, moving from RDB to a graph database just shifts relationship resolution from query time to insert/update time. Sometimes this can be an advantage. I’d even wager usually.
  - deleted 4y ago
    [deleted]
  - bavell 4y ago
    Well said and agreed!
  - mistermann 4y ago
    This is very good news for my purposes, thank you!
  - lolive 4y ago
    It also switches you from schema-based data model to schemaless data model.
    - deleted 4y ago
      [deleted]
- tejtm 4y ago
  Apologies in advance; the R for relational is with respect to the tuple, that is amongst the unordered items which are serialized within a row in a table and not the keys which link different tables, the lie you find does not originate in the definition. As someone who has wanted to love the graph dbs, keep with narrow focus top down queries where the index free adjacency lets you avoid touching the bulk of your data, and stay away from bulk queries that may need to visit everywhere multiple times, which is unfortunately where things get interesting.
- WolfOliver 4y ago
  The R in RDBMS stands for the mathematical term of relations which is defined as a subset of a cartesian product.
henryfjordan 4y ago
Neo4j has a self-hosted community edition that is free, though you need to update to Enterprise to expand your DB beyond one node. It's worth noting though that scaling from a single node to a cluster is going to have some pretty huge performance issues whether you are using SQL or Neo4j to store your graph. The power of graph databases is that it should be very inexpensive to follow an edge in the graph but when that leads to a network hop you lose a lot of performance. If you are comfortable fitting your data onto one box, then development speed is probably more important than other factors and I would just try a few databases out and especially pay attention to how good the libraries are in your programming language of choice. Neo4j for instance had high quality libraries in Java but the experience in Python was not as good. If you have a lot of data or need next-level performance, I would start by doing some research on the various ways to store graph data and then pick a DB that supports the approach you want to take.
- j45 4y ago
  Vaticles typed bid intriguing
taubek 4y ago
For some basics of graph modeling take a look at https://memgraph.com/docs/memgraph/tutorials/graph-modeling https://memgraph.com/docs/memgraph/tutorials/graph-modeling. Some differences between SQL and graph databases are covered at https://memgraph.com/blog/graph-database-vs-relational-database https://memgraph.com/blog/graph-database-vs-relational-datab....
pottertheotter 4y ago
I was looking at graph databases/Neo4j a lot early last year and found Neo4j has a lot of good resources. You can get this book for free through their website. [1] This have some good documentation[2], including this page "Transition from relational to graph database".[3] [1] https://neo4j.com/graph-databases-book/ https://neo4j.com/graph-databases-book/ [2] https://neo4j.com/docs/getting-started/current/ https://neo4j.com/docs/getting-started/current/ [3] https://neo4j.com/docs/getting-started/current/appendix/graphdb-concepts/graphdb-vs-rdbms/ https://neo4j.com/docs/getting-started/current/appendix/grap...
bavell 4y ago
I've been using EdgeDB in a greenfield project and loving it. Some rough edges (forgive the pun) but fantastic for prototyping / smaller projects and could be a good candidate for large projecs. I like it better than when I used neo4j a few years back for a client project. Much better license and I really like that it's built on top on postgres.
deleted 4y ago
[deleted]
xpe 4y ago
> how to choose a platform among the offerings (licensing being I think a key issue, I've heard Neo4J can get expensive) Safest bet (knowing nothing else about your needs): Build at least three prototypes, preferably rapidly and in parallel. Only after doing so will you be able to ask the right questions that speak accurately to your specific needs and therefore be able to map actual requirements to the different graph databases. I’d be curious what other experienced graph database people would say about their first bigger projects. Looking back, my first graph projects involved significant iteration and several restarts. And these were still necessary after having significant background knowledge about the algorithms, offerings, and trade offs. There are many people in roles/orgs that don’t have the humility or organizational cover to admit that these early projects are largely experimental learning experiences. > ...assume a large, "social media" scale system, to be safe. This broad assumption is going to cost you. Instead, I’d encourage you to reframe this assumption in more specific terms such as read, write patterns; need for consistency, availability; support expectations; your talent pool; and lots more. Estimates can be on a log10 scale: 1, 10, 100, 1000 and so on,

cryptonector 4y ago

> You need to be careful about infinite cycles and things like that. Not really, just never forget this: always use `UNION`, and never `UNION ALL`, in recursive queries!

twic 4y ago
There are still ways to go wrong. Try writing a query which explores a graph from a starting node, building up a path from the start to each node as a string. Even using UNION you will get an infinite loop, because every visit to a node in the loop has a different path, so it makes a unique row.
- cryptonector 4y ago
  The aggregation needs to be done after you've computed the transitive closure, not while you're computing it.
  - blowski 4y ago
    To me, that all sounds like you need to be careful.
    - cryptonector 4y ago
      I mean, yes? This is easy to learn the hard way, but it's also easy to learn. Once learned you don't forget.
      - misnome 4y ago
        You started the thread by disagreeing with “you need to be careful”, and the OP is pointing out that your thread conclusion is “you need to be careful”.
        xpe 4y ago
        Maybe, but both shared useful points along the way. Elaboration IOW. All good.