4 ms·

I think graph processing has great potential for use cases where the data model is heterogeneous and the structure of a problem really can be modeled as a graph

I think graph processing has great potential for use cases where the data model is heterogeneous and the structure of a problem really can be modeled as a graph problem. I think the tooling and the technologies are still not nearly as mature as e.g. relational databases and most developers have very little exposure to graph models, so adoption is still quite low, but will probably rise as the cloud providers offer hosted graph databases. In 2015 I was quite excited for TitanDB and the TinkerPop stack but I think the acquisition of the team behind it (Aurelius) by DataStax kind of sucked the air out of the community edition. As far as I understand Amazon is using many ideas and concepts from TitanDB in their Neptune graph database, but unfortunately they don't seem to care to open-source it. I don't think graph databases & graph processing will become mainstream tough, as they're just not a good fit for most problems.

ta988 5y ago
Neptune sucked the air out of blazegraph as well... It is a constant problem. Graph databases will only ever become more mainstream when we get a real postgresql-level database in term of reliability, maintenance and trust in its evolution.
- zozbot234 5y ago
  PostgreSQL is a fine graph database. It would be nice if it supported the graph-like query language that's now being standardized as an extension to SQL, but other than that it works quite well.
  - ta988 5y ago
    It is pretty slow for most graph operations though. It works ok for simple things, especially with the recently added functions.
    - zozbot234 5y ago
      Well yes, if you need to run actual graph algorithms on network data I agree that pgSQL (or any general purpose database, including "semantics" oriented graph db's) is not the best.
  - rambojazz 5y ago
    What is that "graph-like query language that's now being standardized as an extension to SQL"? I don't think I've ever heard of it.
lumost 5y ago
Unfortunately graph DBs are a solution in search of a problem. The performance of a graph traversal in all current implementations will be at best equivalent to a well crafted join at worst much slower. I’m unclear on where truly heterogeneous data problems would emerge outside of Q&A applications and potentially CRM. Usually the work to acquire a dataset is so large that there is limited reason not to properly model the data.
- amw-zero 5y ago
  > The performance of a graph traversal in all current implementations will be at best equivalent to a well crafted join at worst much slower. A join must be implemented with a loop / scan to find the relevant target rows by foreign key - This is why foreign keys pretty much have to be indexed, to give the scan any hope of acceptable performance. So, remember, a join in a relational database generally means a join PLUS an index lookup. If it's not clear, indexes are not free - they have storage cost as well as imposing a cost on writes to the table. Indexes also make lookup very very fast, but still not free. On a large table, you may be incurring the cost of 20-30 comparisons to find the desired rows (note that this is not a made up number. Assuming a btree index which reduces lookups to O(log_2(n)), you will need 20 comparisons to find a record in a table with 1,000,000 rows, and 26 comparisons to find a record in a table with 100,000,000 rows.) Graph databases (at least neo4j) always talk about index-free adjacency. Now you hopefully understand what this means. The edge between two nodes in a graph database is stored directly with no need for an index to find the destination node - the traversal is index-free. This can be a tremendous advantage, and that's also not made-up. You can subtract the cost of index lookups that I gave earlier .
  - jandrewrogers 5y ago
    These methods only work for very small graphs though. Secondary indexing structures are inefficient and don't scale at all regardless of the implementation details, hence why even non-graph systems don't use them at large scales on single machines. The scale threshold where using indexes is worse than the alternatives is low. Direct representation without indexes (like your Neo4j example) are even worse due to the latency introduced for each step by very poor locality. There is no possible version of these methods that will work well; this would have been a solved problem 30 years ago otherwise. Latency is the performance killer for graph traversals, so locality is everything, and preserving locality across recursive traversals is the critical computer science trick that is often missing. The computational cost of a comparison is irrelevant to the performance of graph traversal. The way most "pure" graph systems implement graph traversals is unusably slow outside of trivial cases because they mitigate latency poorly. In practical graph systems, common traversals are typically materialized ahead of time so that a graph traversal operation is not required to satisfy queries, which defeats the point in many cases.

ta988 5y ago

Neptune sucked the air out of blazegraph as well... It is a constant problem. Graph databases will only ever become more mainstream when we get a real postgresql-level database in term of reliability, maintenance and trust in its evolution.

zozbot234 5y ago
PostgreSQL is a fine graph database. It would be nice if it supported the graph-like query language that's now being standardized as an extension to SQL, but other than that it works quite well.
- ta988 5y ago
  It is pretty slow for most graph operations though. It works ok for simple things, especially with the recently added functions.
  - zozbot234 5y ago
    Well yes, if you need to run actual graph algorithms on network data I agree that pgSQL (or any general purpose database, including "semantics" oriented graph db's) is not the best.
- rambojazz 5y ago
  What is that "graph-like query language that's now being standardized as an extension to SQL"? I don't think I've ever heard of it.

lumost 5y ago

Unfortunately graph DBs are a solution in search of a problem. The performance of a graph traversal in all current implementations will be at best equivalent to a well crafted join at worst much slower. I’m unclear on where truly heterogeneous data problems would emerge outside of Q&A applications and potentially CRM. Usually the work to acquire a dataset is so large that there is limited reason not to properly model the data.

amw-zero 5y ago
> The performance of a graph traversal in all current implementations will be at best equivalent to a well crafted join at worst much slower. A join must be implemented with a loop / scan to find the relevant target rows by foreign key - This is why foreign keys pretty much have to be indexed, to give the scan any hope of acceptable performance. So, remember, a join in a relational database generally means a join PLUS an index lookup. If it's not clear, indexes are not free - they have storage cost as well as imposing a cost on writes to the table. Indexes also make lookup very very fast, but still not free. On a large table, you may be incurring the cost of 20-30 comparisons to find the desired rows (note that this is not a made up number. Assuming a btree index which reduces lookups to O(log_2(n)), you will need 20 comparisons to find a record in a table with 1,000,000 rows, and 26 comparisons to find a record in a table with 100,000,000 rows.) Graph databases (at least neo4j) always talk about index-free adjacency. Now you hopefully understand what this means. The edge between two nodes in a graph database is stored directly with no need for an index to find the destination node - the traversal is index-free. This can be a tremendous advantage, and that's also not made-up. You can subtract the cost of index lookups that I gave earlier .
- jandrewrogers 5y ago
  These methods only work for very small graphs though. Secondary indexing structures are inefficient and don't scale at all regardless of the implementation details, hence why even non-graph systems don't use them at large scales on single machines. The scale threshold where using indexes is worse than the alternatives is low. Direct representation without indexes (like your Neo4j example) are even worse due to the latency introduced for each step by very poor locality. There is no possible version of these methods that will work well; this would have been a solved problem 30 years ago otherwise. Latency is the performance killer for graph traversals, so locality is everything, and preserving locality across recursive traversals is the critical computer science trick that is often missing. The computational cost of a comparison is irrelevant to the performance of graph traversal. The way most "pure" graph systems implement graph traversals is unusably slow outside of trivial cases because they mitigate latency poorly. In practical graph systems, common traversals are typically materialized ahead of time so that a graph traversal operation is not required to satisfy queries, which defeats the point in many cases.