8 ms·

Question as someone new to graph databases: Are there any open source graph databases worth looking into?

by valine 11y ago

timClicks 11y ago
Neo4j is a very good option.
iod 11y ago
ArangoDB is free open source multi model no-sql db that has decent¹ perfomance with graph support: https://www.arangodb.com https://www.arangodb.com ¹ https://www.arangodb.com/2015/10/benchmark-postgresql-mongodb-arangodb/ https://www.arangodb.com/2015/10/benchmark-postgresql-mongod...
- rspeer 11y ago
  "The performance will suffer if the dataset is much bigger than the memory." That is a huge drawback when compared to relational databases. A good follow-up question would be: which open-source graph databases can reasonably import and store graph data that's not small -- that is, more data than fits in than RAM? Without proprietary extensions?
  - jerven 11y ago
    Blazegraph Virtuoso Jena tdb Can easily load large to very large graphs.
  - reactor 11y ago
    Looks like they have persistent indexes in the roadmap (ver 3.0) https://www.arangodb.com/roadmap/ https://www.arangodb.com/roadmap/ which might help.
  - LyndsySimon 11y ago
    How much data are you dealing with? In my (admittedly limited) experience, it's usually cheaper to throw money at hardware than other options.
    - rspeer 11y ago
      Throwing money on RAM because most graph databases haven't figured out how to use the disk effectively is not a good use of money. What's cheaper than buying all the RAM in the universe is figuring out a different system besides a graph database that does the job. A good example would be the graph of Wikipedia links. About 100 million edges among 5 million nodes, last I checked. The nodes have large differences in degree. The raw data for this is not the slightest bit large. We're only talking about gigabytes. But it would absolutely destroy Neo4J to even try to import it, to say nothing of running an interesting algorithm that justifies using a graph database on it, and Neo4J seems to be everyone's favorite open-source graph database for some reason.
  - neunhoef 11y ago
    One of the developers of ArangoDB here. Let me explain this quotation. When your graph data (including indices) do no longer fit into the RAM of a single server, you can either live with the higher latency of loading data from disk or you can use sharding, which will lead to communication and therefore slower traversals. That does not mean that things stop working, but performance will be less good, you can no longer visit tens of millions of nodes per second in a traversal as in RAM on a single server. If you actually only traverse a much smaller hot subgraph, I would probably go for the disk based single server approach. If your graph has a natural known clustering, then an optimized sharding solution with fine tuned sharding keys us probably your best bet. You can do all this with ArangoDB. However, graph traversals vary greatly in many respects, and your mileage may vary accordingly, with any approach. I would love to chat in more detail about your use case.
kinow 11y ago
Depends on what kind of data and graph you are going to store/use. Neo4j is quite popular, cypher isn't very hard to learn, and it has lots of examples. Might be a good choice for a beginner. https://en.wikipedia.org/wiki/Graph_database#List_of_graph_databases https://en.wikipedia.org/wiki/Graph_database#List_of_graph_d...
- valine 11y ago
  > cypher isn't very hard to learn Oh but I love a challenge. Are there reasons to choose cypher besides a gentle learning curve?
  - kinow 11y ago
    Not really, if you learn Cypher you should be fine learning the basics of Gremlin, SPARQL, or other languages to operate on graphs in a few hours. There was some post about enabling SPARQL in Neo4J, but when you install Neo4J it comes with cypher by default (not sure if it supports anything else). I use Apache Jena + SPARQL, but had to use Neo4J to help in a master thesis. Took me a few hours of "How the heck can I do that same thing I'd do in SPARQL that way?", plus some reading of the tutorials. Edit: some old post with example of Cypher, Gremlin and SPARQL: http://kinoshita.eti.br/2014/09/09/cypher-gremlin-and-sparql-graph-dialects.html http://kinoshita.eti.br/2014/09/09/cypher-gremlin-and-sparql...
    - kevinschumacher 11y ago
      You can definitely run gremlin queries against Neo4j by a couple of methods. One example: https://github.com/thinkaurelius/neo4j-gremlin-plugin https://github.com/thinkaurelius/neo4j-gremlin-plugin Also can use the Tinkerpop3 or Blueprints APIs to access your graph with Gremlin.
  - jexp 11y ago
    Cypher expressses the graph patterns that you're looking for in an ASCII-art-syntax, so you don't loose sight of the core of your question. On top of that you get filtering, projection, aggregation, pagination. The most fun things are in-query dataflow which allows you to pass information from one query part to the next (projected, aggregated, ordered etc). And the really cool collection and map functions, so you save a lot of roundtrips between client and server. See: http://neo4j.com/developer/guide-sql-to-cypher/ http://neo4j.com/developer/guide-sql-to-cypher/
kawera 11y ago
Cayley is a good option; we use it in production. https://github.com/google/cayley https://github.com/google/cayley
- jalfresi 11y ago
  Might I ask what sort of dataset size, servers etc you are using? I'm looking for an graph database and Cayley seems the best fit, though I'm not sure what sort of limits on the data there would be in the real world.
  - owen11 11y ago
    Cayley can store 130 million quads (2 nodes + connecting edge) on 20 GB harddrive. Join #cayley on freenode and https://groups.google.com/forum/#!forum/cayley-users https://groups.google.com/forum/#!forum/cayley-users and be part of our community!
    - jerven 11y ago
      Virtuoso, does 2 billion in 50GB. Just so you know. And hard graphs like the complete UniProt data of 20 billion+ in 800GB.
  - kawera 11y ago
    Cayley has been stable for us so far but I can't vault for it's scalability as our database is very small, less than 6M quads, so an 8GB machine is more than enough.
- jedc 11y ago
  Is Cayley mature enough for production? I thought it was still relatively new. Would love to know a little more about how you're using it.
  - owen11 11y ago
    There is a similar question on Cayley's google group - https://groups.google.com/forum/#!topic/cayley-users/nirhCbqwnfU https://groups.google.com/forum/#!topic/cayley-users/nirhCbq... Also, I know it's being used by some companies. you can ask directly the people who uses it on IRC - #cayley (freenode).
emehrkay 11y ago
Yes. I love the Tinkerpop stack (http://tinkerpop.incubator.apache.org http://tinkerpop.incubator.apache.org). I am currently writing a Python library as I develop an application around it called gizmo (https://github.com/emehrkay/gizmo https://github.com/emehrkay/gizmo).
jerven 11y ago
In the rdf space there are a whole bunch. Graph as in sparkling gives you : Virtuoso Blazegraph Jena Rdf4j Ruby.rdf There are more but these are opensource and I know them. And money more commercial ones.
rusabd 11y ago
Virtuoso
SanderMak 11y ago
We use http://orientdb.com/orientdb/ http://orientdb.com/orientdb/, seems decent so far.
whazor 11y ago
There are multiple systems out there, however I have my doubts. It is important that your data does not get corrupted, and that your transactions will not get lost. Furthermore, speedups are possible with certain indices. That is why I personally would want to see some more safety/speed analysis and comparisons between the different systems.
marknadal 11y ago
(Full disclosure: I'm the author, we are VC backed) https://github.com/amark/gun https://github.com/amark/gun is an Open Source graph database with Firebase like realtime synchronization.
- phpnode 11y ago
  We've had this discussion before but that product is not a graph database, it has no graph traversal features
  - marknadal 11y ago
    Dijkstra's algorithm is wonderful, but by no means is a requirement for being a graph database. A graph database is that, a database composed of nodes that can interconnect into a graph. GUN supports this and allows for traversing the graph. We haven't implemented Dijkstra's algorithm, which is what your "discussion" refers to.
    - phpnode 11y ago
      We've never talked about Dijkstra as far as I recall. Your product doesn't support graphs any more than, say, mongodb does because ultimately all you are doing is loading an object from JSON and then sending it to the consumer. This is in contrast to true graph databases whose main selling point is their ability to efficiently traverse, filter and aggregate large graphs to find the answer to some question, and then send the answer to the consumer. Gun doesn't meet anyone's definition of "graph database" other than your own. If I load some JSON from a URL, and use lodash to pluck some data out of it, is it a graph database?
      - marknadal 11y ago
        Dijkstra was your complaint about shortest path. GUN can do efficient traversal and filtering, and this is going to be even better in our 0.5.x release with lexical cursor support. By "anyone" do you mean Wikipedia's? https://en.m.wikipedia.org/wiki/Graph_database https://en.m.wikipedia.org/wiki/Graph_database , because GUN does match its definition. Although we haven't implemented Dijkstra's. I'm out in France right now and just boarded a plane to Slovenia, so I won't be able to reply again. Have a good one.
rail2rail 11y ago
We're using TitanDB. One of the main benefits for us is that AWS has provided backend integration with DynamoDB. This affords you practically infinite and painless scaling on a pay-as-you-go model. Love it. https://aws.amazon.com/blogs/aws/new-store-and-process-graph-data-using-the-dynamodb-storage-backend-for-titan/ https://aws.amazon.com/blogs/aws/new-store-and-process-graph...
- deleted 11y ago
  [deleted]
espeed 11y ago
Look at Blazegraph, an open-source GPU-accelerated distributed graph database. See previous discussion: https://news.ycombinator.com/item?id=11197880 https://news.ycombinator.com/item?id=11197880
karussell 11y ago
Here is an (old) overview: https://docs.google.com/spreadsheets/d/1XGapLHpSd2Ta8019VlwYX-K-n7QqgrVkmxvU4Ws8lHc/edit?pref=2&pli=1 https://docs.google.com/spreadsheets/d/1XGapLHpSd2Ta8019VlwY...
d0ne 11y ago
http://stingergraph.com/ http://stingergraph.com/ - From Georgia Tech

Question as someone new to graph databases: Are there any open source graph databases worth looking into?

by valine 11y ago

Question as someone new to graph databases: Are there any open source graph databases worth looking into?

timClicks 11y ago

Neo4j is a very good option.

iod 11y ago

ArangoDB is free open source multi model no-sql db that has decent¹ perfomance with graph support: https://www.arangodb.com https://www.arangodb.com ¹ https://www.arangodb.com/2015/10/benchmark-postgresql-mongodb-arangodb/ https://www.arangodb.com/2015/10/benchmark-postgresql-mongod...

rspeer 11y ago
"The performance will suffer if the dataset is much bigger than the memory." That is a huge drawback when compared to relational databases. A good follow-up question would be: which open-source graph databases can reasonably import and store graph data that's not small -- that is, more data than fits in than RAM? Without proprietary extensions?
- jerven 11y ago
  Blazegraph Virtuoso Jena tdb Can easily load large to very large graphs.
- reactor 11y ago
  Looks like they have persistent indexes in the roadmap (ver 3.0) https://www.arangodb.com/roadmap/ https://www.arangodb.com/roadmap/ which might help.
- LyndsySimon 11y ago
  How much data are you dealing with? In my (admittedly limited) experience, it's usually cheaper to throw money at hardware than other options.
  - rspeer 11y ago
    Throwing money on RAM because most graph databases haven't figured out how to use the disk effectively is not a good use of money. What's cheaper than buying all the RAM in the universe is figuring out a different system besides a graph database that does the job. A good example would be the graph of Wikipedia links. About 100 million edges among 5 million nodes, last I checked. The nodes have large differences in degree. The raw data for this is not the slightest bit large. We're only talking about gigabytes. But it would absolutely destroy Neo4J to even try to import it, to say nothing of running an interesting algorithm that justifies using a graph database on it, and Neo4J seems to be everyone's favorite open-source graph database for some reason.
- neunhoef 11y ago
  One of the developers of ArangoDB here. Let me explain this quotation. When your graph data (including indices) do no longer fit into the RAM of a single server, you can either live with the higher latency of loading data from disk or you can use sharding, which will lead to communication and therefore slower traversals. That does not mean that things stop working, but performance will be less good, you can no longer visit tens of millions of nodes per second in a traversal as in RAM on a single server. If you actually only traverse a much smaller hot subgraph, I would probably go for the disk based single server approach. If your graph has a natural known clustering, then an optimized sharding solution with fine tuned sharding keys us probably your best bet. You can do all this with ArangoDB. However, graph traversals vary greatly in many respects, and your mileage may vary accordingly, with any approach. I would love to chat in more detail about your use case.

kinow 11y ago

Depends on what kind of data and graph you are going to store/use. Neo4j is quite popular, cypher isn't very hard to learn, and it has lots of examples. Might be a good choice for a beginner. https://en.wikipedia.org/wiki/Graph_database#List_of_graph_databases https://en.wikipedia.org/wiki/Graph_database#List_of_graph_d...

valine 11y ago
> cypher isn't very hard to learn Oh but I love a challenge. Are there reasons to choose cypher besides a gentle learning curve?
- kinow 11y ago
  Not really, if you learn Cypher you should be fine learning the basics of Gremlin, SPARQL, or other languages to operate on graphs in a few hours. There was some post about enabling SPARQL in Neo4J, but when you install Neo4J it comes with cypher by default (not sure if it supports anything else). I use Apache Jena + SPARQL, but had to use Neo4J to help in a master thesis. Took me a few hours of "How the heck can I do that same thing I'd do in SPARQL that way?", plus some reading of the tutorials. Edit: some old post with example of Cypher, Gremlin and SPARQL: http://kinoshita.eti.br/2014/09/09/cypher-gremlin-and-sparql-graph-dialects.html http://kinoshita.eti.br/2014/09/09/cypher-gremlin-and-sparql...
  - kevinschumacher 11y ago
    You can definitely run gremlin queries against Neo4j by a couple of methods. One example: https://github.com/thinkaurelius/neo4j-gremlin-plugin https://github.com/thinkaurelius/neo4j-gremlin-plugin Also can use the Tinkerpop3 or Blueprints APIs to access your graph with Gremlin.
- jexp 11y ago
  Cypher expressses the graph patterns that you're looking for in an ASCII-art-syntax, so you don't loose sight of the core of your question. On top of that you get filtering, projection, aggregation, pagination. The most fun things are in-query dataflow which allows you to pass information from one query part to the next (projected, aggregated, ordered etc). And the really cool collection and map functions, so you save a lot of roundtrips between client and server. See: http://neo4j.com/developer/guide-sql-to-cypher/ http://neo4j.com/developer/guide-sql-to-cypher/

kawera 11y ago

Cayley is a good option; we use it in production. https://github.com/google/cayley https://github.com/google/cayley

jalfresi 11y ago
Might I ask what sort of dataset size, servers etc you are using? I'm looking for an graph database and Cayley seems the best fit, though I'm not sure what sort of limits on the data there would be in the real world.
- owen11 11y ago
  Cayley can store 130 million quads (2 nodes + connecting edge) on 20 GB harddrive. Join #cayley on freenode and https://groups.google.com/forum/#!forum/cayley-users https://groups.google.com/forum/#!forum/cayley-users and be part of our community!
  - jerven 11y ago
    Virtuoso, does 2 billion in 50GB. Just so you know. And hard graphs like the complete UniProt data of 20 billion+ in 800GB.
- kawera 11y ago
  Cayley has been stable for us so far but I can't vault for it's scalability as our database is very small, less than 6M quads, so an 8GB machine is more than enough.
jedc 11y ago
Is Cayley mature enough for production? I thought it was still relatively new. Would love to know a little more about how you're using it.
- owen11 11y ago
  There is a similar question on Cayley's google group - https://groups.google.com/forum/#!topic/cayley-users/nirhCbqwnfU https://groups.google.com/forum/#!topic/cayley-users/nirhCbq... Also, I know it's being used by some companies. you can ask directly the people who uses it on IRC - #cayley (freenode).

emehrkay 11y ago

Yes. I love the Tinkerpop stack (http://tinkerpop.incubator.apache.org http://tinkerpop.incubator.apache.org). I am currently writing a Python library as I develop an application around it called gizmo (https://github.com/emehrkay/gizmo https://github.com/emehrkay/gizmo).

jerven 11y ago

In the rdf space there are a whole bunch. Graph as in sparkling gives you : Virtuoso Blazegraph Jena Rdf4j Ruby.rdf There are more but these are opensource and I know them. And money more commercial ones.

rusabd 11y ago

Virtuoso

SanderMak 11y ago

We use http://orientdb.com/orientdb/ http://orientdb.com/orientdb/, seems decent so far.

whazor 11y ago

There are multiple systems out there, however I have my doubts. It is important that your data does not get corrupted, and that your transactions will not get lost. Furthermore, speedups are possible with certain indices. That is why I personally would want to see some more safety/speed analysis and comparisons between the different systems.

marknadal 11y ago

(Full disclosure: I'm the author, we are VC backed) https://github.com/amark/gun https://github.com/amark/gun is an Open Source graph database with Firebase like realtime synchronization.

phpnode 11y ago
We've had this discussion before but that product is not a graph database, it has no graph traversal features
- marknadal 11y ago
  Dijkstra's algorithm is wonderful, but by no means is a requirement for being a graph database. A graph database is that, a database composed of nodes that can interconnect into a graph. GUN supports this and allows for traversing the graph. We haven't implemented Dijkstra's algorithm, which is what your "discussion" refers to.
  - phpnode 11y ago
    We've never talked about Dijkstra as far as I recall. Your product doesn't support graphs any more than, say, mongodb does because ultimately all you are doing is loading an object from JSON and then sending it to the consumer. This is in contrast to true graph databases whose main selling point is their ability to efficiently traverse, filter and aggregate large graphs to find the answer to some question, and then send the answer to the consumer. Gun doesn't meet anyone's definition of "graph database" other than your own. If I load some JSON from a URL, and use lodash to pluck some data out of it, is it a graph database?
    - marknadal 11y ago
      Dijkstra was your complaint about shortest path. GUN can do efficient traversal and filtering, and this is going to be even better in our 0.5.x release with lexical cursor support. By "anyone" do you mean Wikipedia's? https://en.m.wikipedia.org/wiki/Graph_database https://en.m.wikipedia.org/wiki/Graph_database , because GUN does match its definition. Although we haven't implemented Dijkstra's. I'm out in France right now and just boarded a plane to Slovenia, so I won't be able to reply again. Have a good one.

rail2rail 11y ago

We're using TitanDB. One of the main benefits for us is that AWS has provided backend integration with DynamoDB. This affords you practically infinite and painless scaling on a pay-as-you-go model. Love it. https://aws.amazon.com/blogs/aws/new-store-and-process-graph-data-using-the-dynamodb-storage-backend-for-titan/ https://aws.amazon.com/blogs/aws/new-store-and-process-graph...

deleted 11y ago
[deleted]

espeed 11y ago

Look at Blazegraph, an open-source GPU-accelerated distributed graph database. See previous discussion: https://news.ycombinator.com/item?id=11197880 https://news.ycombinator.com/item?id=11197880

karussell 11y ago

Here is an (old) overview: https://docs.google.com/spreadsheets/d/1XGapLHpSd2Ta8019VlwYX-K-n7QqgrVkmxvU4Ws8lHc/edit?pref=2&pli=1 https://docs.google.com/spreadsheets/d/1XGapLHpSd2Ta8019VlwY...

d0ne 11y ago

http://stingergraph.com/ http://stingergraph.com/ - From Georgia Tech