7 ms·

Cayley – An open-source graph database

167 points by indatawetrust 10y ago

ocdtrekkie 10y ago
"Not a Google project, but created and maintained by a Googler" Interesting location, being in the main Google organization, given this disclaimer. I've seen other projects with this disclaimer under Googlers' personal pages before, and I always figured a /google/ URL probably meant it was officially a Google project.
- losvedir 10y ago
  Nah, I think that's the default place that Googler's code goes, if you spend 20% time on it (so it's the company's code). There was a discussion about it a couple months ago [0] when that rust text editor came up on HN. [0] https://news.ycombinator.com/item?id=11576703 https://news.ycombinator.com/item?id=11576703
- deleted 10y ago
  [deleted]
sidi 10y ago
The project has no activity in the last 6 months. I would like to see an open-source graph database that's being actively developed and maintained.
- stiangrindvoll 10y ago
  there is also: https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph also a golang project as the title mentions, if this matters to anyone.
  - worldsayshi 10y ago
    Looks promising. Graphql support!
  - mwambua 10y ago
    Their demo is very pretty... and they seem pretty keen on performance.
  - lolive 10y ago
    Is there a benchmark of a DBPedia import in dgraph?
- indatawetrust 10y ago
  https://github.com/neo4j/neo4j https://github.com/neo4j/neo4j
  - rspeer 10y ago
    Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will generally make it clear that this is not a process that really generalizes, just something that worked once for the developer. [1] http://stackoverflow.com/questions/12212015/how-to-setup-neo4j-with-dbpedia-ontop-of-ruby-on-rails-application http://stackoverflow.com/questions/12212015/how-to-setup-neo... Now suppose you want to load different medium-sized graph-structured data into Neo4j. You're basically going to have to reinvent these hacks for your data. And the last time I tried to load my multi-million-edge dataset into Neo4j through its documented API, I estimated that it would have taken several weeks to finish. Don't tell me that I need some sort of enterprise distributed system to import a few million edges. Right now I keep these edges in a hashtable that I wrote myself, in not-very-optimized Python, that shells out to "sort" for the computationally expensive part of indexing. It's not a very good DB but it gets the job done for now. It takes about 2 hours to import.
    - mwambua 10y ago
      Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran into a similar problem trying to explore Wikidata's json dumps. It's a lot simpler to load it into MongoDB and create indices where you need them, rather than figuring out how to interface to a proprietary system that you may or may not end up using in the long run. I'm still having trouble keeping my indices in memory though, and would be keen to know what sort of latency you encounter hitting an on-disk hashtable.
      - 10y ago
- impostervt 10y ago
  Latest commit is March 23rd.
- moondowner 10y ago
  http://orientdb.com/orientdb/ http://orientdb.com/orientdb/
  - baq 10y ago
    have you used it on non-trivial scales? did it turn out ok for you? any major outstanding issues? asking because i've looked at it a couple of years back and decided to go with SQL but the project looked reeeaally interesting back then and it still might fit for new development.
    - reactor 10y ago
      http://orientdbleaks.blogspot.com/ http://orientdbleaks.blogspot.com/
      - baq 10y ago
        thanks!
    - rspeer 10y ago
      I tried OrientDB. It's not clear how to use their fast data importer on data that's actually structured as a graph (instead of "hey, I've got a SQL database that I want to put into a graph database for some reason"). A couple of their employees have responded to me once but haven't actually answered the question. I also tried it before they had a fast data importer and... well, you need a fast data importer.
      - lvca 10y ago
        What kind of data did you try to import into OrientDB? CSV? Or from any other GraphDB (GraphML format)? Have you already tried http://orientdb.com/docs/last/Graph-Batch-Insert.html http://orientdb.com/docs/last/Graph-Batch-Insert.html? In case you import data from a RDBMS, you can use Teleporter: http://orientdb.com/docs/last/Teleporter-Home.html http://orientdb.com/docs/last/Teleporter-Home.html.
- kawera 10y ago
  OTOH, I've been using it in production for several months with good results. Small dataset though, only 9M quads. Cayley's codebase is not large and very clean.
- reactor 10y ago
  https://arangodb.com https://arangodb.com
- stanislavb 10y ago
  Yes, the dev activity seems to be declining - https://go.libhunt.com/project/cayley https://go.libhunt.com/project/cayley
- deleted 10y ago
  [deleted]
- qw 10y ago
  Neo4J had activity an hour ago
- jerven 10y ago
  Blazegraph. As used for wikidata among others. Has lots of property graph support not just semantic graphs.
  - rspeer 10y ago
    While I've been pointing out the problems I've encountered with Neo4J and OrientDB, I should say as a counterpoint that I just tried Blazegraph based on this recommendation, and so far, it works. Its importer was not the most intuitive thing to use -- I had to dig up configuration items that were scattered across its documentation and Stack Overflow -- but I got it to work, it imported 25 million edges in less than 8 minutes, and it's providing reasonably quick access to those edges now.
- willcodeforfoo 10y ago
  I know neo4j is the 800 lb. gorilla in this space, but it's interesting it has way more stars (almost 8K) than neo4j's almost 3K.
- kajecounterhack 10y ago
  http://github.com/google/badwolf http://github.com/google/badwolf Graph store currently being used by Google's Spam & Abuse Team. Last commit last week.
- gaffer 10y ago
  What about: https://github.com/GovernmentCommunicationsHeadquarters/Gaffer https://github.com/GovernmentCommunicationsHeadquarters/Gaff...
- nl 10y ago
  I'm pretty excited about https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph Distributed (yes, really!) graph database, from a guy who was on Google's knowledge graph team.
- marknadal 10y ago
  https://github.com/amark/gun https://github.com/amark/gun Open Source, graph database, P2P/decentralized, 1800+ stars, browser/javascript friendly, active community, does realtime updates like Firebase. With some INSANE performance specs: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
- ci5er 10y ago
  Can you explain the concern over this maybe not being maintained any more? My perspective (for reference) is that it meets your requirements for a project or it does not. Some of my favorite libraries haven't been updated in over a decade. They are in C, though, and I recognize that this removes a lot of worries about environmental churn someone might natively have if they come from (say) Javascript.
flaviuspopan 10y ago
What are the main reasons someone would use this over Neo4j? Is being open source the primary differentiator?
- kawera 10y ago
  Uses little memory, no JVM. Also interesting for golang shops since it's codebase is relatively small and clean.
- elcct 10y ago
  You don't have to pollute your environment with Java.
  - nugator 10y ago
    How is your environment "polluted" when running something on the JVM?
    - elcct 10y ago
      It is hard to explain - having JVM on the server is like having empty cans of beer laying around in the living room.
- LukaAl 10y ago
  I've used it in the past (and also contributed some patches), it has some roughness around the edges, but it works far better than Neo4J. It handles far bigger dataset than Neo4J (even if performances are not great) and it is easier to use and maintain.
- sciurus 10y ago
  Neo4J is open source. > The Neo4j Community Edition is licensed under the free GNU General Public License (GPL) v3 while the Neo4j Enterprise Edition is dual licensed under Neo4j commercial license as well as under the free Affero General Public License (AGPL) v3. http://neo4j.com/open-source-project/ http://neo4j.com/open-source-project/
deleted 10y ago
[deleted]
scrollaway 10y ago
I've been looking at Cayley recently, for a project of ours. We want to ingest millions of Hearthstone replays (simple XML documents describing thousands of key/value deltas per game) and analyze the game state for every turn, etc. Cayley seems uniquely suited for that, but its development activity is concerning. If anyone has any other suggestions, I'm all ears.
- smnscu 10y ago
  Lots of answers already in this thread. The ones I'd recommend: https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph http://titan.thinkaurelius.com/ http://titan.thinkaurelius.com/ https://www.arangodb.com/ https://www.arangodb.com/ http://orientdb.com/orientdb/ http://orientdb.com/orientdb/
  - arachnids 10y ago
    TitanDB is dead in the water. They have been unresponsive since getting acquired by Datastax, it looks like the team is now working on their commercial offering. Everyone I know is looking for alternatives.
    - wrabbitfoot 10y ago
      the new Datastax product may be built on top of Titan. http://intsantglobalnews.com/datastax-adds-graph-databases-to-enterprise-cassandra-product-set/ http://intsantglobalnews.com/datastax-adds-graph-databases-t...
  - scrollaway 10y ago
    Dgraph looks pretty great.
dang 10y ago
Previously discussed at https://news.ycombinator.com/item?id=7946024 https://news.ycombinator.com/item?id=7946024.
j2kun 10y ago
What exactly is an open source graph?
placeybordeaux 10y ago
For people that are concerned about lack of development on Cayley it looks like there is a fork that is actively merging PRs if nothing else: https://github.com/dennwc/cayley https://github.com/dennwc/cayley Found from the mailing list: https://groups.google.com/forum/?hl=en#!topic/cayley-users/DEh24z8pA7M https://groups.google.com/forum/?hl=en#!topic/cayley-users/D...
erichocean 10y ago
I like https://github.com/GraphChi/graphchi-cpp https://github.com/GraphChi/graphchi-cpp quite a bit. Handles out-of-core graphs really well. Fast.
bogomipz 10y ago
Clair the vulnerability database for docker images uses Cayley which is how I found about it, both neat projects and worth a look: https://github.com/coreos/clair https://github.com/coreos/clair
- wrabbitfoot 10y ago
  I looked at the history of that project a while back. My recollection is that Cayley was pulled out last year. Barak Michener works at CoreOS now, but he may be quite busy.

167 points by indatawetrust 10y ago

"Not a Google project, but created and maintained by a Googler" Interesting location, being in the main Google organization, given this disclaimer. I've seen other projects with this disclaimer under Googlers' personal pages before, and I always figured a /google/ URL probably meant it was officially a Google project.

losvedir 10y ago
Nah, I think that's the default place that Googler's code goes, if you spend 20% time on it (so it's the company's code). There was a discussion about it a couple months ago [0] when that rust text editor came up on HN. [0] https://news.ycombinator.com/item?id=11576703 https://news.ycombinator.com/item?id=11576703
deleted 10y ago
[deleted]

sidi 10y ago

The project has no activity in the last 6 months. I would like to see an open-source graph database that's being actively developed and maintained.

stiangrindvoll 10y ago
there is also: https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph also a golang project as the title mentions, if this matters to anyone.
- worldsayshi 10y ago
  Looks promising. Graphql support!
- mwambua 10y ago
  Their demo is very pretty... and they seem pretty keen on performance.
- lolive 10y ago
  Is there a benchmark of a DBPedia import in dgraph?
indatawetrust 10y ago
https://github.com/neo4j/neo4j https://github.com/neo4j/neo4j
- rspeer 10y ago
  Neo4j, like all graph databases I've tried, is only okay with small data. Suppose I want to import a medium-sized graph into Neo4j. Medium-sized as in "fits on a hard disk and doesn't quite fit in RAM". One example would be importing DBPedia. Some people have come up with not-very-supported hacks for loading DBPedia into Neo4j. Some StackOverflow comments such as [1] will point you to them, and the GitHub pages will generally make it clear that this is not a process that really generalizes, just something that worked once for the developer. [1] http://stackoverflow.com/questions/12212015/how-to-setup-neo4j-with-dbpedia-ontop-of-ruby-on-rails-application http://stackoverflow.com/questions/12212015/how-to-setup-neo... Now suppose you want to load different medium-sized graph-structured data into Neo4j. You're basically going to have to reinvent these hacks for your data. And the last time I tried to load my multi-million-edge dataset into Neo4j through its documented API, I estimated that it would have taken several weeks to finish. Don't tell me that I need some sort of enterprise distributed system to import a few million edges. Right now I keep these edges in a hashtable that I wrote myself, in not-very-optimized Python, that shells out to "sort" for the computationally expensive part of indexing. It's not a very good DB but it gets the job done for now. It takes about 2 hours to import.
  - mwambua 10y ago
    Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran into a similar problem trying to explore Wikidata's json dumps. It's a lot simpler to load it into MongoDB and create indices where you need them, rather than figuring out how to interface to a proprietary system that you may or may not end up using in the long run. I'm still having trouble keeping my indices in memory though, and would be keen to know what sort of latency you encounter hitting an on-disk hashtable.
    - 10y ago
impostervt 10y ago
Latest commit is March 23rd.
moondowner 10y ago
http://orientdb.com/orientdb/ http://orientdb.com/orientdb/
- baq 10y ago
  have you used it on non-trivial scales? did it turn out ok for you? any major outstanding issues? asking because i've looked at it a couple of years back and decided to go with SQL but the project looked reeeaally interesting back then and it still might fit for new development.
  - reactor 10y ago
    http://orientdbleaks.blogspot.com/ http://orientdbleaks.blogspot.com/
    - baq 10y ago
      thanks!
  - rspeer 10y ago
    I tried OrientDB. It's not clear how to use their fast data importer on data that's actually structured as a graph (instead of "hey, I've got a SQL database that I want to put into a graph database for some reason"). A couple of their employees have responded to me once but haven't actually answered the question. I also tried it before they had a fast data importer and... well, you need a fast data importer.
    - lvca 10y ago
      What kind of data did you try to import into OrientDB? CSV? Or from any other GraphDB (GraphML format)? Have you already tried http://orientdb.com/docs/last/Graph-Batch-Insert.html http://orientdb.com/docs/last/Graph-Batch-Insert.html? In case you import data from a RDBMS, you can use Teleporter: http://orientdb.com/docs/last/Teleporter-Home.html http://orientdb.com/docs/last/Teleporter-Home.html.
kawera 10y ago
OTOH, I've been using it in production for several months with good results. Small dataset though, only 9M quads. Cayley's codebase is not large and very clean.
reactor 10y ago
https://arangodb.com https://arangodb.com
stanislavb 10y ago
Yes, the dev activity seems to be declining - https://go.libhunt.com/project/cayley https://go.libhunt.com/project/cayley
deleted 10y ago
[deleted]
qw 10y ago
Neo4J had activity an hour ago
jerven 10y ago
Blazegraph. As used for wikidata among others. Has lots of property graph support not just semantic graphs.
- rspeer 10y ago
  While I've been pointing out the problems I've encountered with Neo4J and OrientDB, I should say as a counterpoint that I just tried Blazegraph based on this recommendation, and so far, it works. Its importer was not the most intuitive thing to use -- I had to dig up configuration items that were scattered across its documentation and Stack Overflow -- but I got it to work, it imported 25 million edges in less than 8 minutes, and it's providing reasonably quick access to those edges now.
willcodeforfoo 10y ago
I know neo4j is the 800 lb. gorilla in this space, but it's interesting it has way more stars (almost 8K) than neo4j's almost 3K.
kajecounterhack 10y ago
http://github.com/google/badwolf http://github.com/google/badwolf Graph store currently being used by Google's Spam & Abuse Team. Last commit last week.
gaffer 10y ago
What about: https://github.com/GovernmentCommunicationsHeadquarters/Gaffer https://github.com/GovernmentCommunicationsHeadquarters/Gaff...
nl 10y ago
I'm pretty excited about https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph Distributed (yes, really!) graph database, from a guy who was on Google's knowledge graph team.
marknadal 10y ago
https://github.com/amark/gun https://github.com/amark/gun Open Source, graph database, P2P/decentralized, 1800+ stars, browser/javascript friendly, active community, does realtime updates like Firebase. With some INSANE performance specs: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
ci5er 10y ago
Can you explain the concern over this maybe not being maintained any more? My perspective (for reference) is that it meets your requirements for a project or it does not. Some of my favorite libraries haven't been updated in over a decade. They are in C, though, and I recognize that this removes a lot of worries about environmental churn someone might natively have if they come from (say) Javascript.

flaviuspopan 10y ago

What are the main reasons someone would use this over Neo4j? Is being open source the primary differentiator?

kawera 10y ago
Uses little memory, no JVM. Also interesting for golang shops since it's codebase is relatively small and clean.
elcct 10y ago
You don't have to pollute your environment with Java.
- nugator 10y ago
  How is your environment "polluted" when running something on the JVM?
  - elcct 10y ago
    It is hard to explain - having JVM on the server is like having empty cans of beer laying around in the living room.
LukaAl 10y ago
I've used it in the past (and also contributed some patches), it has some roughness around the edges, but it works far better than Neo4J. It handles far bigger dataset than Neo4J (even if performances are not great) and it is easier to use and maintain.
sciurus 10y ago
Neo4J is open source. > The Neo4j Community Edition is licensed under the free GNU General Public License (GPL) v3 while the Neo4j Enterprise Edition is dual licensed under Neo4j commercial license as well as under the free Affero General Public License (AGPL) v3. http://neo4j.com/open-source-project/ http://neo4j.com/open-source-project/

deleted 10y ago

[deleted]

scrollaway 10y ago

I've been looking at Cayley recently, for a project of ours. We want to ingest millions of Hearthstone replays (simple XML documents describing thousands of key/value deltas per game) and analyze the game state for every turn, etc. Cayley seems uniquely suited for that, but its development activity is concerning. If anyone has any other suggestions, I'm all ears.

smnscu 10y ago
Lots of answers already in this thread. The ones I'd recommend: https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph http://titan.thinkaurelius.com/ http://titan.thinkaurelius.com/ https://www.arangodb.com/ https://www.arangodb.com/ http://orientdb.com/orientdb/ http://orientdb.com/orientdb/
- arachnids 10y ago
  TitanDB is dead in the water. They have been unresponsive since getting acquired by Datastax, it looks like the team is now working on their commercial offering. Everyone I know is looking for alternatives.
  - wrabbitfoot 10y ago
    the new Datastax product may be built on top of Titan. http://intsantglobalnews.com/datastax-adds-graph-databases-to-enterprise-cassandra-product-set/ http://intsantglobalnews.com/datastax-adds-graph-databases-t...
- scrollaway 10y ago
  Dgraph looks pretty great.

dang 10y ago

Previously discussed at https://news.ycombinator.com/item?id=7946024 https://news.ycombinator.com/item?id=7946024.

j2kun 10y ago

What exactly is an open source graph?

placeybordeaux 10y ago

For people that are concerned about lack of development on Cayley it looks like there is a fork that is actively merging PRs if nothing else: https://github.com/dennwc/cayley https://github.com/dennwc/cayley Found from the mailing list: https://groups.google.com/forum/?hl=en#!topic/cayley-users/DEh24z8pA7M https://groups.google.com/forum/?hl=en#!topic/cayley-users/D...

erichocean 10y ago

I like https://github.com/GraphChi/graphchi-cpp https://github.com/GraphChi/graphchi-cpp quite a bit. Handles out-of-core graphs really well. Fast.

bogomipz 10y ago

Clair the vulnerability database for docker images uses Cayley which is how I found about it, both neat projects and worth a look: https://github.com/coreos/clair https://github.com/coreos/clair

wrabbitfoot 10y ago
I looked at the history of that project a while back. My recollection is that Cayley was pulled out last year. Barak Michener works at CoreOS now, but he may be quite busy.