9 ms·
Cayley – An open-source graph database
- ForHackernews 12y ago> Not a Google project
- riffraff 12y agoand obviously, not "google's new graph database".
- deleted 12y ago[deleted]
- nostrademons 12y agoIt looks like a Googler's 20% project. However, I'll point out that most of the actually-used open-source projects coming out of Google aren't actually "Google projects", they are "projects by Googlers released under the OSPO process". LevelDB (Jeff Dean & Sanjay Ghemawat), Protocol Buffers (Kenton Varda), Guice (Bob Lee, Jesse Wilson, and Kevin Bourillion), Gumbo (myself), and angular.js (a team within DoubleClick) all started out as small internal projects built to scratch an individual's itch that were then released externally because hey, why not. The "corporate" open-source projects have been things like Android, Chrome, GWT, Closure, Polymer, etc.
- barakm 12y agoExactly this. And I'm the Googler. Hi!
- zaphar 12y agoHi. Former googler who does work that Graph Databases would be useful for. If you don't mind I'd like to ask: Does it have bulk import and if so what is it's speed for bulk import rougly speaking?
- barakm 12y agoIt does have bulk import; I've been loading largish subsets of the Freebase dumps. Load speed is pretty good (into persistent storage, I assume) and can be improved with some of the database parameters. A rough estimate is that a million triples or so takes about 5 minutes, but that slows down as it gets bigger. 134m triples took me 6-8hrs, so I slept on it.
- krato 12y agoSince no one's mentioned it yet, another alternative to Neo4J is AllegroGraph (though you need to pay for it; the free version supports 5 million tuples). It _does_ support bulk loading, with over 500K triples per second. According to http://franz.com/agraph/allegrograph/agraph_benchmarks.lhtml http://franz.com/agraph/allegrograph/agraph_benchmarks.lhtml, given enough RAM, it can load over a billion tuples in just over half an hour.
- jerven 12y agoVirtuoso 7.1. and OWLIM 5.5 have similar loading speeds. In this case the decompression algorithm is often the bottleneck needing multiple files to be read in parallel to go faster. Oracle 12c Semantic Network and Yarcdata uRiKA can also go faster in loading if correctly set up.
- _pius 12y agoWhat kind of usage is this seeing inside Google?
- panarky 12y agoNot a Google project, but created and maintained by a Googler, with permission from and assignment to Google, under the Apache License, version 2.0.
- cnbuff410 12y agoHere is a brief introduction http://google-opensource.blogspot.hk/2014/06/cayley-graphs-in-go.html http://google-opensource.blogspot.hk/2014/06/cayley-graphs-i...
- nicklovescode 12y agoI've been using a graph db as my primary database with my last project(neo4j) and it has been a pleasure. I wish a good graph database was hooked in with the new Google Cloud stuff so it could be queried/visualized/performance analyzed in a similar fashion to the BigTable demo earlier
- cihangirsavas 12y agoare you using it for production purposes or for your small projects? if it is for production, how is your read/write performance?
- kaonashi 12y agoNeo4j wants everything in memory, so the bottlenecks would come after your data-set size outstrips the memory available.
- bunkat 12y agoFast disks are also important for write performance since Neo4j syncs every change - big RAID arrays of SSDs help.
- jmlvanre 12y agoThis is one place where Log-Structured-Merge systems can really prove themselves. There is rarely a need to sync on every change as most of them are independent. You can usually gain lots of write-throughput by syncing at the speed the hardware is optimized for and squeezing as many append-only changes in to those logs. At Orly we've spent quite some time looking at ways to deal with these bottle-necks.
- deleted 12y ago[deleted]
- cihangirsavas 12y ago
- hugofirth 12y agoAwesome - all my work is with graph data and graph databases any additions to the space are great news.
- hugofirth 12y agoOne question: I'm curious what your motivation was for providing a Gremlin style query language? As opposed to something like Cypher? Was it a case of expressiveness, personal preference etc....?
- barakm 12y agoI wanted something easy to pick up -- Javascript seemed natural. I also wanted to be as agnostic as possible, because graph query languages are interesting and worth experimenting with. That's why it has submodules per-language and it's (relatively) straight-forward to write another. Cypher's not bad either, but it's not "just Javascript". But I'm totally taking commits if someone wants to port it :)
- JasonL9000 12y agoGetting performance out of a graph often depends on being able to express your query effectively. The last thing you want to do is plunk down a cursor on a node and have your client start wandering around the graph, following edges. All that back-and-forth chattiness is a non-starter network-wise and it gives the database engine essentially no chance to optimize the query. I've been favoring scripts that let me use free variables--like Prolog logic variables--to describe shapes in a graph and then let the database server find bindings that match. Like 'print a.name, b.name for a, b, x, y where a is_friend_of b and a is_friend_of x and b is friend_of y and not x is_friend_of y and x.city == y.city'
- mpark 12y agoI'm a developer on Orly and I focus on the language portion of the project. Orly takes a different approach in this regard by providing a general-purpose functional language as its "query" language, rather than providing a language specifically designed to traverse/query the graph. This approach allows us to perform not only queries but also arbitrary computations on the server-side and take advantage of the powerful resources available on the server such as large memory, CPUs and perhaps most interesting are GPUs.
- dingdingdang 12y agoDemo version somewhere?
- barakm 12y agohttp://cayley-graph.appspot.com http://cayley-graph.appspot.com is a live demo on App Engine.
- bratsche 12y agoFor people here experienced with graph databases, do you typically use the graph db as your primary data store or do you use it in combination with something like postgresql? If you're using both, can you talk about how that works and if it's been successful for you? I'm curious because I've had a couple situations where I thought using neo4j (or some graph db) would be a natural fit for something I wanted to do, but otherwise I thought most of my other data fit into postgresql just fine. My instinct is that if I'm doing this in a web app then querying from two different databases is going to slow down my responses a lot.
- klapinat0r 12y agoI don't currently use graph databases in production, but I do, however, have some experience. I use both, in a similar vein of "using Elastic Search". It could be your primary store, but it's sometimes it's more pragmatic to have two sets, a "solid" base. This is not to say that it can't be done. What I'm stressing is that larger "changes" are hard and difficult to handle - which means a lot in the start of your process, and less in the end, as in, when you're deciding how to model your data. For instance node layout (new properties? different type? other constraints?), and mass updates are also a bit cumbersome. Usually I have more than one SQL table (naturally) since the data I've used in graph databases is mix and match (otherwise I'd just use a fixed schema and some relational DB). -- As for "how that works", for me it's: routinely update from my base database with queries alike: ID > last ID. This has worked as expected, in terms of what data you get in, and which limitations you impose (e.g. timeliness). I'm currently making a shift to running all data in my graph database as I've settled on a model (which edges, which nodes, which properties). > querying from two different databases is going to slow down my responses True, but depending on your data (do you know one of the queries beforehand - e.g. is your postgresql query enriching whatever your graph query returns) you might have success tying (inserting) some of the SQL data to your graph database.
- nchuhoai 12y agoI have actually started with using neo4j as a primary store and then moved to using it as a secondary store next to postgres. I wrote some stuff up here: http://nambrot.com/posts/14-credport-technical-post-mortem/ http://nambrot.com/posts/14-credport-technical-post-mortem/ It led to itself as neo4j was only useful for parts of our queries and fitting everything into neo4j was just hassle when most of our data was relational.
- preillyme 12y agoHas anybody tried out Orly? https://github.com/orlyatomics/orly https://github.com/orlyatomics/orly
- JasonL9000 12y agoI have. I'm on the team that's been developing it for the past four years. It's nice to see graph databases getting some popular traction at last. (I was into them before they were cool, of course.) I've only just started looking at Cayley but it looks like there are some significant differences between the projects. Orly is designed for high-speed, high-volume applications that need large-scale storage and consistent transactions. We're more OLTP than OLAP, which seems to be the way Cayley leans. Most graph systems tend toward analytic applications.
- preillyme 12y agoCould Cayley use Orly for a storage engine?
- JasonL9000 12y agoMaybe so. Orly has a somewhat leveldb-like component called a Repo which might slot in well. A Repo is a log-structured merge storage system that also provides indexing, access to previous values, and, most importantly, consistent reads and writes, even across multiple Repos. It also makes good use of resources (RAM, SSD, HDD) to optimize performance and cost. (Also working on letting it run on GPUs, for insanely high performance for people with the power and air conditioning budgets.)
- e12e 12y agoI've seen a couple of mentions of graph dbs on GPUs in this thread, and it does seem like a (somewhat) obvious fit. Anyone aware of any projects that make (good) use of that right now? Not necessarily a stand-alone service, but also things like an embedded graph db ("berkley db for graphs") or something like that?
- slapresta 12y agoI can't get over the logo being a 3-colored version of a 2-colorable graph.
- JasonL9000 12y agoEven Arthur Cayley might have smirked at that one. :)
- msutherl 12y agoIt would probably look better without the grey nodes on the ends anyway.
- nandemo 12y agoThat's intentional. It represents robustness and scalability: you can add an edge between a red and yellow node, without having to re-color. (actually I've just made that up)
- jperras 12y ago
- leccine 12y agoWoo, this is probably the coolest project from Google in the last while. Is there anybody using a graph database here? What is your use case? Would you mind sharing?
- jonathanmarvens 12y agoAt my company, we're using graph-based models for most of our NLP stuff (mainly entity extraction ATM) and our ANNs (mainly object classifiers). - Jonathan
- leccine 12y agoThanks Jonathan. I am not terribly familiar with NLP would you mind resolving the 3 letter acronyms (ATM, ANN)?
- dajohnson89 12y agoHas anyone tried out Tinkerpop? http://www.tinkerpop.com http://www.tinkerpop.com I played with it, and it was kinda fun.
- bobbriody 12y agoCan anyone provide a working example for the visualization feature? The docs say that you can use the Tag functionality to label source/target nodes for sigma.js rendering, but bridging the gap between that suggestion and the actual query does not seem trivial.
- tempodox 12y agoOS X: Homebrew is the preferred method. Over my dead body. Homebrew messes up my /usr/local, leaving its heaps of crap everywhere. I don't find that acceptable. MacPorts at least has the decency to put itself into /opt/local, which isn't normally used by anything else.
- sfvisser 12y agoInteresting 'triple' they use :) // Our triple struct, used throughout. type Triple struct { Sub string `json:"subject"` Pred string `json:"predicate"` Obj string `json:"object"` Provenance string `json:"provenance,omitempty"` }
- AlisdairO 12y agoThis is weirdly ubiquitous in the RDF store world as well. I eventually gave up talking about 'triple stores' and called them 'RDF stores' instead :-)
- jerven 12y agoIt is basically required for SPARQL 1.1. (Named) graph support. Often people call them quad stores.
- webmaven 12y agoSeems like provenance ought to just be it's own triple describing the relation of another triple to some source. Is this just a shortcut?
- philjohn 12y agoSad to see no SPARQL support as of yet, it looks like it's on their longer term goals though as they are query language agnostic. Interesting to see more products entering this space.
- brickcap 12y agoAlmost all graph databases use gremlin as querying language but I really love orient db's approach of using sql[1] for querying graphs. It feels more natural in my opinion plus it lowers the barrier to entry for people who already know about sql. Some of the things I like about cayley 1. Switchable backends(I wonder if I can configure it to use couchdb as a store) 2. Documentation get right to the point. When I first tried my hand at graph databases I could not understand where to start but cayley's approach is pretty straight forward and it wins plus points from me for including a big dataset :) A question: I see no mention in the docs about running it on multiple nodes (where does it stand with regards to CAP etc) [1](https://github.com/orientechnologies/orientdb/wiki/SQL https://github.com/orientechnologies/orientdb/wiki/SQL)
- weihuan 12y agozxcv