6 ms·

Titan: A Highly Scalable, Distributed Graph Database

170 points by d0ugal 14y ago

nchuhoai 14y ago
Interesting, does anyone know how this compares with the popular Neo4j Database?
- norkakn 14y ago
  Each Neo4j node stores all of the data, and it doesn't scale write horizontally well. OrientDB tries to scale writes (I'll be testing this in a few months), but still stores all of the data everywhere. This looks like it shards the data automagically. If it works well, I might be able to bang on it a bit, but I'm guessing that it gives shit performance for complex graph questions.
  - okram 14y ago
    Titan exposes graph data over a machine cluster. It is an OLTP system that allows you to do local neighborhood graph traversals in sub-second time. For OLAP processing (e.g. global graph algorithms), Aurelius will be releasing two projects named Faunus and Fulgora in the coming months. These provide Hadoop connectivity and compressed in-memory representations of "graph slices." We will be publishing our talk slides tonight that discuss this eco-system of graph technologies. See http://titanbiggraphdata.eventbrite.com/ http://titanbiggraphdata.eventbrite.com/
    - jbellis 14y ago
      Congrats on the release, Marko!
    - norkakn 14y ago
      I really wish that I could attend. We ask a lot of questions like: Find all Nodes with a property in a tree Find all leaves L of those nodes Find all annotations in a DAG of those leaves Collapse similar DAG entries by backtracking up the graph based on edge weights Writes are bulk loaded, and right now, we are just trying to push all of the graph stuff offline, but there are some limitations to that, and we could really up our accuracy by being able to perform these queries quickly.
  - okram 14y ago
    Also, Aurelius will be posting a blog post in ~2 weeks where we will be simulating Twitter. We replay Twitter from day 1 to June 2009 and slam it with ~10,000 concurrent users writing/reading follows relationships, tweets, stream constructions -- ultimately growing (what we think will be) a 3 billion edge graph when the simulation is complete.
oacgnol 14y ago
Very interesting. Since it implements Blueprints, does it also have support for Furnace (graph algorithms)? If so, does this imply that graph processing is done on disk rather than in-memory? I am rather unfamiliar with Blueprints but I'm wondering how Titan implements that aspect.
joe_the_user 14y ago
Also, What is interesting about a graph database relative to simple key-value database? Storing edges of a graph is trivial for a key-value store and so it seems like any key-value store could let store the basic graph structure? Do graph databases support graphic-specific queries and indices?
- netghost 14y ago
  They seem to implement a variety of graph centric APIs. From a query standpoint, it's easier to use a graph database to do something like find all the people who share interests with someone (person <--> interest <--> person), or find all the grand children in a family tree. Now whether or not Titan implements these performantly, I have no idea.
- JPKab 14y ago
  I can only answer for one advantage I specifically know of regarding graph DB's over key value: dynamic, mergeable schemas which enforce data integrity WITHIN the database rather than with code on top of it. There are many, many people on HN who are much more knowledgeable than I am on graph DB's, and I sure as hell hope they answer on this question. I'm curious if this supports the RDF, OWL, and SPARQL standards? I'm a little tired of graph DB's that focus on scale, rather than speed and flexibility though. A good one to check out is Stardog. http://stardog.com/ http://stardog.com/ I think it just went into 1.0.
  - philjohn 14y ago
    It looks like it's Gremlin only. What triple stores have you looked at? 4store is performant, but doesn't support reasoning. There's also BigData and Virtuoso which support various levels of it, and Franz are apparently working on a clustered version of Allegrograph.
- dkersten 14y ago
  Sure you can easily store graphs in a KV store, but graph databases allow you to efficiently traverse the graph too. Look up how to traverse graph structures in hadoop for example - I was looking it up recently for something Im building and in the end I opted for using hadoop to filter, aggregate and format the raw data and then input the preprocessed data into graph databases (Im using orientdb at the moment) for traversal queries. I duplicate the data on multiple machines and use storm bolts to do geaph queries (so each storm bolt has its own local copy of the graph database). Seems to be working well so far and allows me to do realtime traversals (though inserts wait hadoop the next hadoop batch job). Seemed a lot easier than what Id have to do to efficiently process using hadoop.
PaulHoule 14y ago
Where's the SPARQL support?
- okram 14y ago
  Titan currently does not have "edge indexing" and thus can not implement Blueprints' GraphSail interface [https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementation https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementa...]. If it did implement GraphSail, then SPARQL would be supported via the Sail SPARQL engine.
jandrewrogers 14y ago
What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site suggests this is the case. Many graph databases have claimed high scalability and distributability but none of those claims have held up over time due to the aforementioned computer science problems. This may be a very nice graph database but I am skeptical of the claims of "highly scalable, distributed" unless there is evidence that it uses fundamentally new theoretical computer science to achieve that.
- th0ma5 14y ago
  You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.
  - tptacek 14y ago
    Worth a re-read: http://codahale.com/you-cant-sacrifice-partition-tolerance/ http://codahale.com/you-cant-sacrifice-partition-tolerance/
    - th0ma5 14y ago
      Thank you, this is really great to show AP or CP but never CA and call it distributed.
  - okram 14y ago
    See https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-graph-data?slide=126 https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap...
    - th0ma5 14y ago
okram 14y ago
We are going to address alot of these questions in our presentation tonight. However, for the Hacker News crew that won't be there tonight, here is an early release of the Titan talk. https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-graph-data https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap... http://titanbiggraphdata.eventbrite.com/ http://titanbiggraphdata.eventbrite.com/ Enjoy!, Marko.
- hendler 14y ago
  Great presentation.
  - benjaminRRR 14y ago
    Also it appears like a great way to take speakerdeck down...
niho 14y ago
I don't want to spam this thread, but I feel I have to plug my personal pet project Related (https://github.com/sutajio/related https://github.com/sutajio/related), since it is, well.. yeah... "related" to the topic being discussed. It can't do half of what Titan does and is a much, much simpler design. But it is fast, easy to use and works really well for 80% of the use cases you might have for a graph database on the web (social graphs, semantic web stuff, etc.)
moondowner 14y ago
Anyone tried to use this with Spring?
- moondowner 14y ago
  Why the downvote? Spring Data Neo4j [1] works excellent, it's pretty popular and I'm interested has anyone tried working with TITAN in a Spring project(probably using the Blueprints API). [1] http://www.springsource.org/spring-data/neo4j http://www.springsource.org/spring-data/neo4j
sravfeyn 14y ago
"Vertex-centric indices provide vertex-level querying to alleviate issues with the infamous super node problem" Does that mean, for each vertex, is it's sub-graph indexed?!

170 points by d0ugal 14y ago

nchuhoai 14y ago

Interesting, does anyone know how this compares with the popular Neo4j Database?

norkakn 14y ago
Each Neo4j node stores all of the data, and it doesn't scale write horizontally well. OrientDB tries to scale writes (I'll be testing this in a few months), but still stores all of the data everywhere. This looks like it shards the data automagically. If it works well, I might be able to bang on it a bit, but I'm guessing that it gives shit performance for complex graph questions.
- okram 14y ago
  Titan exposes graph data over a machine cluster. It is an OLTP system that allows you to do local neighborhood graph traversals in sub-second time. For OLAP processing (e.g. global graph algorithms), Aurelius will be releasing two projects named Faunus and Fulgora in the coming months. These provide Hadoop connectivity and compressed in-memory representations of "graph slices." We will be publishing our talk slides tonight that discuss this eco-system of graph technologies. See http://titanbiggraphdata.eventbrite.com/ http://titanbiggraphdata.eventbrite.com/
  - jbellis 14y ago
    Congrats on the release, Marko!
  - norkakn 14y ago
    I really wish that I could attend. We ask a lot of questions like: Find all Nodes with a property in a tree Find all leaves L of those nodes Find all annotations in a DAG of those leaves Collapse similar DAG entries by backtracking up the graph based on edge weights Writes are bulk loaded, and right now, we are just trying to push all of the graph stuff offline, but there are some limitations to that, and we could really up our accuracy by being able to perform these queries quickly.
- okram 14y ago
  Also, Aurelius will be posting a blog post in ~2 weeks where we will be simulating Twitter. We replay Twitter from day 1 to June 2009 and slam it with ~10,000 concurrent users writing/reading follows relationships, tweets, stream constructions -- ultimately growing (what we think will be) a 3 billion edge graph when the simulation is complete.

oacgnol 14y ago

Very interesting. Since it implements Blueprints, does it also have support for Furnace (graph algorithms)? If so, does this imply that graph processing is done on disk rather than in-memory? I am rather unfamiliar with Blueprints but I'm wondering how Titan implements that aspect.

joe_the_user 14y ago

Also, What is interesting about a graph database relative to simple key-value database? Storing edges of a graph is trivial for a key-value store and so it seems like any key-value store could let store the basic graph structure? Do graph databases support graphic-specific queries and indices?

netghost 14y ago
They seem to implement a variety of graph centric APIs. From a query standpoint, it's easier to use a graph database to do something like find all the people who share interests with someone (person <--> interest <--> person), or find all the grand children in a family tree. Now whether or not Titan implements these performantly, I have no idea.
JPKab 14y ago
I can only answer for one advantage I specifically know of regarding graph DB's over key value: dynamic, mergeable schemas which enforce data integrity WITHIN the database rather than with code on top of it. There are many, many people on HN who are much more knowledgeable than I am on graph DB's, and I sure as hell hope they answer on this question. I'm curious if this supports the RDF, OWL, and SPARQL standards? I'm a little tired of graph DB's that focus on scale, rather than speed and flexibility though. A good one to check out is Stardog. http://stardog.com/ http://stardog.com/ I think it just went into 1.0.
- philjohn 14y ago
  It looks like it's Gremlin only. What triple stores have you looked at? 4store is performant, but doesn't support reasoning. There's also BigData and Virtuoso which support various levels of it, and Franz are apparently working on a clustered version of Allegrograph.
dkersten 14y ago
Sure you can easily store graphs in a KV store, but graph databases allow you to efficiently traverse the graph too. Look up how to traverse graph structures in hadoop for example - I was looking it up recently for something Im building and in the end I opted for using hadoop to filter, aggregate and format the raw data and then input the preprocessed data into graph databases (Im using orientdb at the moment) for traversal queries. I duplicate the data on multiple machines and use storm bolts to do geaph queries (so each storm bolt has its own local copy of the graph database). Seems to be working well so far and allows me to do realtime traversals (though inserts wait hadoop the next hadoop batch job). Seemed a lot easier than what Id have to do to efficiently process using hadoop.

PaulHoule 14y ago

Where's the SPARQL support?

okram 14y ago
Titan currently does not have "edge indexing" and thus can not implement Blueprints' GraphSail interface [https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementation https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementa...]. If it did implement GraphSail, then SPARQL would be supported via the Sail SPARQL engine.

jandrewrogers 14y ago

What makes this graph database "highly scalable, distributed"? There are difficult theoretical computer science problems that effectively limit the parallelization/distribution of generalized graph operators. To achieve high scalability you have to solve these computer science problems first. If this design offers a novel solution to the longstanding computer science problem then kudos, but nothing at the site suggests this is the case. Many graph databases have claimed high scalability and distributability but none of those claims have held up over time due to the aforementioned computer science problems. This may be a very nice graph database but I am skeptical of the claims of "highly scalable, distributed" unless there is evidence that it uses fundamentally new theoretical computer science to achieve that.

th0ma5 14y ago
You mean CAP theorem? http://en.wikipedia.org/wiki/CAP_theorem http://en.wikipedia.org/wiki/CAP_theorem I imagine it is either the A or the P that gets to be the victim, but I'm not sure which in this case.
- tptacek 14y ago
  Worth a re-read: http://codahale.com/you-cant-sacrifice-partition-tolerance/ http://codahale.com/you-cant-sacrifice-partition-tolerance/
  - th0ma5 14y ago
    Thank you, this is really great to show AP or CP but never CA and call it distributed.
- okram 14y ago
  See https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-graph-data?slide=126 https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap...
  - th0ma5 14y ago

okram 14y ago

We are going to address alot of these questions in our presentation tonight. However, for the Hacker News crew that won't be there tonight, here is an early release of the Titan talk. https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-graph-data https://speakerdeck.com/u/okram/p/titan-the-rise-of-big-grap... http://titanbiggraphdata.eventbrite.com/ http://titanbiggraphdata.eventbrite.com/ Enjoy!, Marko.

hendler 14y ago
Great presentation.
- benjaminRRR 14y ago
  Also it appears like a great way to take speakerdeck down...

niho 14y ago

I don't want to spam this thread, but I feel I have to plug my personal pet project Related (https://github.com/sutajio/related https://github.com/sutajio/related), since it is, well.. yeah... "related" to the topic being discussed. It can't do half of what Titan does and is a much, much simpler design. But it is fast, easy to use and works really well for 80% of the use cases you might have for a graph database on the web (social graphs, semantic web stuff, etc.)

moondowner 14y ago

Anyone tried to use this with Spring?

moondowner 14y ago
Why the downvote? Spring Data Neo4j [1] works excellent, it's pretty popular and I'm interested has anyone tried working with TITAN in a Spring project(probably using the Blueprints API). [1] http://www.springsource.org/spring-data/neo4j http://www.springsource.org/spring-data/neo4j

sravfeyn 14y ago

"Vertex-centric indices provide vertex-level querying to alleviate issues with the infamous super node problem" Does that mean, for each vertex, is it's sub-graph indexed?!