5 ms·

I think the wikidata was the right decision :) SPARQL really is lovely when dealing with public databases over http. But then I am biased by the success of spar

by jerven 11y ago

I think the wikidata was the right decision :) SPARQL really is lovely when dealing with public databases over http. But then I am biased by the success of sparql.uniprot.org :) On the other hand the usecase of wikidata and uniprot are quite different from most graph database deployments who have only internal or controlled access via APIs. Still UniProt is a graph with 3 billion nodes and 15 billion edges so not tiny but not humongous either. Wikidata is a bit smaller if I recall. Titan seems to have a different use case, much more orientated to graph traversal than analytics on graph modelled data. So I can understand that many systems need something like titan.

lobster_johnson 11y ago
For my part I'm evaluating graph databases for content storage, as backing store for webapps, as an alternative to relational databases. Number of nodes less than a million, so very small datasets by most standards, and I do need fast ad-hoc queries (what you call graph traversal, not analytics), sharding and transaction support.
- rspeer 11y ago
  Other people may convince you otherwise, but I believe there are no mature graph databases. You would use them if you want to experience the bleeding edge of databases and exploring uncharted territory excites you, and not if you want to get webapps built.
  - emileifrem 11y ago
    I'm curious, what is lacking from Neo4j to qualify it as a mature graph database? We've worked on it for over a decade, it's used in production by thousands of community users, hundreds of customers and 75+ Global 2000 companies (see http://neo4j.com/customers http://neo4j.com/customers). For many of those Neo4j is used in business critical use cases, i.e. they require Neo4j to be up and running every minute of every day or it'll show up in their next earnings call. If you've shopped online or in a US retail store this week for example, it's very likely that you've used Neo4j. There's rich support for pretty much any programming language and framework out there, an ecosystem of consulting partners whose sole business it is to do Neo4j implementations, 10+ books written specifically about Neo4j, rich online training, formal enterprise support backed by a global commercial organization, an active community. What's missing? I'm not trying to be facetious -- I'm genuinely curious as to what you feel is missing to consider it mature.
    - rspeer 11y ago
      The last time I tried Neo4j was in 2010 or 2011, when I was trying to build ConceptNet 5 (http://conceptnet5.media.mit.edu http://conceptnet5.media.mit.edu) on it. It had showstopping security problems when bound to anything but 127.0.0.1, so I came up with a software firewall to put around it and hoped for the best. It promised Lucene search but its implementation was full of Lucene injections, unless I escaped every special character I could think of like a freaking PHP programmer. There was no way to get data in faster than a slow trickle, unless that data was somehow already in another Neo4j database. Doing any interesting graph operations led to interesting messages about running out of "PermGen". And before I could even get all the data in, it had consumed enough resources to blow my academic AWS budget for months. I was on the mailing list looking for support, and found it pretty lacking. The best I ever got was a bunch of Java code to try (my code is in Python). I use SQLite now. It doesn't do very much, but it does what it's supposed to, and that's great. If Neo4J has improved significantly since then, forgive me that I'm not rushing back to try it again.
      - emileifrem 11y ago
        That sucks. :( Sorry about that. Neo4j isn't perfect today and it certainly wasn't perfect 4-5 years ago. We're working hard on it tho! And thanks for being specific (amazed that you remember specific issues from five years ago!). I don't remember the 127.0.0.1 security problems, but I don't hear anything about them so my guess is they've been addressed. We have a lot of finance and government customers that have high requirements on security. As for your Lucene issues, we did a complete overhaul of our search and indexing story in Neo4j 2.0 (released late 2013). We've continuously improved import performance (which has traditionally been a weak spot) and Neo4j 2.2 includes a batch importer which injects >1M records / sec sustained pace at scale (10s of billions of records) on commodity hardware. As for the memory management issues, we like many other data products written in Java struggled with GC for a long time, and like many others we ultimately concluded that we had to move a lot of the critical parts off heap / manage the memory ourselves, which significantly improved memory utilization. I understand that you got stung historically and therefore hesitate to check us out again. And if SQLite is working well for you, there's no need to! But Neo4j and the graph space has matured a LOT since 2010 and fortunately I don't think your "bleeding edge" experience from 4-5 years ago will be replicated anymore for someone coming new into the space. Thanks for the feedback.
- aurelianito 11y ago
  Why do you need sharding on a dataset so small?
  - lobster_johnson 11y ago
    I should perhaps have said replicas (ie., redundant shards).
rusabd 11y ago
what store sparql.uniprot.org uses? Virtuoso?
- jerven 11y ago
  Yes and it has gotten much better in recent releases. On the other hand switching to an other sparql impl takes about 2 weeks of work in evaluation and deployment. We can currently switch to GraphDB or oracle sem net without further effort. Blazegraph stardog are easy to switch too. Others would require a bit more work.

by jerven 11y ago

lobster_johnson 11y ago

For my part I'm evaluating graph databases for content storage, as backing store for webapps, as an alternative to relational databases. Number of nodes less than a million, so very small datasets by most standards, and I do need fast ad-hoc queries (what you call graph traversal, not analytics), sharding and transaction support.

rspeer 11y ago
Other people may convince you otherwise, but I believe there are no mature graph databases. You would use them if you want to experience the bleeding edge of databases and exploring uncharted territory excites you, and not if you want to get webapps built.
- emileifrem 11y ago
  I'm curious, what is lacking from Neo4j to qualify it as a mature graph database? We've worked on it for over a decade, it's used in production by thousands of community users, hundreds of customers and 75+ Global 2000 companies (see http://neo4j.com/customers http://neo4j.com/customers). For many of those Neo4j is used in business critical use cases, i.e. they require Neo4j to be up and running every minute of every day or it'll show up in their next earnings call. If you've shopped online or in a US retail store this week for example, it's very likely that you've used Neo4j. There's rich support for pretty much any programming language and framework out there, an ecosystem of consulting partners whose sole business it is to do Neo4j implementations, 10+ books written specifically about Neo4j, rich online training, formal enterprise support backed by a global commercial organization, an active community. What's missing? I'm not trying to be facetious -- I'm genuinely curious as to what you feel is missing to consider it mature.
  - rspeer 11y ago
    The last time I tried Neo4j was in 2010 or 2011, when I was trying to build ConceptNet 5 (http://conceptnet5.media.mit.edu http://conceptnet5.media.mit.edu) on it. It had showstopping security problems when bound to anything but 127.0.0.1, so I came up with a software firewall to put around it and hoped for the best. It promised Lucene search but its implementation was full of Lucene injections, unless I escaped every special character I could think of like a freaking PHP programmer. There was no way to get data in faster than a slow trickle, unless that data was somehow already in another Neo4j database. Doing any interesting graph operations led to interesting messages about running out of "PermGen". And before I could even get all the data in, it had consumed enough resources to blow my academic AWS budget for months. I was on the mailing list looking for support, and found it pretty lacking. The best I ever got was a bunch of Java code to try (my code is in Python). I use SQLite now. It doesn't do very much, but it does what it's supposed to, and that's great. If Neo4J has improved significantly since then, forgive me that I'm not rushing back to try it again.
    - emileifrem 11y ago
      That sucks. :( Sorry about that. Neo4j isn't perfect today and it certainly wasn't perfect 4-5 years ago. We're working hard on it tho! And thanks for being specific (amazed that you remember specific issues from five years ago!). I don't remember the 127.0.0.1 security problems, but I don't hear anything about them so my guess is they've been addressed. We have a lot of finance and government customers that have high requirements on security. As for your Lucene issues, we did a complete overhaul of our search and indexing story in Neo4j 2.0 (released late 2013). We've continuously improved import performance (which has traditionally been a weak spot) and Neo4j 2.2 includes a batch importer which injects >1M records / sec sustained pace at scale (10s of billions of records) on commodity hardware. As for the memory management issues, we like many other data products written in Java struggled with GC for a long time, and like many others we ultimately concluded that we had to move a lot of the critical parts off heap / manage the memory ourselves, which significantly improved memory utilization. I understand that you got stung historically and therefore hesitate to check us out again. And if SQLite is working well for you, there's no need to! But Neo4j and the graph space has matured a LOT since 2010 and fortunately I don't think your "bleeding edge" experience from 4-5 years ago will be replicated anymore for someone coming new into the space. Thanks for the feedback.
aurelianito 11y ago
Why do you need sharding on a dataset so small?
- lobster_johnson 11y ago
  I should perhaps have said replicas (ie., redundant shards).

rusabd 11y ago

what store sparql.uniprot.org uses? Virtuoso?

jerven 11y ago
Yes and it has gotten much better in recent releases. On the other hand switching to an other sparql impl takes about 2 weeks of work in evaluation and deployment. We can currently switch to GraphDB or oracle sem net without further effort. Blazegraph stardog are easy to switch too. Others would require a bit more work.