4 ms·

I abandoned Neo4j -- and graph databases in general -- years ago because there was no reasonable way to load in several million edges that were not already in a

by rspeer 11y ago

I abandoned Neo4j -- and graph databases in general -- years ago because there was no reasonable way to load in several million edges that were not already in a graph database. Does Neo4j have a better importing story now? I see a blog post from 2014 [1] that makes importing merely a million edges between 500 nodes sound like it's still a terribly difficult operation, giving me the impression that graph databases aren't quite ready for <s>big</s> medium data yet. [1] http://jexp.de/blog/2014/06/load-csv-into-neo4j-quickly-and-successfully/ http://jexp.de/blog/2014/06/load-csv-into-neo4j-quickly-and-... If, for example, I wanted to load an N-Triples file that's approximately the size and shape of DBPedia, can I reasonably do so? What tools should I use to get the job done quickly without descending into a Java nightmare?

lmeyerov 11y ago
We settled on "medium"-batching into Titan, and sharing your experience, I'm hoping the Datastax acquisition means ingest will improve. For the bulk of our work, we do what you'd expect -- load terabytes into HDFS, (Py)Spark for straight SQL and some join helper functions, and occasionally, add some GraphX scala libraries. I'm curious -- what did you end up doing, & for what?
- rspeer 11y ago
  I maintain ConceptNet [1]. It's a large-ish, messy semantic graph linking together crowd-sourced knowledge (from Open Mind Common Sense, Wiktionary, some Games with a Purpose, etc.) with expert-created resources (WordNet, OpenCyc, etc.) [1] http://conceptnet5.media.mit.edu http://conceptnet5.media.mit.edu It's turned out to be a good input for machine learning about semantics, which has changed the goals of its representation a bit -- not only do I need to be able to load in data easily, I also need to be able to iterate over all of it. But some graph operations would be nice to have, too. Many technical people I describe the project to immediately ask me what graph database I'm using, both before and after the ill-fated semester of grad school where I actually tried to use graph databases. The answer to what I use now is: a bit of SQLite and some flat files. No need for HDFS, it still fits easily on a hard disk.
schwarzmx 11y ago
Have you tried Stardog [1]? Stardog can handle billions of triples and the upload process is pretty painless. Disclaimer: I'm one of the developers of Stardog. [1] http://stardog.com/ http://stardog.com/
- rspeer 11y ago
  It needs to be free, open-source, and available for commercial use. I'm not the only person who ever builds ConceptNet, and my company is not the only company who ever builds ConceptNet. It would be unreasonable to ask downstream users to get a Stardog license. Stardog doesn't even show me a price for putting a billion edges into it, just an e-mail link marked "INQUIRE", so I have to assume it would be very, very expensive.
  - schwarzmx 11y ago
    I guess that answers my question. FWIW Developer/Enterprise versions are free to try and Community doesn't expire.
    - rspeer 11y ago
      Free to try, expensive to succeed. Thanks for the offer, but I'd only go with that model if there were no other options, and right now you're competing with SQLite and a filesystem, which have no additional costs.
david_p 11y ago
To my knowledge, Neo4j has been working on performances a lot. Neo4j v2.2+ does a much better job with large-scale graphs. The use-case you describe (a couple million nodes + edges) sounds like a fairly reasonable task for Neo4j. If Neo4j has been a bad experience for you, I also recommend looking into TitanDB v1+, which scales horizontally (backed by Cassandra), although the query language (Gremlin) is not as easy to learn as Cypher.
- amirouche 11y ago
  not sure about Cypher being easier than Gremlin.
nicolewhite 11y ago
I understand your pain around Neo4j's past import capabilities; when I found it around two years ago I had a lot of trouble with this as well. But with the new neo4j-import tool [0] that's no longer an issue. I've imported ~50 million nodes and ~100 million relationships into Neo4j in under four minutes with neo4j-import. [0] http://neo4j.com/docs/stable/import-tool-examples.html http://neo4j.com/docs/stable/import-tool-examples.html
- rspeer 11y ago
  Thanks, that sounds useful. I'll look into that when the site is back up.

by rspeer 11y ago

lmeyerov 11y ago

We settled on "medium"-batching into Titan, and sharing your experience, I'm hoping the Datastax acquisition means ingest will improve. For the bulk of our work, we do what you'd expect -- load terabytes into HDFS, (Py)Spark for straight SQL and some join helper functions, and occasionally, add some GraphX scala libraries. I'm curious -- what did you end up doing, & for what?

rspeer 11y ago
I maintain ConceptNet [1]. It's a large-ish, messy semantic graph linking together crowd-sourced knowledge (from Open Mind Common Sense, Wiktionary, some Games with a Purpose, etc.) with expert-created resources (WordNet, OpenCyc, etc.) [1] http://conceptnet5.media.mit.edu http://conceptnet5.media.mit.edu It's turned out to be a good input for machine learning about semantics, which has changed the goals of its representation a bit -- not only do I need to be able to load in data easily, I also need to be able to iterate over all of it. But some graph operations would be nice to have, too. Many technical people I describe the project to immediately ask me what graph database I'm using, both before and after the ill-fated semester of grad school where I actually tried to use graph databases. The answer to what I use now is: a bit of SQLite and some flat files. No need for HDFS, it still fits easily on a hard disk.

schwarzmx 11y ago

Have you tried Stardog [1]? Stardog can handle billions of triples and the upload process is pretty painless. Disclaimer: I'm one of the developers of Stardog. [1] http://stardog.com/ http://stardog.com/

rspeer 11y ago
It needs to be free, open-source, and available for commercial use. I'm not the only person who ever builds ConceptNet, and my company is not the only company who ever builds ConceptNet. It would be unreasonable to ask downstream users to get a Stardog license. Stardog doesn't even show me a price for putting a billion edges into it, just an e-mail link marked "INQUIRE", so I have to assume it would be very, very expensive.
- schwarzmx 11y ago
  I guess that answers my question. FWIW Developer/Enterprise versions are free to try and Community doesn't expire.
  - rspeer 11y ago
    Free to try, expensive to succeed. Thanks for the offer, but I'd only go with that model if there were no other options, and right now you're competing with SQLite and a filesystem, which have no additional costs.

david_p 11y ago

To my knowledge, Neo4j has been working on performances a lot. Neo4j v2.2+ does a much better job with large-scale graphs. The use-case you describe (a couple million nodes + edges) sounds like a fairly reasonable task for Neo4j. If Neo4j has been a bad experience for you, I also recommend looking into TitanDB v1+, which scales horizontally (backed by Cassandra), although the query language (Gremlin) is not as easy to learn as Cypher.

amirouche 11y ago
not sure about Cypher being easier than Gremlin.

nicolewhite 11y ago

I understand your pain around Neo4j's past import capabilities; when I found it around two years ago I had a lot of trouble with this as well. But with the new neo4j-import tool [0] that's no longer an issue. I've imported ~50 million nodes and ~100 million relationships into Neo4j in under four minutes with neo4j-import. [0] http://neo4j.com/docs/stable/import-tool-examples.html http://neo4j.com/docs/stable/import-tool-examples.html

rspeer 11y ago
Thanks, that sounds useful. I'll look into that when the site is back up.