3 ms·

Regarding graph databases: does anyone have specific recommendations regarding something suitable to hold 2-digit TB of graph data, and allow many queries regar

by namibj 9y ago

Regarding graph databases: does anyone have specific recommendations regarding something suitable to hold 2-digit TB of graph data, and allow many queries regarding how datapoints that come up are connected to some areas of interest, through the big knowledge graph[0]? I'm looking into doing some prediction work based on it, and have not found any suitable data store so far, which would work well with it, due to the size and budged restraints limiting storage to NVMe modules, instead of DRAM. Any help finding a suitable datastore would be greatly appreciated. [0]: http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook-V2.1.pdf http://data.gdeltproject.org/documentation/GDELT-Global_Know... https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...

kawera 9y ago
Have you checked https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph ?
- namibj 9y ago
  Thank you for the suggestion, this seems to be exactly what I needed. It seems that the new-ness of it prevented me from finding it earlier. Edit: I just read some more, and it seem like this is a graph-native "brother" of cockroachdb. Which was the one I previously cosndidered, until learning of the inadequate down-pushing ability of graph queries, which would increase network needs a lot. Let me know if you want me to try and notify you once there are benchmarks or production tests successfully completed.
  - porker 9y ago
    Dgraph is very, very immature. I'd monitor their forum for a bit before deciding whether to use - you quickly get a good feel for the issues and raw edges: https://discuss.dgraph.io/ https://discuss.dgraph.io/
sandGorgon 9y ago
The production stack for this is almost always postgresql/Cassandra+spark. Some people also use stuff like Janus graph, but that is still too new.
doh 9y ago
Haven't tried in production, but Janus Graph[0] supposed to be able to scale to the sizes you're describing. The another one I looked at is ArangoDB[1] but again, never used in production. [0] http://janusgraph.org http://janusgraph.org [1] https://www.arangodb.com https://www.arangodb.com
mikestaub 9y ago
https://github.com/arangodb/arangodb https://github.com/arangodb/arangodb
janemanos 9y ago
Hi Jan from ArangoDB here. 2-digit TB of graph data is for sure a serious graph :) Depending on what you actually want to analyze and how the graph is structured, you can either use our Pregel integration for high-level analytics of the data set or use SmartGraphs to shard the data to a cluster and perform in-depth traversals, pattern matching and such stuff. SmartGraphs is an Enterprise feature but free for evaluation and testing. Please note that the suitability of SmartGraphs depends on the structure of the graph itself (e.g. do you have or can identify communities to shard by efficiently) We created a series of tutorials which guide you through such a process https://www.arangodb.com/pregel-community-detection/ https://www.arangodb.com/pregel-community-detection/ (next step always at the end of each tutorial). You can choose between two storage engines. Think RocksDB is the better choice in your case, as everything is persisted to disk (data/indexes) and you can configure how much main memory should be used, so you can configure the trade-off between performance and main memory yourself. If you have fast SSDs it's even better, as RocksDB is optimized for that.

by namibj 9y ago

kawera 9y ago

Have you checked https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph ?

namibj 9y ago
Thank you for the suggestion, this seems to be exactly what I needed. It seems that the new-ness of it prevented me from finding it earlier. Edit: I just read some more, and it seem like this is a graph-native "brother" of cockroachdb. Which was the one I previously cosndidered, until learning of the inadequate down-pushing ability of graph queries, which would increase network needs a lot. Let me know if you want me to try and notify you once there are benchmarks or production tests successfully completed.
- porker 9y ago
  Dgraph is very, very immature. I'd monitor their forum for a bit before deciding whether to use - you quickly get a good feel for the issues and raw edges: https://discuss.dgraph.io/ https://discuss.dgraph.io/

sandGorgon 9y ago

The production stack for this is almost always postgresql/Cassandra+spark. Some people also use stuff like Janus graph, but that is still too new.

doh 9y ago

Haven't tried in production, but Janus Graph[0] supposed to be able to scale to the sizes you're describing. The another one I looked at is ArangoDB[1] but again, never used in production. [0] http://janusgraph.org http://janusgraph.org [1] https://www.arangodb.com https://www.arangodb.com

mikestaub 9y ago

https://github.com/arangodb/arangodb https://github.com/arangodb/arangodb

janemanos 9y ago

Hi Jan from ArangoDB here. 2-digit TB of graph data is for sure a serious graph :) Depending on what you actually want to analyze and how the graph is structured, you can either use our Pregel integration for high-level analytics of the data set or use SmartGraphs to shard the data to a cluster and perform in-depth traversals, pattern matching and such stuff. SmartGraphs is an Enterprise feature but free for evaluation and testing. Please note that the suitability of SmartGraphs depends on the structure of the graph itself (e.g. do you have or can identify communities to shard by efficiently) We created a series of tutorials which guide you through such a process https://www.arangodb.com/pregel-community-detection/ https://www.arangodb.com/pregel-community-detection/ (next step always at the end of each tutorial). You can choose between two storage engines. Think RocksDB is the better choice in your case, as everything is persisted to disk (data/indexes) and you can configure how much main memory should be used, so you can configure the trade-off between performance and main memory yourself. If you have fast SSDs it's even better, as RocksDB is optimized for that.