7 ms·

RedisGraph: A High Performance In-Memory Graph Database as a Redis Module

237 points by wll 9y ago

deadwisdom 9y ago
Oh man. This is my idea of nirvana. Basically done with any other data-store except maybe for full-text search.
- 9a5351b7-c78e 9y ago
  Seriously. Have you tried this out for full-text search? https://github.com/RedisLabsModules/RediSearch https://github.com/RedisLabsModules/RediSearch
- coleifer 9y ago
  Lightweight python implementation of both graph and full-text search (plus other fun stuff): https://walrus.readthedocs.io/en/latest/ https://walrus.readthedocs.io/en/latest/
- ahtu123 9y ago
  Memory storage is fun but there's a very hard max limit on how much data you can have. For many use cases with constant storage it's awesome though.
  - ricardobeat 9y ago
    How much data do you have? You can get 24TB RAM on a server these days.
    - jazoom 9y ago
      That sounds extremely expensive
      - eat_veggies 9y ago
        They're also really high latency, unless ridiculous-amounts-of-ram server technology has gotten better in the past few years. Here's one I found that supports up to 48 TB: https://www.hpe.com/us/en/product-catalog/servers/integrity-servers/pip.hpe-integrity-mc990-x-server.1008798952.html https://www.hpe.com/us/en/product-catalog/servers/integrity-... They don't include a price, so I'd assume it's in the low millions or high six figures.
        alexforster 9y ago
        The latency doesn't sound that bad. It's 200ns to access memory on another local NUMA node, and 500ns to access memory on another machine using NUMAlink. Fascinating tech. I had no idea this could be done with Xeons.
  - vidarh 9y ago
    While there are certainly plenty of people who need more storage than is practical to put in-memory, most people don't. E.g. it's rare enough to find people with single datasets in the 1TB+ range. Most databases I've come across at clients max out in the tens to hundreds of GB range. Many more could be easily split (e.g. they may be multi-TB or above, but are that way because of aggregation of data that is logically separate, such as relating to different customers, and can easily be sharded) - though whether or not that's worthwhile is a separate issue. So while you're right, for most people it's not going to be a practical problem. Query performance tends to be a bigger problem in practice, and IO bandwidth is often the main cost driver for servers (with NVMe SSDs providing so much additional IO bandwidth that they often pay for themselves several times over by reducing number of servers needed for people who insist on "traditional" disk focused databases), pushing server costs into ranges where spending more on RAM but needing fewer servers to handle the IO load, is more and more often a good tradeoff.
Belphemur 9y ago
This is great ! I'm curious how does it perform in comparison of neo4j. Especially in memory usage.
- namibj 9y ago
  Regarding graph databases: does anyone have specific recommendations regarding something suitable to hold 2-digit TB of graph data, and allow many queries regarding how datapoints that come up are connected to some areas of interest, through the big knowledge graph[0]? I'm looking into doing some prediction work based on it, and have not found any suitable data store so far, which would work well with it, due to the size and budged restraints limiting storage to NVMe modules, instead of DRAM. Any help finding a suitable datastore would be greatly appreciated. [0]: http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook-V2.1.pdf http://data.gdeltproject.org/documentation/GDELT-Global_Know... https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...
  - kawera 9y ago
    Have you checked https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph ?
    - namibj 9y ago
      Thank you for the suggestion, this seems to be exactly what I needed. It seems that the new-ness of it prevented me from finding it earlier. Edit: I just read some more, and it seem like this is a graph-native "brother" of cockroachdb. Which was the one I previously cosndidered, until learning of the inadequate down-pushing ability of graph queries, which would increase network needs a lot. Let me know if you want me to try and notify you once there are benchmarks or production tests successfully completed.
      - porker 9y ago
        Dgraph is very, very immature. I'd monitor their forum for a bit before deciding whether to use - you quickly get a good feel for the issues and raw edges: https://discuss.dgraph.io/ https://discuss.dgraph.io/
weyland108 9y ago
Would this work if I wanted to store the graph forever we generally use redis as memcache to store data with expectation that it will gone at any point in time.
- ahtu123 9y ago
  As long as you have the recommended 1:1 backup set up, it should be almost forever.
- swilly2 9y ago
  This is a wrong misconception as Redis data can be persistent https://redis.io/topics/persistence https://redis.io/topics/persistence Modules extending Redis with new capabilities and data types can and probably should use Redis persistency.
shaolincoder 9y ago
Graph on Redis is cool and I get that hexastore is the way to go. Its very different from the Neo4J like graphs where the nodes and edges have properties. Hexastore is more based on the W3C RDF model. What confuses me is then why not have SPARQL as the query language?
- swilly2 9y ago
  Although RedisGraph current version uses the concept of a Hexastore to represent the Graph I'm not sure this is the best way to go both memory (X6) and CPUwise (when traversing the graph). That said RedisGraph does supports both nodes and edges properties as these are not stored within the Hexastore but on the actual entity object. The new version of RedisGraph which will be presented at RedisConf2018 will completely replace the Hexastore with a new data structure which is better suitable in terms of both memory and cpu.
  - wll 9y ago
    That’s great to hear! I see RedisConf is at April’s end. Would it be possible to get early access? RedisGraph is a crucial component of a system I am designing.
    - swilly2 9y ago
      Well, RedisGraph is an open source project, you can browse its repository at https://github.com/RedisLabsModules/redis-graph https://github.com/RedisLabsModules/redis-graph Although the work which will be presented at Redis Conf has yet to be published.
      - wll 9y ago
        I was indeed wondering about the upcoming release.
jazoom 9y ago
>RediSearch has a distributed cluster version that can scale to billions of documents and hundreds of servers. However, it is only available as part of Redis Labs Enterprise.
kuwze 9y ago
I believe SAP has had this for a while[0] (but they are columnar while Redis is key-value). [0]: http://www.ziti.uni-heidelberg.de/ziti/uploads/ce_group/2015-PELGA.pdf http://www.ziti.uni-heidelberg.de/ziti/uploads/ce_group/2015...
- godelmachine 9y ago
  SAP HANA is a full fledged in memory computing DB.
antirez 9y ago
A week ago while in Tel Aviv I talked with my colleague which is the lead developer of RedisGraph. He will be in San Francisco at Redis Conf with us and will release exiting news about RedisGraph, currently the development is very active.
emehrkay 9y ago
This is really cool. I wrote something similar with Python (it was slow ha) as an exercise to learn Redis a few years back. I used Gremlin/Groovy as an inspiration for the query lang, not Cypher as this project does. https://github.com/emehrkay/rgp https://github.com/emehrkay/rgp
antpls 9y ago
Looks good! Two questions come to my mind : - Is it possible to embed RedisGraph inside an Android application ? - Given it is memory only, is it possible to "dump" the graph to local storage then restore it to memory later ? Another question, more general : Given an X-only application (X being memory or disk), it is possible to emulate the API on top on the other, for example filesystem-in-memory and memory-on-disk (basically, swap I guess). Would you rather build an application using exclusively a Filesystem API, or exclusively a Memory-oriented API ? Best Regards
- dvirsky 9y ago
  > Given it is memory only, is it possible to "dump" the graph to local storage then restore it to memory later ? That's how Redis operates. It happens automatically. You can also back every command that changes the graph to a durable transaction log.
- swilly2 9y ago
  Can RedisGraph be embedded inside an Android application? I don't have much experience developing applications for Android but it mostly comes down to the underline architecture you're aiming for, although Redis 4.0 can run on ARM, RedisGraph currently can't as it access unaligned memory. But incase that's resolved or in the case of Intel CPU you'll have to embed a binary executable within your Android application, i'm pretty sure that's possible. With regard to your second question, I think this a personal preference question, as I've been dealing mostly with in-memory applications for the past couple of years I'll prefer a memory oriented API.
- MR4D 9y ago
  The main features list includes “On disk persistence”. Is that what you were looking for?

237 points by wll 9y ago

deadwisdom 9y ago

Oh man. This is my idea of nirvana. Basically done with any other data-store except maybe for full-text search.

9a5351b7-c78e 9y ago
Seriously. Have you tried this out for full-text search? https://github.com/RedisLabsModules/RediSearch https://github.com/RedisLabsModules/RediSearch
coleifer 9y ago
Lightweight python implementation of both graph and full-text search (plus other fun stuff): https://walrus.readthedocs.io/en/latest/ https://walrus.readthedocs.io/en/latest/
ahtu123 9y ago
Memory storage is fun but there's a very hard max limit on how much data you can have. For many use cases with constant storage it's awesome though.
- ricardobeat 9y ago
  How much data do you have? You can get 24TB RAM on a server these days.
  - jazoom 9y ago
    That sounds extremely expensive
    - eat_veggies 9y ago
      They're also really high latency, unless ridiculous-amounts-of-ram server technology has gotten better in the past few years. Here's one I found that supports up to 48 TB: https://www.hpe.com/us/en/product-catalog/servers/integrity-servers/pip.hpe-integrity-mc990-x-server.1008798952.html https://www.hpe.com/us/en/product-catalog/servers/integrity-... They don't include a price, so I'd assume it's in the low millions or high six figures.
      - alexforster 9y ago
        The latency doesn't sound that bad. It's 200ns to access memory on another local NUMA node, and 500ns to access memory on another machine using NUMAlink. Fascinating tech. I had no idea this could be done with Xeons.
- vidarh 9y ago
  While there are certainly plenty of people who need more storage than is practical to put in-memory, most people don't. E.g. it's rare enough to find people with single datasets in the 1TB+ range. Most databases I've come across at clients max out in the tens to hundreds of GB range. Many more could be easily split (e.g. they may be multi-TB or above, but are that way because of aggregation of data that is logically separate, such as relating to different customers, and can easily be sharded) - though whether or not that's worthwhile is a separate issue. So while you're right, for most people it's not going to be a practical problem. Query performance tends to be a bigger problem in practice, and IO bandwidth is often the main cost driver for servers (with NVMe SSDs providing so much additional IO bandwidth that they often pay for themselves several times over by reducing number of servers needed for people who insist on "traditional" disk focused databases), pushing server costs into ranges where spending more on RAM but needing fewer servers to handle the IO load, is more and more often a good tradeoff.

Belphemur 9y ago

This is great ! I'm curious how does it perform in comparison of neo4j. Especially in memory usage.

namibj 9y ago
Regarding graph databases: does anyone have specific recommendations regarding something suitable to hold 2-digit TB of graph data, and allow many queries regarding how datapoints that come up are connected to some areas of interest, through the big knowledge graph[0]? I'm looking into doing some prediction work based on it, and have not found any suitable data store so far, which would work well with it, due to the size and budged restraints limiting storage to NVMe modules, instead of DRAM. Any help finding a suitable datastore would be greatly appreciated. [0]: http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook-V2.1.pdf http://data.gdeltproject.org/documentation/GDELT-Global_Know... https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/ https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-...
- kawera 9y ago
  Have you checked https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph ?
  - namibj 9y ago
    Thank you for the suggestion, this seems to be exactly what I needed. It seems that the new-ness of it prevented me from finding it earlier. Edit: I just read some more, and it seem like this is a graph-native "brother" of cockroachdb. Which was the one I previously cosndidered, until learning of the inadequate down-pushing ability of graph queries, which would increase network needs a lot. Let me know if you want me to try and notify you once there are benchmarks or production tests successfully completed.
    - porker 9y ago
      Dgraph is very, very immature. I'd monitor their forum for a bit before deciding whether to use - you quickly get a good feel for the issues and raw edges: https://discuss.dgraph.io/ https://discuss.dgraph.io/

weyland108 9y ago

Would this work if I wanted to store the graph forever we generally use redis as memcache to store data with expectation that it will gone at any point in time.

ahtu123 9y ago
As long as you have the recommended 1:1 backup set up, it should be almost forever.
swilly2 9y ago
This is a wrong misconception as Redis data can be persistent https://redis.io/topics/persistence https://redis.io/topics/persistence Modules extending Redis with new capabilities and data types can and probably should use Redis persistency.

shaolincoder 9y ago

Graph on Redis is cool and I get that hexastore is the way to go. Its very different from the Neo4J like graphs where the nodes and edges have properties. Hexastore is more based on the W3C RDF model. What confuses me is then why not have SPARQL as the query language?

swilly2 9y ago
Although RedisGraph current version uses the concept of a Hexastore to represent the Graph I'm not sure this is the best way to go both memory (X6) and CPUwise (when traversing the graph). That said RedisGraph does supports both nodes and edges properties as these are not stored within the Hexastore but on the actual entity object. The new version of RedisGraph which will be presented at RedisConf2018 will completely replace the Hexastore with a new data structure which is better suitable in terms of both memory and cpu.
- wll 9y ago
  That’s great to hear! I see RedisConf is at April’s end. Would it be possible to get early access? RedisGraph is a crucial component of a system I am designing.
  - swilly2 9y ago
    Well, RedisGraph is an open source project, you can browse its repository at https://github.com/RedisLabsModules/redis-graph https://github.com/RedisLabsModules/redis-graph Although the work which will be presented at Redis Conf has yet to be published.
    - wll 9y ago
      I was indeed wondering about the upcoming release.

jazoom 9y ago

>RediSearch has a distributed cluster version that can scale to billions of documents and hundreds of servers. However, it is only available as part of Redis Labs Enterprise.

kuwze 9y ago

I believe SAP has had this for a while[0] (but they are columnar while Redis is key-value). [0]: http://www.ziti.uni-heidelberg.de/ziti/uploads/ce_group/2015-PELGA.pdf http://www.ziti.uni-heidelberg.de/ziti/uploads/ce_group/2015...

godelmachine 9y ago
SAP HANA is a full fledged in memory computing DB.

antirez 9y ago

A week ago while in Tel Aviv I talked with my colleague which is the lead developer of RedisGraph. He will be in San Francisco at Redis Conf with us and will release exiting news about RedisGraph, currently the development is very active.

emehrkay 9y ago

This is really cool. I wrote something similar with Python (it was slow ha) as an exercise to learn Redis a few years back. I used Gremlin/Groovy as an inspiration for the query lang, not Cypher as this project does. https://github.com/emehrkay/rgp https://github.com/emehrkay/rgp

antpls 9y ago

Looks good! Two questions come to my mind : - Is it possible to embed RedisGraph inside an Android application ? - Given it is memory only, is it possible to "dump" the graph to local storage then restore it to memory later ? Another question, more general : Given an X-only application (X being memory or disk), it is possible to emulate the API on top on the other, for example filesystem-in-memory and memory-on-disk (basically, swap I guess). Would you rather build an application using exclusively a Filesystem API, or exclusively a Memory-oriented API ? Best Regards

dvirsky 9y ago
> Given it is memory only, is it possible to "dump" the graph to local storage then restore it to memory later ? That's how Redis operates. It happens automatically. You can also back every command that changes the graph to a durable transaction log.
swilly2 9y ago
Can RedisGraph be embedded inside an Android application? I don't have much experience developing applications for Android but it mostly comes down to the underline architecture you're aiming for, although Redis 4.0 can run on ARM, RedisGraph currently can't as it access unaligned memory. But incase that's resolved or in the case of Intel CPU you'll have to embed a binary executable within your Android application, i'm pretty sure that's possible. With regard to your second question, I think this a personal preference question, as I've been dealing mostly with in-memory applications for the past couple of years I'll prefer a memory oriented API.
MR4D 9y ago
The main features list includes “On disk persistence”. Is that what you were looking for?