8 ms·

Neo4J has been around so long -- is anyone here using it in production and really happy with it, where it's like really crushing a use-case? I feel like 90% of

Neo4J has been around so long -- is anyone here using it in production and really happy with it, where it's like really crushing a use-case? I feel like 90% of the applications in existence can go so far with a regular RDBMS that they never try out Neo4j... I know that's the case with me. Half the time I think I'd just try throwing Agensgraph[0] at the problem instead of jumping to the community version of Neo4J. [0]: https://github.com/bitnine-oss/agensgraph https://github.com/bitnine-oss/agensgraph

latchkey 8y ago
I used it heavily in production in 2013, for a dating site. I'm sure it has gotten better since then (or at least I hope so). At the time, it was really just a single node. The issue with graph databases where RDBMS doesn't scale is that the graph can grow quite quickly. I've kept a loose eye on the space since then and I think there is quite a few competitors.
- philjohn 8y ago
  This. It's the reason why the heavy hitting products in this space (MarkLogic, OpenLink Virtuoso, AllegroGraph) are all scale out products. A graph can get HUGE very easily.
  - emileifrem 8y ago
    Indeed, and Neo4j has scaled out horizontally for reads since 2011. However, the really hard problem with scaling graphs is scaling writes, i.e. partitioning. In my mind, no one has solved that well today because you can't just slap it on top of a partitioning algorithm that's designed for data without explicit relationships (think documents, key value pairs). In order to gain something other than a checkbox feature and "sharding claims" you have to partition based on the shape of the graph at the time of insertion, but also revise it continuously as the graph evolves over time. That's a non-trivial problem that no one has solved today. (Yes, we're obviously working on it here at Neo4j.) The good news is that you can get really really far with the replicated horizontal scale out model. "Big RAM is growing faster than big data" as you've probably heard and today there are massive Neo4j deployments in production using our third generation scale out architecture (Raft based, multi clustering, causal consistency).
    - espeed 8y ago
      Hi Emil - I don't understand this comment. Last year, the MIT team showed the GraphBLAS/D4M model can achieve 100M inserts per second on a cluster [1], and it's improved since then. The GraphBLAS [2] standard has been in the works for more than 10 years. It's the culmination of the initial D4M matrix model design by Jeremy Kepner [3] and his team at MIT Lincoln Laboratory Supercomputing Center, which models graphs in the language of linear algebra. And for the last ~5 years or so the GraphBLAS software model has been designed in collaboration with hardware teams at Intel, NVIDIA, IBM, and the labs to make chips and architectures optimized for these new matrix models and capable of scaling out to exascale [4]. From a query perspective the GraphBLAS model is even better now that GPU and TPU accelerators are populating the data centers since it means you can now run local graph queries and global graph analytics algorithms on the same system and both return results in less than a second. * Google Cloud GPUs https://cloud.google.com/gpu/ https://cloud.google.com/gpu/ * Google Cloud TPUs https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ For an overview of GraphBLAS in the context Heterogeneous High-Performance Computing (HHPC) systems running on NVIDIA GPUs and Intel Xeon Phis, see the 2015 talk Scott McMillan [5] gave at the CMU Software Engineering Institute [6]. GraphBLAS is in RedisGraph now [7] -- it uses the official GraphBLAS C implementation written by Tim Davis [8], who as you know implements the underlying sparse matrix algos used in everything from MATLAB to Google Maps -- see his recent talk: * RedisGraph in the Language of Linear Algebra with GraphBLAS, co-presented by RedisLabs and Tim Davis [video] https://www.youtube.com/watch?v=xnez6tloNSQ https://www.youtube.com/watch?v=xnez6tloNSQ Adoption of the GraphBLAS standard by hardware chip manufacturers is a sign that the golden-age of graphs is upon us -- the the graph hardware/software model has effectively been solved, and the preceding wave created by AI/deep-learning demand led the way for the emergence of GPU/TPU accelerators in the cloud -- all the forces have aligned. But for whatever reason, I haven't heard anything about Neo4j in terms of GraphBLAS. What's Neo4j's official position on this and the adoption of the GraphBLAS standard? I realize it's a big change and would require a big architectural overhaul, but it's been in the works for 10 years, and most of the vendors have been involved for the last 5 years. Has Neo4j been involved in the GraphBLAS design process and/or are you moving toward adopting the standard? ---//--- [1] Achieving 100M database inserts per second using Apache Accumulo and D4M [pdf] http://www.ieee-hpec.org/2014/CD/index_htm_files/FinalPapers/31.pdf http://www.ieee-hpec.org/2014/CD/index_htm_files/FinalPapers... Previous Discussion: https://news.ycombinator.com/item?id=13465141 https://news.ycombinator.com/item?id=13465141 [2] GraphBLAS Standard http://graphblas.org http://graphblas.org [3] Jeremy Kepner http://www.mit.edu/~kepner/ http://www.mit.edu/~kepner/ [4] GraphBLAS: Building Blocks For High Performance Graph Analytics https://crd.lbl.gov/news-and-publications/news/2017/graphblas-building-blocks-for-high-performance-graph-analytics/ https://crd.lbl.gov/news-and-publications/news/2017/graphbla... [5] Scott McMillan https://insights.sei.cmu.edu/author/scott-mcmillan/ https://insights.sei.cmu.edu/author/scott-mcmillan/ [6] Graph Algorithms on Future Architectures [video] https://www.youtube.com/watch?v=-sIdS4cz7-4 https://www.youtube.com/watch?v=-sIdS4cz7-4 [7] RedisGraph https://oss.redislabs.com/redisgraph/ https://oss.redislabs.com/redisgraph/ [8] Tim Davis http://faculty.cse.tamu.edu/davis/ http://faculty.cse.tamu.edu/davis/ Previous discussion: https://news.ycombinator.com/item?id=18081978 https://news.ycombinator.com/item?id=18081978
threeseed 8y ago
Yes I've seen it used in many situations where any other non-graph database would struggle. If you're doing security or fraud detection, geospatial analytics, data lineage, anything social, dynamic relationship modelling etc. then graph works well. RDBMS has its strengths but it isn't suitable for every use case.
emileifrem 8y ago
We just did our annual conference GraphConnect in NYC last month. There's a BUNCH of talks online of people who are very successfully and very happily using it in production deployments. Check it out here: https://neo4j.com/graphconnect-2018 https://neo4j.com/graphconnect-2018 Btw, if you've ever stayed at a Marriott (or anyone they now own: Starwood, Hilton, Ritz Carlton) then you've used Neo4j in production (to book that room). If you've ever purchased a flight ticket, then you've used Neo4j in production (over 99% of all fare calculations are done with Neo4j). Etc.
- weavie 8y ago
  Out of curiosity, how would a graph database help when booking a room? I would have thought a simple table: create table room_booking { roomid, booking_started, booking_ended } and some simple sql to check if any rows exist within the required date range would have covered it? What other aspects am I not considering?
  - emileifrem 8y ago
    Oh, there's a bunch of stuff in addition to that. I could give it a shot, but would probably mess it up. :) Way better if you hear it from the horse's mouth. Here's Scott Grimes from Marriott at GraphConnect 2016: https://www.youtube.com/watch?v=wdFYEuWWpzo https://www.youtube.com/watch?v=wdFYEuWWpzo Pretty amazing stats in there!
    - weavie 8y ago
      Thanks, I'll have a watch when I get the time.
  - jeletonskelly 8y ago
    I used to work at an OTA (online travel agency, ie Booking.com, Hotels.com). There's all kinds of complications to what seems to be a pretty simple thing: a building with rooms that are either occupied or not. There's multiple rates per room for a single night, then there's something called room rate plans which I won't get into. In other words: it's more complicated that we think (or than it needs to be).
ThePhysicist 8y ago
In general I think that graphs are a very good fit for many problems in analytics (e.g. fraud detection, logistics) and having a database that treats graphs as first-class citizens is a good thing as well. I've used Neo4j in the past and it seemed to be stable and efficient, not sure about how well it scales though. Before, TitanDB (now JanusDB) in conjunction with the Tinkerpop stack was probably my favorite graph DB/stack, but not sure how seriously Datastacks (who owns TitanDB now) continues developing it. And JanusDB as a project didn't seem very active to me (I could be wrong of course). That said, you can construct and handle graphs using relational databases, graph databases give you advantages in that they have (often) better indexing (i.e. O(1) lookups of vertices & adjacent edges) and come with graph querying languages (e.g. Gremlin), which make it much easier to work with graphs compared to SQL (you can use recursive CTEs to walk graphs on the database side as opposed the client side but complex queries are hard/impossible to write like that). I've written a graph DB abstraction layer in the past that also supports SQL backends: https://github.com/7scientists/vortex https://github.com/7scientists/vortex.
- deepGem 8y ago
  Wow, we had used the same stack. Neo4j - gave up because of horizontal scaling issues, moved to TitanDB and then our startup folded. Fast forward 3 years and am back in the graph world, this time dealing with money laundering and fraud rings. It stumps me as to why graphs are not used more in the financial world. Financial transactions are no different from social interactions. So concepts such as community detection etc apply to the financial realm as well and in fact we are using such concepts to determine fraud rings. I may get back to neo4j.
  - emileifrem 8y ago
    Feel free to have a look again and let me know your thoughts. emil@neo4j.com or @emileifrem. 20 of the 25 biggest banks in the world use Neo4j today, and fraud detection (specifically fraud rings) is a really popular use case. (So much so that if you search for "fraud detection" in an incog chrome window, https://neo4j.com/use-cases/fraud-detection/ https://neo4j.com/use-cases/fraud-detection/ is on the first page of google). Btw, we've scaled out horizontally for reads since 2011 but our initial Paxos based implementation was shaky. Since then we've revised it twice and our third generation Raft based architecture is significantly better. If you add multi clustering support + causal consistency and a bunch of other improvements I think our horizontal scalability story is very different today from a few years ago. Lemme know if you try it out and disagree!
gibsonf1 8y ago
There are two kinds of graph databases, property graph and rdf semantic graph. Neo 4J is the former, which I don't think is that useful in primarily only showing some kind of relation between things. The rdf semantic graph approach, however, captures the semantics of the relationship allowing inference and queries not normally possible with rdbms. It's an easier cultural change in thinking from rdbms to property graph, but also not a huge improvement in terms of what you can do vs an rdbms. Going with a full semantic graph where the relationship is also represented by a node with unlimited relationships, and moving and thinking in hierarchies and inference, is a complete cultural change with an impressive productivity and capability improvement not possible with rdbms. Allegrograph is a good example of a semantic graph which can handle trillions of triples.
- castlecrasher2 8y ago
  Do you have a good example of when an RDF is superior? My looking into RDF's was summarized as "way too complex for little gain" but I admittedly was looking at it through my SQL/Neo4j lens.
  - 131012 8y ago
    I'm working on a project with old school RDF & semantic web, and I tend to concur with you. I often miss my usual RDBMS. Maybe it's the way my data is structured, but any complex display of data quickly becomes a weird maze of async calls. I'd be happy to be proven wrong, I love graphs!
    - gibsonf1 8y ago
      You have to think completely differently from rdbms to use graph well. It's a hard cultural change to make.
      - technige 8y ago
        True, but a very worthwhile one. Before getting into Neo4j (and later joining the company), I'd worked with SQL databases a lot and, at one point, as a DBA. I spent the first few months trying to draw direct analogies between graphs and tables but after a while realised that I needed to "unlearn" a load of what I thought I already knew. It's now pretty refreshing to be able to look from a higher vantage point at the different types of data store available, and work out which is the right one for each job. It turns out that the graph is actually the best solution more often than you'd think!
mschaef 8y ago
> is anyone here using it in production and really happy with it, where it's like really crushing a use-case? I've used it in production settings where it was NOT crushing a use case. (A couple small internal HR-related sites that would've been better suited to an RDBMS.) Based on that experience, my impression is that it would take a fairly specific graph-oriented problem to get me seriously thinking about paying the costs associated with something like Neo4J when compared to a traditional RDBMS. (Less mature compared to RDBMS systems, less well understood data model, less common query language, less support from other tooling.... all of which can be quite important to the overall costs of a system.)
lolive 8y ago
Neo4J is used in production at my company. We have had problems with the clustering architecture known as HA. It was superseded by causal clustering in latest Neo4J releases. We are still requalifying our app with this new system, but we are reasonably confident that it will improve things significantly. The biggest selling point is that business people have no problem discussing the graph data model with us, data engineers. And their OGM layer is easy to use (but it lacks support for multiple inheritance, at least in its Java version).

latchkey 8y ago

I used it heavily in production in 2013, for a dating site. I'm sure it has gotten better since then (or at least I hope so). At the time, it was really just a single node. The issue with graph databases where RDBMS doesn't scale is that the graph can grow quite quickly. I've kept a loose eye on the space since then and I think there is quite a few competitors.

philjohn 8y ago
This. It's the reason why the heavy hitting products in this space (MarkLogic, OpenLink Virtuoso, AllegroGraph) are all scale out products. A graph can get HUGE very easily.
- emileifrem 8y ago
  Indeed, and Neo4j has scaled out horizontally for reads since 2011. However, the really hard problem with scaling graphs is scaling writes, i.e. partitioning. In my mind, no one has solved that well today because you can't just slap it on top of a partitioning algorithm that's designed for data without explicit relationships (think documents, key value pairs). In order to gain something other than a checkbox feature and "sharding claims" you have to partition based on the shape of the graph at the time of insertion, but also revise it continuously as the graph evolves over time. That's a non-trivial problem that no one has solved today. (Yes, we're obviously working on it here at Neo4j.) The good news is that you can get really really far with the replicated horizontal scale out model. "Big RAM is growing faster than big data" as you've probably heard and today there are massive Neo4j deployments in production using our third generation scale out architecture (Raft based, multi clustering, causal consistency).
  - espeed 8y ago
    Hi Emil - I don't understand this comment. Last year, the MIT team showed the GraphBLAS/D4M model can achieve 100M inserts per second on a cluster [1], and it's improved since then. The GraphBLAS [2] standard has been in the works for more than 10 years. It's the culmination of the initial D4M matrix model design by Jeremy Kepner [3] and his team at MIT Lincoln Laboratory Supercomputing Center, which models graphs in the language of linear algebra. And for the last ~5 years or so the GraphBLAS software model has been designed in collaboration with hardware teams at Intel, NVIDIA, IBM, and the labs to make chips and architectures optimized for these new matrix models and capable of scaling out to exascale [4]. From a query perspective the GraphBLAS model is even better now that GPU and TPU accelerators are populating the data centers since it means you can now run local graph queries and global graph analytics algorithms on the same system and both return results in less than a second. * Google Cloud GPUs https://cloud.google.com/gpu/ https://cloud.google.com/gpu/ * Google Cloud TPUs https://cloud.google.com/tpu/ https://cloud.google.com/tpu/ For an overview of GraphBLAS in the context Heterogeneous High-Performance Computing (HHPC) systems running on NVIDIA GPUs and Intel Xeon Phis, see the 2015 talk Scott McMillan [5] gave at the CMU Software Engineering Institute [6]. GraphBLAS is in RedisGraph now [7] -- it uses the official GraphBLAS C implementation written by Tim Davis [8], who as you know implements the underlying sparse matrix algos used in everything from MATLAB to Google Maps -- see his recent talk: * RedisGraph in the Language of Linear Algebra with GraphBLAS, co-presented by RedisLabs and Tim Davis [video] https://www.youtube.com/watch?v=xnez6tloNSQ https://www.youtube.com/watch?v=xnez6tloNSQ Adoption of the GraphBLAS standard by hardware chip manufacturers is a sign that the golden-age of graphs is upon us -- the the graph hardware/software model has effectively been solved, and the preceding wave created by AI/deep-learning demand led the way for the emergence of GPU/TPU accelerators in the cloud -- all the forces have aligned. But for whatever reason, I haven't heard anything about Neo4j in terms of GraphBLAS. What's Neo4j's official position on this and the adoption of the GraphBLAS standard? I realize it's a big change and would require a big architectural overhaul, but it's been in the works for 10 years, and most of the vendors have been involved for the last 5 years. Has Neo4j been involved in the GraphBLAS design process and/or are you moving toward adopting the standard? ---//--- [1] Achieving 100M database inserts per second using Apache Accumulo and D4M [pdf] http://www.ieee-hpec.org/2014/CD/index_htm_files/FinalPapers/31.pdf http://www.ieee-hpec.org/2014/CD/index_htm_files/FinalPapers... Previous Discussion: https://news.ycombinator.com/item?id=13465141 https://news.ycombinator.com/item?id=13465141 [2] GraphBLAS Standard http://graphblas.org http://graphblas.org [3] Jeremy Kepner http://www.mit.edu/~kepner/ http://www.mit.edu/~kepner/ [4] GraphBLAS: Building Blocks For High Performance Graph Analytics https://crd.lbl.gov/news-and-publications/news/2017/graphblas-building-blocks-for-high-performance-graph-analytics/ https://crd.lbl.gov/news-and-publications/news/2017/graphbla... [5] Scott McMillan https://insights.sei.cmu.edu/author/scott-mcmillan/ https://insights.sei.cmu.edu/author/scott-mcmillan/ [6] Graph Algorithms on Future Architectures [video] https://www.youtube.com/watch?v=-sIdS4cz7-4 https://www.youtube.com/watch?v=-sIdS4cz7-4 [7] RedisGraph https://oss.redislabs.com/redisgraph/ https://oss.redislabs.com/redisgraph/ [8] Tim Davis http://faculty.cse.tamu.edu/davis/ http://faculty.cse.tamu.edu/davis/ Previous discussion: https://news.ycombinator.com/item?id=18081978 https://news.ycombinator.com/item?id=18081978

threeseed 8y ago

Yes I've seen it used in many situations where any other non-graph database would struggle. If you're doing security or fraud detection, geospatial analytics, data lineage, anything social, dynamic relationship modelling etc. then graph works well. RDBMS has its strengths but it isn't suitable for every use case.

emileifrem 8y ago

We just did our annual conference GraphConnect in NYC last month. There's a BUNCH of talks online of people who are very successfully and very happily using it in production deployments. Check it out here: https://neo4j.com/graphconnect-2018 https://neo4j.com/graphconnect-2018 Btw, if you've ever stayed at a Marriott (or anyone they now own: Starwood, Hilton, Ritz Carlton) then you've used Neo4j in production (to book that room). If you've ever purchased a flight ticket, then you've used Neo4j in production (over 99% of all fare calculations are done with Neo4j). Etc.

weavie 8y ago
Out of curiosity, how would a graph database help when booking a room? I would have thought a simple table: create table room_booking { roomid, booking_started, booking_ended } and some simple sql to check if any rows exist within the required date range would have covered it? What other aspects am I not considering?
- emileifrem 8y ago
  Oh, there's a bunch of stuff in addition to that. I could give it a shot, but would probably mess it up. :) Way better if you hear it from the horse's mouth. Here's Scott Grimes from Marriott at GraphConnect 2016: https://www.youtube.com/watch?v=wdFYEuWWpzo https://www.youtube.com/watch?v=wdFYEuWWpzo Pretty amazing stats in there!
  - weavie 8y ago
    Thanks, I'll have a watch when I get the time.
- jeletonskelly 8y ago
  I used to work at an OTA (online travel agency, ie Booking.com, Hotels.com). There's all kinds of complications to what seems to be a pretty simple thing: a building with rooms that are either occupied or not. There's multiple rates per room for a single night, then there's something called room rate plans which I won't get into. In other words: it's more complicated that we think (or than it needs to be).

ThePhysicist 8y ago

In general I think that graphs are a very good fit for many problems in analytics (e.g. fraud detection, logistics) and having a database that treats graphs as first-class citizens is a good thing as well. I've used Neo4j in the past and it seemed to be stable and efficient, not sure about how well it scales though. Before, TitanDB (now JanusDB) in conjunction with the Tinkerpop stack was probably my favorite graph DB/stack, but not sure how seriously Datastacks (who owns TitanDB now) continues developing it. And JanusDB as a project didn't seem very active to me (I could be wrong of course). That said, you can construct and handle graphs using relational databases, graph databases give you advantages in that they have (often) better indexing (i.e. O(1) lookups of vertices & adjacent edges) and come with graph querying languages (e.g. Gremlin), which make it much easier to work with graphs compared to SQL (you can use recursive CTEs to walk graphs on the database side as opposed the client side but complex queries are hard/impossible to write like that). I've written a graph DB abstraction layer in the past that also supports SQL backends: https://github.com/7scientists/vortex https://github.com/7scientists/vortex.

deepGem 8y ago
Wow, we had used the same stack. Neo4j - gave up because of horizontal scaling issues, moved to TitanDB and then our startup folded. Fast forward 3 years and am back in the graph world, this time dealing with money laundering and fraud rings. It stumps me as to why graphs are not used more in the financial world. Financial transactions are no different from social interactions. So concepts such as community detection etc apply to the financial realm as well and in fact we are using such concepts to determine fraud rings. I may get back to neo4j.
- emileifrem 8y ago
  Feel free to have a look again and let me know your thoughts. emil@neo4j.com or @emileifrem. 20 of the 25 biggest banks in the world use Neo4j today, and fraud detection (specifically fraud rings) is a really popular use case. (So much so that if you search for "fraud detection" in an incog chrome window, https://neo4j.com/use-cases/fraud-detection/ https://neo4j.com/use-cases/fraud-detection/ is on the first page of google). Btw, we've scaled out horizontally for reads since 2011 but our initial Paxos based implementation was shaky. Since then we've revised it twice and our third generation Raft based architecture is significantly better. If you add multi clustering support + causal consistency and a bunch of other improvements I think our horizontal scalability story is very different today from a few years ago. Lemme know if you try it out and disagree!

gibsonf1 8y ago

There are two kinds of graph databases, property graph and rdf semantic graph. Neo 4J is the former, which I don't think is that useful in primarily only showing some kind of relation between things. The rdf semantic graph approach, however, captures the semantics of the relationship allowing inference and queries not normally possible with rdbms. It's an easier cultural change in thinking from rdbms to property graph, but also not a huge improvement in terms of what you can do vs an rdbms. Going with a full semantic graph where the relationship is also represented by a node with unlimited relationships, and moving and thinking in hierarchies and inference, is a complete cultural change with an impressive productivity and capability improvement not possible with rdbms. Allegrograph is a good example of a semantic graph which can handle trillions of triples.

castlecrasher2 8y ago
Do you have a good example of when an RDF is superior? My looking into RDF's was summarized as "way too complex for little gain" but I admittedly was looking at it through my SQL/Neo4j lens.
- 131012 8y ago
  I'm working on a project with old school RDF & semantic web, and I tend to concur with you. I often miss my usual RDBMS. Maybe it's the way my data is structured, but any complex display of data quickly becomes a weird maze of async calls. I'd be happy to be proven wrong, I love graphs!
  - gibsonf1 8y ago
    You have to think completely differently from rdbms to use graph well. It's a hard cultural change to make.
    - technige 8y ago
      True, but a very worthwhile one. Before getting into Neo4j (and later joining the company), I'd worked with SQL databases a lot and, at one point, as a DBA. I spent the first few months trying to draw direct analogies between graphs and tables but after a while realised that I needed to "unlearn" a load of what I thought I already knew. It's now pretty refreshing to be able to look from a higher vantage point at the different types of data store available, and work out which is the right one for each job. It turns out that the graph is actually the best solution more often than you'd think!

mschaef 8y ago

> is anyone here using it in production and really happy with it, where it's like really crushing a use-case? I've used it in production settings where it was NOT crushing a use case. (A couple small internal HR-related sites that would've been better suited to an RDBMS.) Based on that experience, my impression is that it would take a fairly specific graph-oriented problem to get me seriously thinking about paying the costs associated with something like Neo4J when compared to a traditional RDBMS. (Less mature compared to RDBMS systems, less well understood data model, less common query language, less support from other tooling.... all of which can be quite important to the overall costs of a system.)

lolive 8y ago

Neo4J is used in production at my company. We have had problems with the clustering architecture known as HA. It was superseded by causal clustering in latest Neo4J releases. We are still requalifying our app with this new system, but we are reasonably confident that it will improve things significantly. The biggest selling point is that business people have no problem discussing the graph data model with us, data engineers. And their OGM layer is easy to use (but it lacks support for multiple inheritance, at least in its Java version).