7 ms·

Scaling nodes in a Dgraph cluster

69 points by mrjn 10y ago

trishume 10y ago
The thing is this benchmark uses a small data set and no concurrent writes. This means that the data set can be trivially cached in memory on every single instance. In this benchmark there is theoretically no need for different database nodes to even communicate with each other after the initial data sync. It is embarrassingly parallel. In fact it is somewhat alarming that going from 2 to 5 nodes doesn't at least double the performance, given that there should be absolutely zero need for coordination between servers.
- ashwin95r 10y ago
  While some data is cached at an instance level by rocksDB, it wouldn't be cached across instances. So, there is a need for coordination/communication among the servers for every request.
- mrjn 10y ago
  The writes for Dgraph actually happen in O(log N) time where N = number of subjects for a given predicate; and a background job then syncs the posting lists to disk later. So, that performance wouldn't be any worse than what we observe here. However, to do writes like you would do them in production would require you to sync a mutation log to disk, before applying it in memory and returning an OK. This particular operation bounded by disk I/O would then become the bottleneck, and not help with doing benchmarking for the database. Which is why we didn't make writes part of this particular benchmark. Dgraph does concurrent writes very well. To determine that, you can run Dgraph batch loader. It can load up the entire 21M triples from the blog post, in 20mins on an n1-standard GCE instance. https://github.com/dgraph-io/dgraph#loader-performance https://github.com/dgraph-io/dgraph#loader-performance When jumping from 2 to 5 nodes, we did want to ensure that the nodes have to communicate to run the queries, as they would in a real-world scenario; which is why the performance wouldn't just double. I think you're putting a lot of emphasis on the size of the data. While that is surely important, the size itself isn't the only thing which proves scalability. It's about the concurrency, throughput and latency -- and determining that they do improve when you add more hardware power to the cluster.
  - trishume 10y ago
    Thanks for the info. I guess my follow up question is why doesn't Dgraph end up caching all the data on every node when it has the memory to do so? Is this a planned feature that you just haven't implemented yet?
    - mrjn 10y ago
      It depends upon the memory available. We have a background goroutine which checks how much memory we're using and evicts posting lists if we exceed a predefined usage. If we have a lot of memory then we just won't evict the posting lists, which means they'll be in RAM. All this "caching" happens close to the disk level, just above RocksDB. We don't cache any results -- to me caching is cheating -- and we want to build something truly low latency. A user can easily add caching if they want to decrease the load on Dgraph, but it's not something I think we should be doing at the db level.
qaq 10y ago
OK if you load this dataset into Postgres it will be about 10-20X this speed for the queries performed in this test. What is the point of benchmarking a graph database on the tiny dataset and using queries that can be run easily on RDBMS? Isn't whole point of Graph Databases that they can perform well on queries that would not run well on RDBMS and handle datasets that RDBMS would choke on?
- lobster_johnson 10y ago
  Not the entire point. Relational databases have schemas that are not well suited to graph structures, and SQL is also a barrier. A better question is whether you need to implement your own storage backend or whether something like Postgres would suffice.
  - qaq 10y ago
    OK then what type of graph would be hard to store in RDBMS? There are obviously queries that one would want to run on graph data that RDBMS will not run efficiently (but the queries in this test are actually trivial for RDBMS). In what way is SQL a barrier compared to learning query language used by some graph database that is not applicable to anything else you do? "A better question is whether you need to implement your own storage backend or whether something like Postgres would suffice". You could just use a more established Graph Database that actually performs well on complex graphs?
    - dominotw 10y ago
      this project https://github.com/google/cayley https://github.com/google/cayley seems to have gone the route of using pluggable backends.
    - nl 10y ago
      Here's a thing that most RDBMS don't do well but graph DBs do: find me nodes connected to at least X nodes of type T where those nodes have attribute A and are also connected to node N. Now duplicate that filter a potentially arbitrary number of times. The issue here is that the number of joins explodes, and depending on your schema you may be doing lots of self joins. An additional complication is if your dataset is too big for a single host. In Postures you shard, but that is manual and has significant cost. In DGraph you lose some performance but (hopefully) if you know something about your queries you can optimize the distribution function to minimize cross node queries. This is a pretty hard problem to generalize, but even a partial solution is good.
mikkom 10y ago
> Freebase is an online collection of structured data which includes contributions from many sources including individual and user-generated contributions. Currently, it has 1.9 Billion RDF N-Triples worth 250GB of uncompressed data. On top of that, this dataset is over 95% accurate with a complex and rich real world schema. It is an ideal data set to test the performance of Dgraph. We decided not to use the entire data set as it wasn’t necessary for our goal here. I don't follow. Wasn't using as much data as possible exactly what scaling is about?
- mrjn 10y ago
  Not exactly. Given enough time and a lot of hard disk, you can put the entire 1.9B triples into any database -- that doesn't prove that it scales. Also, if the database uses a single global mutex lock for all reads and writes, again it's not a scalable architecture. Scaling is about throughput and latency. This post is trying to determine whether adding more machine power actually lets the db perform better.
  - LoSboccacc 10y ago
    > Scaling is about throughput and latency. under reasonable constraint. throughput and latency from a cacheable dataset is moot for big data.
jerven 10y ago
If you are in graph benchmarking you should really have a good look at http://ldbcouncil.org/ http://ldbcouncil.org/. Especially the fact that this is a read only benchmark. Plus your dataset is tiny. So all you showed is that if there is no contention and all data fits on the local machine there is no communication overhead and near perfect scaling. That is not a benchmark that is hard to beat, at all.
- mrjn 10y ago
  This is not a benchmark. But, there is network communication going on in the multi node cluster tests.
nl 10y ago
It seems that a lot of people don't realize that a scalability test is not a performance test! They are close to orthogonal concerns. For example I'll take lower absolute performance if I can scale out. Performance itself can be optimized separately, and usually that optimisation also applies to the scaled out version.
- qaq 10y ago
  They are orthogonal to a point :) Thats why people want to see a real world dataset and queries. If it's performance is 100 times less than an alternative I don't care if it can scale out. Also if it can scale out to 10 nodes it tell's me little about how it will perform with 100 nodes. If It takes 30 sec for complex query on a production dataset I really don't care if I can run 1000 30 sec. queries in parallel if I need that query to run in 50ms on avg and to never run more than 200ms
mrjn 10y ago
Thanks for your comments! I think people feel that the data set wasn't big enough to prove scalability. While I disagree, I think this is a great opportunity as well -- to hear from people. We can actually bring up the entire 1.9B triples from Freebase and run some queries on them -- get some more real world numbers. So, tell us what to run here: https://discuss.dgraph.io/t/give-us-freebase-graphql-queries-to-run-on-dgraph/463 https://discuss.dgraph.io/t/give-us-freebase-graphql-queries...
- lunula 10y ago
  You should run the whole of Freebase, or really the biggest thing you can get your hands on. Hope you do. I'd love something that can comfortably handle billion order graphs!
marknadal 10y ago
For everyone complaining about this not being a performance benchmark, I invite you to check out this test: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . Doing about 30M ops/sec on a Macbook Air. As for the authors of Dgraph, keep it up! Your tech looks promising and I'm excited to see more and more Open Source graph databases in the market. Would love to compare notes, shoot me a message.

69 points by mrjn 10y ago

The thing is this benchmark uses a small data set and no concurrent writes. This means that the data set can be trivially cached in memory on every single instance. In this benchmark there is theoretically no need for different database nodes to even communicate with each other after the initial data sync. It is embarrassingly parallel. In fact it is somewhat alarming that going from 2 to 5 nodes doesn't at least double the performance, given that there should be absolutely zero need for coordination between servers.

ashwin95r 10y ago
While some data is cached at an instance level by rocksDB, it wouldn't be cached across instances. So, there is a need for coordination/communication among the servers for every request.
mrjn 10y ago
The writes for Dgraph actually happen in O(log N) time where N = number of subjects for a given predicate; and a background job then syncs the posting lists to disk later. So, that performance wouldn't be any worse than what we observe here. However, to do writes like you would do them in production would require you to sync a mutation log to disk, before applying it in memory and returning an OK. This particular operation bounded by disk I/O would then become the bottleneck, and not help with doing benchmarking for the database. Which is why we didn't make writes part of this particular benchmark. Dgraph does concurrent writes very well. To determine that, you can run Dgraph batch loader. It can load up the entire 21M triples from the blog post, in 20mins on an n1-standard GCE instance. https://github.com/dgraph-io/dgraph#loader-performance https://github.com/dgraph-io/dgraph#loader-performance When jumping from 2 to 5 nodes, we did want to ensure that the nodes have to communicate to run the queries, as they would in a real-world scenario; which is why the performance wouldn't just double. I think you're putting a lot of emphasis on the size of the data. While that is surely important, the size itself isn't the only thing which proves scalability. It's about the concurrency, throughput and latency -- and determining that they do improve when you add more hardware power to the cluster.
- trishume 10y ago
  Thanks for the info. I guess my follow up question is why doesn't Dgraph end up caching all the data on every node when it has the memory to do so? Is this a planned feature that you just haven't implemented yet?
  - mrjn 10y ago
    It depends upon the memory available. We have a background goroutine which checks how much memory we're using and evicts posting lists if we exceed a predefined usage. If we have a lot of memory then we just won't evict the posting lists, which means they'll be in RAM. All this "caching" happens close to the disk level, just above RocksDB. We don't cache any results -- to me caching is cheating -- and we want to build something truly low latency. A user can easily add caching if they want to decrease the load on Dgraph, but it's not something I think we should be doing at the db level.

qaq 10y ago

OK if you load this dataset into Postgres it will be about 10-20X this speed for the queries performed in this test. What is the point of benchmarking a graph database on the tiny dataset and using queries that can be run easily on RDBMS? Isn't whole point of Graph Databases that they can perform well on queries that would not run well on RDBMS and handle datasets that RDBMS would choke on?

lobster_johnson 10y ago
Not the entire point. Relational databases have schemas that are not well suited to graph structures, and SQL is also a barrier. A better question is whether you need to implement your own storage backend or whether something like Postgres would suffice.
- qaq 10y ago
  OK then what type of graph would be hard to store in RDBMS? There are obviously queries that one would want to run on graph data that RDBMS will not run efficiently (but the queries in this test are actually trivial for RDBMS). In what way is SQL a barrier compared to learning query language used by some graph database that is not applicable to anything else you do? "A better question is whether you need to implement your own storage backend or whether something like Postgres would suffice". You could just use a more established Graph Database that actually performs well on complex graphs?
  - dominotw 10y ago
    this project https://github.com/google/cayley https://github.com/google/cayley seems to have gone the route of using pluggable backends.
  - nl 10y ago
    Here's a thing that most RDBMS don't do well but graph DBs do: find me nodes connected to at least X nodes of type T where those nodes have attribute A and are also connected to node N. Now duplicate that filter a potentially arbitrary number of times. The issue here is that the number of joins explodes, and depending on your schema you may be doing lots of self joins. An additional complication is if your dataset is too big for a single host. In Postures you shard, but that is manual and has significant cost. In DGraph you lose some performance but (hopefully) if you know something about your queries you can optimize the distribution function to minimize cross node queries. This is a pretty hard problem to generalize, but even a partial solution is good.

mikkom 10y ago

> Freebase is an online collection of structured data which includes contributions from many sources including individual and user-generated contributions. Currently, it has 1.9 Billion RDF N-Triples worth 250GB of uncompressed data. On top of that, this dataset is over 95% accurate with a complex and rich real world schema. It is an ideal data set to test the performance of Dgraph. We decided not to use the entire data set as it wasn’t necessary for our goal here. I don't follow. Wasn't using as much data as possible exactly what scaling is about?

mrjn 10y ago
Not exactly. Given enough time and a lot of hard disk, you can put the entire 1.9B triples into any database -- that doesn't prove that it scales. Also, if the database uses a single global mutex lock for all reads and writes, again it's not a scalable architecture. Scaling is about throughput and latency. This post is trying to determine whether adding more machine power actually lets the db perform better.
- LoSboccacc 10y ago
  > Scaling is about throughput and latency. under reasonable constraint. throughput and latency from a cacheable dataset is moot for big data.

jerven 10y ago

If you are in graph benchmarking you should really have a good look at http://ldbcouncil.org/ http://ldbcouncil.org/. Especially the fact that this is a read only benchmark. Plus your dataset is tiny. So all you showed is that if there is no contention and all data fits on the local machine there is no communication overhead and near perfect scaling. That is not a benchmark that is hard to beat, at all.

mrjn 10y ago
This is not a benchmark. But, there is network communication going on in the multi node cluster tests.

nl 10y ago

It seems that a lot of people don't realize that a scalability test is not a performance test! They are close to orthogonal concerns. For example I'll take lower absolute performance if I can scale out. Performance itself can be optimized separately, and usually that optimisation also applies to the scaled out version.

qaq 10y ago
They are orthogonal to a point :) Thats why people want to see a real world dataset and queries. If it's performance is 100 times less than an alternative I don't care if it can scale out. Also if it can scale out to 10 nodes it tell's me little about how it will perform with 100 nodes. If It takes 30 sec for complex query on a production dataset I really don't care if I can run 1000 30 sec. queries in parallel if I need that query to run in 50ms on avg and to never run more than 200ms

mrjn 10y ago

Thanks for your comments! I think people feel that the data set wasn't big enough to prove scalability. While I disagree, I think this is a great opportunity as well -- to hear from people. We can actually bring up the entire 1.9B triples from Freebase and run some queries on them -- get some more real world numbers. So, tell us what to run here: https://discuss.dgraph.io/t/give-us-freebase-graphql-queries-to-run-on-dgraph/463 https://discuss.dgraph.io/t/give-us-freebase-graphql-queries...

lunula 10y ago
You should run the whole of Freebase, or really the biggest thing you can get your hands on. Hope you do. I'd love something that can comfortably handle billion order graphs!

marknadal 10y ago

For everyone complaining about this not being a performance benchmark, I invite you to check out this test: https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . Doing about 30M ops/sec on a Macbook Air. As for the authors of Dgraph, keep it up! Your tech looks promising and I'm excited to see more and more Open Source graph databases in the market. Would love to compare notes, shoot me a message.