4 ms·
The writes for Dgraph actually happen in O(log N) time where N = number of subjects for a given predicate; and a background job then syncs the posting lists to
by mrjn 10y ago
The writes for Dgraph actually happen in O(log N) time where N = number of subjects for a given predicate; and a background job then syncs the posting lists to disk later. So, that performance wouldn't be any worse than what we observe here.
However, to do writes like you would do them in production would require you to sync a mutation log to disk, before applying it in memory and returning an OK. This particular operation bounded by disk I/O would then become the bottleneck, and not help with doing benchmarking for the database. Which is why we didn't make writes part of this particular benchmark.
Dgraph does concurrent writes very well. To determine that, you can run Dgraph batch loader. It can load up the entire 21M triples from the blog post, in 20mins on an n1-standard GCE instance.
https://github.com/dgraph-io/dgraph#loader-performance https://github.com/dgraph-io/dgraph#loader-performance
When jumping from 2 to 5 nodes, we did want to ensure that the nodes have to communicate to run the queries, as they would in a real-world scenario; which is why the performance wouldn't just double.
I think you're putting a lot of emphasis on the size of the data. While that is surely important, the size itself isn't the only thing which proves scalability. It's about the concurrency, throughput and latency -- and determining that they do improve when you add more hardware power to the cluster.
- trishume 10y agoThanks for the info. I guess my follow up question is why doesn't Dgraph end up caching all the data on every node when it has the memory to do so? Is this a planned feature that you just haven't implemented yet?
- mrjn 10y agoIt depends upon the memory available. We have a background goroutine which checks how much memory we're using and evicts posting lists if we exceed a predefined usage. If we have a lot of memory then we just won't evict the posting lists, which means they'll be in RAM. All this "caching" happens close to the disk level, just above RocksDB. We don't cache any results -- to me caching is cheating -- and we want to build something truly low latency. A user can easily add caching if they want to decrease the load on Dgraph, but it's not something I think we should be doing at the db level.