7 ms·

I haven't read the whole piece yet, but I saw some things that already stood out as strange. > Please keep reading to see how diligent we were in creating a fa

I haven't read the whole piece yet, but I saw some things that already stood out as strange. > Please keep reading to see how diligent we were in creating a fair test case I hope this is true, but my cursory reading found several unfair spots that seem at odds with this statement. > 3-node cluster on single DC | RF=3 DynamoDB works across AZs and will continue to be available even when a data center goes down. That cross-AZ operation has benefits as well as latency costs that are not present in the ScyllaDB setup. > We hit errors on ~50% of the YCSB threads causing them to die when using ≥50% of write provisioned capacity That is surprising. It is quite common to use your full provisioned capacity without problems. My guess is that there is something not ideal about the YCSB DynamoDB library or its configuration. I'm not familiar with YCSB: does it give you stack traces that indicate why the threads failed? > Sadly for DynamoDB, each item weighted 1.1kb – YCSB default schema, thus each write originated in two accesses This is what made me come write this comment. You specifically knew this was a pessimistic case which could be easily addressed to allow DynamoDB to operate at a lower cost. Is arbitrarily settling for the default 1.1kB item size on an artificial benchmark fair? Good engineering teams use their tools the way that gives them the most benefit. Calling out that you may have to work to ensure your use case doesn't have pessimistic characteristics would clearly be fair, but I'm not convinced that just picking arbitrary benchmark settings is.

thekozmo 8y ago
Fair questions, let me (Dor) answer: 1. Scylla works within different AZs too We are topology aware and can have as good and even better HA than DynamoDB. Our design is based on C* 2. We were surprised with getting small utilization too. As I wrote in the article, I think Dynamo had a hard time reaching to 1TB that quick. The population started fine and deeper in the the population it failed. Only a decrease in the rate solved it. 3. 1.1kb The is the default table setup by YCSB. It isn't optimal for Dynamo but that's life.. think about if you store a blob of 1kb - together with the key it will be more than 1kb. We were upfront about it and you can make your own calculation. Scylla will still be better. The only use case where Dynamo is better in price is when you store lots of data but with a tiny IOPS reservation. We'll address this case over time too. Cheers!
- planckscnst 8y ago
  > Scylla works within different AZs too Did your test use this feature of Scylla? As I understand the documentation, it doesn't appear so (maybe that needs to be fixed). If that's the case, do you think this is an example of being diligent in your fairness? > Only a decrease in the rate solved it. I don't think that is the only way to solve it. Something is probably wrong here, and I would expect that diligence to fairness would dictate trying to understand and correct that. This benchmark would be more compelling if problems like this weren't just ignored, but actually solved. Any team that cares about the cost of their database would do that. > It isn't optimal for Dynamo but that's life. Yes, life is unfair and this can happen. This test specifically claimed to be fair, though. Again, for this benchmark to be meaningful and compelling, this kind of thing should be addressed. > The only use case where Dynamo is better in price is when you store lots of data but with a tiny IOPS reservation This is a surprising statement. If this is the case, I highly recommend addressing the issues so that you are using DynamoDB like a typical engineering team would. A blanket statement like this with questionable benchmarks as evidence is just not compelling. It would be much more helpful to see where DynamoDB really can't stand a chance competing against ScyllaDB even when it's fully optimiized. Are you ignoring maintenance costs? Do nodes get added and removed without paying an engineer? How do you keep the nodes up-to-date as far as security updates or adding new features? How much overhead cost is engineering on-call to respond when the cluster has issues (e.g. nearing capacity limits, nodes failing, loss of AZ availability)? I'm also curious if you are only considering use cases that you typically already see on ScyllaDB and Cassandra. How about the use case where I have a load that increases without warning by 10x or where I need regular backups or that requires 5000 nodes?
  - thekozmo 8y ago
    About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit, most of our AWS customers have AZ-aware deployments and it works great. Since I see you're with AWS, you're welcome to help us debug this error 500 on population. I guarantee we'll update the blog if we'll see it works fine I didn't get your surprise, I actually am 100% open about digging deep and come up with a uncommon use case that Dynamo can finally be a good alternative. Believe me, I can come with examples that will put Scylla in a much better light. We do offer a fully managed service on AWS with 4x-6x cost saving vs Dynamo where all the backup and auto scale is done by the Scylla team.
jonstewart 8y ago
I don’t know much about either Dynamo or Scylla, but just a side note to say that YCSB is a well-known benchmark and has been around for years. There have been many bake offs like this. Here’s an example blogpost from 2011 for testing HBase: http://sujee.net/2011/09/12/performance-testing-a-hbase-cluster/#.XBRHvaJOmEc http://sujee.net/2011/09/12/performance-testing-a-hbase-clus...

thekozmo 8y ago

Fair questions, let me (Dor) answer: 1. Scylla works within different AZs too We are topology aware and can have as good and even better HA than DynamoDB. Our design is based on C* 2. We were surprised with getting small utilization too. As I wrote in the article, I think Dynamo had a hard time reaching to 1TB that quick. The population started fine and deeper in the the population it failed. Only a decrease in the rate solved it. 3. 1.1kb The is the default table setup by YCSB. It isn't optimal for Dynamo but that's life.. think about if you store a blob of 1kb - together with the key it will be more than 1kb. We were upfront about it and you can make your own calculation. Scylla will still be better. The only use case where Dynamo is better in price is when you store lots of data but with a tiny IOPS reservation. We'll address this case over time too. Cheers!

planckscnst 8y ago
> Scylla works within different AZs too Did your test use this feature of Scylla? As I understand the documentation, it doesn't appear so (maybe that needs to be fixed). If that's the case, do you think this is an example of being diligent in your fairness? > Only a decrease in the rate solved it. I don't think that is the only way to solve it. Something is probably wrong here, and I would expect that diligence to fairness would dictate trying to understand and correct that. This benchmark would be more compelling if problems like this weren't just ignored, but actually solved. Any team that cares about the cost of their database would do that. > It isn't optimal for Dynamo but that's life. Yes, life is unfair and this can happen. This test specifically claimed to be fair, though. Again, for this benchmark to be meaningful and compelling, this kind of thing should be addressed. > The only use case where Dynamo is better in price is when you store lots of data but with a tiny IOPS reservation This is a surprising statement. If this is the case, I highly recommend addressing the issues so that you are using DynamoDB like a typical engineering team would. A blanket statement like this with questionable benchmarks as evidence is just not compelling. It would be much more helpful to see where DynamoDB really can't stand a chance competing against ScyllaDB even when it's fully optimiized. Are you ignoring maintenance costs? Do nodes get added and removed without paying an engineer? How do you keep the nodes up-to-date as far as security updates or adding new features? How much overhead cost is engineering on-call to respond when the cluster has issues (e.g. nearing capacity limits, nodes failing, loss of AZ availability)? I'm also curious if you are only considering use cases that you typically already see on ScyllaDB and Cassandra. How about the use case where I have a load that increases without warning by 10x or where I need regular backups or that requires 5000 nodes?
- thekozmo 8y ago
  About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit, most of our AWS customers have AZ-aware deployments and it works great. Since I see you're with AWS, you're welcome to help us debug this error 500 on population. I guarantee we'll update the blog if we'll see it works fine I didn't get your surprise, I actually am 100% open about digging deep and come up with a uncommon use case that Dynamo can finally be a good alternative. Believe me, I can come with examples that will put Scylla in a much better light. We do offer a fully managed service on AWS with 4x-6x cost saving vs Dynamo where all the backup and auto scale is done by the Scylla team.

jonstewart 8y ago

I don’t know much about either Dynamo or Scylla, but just a side note to say that YCSB is a well-known benchmark and has been around for years. There have been many bake offs like this. Here’s an example blogpost from 2011 for testing HBase: http://sujee.net/2011/09/12/performance-testing-a-hbase-cluster/#.XBRHvaJOmEc http://sujee.net/2011/09/12/performance-testing-a-hbase-clus...