10 ms·

> Scylla works within different AZs too Did your test use this feature of Scylla? As I understand the documentation, it doesn't appear so (maybe that needs to

> Scylla works within different AZs too Did your test use this feature of Scylla? As I understand the documentation, it doesn't appear so (maybe that needs to be fixed). If that's the case, do you think this is an example of being diligent in your fairness? > Only a decrease in the rate solved it. I don't think that is the only way to solve it. Something is probably wrong here, and I would expect that diligence to fairness would dictate trying to understand and correct that. This benchmark would be more compelling if problems like this weren't just ignored, but actually solved. Any team that cares about the cost of their database would do that. > It isn't optimal for Dynamo but that's life. Yes, life is unfair and this can happen. This test specifically claimed to be fair, though. Again, for this benchmark to be meaningful and compelling, this kind of thing should be addressed. > The only use case where Dynamo is better in price is when you store lots of data but with a tiny IOPS reservation This is a surprising statement. If this is the case, I highly recommend addressing the issues so that you are using DynamoDB like a typical engineering team would. A blanket statement like this with questionable benchmarks as evidence is just not compelling. It would be much more helpful to see where DynamoDB really can't stand a chance competing against ScyllaDB even when it's fully optimiized. Are you ignoring maintenance costs? Do nodes get added and removed without paying an engineer? How do you keep the nodes up-to-date as far as security updates or adding new features? How much overhead cost is engineering on-call to respond when the cluster has issues (e.g. nearing capacity limits, nodes failing, loss of AZ availability)? I'm also curious if you are only considering use cases that you typically already see on ScyllaDB and Cassandra. How about the use case where I have a load that increases without warning by 10x or where I need regular backups or that requires 5000 nodes?

thekozmo 8y ago
About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit, most of our AWS customers have AZ-aware deployments and it works great. Since I see you're with AWS, you're welcome to help us debug this error 500 on population. I guarantee we'll update the blog if we'll see it works fine I didn't get your surprise, I actually am 100% open about digging deep and come up with a uncommon use case that Dynamo can finally be a good alternative. Believe me, I can come with examples that will put Scylla in a much better light. We do offer a fully managed service on AWS with 4x-6x cost saving vs Dynamo where all the backup and auto scale is done by the Scylla team.
- otterley 8y ago
  > About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit Are you suggesting that the metrics would not change? I assure you that is not the case: inter-AZ latency is 1-2 orders of magnitude higher than intra-AZ latency. You'll need to re-run your benchmarks in a multi-AZ cluster in order to provide an apples-to-apples comparison.
  - thekozmo 8y ago
    I don't think this is the case - check this link: https://knowledgebase.progress.com/articles/Article/expected-latency-between-aws-availability-zones https://knowledgebase.progress.com/articles/Article/expected... The intra AZ latency according to AWS is sub millisecond! It's inline with what our customers see. Since all of the numbers in the blog are above a millisecond, the result does not change.
    - otterley 8y ago
      Intra-AZ latency is indeed sub-millisecond. Inter-AZ latency is in the milliseconds range. (The intra prefix means within, inter means between.) And those latencies accumulate quickly when benchmarking. Why don't you try it yourself and see the difference? Relying on some third party website is no substitute for testing.
      - thekozmo 8y ago
        I went and run a ping between 2 VMs, one on us-east1c and the other on a different AZ - us-east-1b As you can see, the ping latency is 0.5ms to 1ms on rare occasions. Of course that ping has a cost even within a single AZ so the above numbers reflect much more than the AZ round trip time. You're welcome to add 1ms to our latency numbers, it's 3x-4x better in the 99th case so there's enough room... 64 bytes from 35.173.249.24: icmp_seq=14 ttl=63 time=0.584 ms 64 bytes from 35.173.249.24: icmp_seq=15 ttl=63 time=0.462 ms 64 bytes from 35.173.249.24: icmp_seq=16 ttl=63 time=0.848 ms 64 bytes from 35.173.249.24: icmp_seq=17 ttl=63 time=0.465 ms 64 bytes from 35.173.249.24: icmp_seq=18 ttl=63 time=0.451 ms 64 bytes from 35.173.249.24: icmp_seq=19 ttl=63 time=1.00 ms 64 bytes from 35.173.249.24: icmp_seq=20 ttl=63 time=0.854 ms 64 bytes from 35.173.249.24: icmp_seq=21 ttl=63 time=0.446 ms
PeterCorless 8y ago
> "...or that requires 5000 nodes?" While there are indeed deployments that large, I've yet to see a 5,000 node _benchmark_. Though if you've seen tests that big, please send me the link!
- planckscnst 8y ago
  That comment was specifically in regards to the claim that ScyllaDB is cheaper for nearly every use case. The 5000 nodes point was to make clear that you cannot disregard automation and maintenance costs. This is relevant to the post's claim about trying to be fair: the operation costs are included in the DynamoDB pricing and not in the ScyllaDB pricing and it was not even mentioned.
  - PeterCorless 8y ago
    Though Amazon's managed service isn't completely managed at that scale either. There are still hot key partitions. There are still other issues that cause people to abandon DynamoDB at scale. And people still have "the blindfold on." q.v., https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyone-and-how-to-decide-when-it-s-for-you-aefc52ea9476 https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyo...
    - Dunedan 8y ago
      This article is already pretty outdated. According to Jim Scharf with DynamoDB On-Demand you don't really have to care anymore about DynamoDB's implementation details, like partitioning: https://twitter.com/jim_scharf/status/1069056907686465537 https://twitter.com/jim_scharf/status/1069056907686465537
      - thekozmo 8y ago
        There are tons of limitation on the on-demand, certainly it cannot go to extent of 120k ops. Check their example, they toy with a tiny workload.

thekozmo 8y ago

About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit, most of our AWS customers have AZ-aware deployments and it works great. Since I see you're with AWS, you're welcome to help us debug this error 500 on population. I guarantee we'll update the blog if we'll see it works fine I didn't get your surprise, I actually am 100% open about digging deep and come up with a uncommon use case that Dynamo can finally be a good alternative. Believe me, I can come with examples that will put Scylla in a much better light. We do offer a fully managed service on AWS with 4x-6x cost saving vs Dynamo where all the backup and auto scale is done by the Scylla team.

otterley 8y ago
> About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit Are you suggesting that the metrics would not change? I assure you that is not the case: inter-AZ latency is 1-2 orders of magnitude higher than intra-AZ latency. You'll need to re-run your benchmarks in a multi-AZ cluster in order to provide an apples-to-apples comparison.
- thekozmo 8y ago
  I don't think this is the case - check this link: https://knowledgebase.progress.com/articles/Article/expected-latency-between-aws-availability-zones https://knowledgebase.progress.com/articles/Article/expected... The intra AZ latency according to AWS is sub millisecond! It's inline with what our customers see. Since all of the numbers in the blog are above a millisecond, the result does not change.
  - otterley 8y ago
    Intra-AZ latency is indeed sub-millisecond. Inter-AZ latency is in the milliseconds range. (The intra prefix means within, inter means between.) And those latencies accumulate quickly when benchmarking. Why don't you try it yourself and see the difference? Relying on some third party website is no substitute for testing.
    - thekozmo 8y ago
      I went and run a ping between 2 VMs, one on us-east1c and the other on a different AZ - us-east-1b As you can see, the ping latency is 0.5ms to 1ms on rare occasions. Of course that ping has a cost even within a single AZ so the above numbers reflect much more than the AZ round trip time. You're welcome to add 1ms to our latency numbers, it's 3x-4x better in the 99th case so there's enough room... 64 bytes from 35.173.249.24: icmp_seq=14 ttl=63 time=0.584 ms 64 bytes from 35.173.249.24: icmp_seq=15 ttl=63 time=0.462 ms 64 bytes from 35.173.249.24: icmp_seq=16 ttl=63 time=0.848 ms 64 bytes from 35.173.249.24: icmp_seq=17 ttl=63 time=0.465 ms 64 bytes from 35.173.249.24: icmp_seq=18 ttl=63 time=0.451 ms 64 bytes from 35.173.249.24: icmp_seq=19 ttl=63 time=1.00 ms 64 bytes from 35.173.249.24: icmp_seq=20 ttl=63 time=0.854 ms 64 bytes from 35.173.249.24: icmp_seq=21 ttl=63 time=0.446 ms

PeterCorless 8y ago

> "...or that requires 5000 nodes?" While there are indeed deployments that large, I've yet to see a 5,000 node _benchmark_. Though if you've seen tests that big, please send me the link!

planckscnst 8y ago
That comment was specifically in regards to the claim that ScyllaDB is cheaper for nearly every use case. The 5000 nodes point was to make clear that you cannot disregard automation and maintenance costs. This is relevant to the post's claim about trying to be fair: the operation costs are included in the DynamoDB pricing and not in the ScyllaDB pricing and it was not even mentioned.
- PeterCorless 8y ago
  Though Amazon's managed service isn't completely managed at that scale either. There are still hot key partitions. There are still other issues that cause people to abandon DynamoDB at scale. And people still have "the blindfold on." q.v., https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyone-and-how-to-decide-when-it-s-for-you-aefc52ea9476 https://read.acloud.guru/why-amazon-dynamodb-isnt-for-everyo...
  - Dunedan 8y ago
    This article is already pretty outdated. According to Jim Scharf with DynamoDB On-Demand you don't really have to care anymore about DynamoDB's implementation details, like partitioning: https://twitter.com/jim_scharf/status/1069056907686465537 https://twitter.com/jim_scharf/status/1069056907686465537
    - thekozmo 8y ago
      There are tons of limitation on the on-demand, certainly it cannot go to extent of 120k ops. Check their example, they toy with a tiny workload.