5 ms·

About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit, most of our AWS customers have AZ-aware deployments and it works great

About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit, most of our AWS customers have AZ-aware deployments and it works great. Since I see you're with AWS, you're welcome to help us debug this error 500 on population. I guarantee we'll update the blog if we'll see it works fine I didn't get your surprise, I actually am 100% open about digging deep and come up with a uncommon use case that Dynamo can finally be a good alternative. Believe me, I can come with examples that will put Scylla in a much better light. We do offer a fully managed service on AWS with 4x-6x cost saving vs Dynamo where all the backup and auto scale is done by the Scylla team.

otterley 8y ago
> About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit Are you suggesting that the metrics would not change? I assure you that is not the case: inter-AZ latency is 1-2 orders of magnitude higher than intra-AZ latency. You'll need to re-run your benchmarks in a multi-AZ cluster in order to provide an apples-to-apples comparison.
- thekozmo 8y ago
  I don't think this is the case - check this link: https://knowledgebase.progress.com/articles/Article/expected-latency-between-aws-availability-zones https://knowledgebase.progress.com/articles/Article/expected... The intra AZ latency according to AWS is sub millisecond! It's inline with what our customers see. Since all of the numbers in the blog are above a millisecond, the result does not change.
  - otterley 8y ago
    Intra-AZ latency is indeed sub-millisecond. Inter-AZ latency is in the milliseconds range. (The intra prefix means within, inter means between.) And those latencies accumulate quickly when benchmarking. Why don't you try it yourself and see the difference? Relying on some third party website is no substitute for testing.
    - thekozmo 8y ago
      I went and run a ping between 2 VMs, one on us-east1c and the other on a different AZ - us-east-1b As you can see, the ping latency is 0.5ms to 1ms on rare occasions. Of course that ping has a cost even within a single AZ so the above numbers reflect much more than the AZ round trip time. You're welcome to add 1ms to our latency numbers, it's 3x-4x better in the 99th case so there's enough room... 64 bytes from 35.173.249.24: icmp_seq=14 ttl=63 time=0.584 ms 64 bytes from 35.173.249.24: icmp_seq=15 ttl=63 time=0.462 ms 64 bytes from 35.173.249.24: icmp_seq=16 ttl=63 time=0.848 ms 64 bytes from 35.173.249.24: icmp_seq=17 ttl=63 time=0.465 ms 64 bytes from 35.173.249.24: icmp_seq=18 ttl=63 time=0.451 ms 64 bytes from 35.173.249.24: icmp_seq=19 ttl=63 time=1.00 ms 64 bytes from 35.173.249.24: icmp_seq=20 ttl=63 time=0.854 ms 64 bytes from 35.173.249.24: icmp_seq=21 ttl=63 time=0.446 ms
      - erik_seaberg 8y ago
        Inter-AZ latency is going to matter more for conditional writes. I'm disappointed to find LWT so far out on Scylla's roadmap since preventing conflicts is a showstopper for correctness and we found it to be Cassandra's weak point in scaling.
        jjirsa 8y ago
        fwiw, work being done on Cassandra to speed this (cas/lwt) up SIGNIFICANTLY
        ddorian43 8y ago
        Link ?
        jjirsa 8y ago
        Started here: https://lists.apache.org/thread.html/15d1b4b5bb8a24e41d51db58538b56d0bab40c17fc9c64e898f2ad3b@%3Cdev.cassandra.apache.org%3E https://lists.apache.org/thread.html/15d1b4b5bb8a24e41d51db5... Continues here: https://issues.apache.org/jira/browse/CASSANDRA-14448 https://issues.apache.org/jira/browse/CASSANDRA-14448
Artemis2 8y ago
Quorum reads will always be much faster in the same AZ :-(
lph 8y ago
> you're welcome to help us debug this error 500 on population Create the table with 4x the provisioned capacity and then dial down to your nominal levels after it's been created.
- biggestdummy 8y ago
  Are you serious? Or is my sarcasm meter faulty? So you solve 500 errors by provisioning 4x your usage? Seems like solving a leak in the gas tank by instructing the user to simply fill up the tank more often. Nice business if you are the gas station!
  - erik_seaberg 8y ago
    This is a kludgy way to get DynamoDB to leave your table split across more shards, even after you dial it down to the capacity you need.
    - thekozmo 8y ago
      It may work at the price of 4x which is already 7x higher, resulting in 28x total.. not to mention the hassle. Man, give us a try. Seen your comment about LWT, agree.. We knownly chose the first make MV/SI great before committing to another big feature. Glad to say that with Scylla 3.0 release this month we'll finally get LWT done
      - lph 8y ago
        It's a price of 4x higher for just long enough to create the table, not for the duration of the test.
  - Shorn 8y ago
    > Are you serious? They really are. In order to make effective (let alone efficient) use of DynamoDB you need to keep in mind exactly how it works behind-the-scenes and exactly how the charging models works at all times. And you have very few, very coarse levers to pull (over-provision, under-provision, etc.) Every in-depth explanation or tutorial of DynamoDB I've seen advises to do things like this in multiple different load scenarios. There's a pretty good intro for DynamoDB over at acloud.guru: https://acloud.guru/learn/aws-dynamodb https://acloud.guru/learn/aws-dynamodb - beware though, it costs money and it's very long (though it's split up so you can cherry pick the bits you care about). I think DynamoDB might be worth investigating when you're hitting a scale where most other off-the-shelf options don't work and you'd be heading down the path of building your own solution on top of something anyway.
planckscnst 8y ago
> Believe me, I can come with examples that will put Scylla in a much better light I know this; it's why I was disappointed in the way this test was designed. It would be great to see as close as possible comparison between the two then mention improvements and trade-offs that you can still make (such as single-AZ or self-service) since you have different knobs to turn as well as DynamoDB quirks (such as item size boundaries) that can make ScyllaDB more attractive for your use-case.

otterley 8y ago

> About multiple AZ, I think we haven't used it in this case but it shouldn't change a bit Are you suggesting that the metrics would not change? I assure you that is not the case: inter-AZ latency is 1-2 orders of magnitude higher than intra-AZ latency. You'll need to re-run your benchmarks in a multi-AZ cluster in order to provide an apples-to-apples comparison.

thekozmo 8y ago
I don't think this is the case - check this link: https://knowledgebase.progress.com/articles/Article/expected-latency-between-aws-availability-zones https://knowledgebase.progress.com/articles/Article/expected... The intra AZ latency according to AWS is sub millisecond! It's inline with what our customers see. Since all of the numbers in the blog are above a millisecond, the result does not change.
- otterley 8y ago
  Intra-AZ latency is indeed sub-millisecond. Inter-AZ latency is in the milliseconds range. (The intra prefix means within, inter means between.) And those latencies accumulate quickly when benchmarking. Why don't you try it yourself and see the difference? Relying on some third party website is no substitute for testing.
  - thekozmo 8y ago
    I went and run a ping between 2 VMs, one on us-east1c and the other on a different AZ - us-east-1b As you can see, the ping latency is 0.5ms to 1ms on rare occasions. Of course that ping has a cost even within a single AZ so the above numbers reflect much more than the AZ round trip time. You're welcome to add 1ms to our latency numbers, it's 3x-4x better in the 99th case so there's enough room... 64 bytes from 35.173.249.24: icmp_seq=14 ttl=63 time=0.584 ms 64 bytes from 35.173.249.24: icmp_seq=15 ttl=63 time=0.462 ms 64 bytes from 35.173.249.24: icmp_seq=16 ttl=63 time=0.848 ms 64 bytes from 35.173.249.24: icmp_seq=17 ttl=63 time=0.465 ms 64 bytes from 35.173.249.24: icmp_seq=18 ttl=63 time=0.451 ms 64 bytes from 35.173.249.24: icmp_seq=19 ttl=63 time=1.00 ms 64 bytes from 35.173.249.24: icmp_seq=20 ttl=63 time=0.854 ms 64 bytes from 35.173.249.24: icmp_seq=21 ttl=63 time=0.446 ms
    - erik_seaberg 8y ago
      Inter-AZ latency is going to matter more for conditional writes. I'm disappointed to find LWT so far out on Scylla's roadmap since preventing conflicts is a showstopper for correctness and we found it to be Cassandra's weak point in scaling.
      - jjirsa 8y ago
        fwiw, work being done on Cassandra to speed this (cas/lwt) up SIGNIFICANTLY
        ddorian43 8y ago
        Link ?
        jjirsa 8y ago
        Started here: https://lists.apache.org/thread.html/15d1b4b5bb8a24e41d51db58538b56d0bab40c17fc9c64e898f2ad3b@%3Cdev.cassandra.apache.org%3E https://lists.apache.org/thread.html/15d1b4b5bb8a24e41d51db5... Continues here: https://issues.apache.org/jira/browse/CASSANDRA-14448 https://issues.apache.org/jira/browse/CASSANDRA-14448

Artemis2 8y ago

Quorum reads will always be much faster in the same AZ :-(

lph 8y ago

> you're welcome to help us debug this error 500 on population Create the table with 4x the provisioned capacity and then dial down to your nominal levels after it's been created.

biggestdummy 8y ago
Are you serious? Or is my sarcasm meter faulty? So you solve 500 errors by provisioning 4x your usage? Seems like solving a leak in the gas tank by instructing the user to simply fill up the tank more often. Nice business if you are the gas station!
- erik_seaberg 8y ago
  This is a kludgy way to get DynamoDB to leave your table split across more shards, even after you dial it down to the capacity you need.
  - thekozmo 8y ago
    It may work at the price of 4x which is already 7x higher, resulting in 28x total.. not to mention the hassle. Man, give us a try. Seen your comment about LWT, agree.. We knownly chose the first make MV/SI great before committing to another big feature. Glad to say that with Scylla 3.0 release this month we'll finally get LWT done
    - lph 8y ago
      It's a price of 4x higher for just long enough to create the table, not for the duration of the test.
- Shorn 8y ago
  > Are you serious? They really are. In order to make effective (let alone efficient) use of DynamoDB you need to keep in mind exactly how it works behind-the-scenes and exactly how the charging models works at all times. And you have very few, very coarse levers to pull (over-provision, under-provision, etc.) Every in-depth explanation or tutorial of DynamoDB I've seen advises to do things like this in multiple different load scenarios. There's a pretty good intro for DynamoDB over at acloud.guru: https://acloud.guru/learn/aws-dynamodb https://acloud.guru/learn/aws-dynamodb - beware though, it costs money and it's very long (though it's split up so you can cherry pick the bits you care about). I think DynamoDB might be worth investigating when you're hitting a scale where most other off-the-shelf options don't work and you'd be heading down the path of building your own solution on top of something anyway.

planckscnst 8y ago

> Believe me, I can come with examples that will put Scylla in a much better light I know this; it's why I was disappointed in the way this test was designed. It would be great to see as close as possible comparison between the two then mention improvements and trade-offs that you can still make (such as single-AZ or self-service) since you have different knobs to turn as well as DynamoDB quirks (such as item size boundaries) that can make ScyllaDB more attractive for your use-case.