4 ms·

Curious if you also looked at Aurora over Cassandra being as you are in the AWS ecosystem? We are an AWS aligned consultancy who are seeing lots of take up of

Curious if you also looked at Aurora over Cassandra being as you are in the AWS ecosystem? We are an AWS aligned consultancy who are seeing lots of take up of Aurora as it’s MySQL API compatible but with better performance and fully managed. There’s an interesting pattern here using Lambda for CDC between Aurora and Redshift, though not sure how the cost would scale at WePay scale https://aws.amazon.com/blogs/big-data/combine-transactional-and-analytical-data-using-amazon-aurora-and-amazon-redshift/ https://aws.amazon.com/blogs/big-data/combine-transactional-...

chucky_z 7y ago
Aurora and Redshift do not scale well enough for companies at their scale (only 64TB for Aurora last I checked, and Redshift falls over near the 100TB level). They'd be looking at DynamoDB.
- gregorygoc 7y ago
  I don’t think any managed AWS service scales well for any bigger enterprise which have to handle a decent amount of traffic. Once you reach certain threshold you soon hit some throttling and limits BS and AWS solution is just to throw more money at them.
  - chucky_z 7y ago
    DynamoDB scales a lot. It falls into the "throw money at it," bucket though, and you need to have a pattern that shards well with it. If you have a reason they will basically always increase your limits, too.
- squirrelly 7y ago
  > Redshift falls over near the 100TB level. This is based on what? They have stated in various presentations that they have customers using 1pb clusters.
  - chucky_z 7y ago
    In my personal experience. There's a re:invent presentation of them having an in-house 1pb+ cluster. What they don't tell you is that 101 8xl nodes is half a million dollars a year in reserved instances, before you include any other costs associated with it; and that particular workflow (log scanning) is very nicely suited for any columnar store. Operations are also a disaster with Redshift; anything where you have to touch the cluster itself at any scale past a handful of nodes typically requires a support ticket with "hey, when this breaks, please work your magic and fix it?" There's also the issue of tuning your queues, which is a whole extra layer that you, the customer, must tune. Their suggestion tools are getting better on that front though. Just use something else if you have more than a few TB, or you have a ton of time and money to just throw around.
    - squirrelly 7y ago
      Ha, $500k/year is the amount a salesperson can knock off the cost of Teradata 100TB setup without calling their boss. As for ops stuff, I guess YMMV. We’ve never had any significant issues. Tuning is the same or less than on-prem systems IME. Perf is still pretty much unmatched in the cloud, especially perf/$. Try to do a join on 100tb in a competing system (e.g. BQ) and it won’t finish.
gunnarmorling 7y ago
Out of curiosity, is the binlog accessible on Aurora/MySQL? We've lots of users of the Debezium MySQL connector on AWS, but would be interesting to know whether it'd work with Aurora, too. (Disclaimer: I'm the lead of Debezium, which is the open-source CDC platform to which WePay are contributing their Cassandra work)

chucky_z 7y ago

Aurora and Redshift do not scale well enough for companies at their scale (only 64TB for Aurora last I checked, and Redshift falls over near the 100TB level). They'd be looking at DynamoDB.

gregorygoc 7y ago
I don’t think any managed AWS service scales well for any bigger enterprise which have to handle a decent amount of traffic. Once you reach certain threshold you soon hit some throttling and limits BS and AWS solution is just to throw more money at them.
- chucky_z 7y ago
  DynamoDB scales a lot. It falls into the "throw money at it," bucket though, and you need to have a pattern that shards well with it. If you have a reason they will basically always increase your limits, too.
squirrelly 7y ago
> Redshift falls over near the 100TB level. This is based on what? They have stated in various presentations that they have customers using 1pb clusters.
- chucky_z 7y ago
  In my personal experience. There's a re:invent presentation of them having an in-house 1pb+ cluster. What they don't tell you is that 101 8xl nodes is half a million dollars a year in reserved instances, before you include any other costs associated with it; and that particular workflow (log scanning) is very nicely suited for any columnar store. Operations are also a disaster with Redshift; anything where you have to touch the cluster itself at any scale past a handful of nodes typically requires a support ticket with "hey, when this breaks, please work your magic and fix it?" There's also the issue of tuning your queues, which is a whole extra layer that you, the customer, must tune. Their suggestion tools are getting better on that front though. Just use something else if you have more than a few TB, or you have a ton of time and money to just throw around.
  - squirrelly 7y ago
    Ha, $500k/year is the amount a salesperson can knock off the cost of Teradata 100TB setup without calling their boss. As for ops stuff, I guess YMMV. We’ve never had any significant issues. Tuning is the same or less than on-prem systems IME. Perf is still pretty much unmatched in the cloud, especially perf/$. Try to do a join on 100tb in a competing system (e.g. BQ) and it won’t finish.

gunnarmorling 7y ago

Out of curiosity, is the binlog accessible on Aurora/MySQL? We've lots of users of the Debezium MySQL connector on AWS, but would be interesting to know whether it'd work with Aurora, too. (Disclaimer: I'm the lead of Debezium, which is the open-source CDC platform to which WePay are contributing their Cassandra work)