5 ms·

Would Google Cloud Spanner work well for a workload like this one? Why or why not?

winrid 6y ago
It'd probably work. They cite cost as a reason for not going third party in the article.
axegon_ 6y ago
By all means yes, I'd argue spanner would be the ideal candidate for the task. It scales beyond anything else I've come across. Mind you, so do the costs of running it. I've only used it for training exercises and numbers add up like nothing else. IIRC running a production instance at the bare minimum would set you back around 9k/month.
- boulos 6y ago
  Disclosure: I work on Google Cloud. That was probably before the change last fall [1] to have an SLA for single-"node" instances. At $0.90/hr per "node" for a Regional setup in Iowa, that's just over $650/month. Note that I put "node" in quotes, because it's hard to remember that "num nodes" != num replicas [2]. We "require" (I don't know if this is a hard requirement, but let's assume it is) a node per 2 TB of storage [3], so the example for Quora of "tens of TB without counting replicas" would probably be 15 nodes at 30T and say 50 nodes at most (100TB). Assuming 50T as a balance, you'll need at least 25 nodes and let's assume regional pricing (since the MySQL example isn't going to be multi-regional by comparison). That's 25 x $.9/hr x 730 hours/month (~$16500) plus 50T of storage at $.3/GB/month (another $15k). So just over $30k/month or $360k/year. As for qps, we quote the multi-regional variants at 7k read qps per node (so 175k in this example) and ~2k write (so 50k). That's all in though, so you'd need to compare to the cost of running a fleet of MySQL boxes too (at a minimum you'll also need ~75-ish boxes, just like the 25x3 for Spanner), 150T of raw storage (assuming 1 master and 2 read replicas per shard, which is pretty common) and then Zone to Zone egress (I believe Quora is on EC2). I'm assuming they're using gp2 volumes at $.10/GB/month, so that's the same $15k/month of storage. Being cost-conservative and assuming r5a's and 3-yr RIs, you pay about $18/month per thread, so we can buy up to 833 threads of r5a for the remaining $15k/month. That's maybe close to feasible (~11 threads per box), but honestly you probably need 16+ or even 32 for traffic this high (depends on query caching). tl;dr: I don't think Spanner's pricing is higher than the sharded mysql fleet in this example! I do think it's not available on AWS or on-premises though, so that's a fine disqualifier for Quora. [1] https://cloud.google.com/spanner/docs/release-notes#September_25_2019 https://cloud.google.com/spanner/docs/release-notes#Septembe... [2] https://cloud.google.com/spanner/docs/instances#available-configurations-regional https://cloud.google.com/spanner/docs/instances#available-co... [3] https://cloud.google.com/spanner/quotas#notes https://cloud.google.com/spanner/quotas#notes
  - exacube 6y ago
    Even at ~$360k/year using Spanner, I wonder if Spanner is cheaper than the amortized cost of dedicating engineers to develop and maintain a custom MySQL sharding solution, on top of the AWS bill.
  - axegon_ 6y ago
    Oooh, didn't know that. Nice.

It'd probably work. They cite cost as a reason for not going third party in the article.

By all means yes, I'd argue spanner would be the ideal candidate for the task. It scales beyond anything else I've come across. Mind you, so do the costs of running it. I've only used it for training exercises and numbers add up like nothing else. IIRC running a production instance at the bare minimum would set you back around 9k/month.

boulos 6y ago
Disclosure: I work on Google Cloud. That was probably before the change last fall [1] to have an SLA for single-"node" instances. At $0.90/hr per "node" for a Regional setup in Iowa, that's just over $650/month. Note that I put "node" in quotes, because it's hard to remember that "num nodes" != num replicas [2]. We "require" (I don't know if this is a hard requirement, but let's assume it is) a node per 2 TB of storage [3], so the example for Quora of "tens of TB without counting replicas" would probably be 15 nodes at 30T and say 50 nodes at most (100TB). Assuming 50T as a balance, you'll need at least 25 nodes and let's assume regional pricing (since the MySQL example isn't going to be multi-regional by comparison). That's 25 x $.9/hr x 730 hours/month (~$16500) plus 50T of storage at $.3/GB/month (another $15k). So just over $30k/month or $360k/year. As for qps, we quote the multi-regional variants at 7k read qps per node (so 175k in this example) and ~2k write (so 50k). That's all in though, so you'd need to compare to the cost of running a fleet of MySQL boxes too (at a minimum you'll also need ~75-ish boxes, just like the 25x3 for Spanner), 150T of raw storage (assuming 1 master and 2 read replicas per shard, which is pretty common) and then Zone to Zone egress (I believe Quora is on EC2). I'm assuming they're using gp2 volumes at $.10/GB/month, so that's the same $15k/month of storage. Being cost-conservative and assuming r5a's and 3-yr RIs, you pay about $18/month per thread, so we can buy up to 833 threads of r5a for the remaining $15k/month. That's maybe close to feasible (~11 threads per box), but honestly you probably need 16+ or even 32 for traffic this high (depends on query caching). tl;dr: I don't think Spanner's pricing is higher than the sharded mysql fleet in this example! I do think it's not available on AWS or on-premises though, so that's a fine disqualifier for Quora. [1] https://cloud.google.com/spanner/docs/release-notes#September_25_2019 https://cloud.google.com/spanner/docs/release-notes#Septembe... [2] https://cloud.google.com/spanner/docs/instances#available-configurations-regional https://cloud.google.com/spanner/docs/instances#available-co... [3] https://cloud.google.com/spanner/quotas#notes https://cloud.google.com/spanner/quotas#notes
- exacube 6y ago
  Even at ~$360k/year using Spanner, I wonder if Spanner is cheaper than the amortized cost of dedicating engineers to develop and maintain a custom MySQL sharding solution, on top of the AWS bill.
- axegon_ 6y ago
  Oooh, didn't know that. Nice.