4 ms·

This sounds super interesting, are you able to share any further detail? I'm currently wrestling with a 35TB Aurora DB and considering options for future growt

This sounds super interesting, are you able to share any further detail? I'm currently wrestling with a 35TB Aurora DB and considering options for future growth. The system is multi-tenant and one of the options on the table is breaking out the tenants and hosting them on either their own system or alongside smaller subset of tenants.

gregwebs 5y ago
The issue with this approach (other than isolation) is that the largest tenant may be orders of magnitude bigger than others and still need more sharding to perform well while their smallest still needs to share a db. Have you looked at TiDB (MySQL) or CRDB (PG)?
- cutthegrass2 5y ago
  Absolutely. Thanks for the suggestion of TiDB, haven't previously considered this but it looks interesting, will take a closer look.
aynyc 5y ago
Done that before, where my largest clients are exponentially bigger than my smallest client. The key is to realize one-size-fits-all doesn't work, not in the long term. I'm talking about feature development, storage, etc.. In the end, we break out large clients onto their own instances of RDS, where smaller clients stay multi-tenants. They share the same schemas, etc.. We wrote a lot of utility scripts to migrate clients from small to big, and vice versa. Smallest clients get automatic minor and major upgrades to latest features almost immediate. Larger clients generally only get major upgrades.
exikyut 5y ago
I must admit I'm very curious as to what real-world thing(s) translate to 35TB of stored data (that presumably isn't large blobs, like video or ML models). (And oh wow I just realized the billing must be amazing...)
- cutthegrass2 5y ago
  Yep, the AWS cost footprint of this system is eye-watering. The compute spend is 2x the RDS spend! Can't say too much, but it's Financial Services related.
  - exikyut 5y ago
    Ah, a gigantic time series event firehose. (That was what I was curious about.) I guess this cues the stereotypical "have you looked at kdb+" question, then, which you've probably already fielded :) (https://kx.com https://kx.com) IIUC, kdb+ does in fact have a real DB hiding in it, and it's not all just "weird programming language" - but it does seem to require/prefer a reasonable amount of buy-in to the way it does things to be valuable. I guess the lateral question is whether its performance obsession extends to smaller on-disk footprint, and whether the savings would be worth throwing out all the the existing Aurora integration work. (Answer: very probably not.)

gregwebs 5y ago

The issue with this approach (other than isolation) is that the largest tenant may be orders of magnitude bigger than others and still need more sharding to perform well while their smallest still needs to share a db. Have you looked at TiDB (MySQL) or CRDB (PG)?

cutthegrass2 5y ago
Absolutely. Thanks for the suggestion of TiDB, haven't previously considered this but it looks interesting, will take a closer look.

aynyc 5y ago

Done that before, where my largest clients are exponentially bigger than my smallest client. The key is to realize one-size-fits-all doesn't work, not in the long term. I'm talking about feature development, storage, etc.. In the end, we break out large clients onto their own instances of RDS, where smaller clients stay multi-tenants. They share the same schemas, etc.. We wrote a lot of utility scripts to migrate clients from small to big, and vice versa. Smallest clients get automatic minor and major upgrades to latest features almost immediate. Larger clients generally only get major upgrades.

exikyut 5y ago

I must admit I'm very curious as to what real-world thing(s) translate to 35TB of stored data (that presumably isn't large blobs, like video or ML models). (And oh wow I just realized the billing must be amazing...)

cutthegrass2 5y ago
Yep, the AWS cost footprint of this system is eye-watering. The compute spend is 2x the RDS spend! Can't say too much, but it's Financial Services related.
- exikyut 5y ago
  Ah, a gigantic time series event firehose. (That was what I was curious about.) I guess this cues the stereotypical "have you looked at kdb+" question, then, which you've probably already fielded :) (https://kx.com https://kx.com) IIUC, kdb+ does in fact have a real DB hiding in it, and it's not all just "weird programming language" - but it does seem to require/prefer a reasonable amount of buy-in to the way it does things to be valuable. I guess the lateral question is whether its performance obsession extends to smaller on-disk footprint, and whether the savings would be worth throwing out all the the existing Aurora integration work. (Answer: very probably not.)