3 ms·

All good advice, but do you also have a rule where our DBs have to be less than a month old in prod? Doesn't look very practical if your DB has >100s of TBs

by aprdm 5y ago

KaiserPro 5y ago
> Doesn't look very practical if your DB has >100s of TBs If that's in one shard, then you've got big issues. with larger DBs you need to be practising rolling replacement replicas, because as you scale the chance that one of your shards cocking up approaches 1. Again, it depends on your use case. RDS solves 95% of your problems (barring high scale and expense) If your running your own DBs then you _must_ be replacing part or all of the cluster regularly to make sure that your backup mechanisms are working. For us, when we were using cassandra (hint: dont) we used to spin up a "b cluster" for large scale performance testing of prod. That allowed us to do one touch deploys from hot snapshots. Eventually. This saved us from a drive by malware infection, which caused our instances to OOM.
- aprdm 5y ago
  I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used rds and other managed DBs it was easier to not have to think about it.
  - mhitza 5y ago
    Why are those so big? Are you storing binary data (assets) within the tables? Or it just has old historical records that were never "warehoused"?
    - aprdm 5y ago
      We run a lot of workflows in our systems, we keep a rolling window of 3 months for the farm data, the rest goes to a warehouse system. For the asset we keep for a show duration and we have many shows being worked from many studios. We don’t have binary data in them
  - KaiserPro 5y ago
    As an ex VFX sysadmin, I feel your pain. Any kind of recovery needed for that sized array takes days, so I feel your pain there.

All good advice, but do you also have a rule where our DBs have to be less than a month old in prod? Doesn't look very practical if your DB has >100s of TBs

by aprdm 5y ago

All good advice, but do you also have a rule where our DBs have to be less than a month old in prod? Doesn't look very practical if your DB has >100s of TBs

KaiserPro 5y ago

> Doesn't look very practical if your DB has >100s of TBs If that's in one shard, then you've got big issues. with larger DBs you need to be practising rolling replacement replicas, because as you scale the chance that one of your shards cocking up approaches 1. Again, it depends on your use case. RDS solves 95% of your problems (barring high scale and expense) If your running your own DBs then you _must_ be replacing part or all of the cluster regularly to make sure that your backup mechanisms are working. For us, when we were using cassandra (hint: dont) we used to spin up a "b cluster" for large scale performance testing of prod. That allowed us to do one touch deploys from hot snapshots. Eventually. This saved us from a drive by malware infection, which caused our instances to OOM.

aprdm 5y ago
I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used rds and other managed DBs it was easier to not have to think about it.
- mhitza 5y ago
  Why are those so big? Are you storing binary data (assets) within the tables? Or it just has old historical records that were never "warehoused"?
  - aprdm 5y ago
    We run a lot of workflows in our systems, we keep a rolling window of 3 months for the farm data, the rest goes to a warehouse system. For the asset we keep for a show duration and we have many shows being worked from many studios. We don’t have binary data in them
- KaiserPro 5y ago
  As an ex VFX sysadmin, I feel your pain. Any kind of recovery needed for that sized array takes days, so I feel your pain there.