3 ms·

> We only merge code that is ready to go live Cool story, but you don't _know_ if its ready until after. Look, staging environments are not great, for the rea

> We only merge code that is ready to go live Cool story, but you don't _know_ if its ready until after. Look, staging environments are not great, for the reasons described. But just killing staging and having done with it isn't the answer either. You need to _know_ when your service is fucked or not performing correctly. The only way that this kind of deployment is practical _at scale_ is to have comprehensive end-to-end testing constantly running on prod. This was the only real way we could be sure that our service was fully working within acceptable parameters. We ran captured real life queries constantly in a random order, at a random time (caching can give you a false sense of security, go on, ask me how I know) At no point is monitoring strategy discussed. Unless you know how your service is supposed to behave, and you can describe that state using metrics, your system isn't monitored. Logging is too shit, slow and expensive to get meaningful near realtime results. Some companies expend billions taming logs into metrics. don't do that, make metrics first. > You’ll reduce cost and complexity in your infrastructure I mean possibly, but you'll need to spend a lot more on making sure that your backups work. I have had a rule for a while that all instances must be younger than a month in prod. This means that you should be able to re-build _from scratch_ all instances and datastores. Instances are trivial to rebuild, databases should also be, but often arn't. If you're going to fuck around an find out in prod, then you need good well practised recovery procedures > If we ever have an issue in production, we always roll forward. I mean that cute and all, but not being able to back out means that you're fucked, you might not think you're fucked, but that's because you've not been fucked yet. its like the old addage, there are two states of system admin: Those who are about to have data loss, and those who have had data loss.

aprdm 5y ago
All good advice, but do you also have a rule where our DBs have to be less than a month old in prod? Doesn't look very practical if your DB has >100s of TBs
- KaiserPro 5y ago
  > Doesn't look very practical if your DB has >100s of TBs If that's in one shard, then you've got big issues. with larger DBs you need to be practising rolling replacement replicas, because as you scale the chance that one of your shards cocking up approaches 1. Again, it depends on your use case. RDS solves 95% of your problems (barring high scale and expense) If your running your own DBs then you _must_ be replacing part or all of the cluster regularly to make sure that your backup mechanisms are working. For us, when we were using cassandra (hint: dont) we used to spin up a "b cluster" for large scale performance testing of prod. That allowed us to do one touch deploys from hot snapshots. Eventually. This saved us from a drive by malware infection, which caused our instances to OOM.
  - aprdm 5y ago
    I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used rds and other managed DBs it was easier to not have to think about it.
    - mhitza 5y ago
      Why are those so big? Are you storing binary data (assets) within the tables? Or it just has old historical records that were never "warehoused"?
      - aprdm 5y ago
        We run a lot of workflows in our systems, we keep a rolling window of 3 months for the farm data, the rest goes to a warehouse system. For the asset we keep for a show duration and we have many shows being worked from many studios. We don’t have binary data in them
    - KaiserPro 5y ago
      As an ex VFX sysadmin, I feel your pain. Any kind of recovery needed for that sized array takes days, so I feel your pain there.
deleted 5y ago
[deleted]

aprdm 5y ago

All good advice, but do you also have a rule where our DBs have to be less than a month old in prod? Doesn't look very practical if your DB has >100s of TBs

KaiserPro 5y ago
> Doesn't look very practical if your DB has >100s of TBs If that's in one shard, then you've got big issues. with larger DBs you need to be practising rolling replacement replicas, because as you scale the chance that one of your shards cocking up approaches 1. Again, it depends on your use case. RDS solves 95% of your problems (barring high scale and expense) If your running your own DBs then you _must_ be replacing part or all of the cluster regularly to make sure that your backup mechanisms are working. For us, when we were using cassandra (hint: dont) we used to spin up a "b cluster" for large scale performance testing of prod. That allowed us to do one touch deploys from hot snapshots. Eventually. This saved us from a drive by malware infection, which caused our instances to OOM.
- aprdm 5y ago
  I work in VFX and we have 1 primary 1 replica for the render farm (MySQL), and another one for an asset system. They both have 100s of TBs many cores and a lot of RAM, we treat them a bit like unicorn machines (they're bare metal), which isn't ideal, but yeah.. our failover and whatnot is to make the primary the replica and vice versa. I cannot imagine reprovisioning it very often, when I worked in startups and used rds and other managed DBs it was easier to not have to think about it.
  - mhitza 5y ago
    Why are those so big? Are you storing binary data (assets) within the tables? Or it just has old historical records that were never "warehoused"?
    - aprdm 5y ago
      We run a lot of workflows in our systems, we keep a rolling window of 3 months for the farm data, the rest goes to a warehouse system. For the asset we keep for a show duration and we have many shows being worked from many studios. We don’t have binary data in them
  - KaiserPro 5y ago
    As an ex VFX sysadmin, I feel your pain. Any kind of recovery needed for that sized array takes days, so I feel your pain there.

deleted 5y ago

[deleted]