6 ms·

One of the big mistakes I see here is that the testing environment A doesn't look and work the same as the prod environment B. The closer both are, the surer yo

by corty 5y ago

One of the big mistakes I see here is that the testing environment A doesn't look and work the same as the prod environment B. The closer both are, the surer you can be that you can catch such mistakes by testing in A. The bigger the differences, the more problems can sneak in undetected. So while I think it is OK for the author to be a little humbled by his mistake, I would actually place the blame on the misdesigned environments. We should try to expect human mistakes and make them avoidable (by noticing them early in the test environment) or prevent them alltogether (by automatically testing and only allowing the change to production, but that is far harder to setup).

cameronh90 5y ago
"One of the big mistakes I see here is that the testing environment A doesn't look and work the same as the prod environment B" This is a nice aspirational goal, one that you should definitely aim for, but I have never seen a situation in a non-trivial app where the test and prod environment are even close to identical. All large apps I've worked on end up with external dependencies that themselves don't have a test system and have real world side effects (like buying things), so straight off the bat, all of them need to be mocked out. Then there's no way we'd get budget for a 1000+ node cluster on our test system, so naturally there are entire classes of high load / large network bugs that are less likely to crop up. Finally the test systems are often not allowed to handle prod data for regulatory reasons. Thus, everything needs to be either synthetically generated or stripped versions of prod data, which unsurprisingly sometimes behaves rather differently. Even if you think your test environment is a pretty good replica of prod, chances are it's probably not as good as you think. So you should always have processes in place to be able to slowly rollout deployments and halt/rollback if there are issues. Everything should always be backed up. And importantly, this should not all be controlled by a single deploy script with global permissions - anything that deletes backups should need explicit multi-step approval.
- habeebtc 5y ago
  Indeed. Duplicating Test and Prod can be tens of thousands of dollars (or more) a month. You have to invest big time in automation to build up and tear down any decent sized environment to save the spend there, and that itself costs money.
- jiggawatts 5y ago
  Over decades I've slowly convinced myself that the best approach is to split production into groups and do rolling updates. One of the smoothest run environments I've ever worked with had ten silos. A failed update would not impact more than 10% of the users, and we could quickly redirect them to the remaining 90% of the platform without a material performance impact.
  - corty 5y ago
    That definitely is an option, but viability depends on what your environment is doing: "Sorry, we lost 10% of your bank transfers due to an update" just won't be acceptable, but "we delivered 10% less catpics today" might be. And I wonder how many platforms can be made to work like that.
- bmm6o 5y ago
  Sure, they will be different in scale. But can't you keep them synched in all the ways that matter for TFA? Like your clusters will be different sizes but will have the same resource definitions, etc.

by corty 5y ago

cameronh90 5y ago

"One of the big mistakes I see here is that the testing environment A doesn't look and work the same as the prod environment B" This is a nice aspirational goal, one that you should definitely aim for, but I have never seen a situation in a non-trivial app where the test and prod environment are even close to identical. All large apps I've worked on end up with external dependencies that themselves don't have a test system and have real world side effects (like buying things), so straight off the bat, all of them need to be mocked out. Then there's no way we'd get budget for a 1000+ node cluster on our test system, so naturally there are entire classes of high load / large network bugs that are less likely to crop up. Finally the test systems are often not allowed to handle prod data for regulatory reasons. Thus, everything needs to be either synthetically generated or stripped versions of prod data, which unsurprisingly sometimes behaves rather differently. Even if you think your test environment is a pretty good replica of prod, chances are it's probably not as good as you think. So you should always have processes in place to be able to slowly rollout deployments and halt/rollback if there are issues. Everything should always be backed up. And importantly, this should not all be controlled by a single deploy script with global permissions - anything that deletes backups should need explicit multi-step approval.

habeebtc 5y ago
Indeed. Duplicating Test and Prod can be tens of thousands of dollars (or more) a month. You have to invest big time in automation to build up and tear down any decent sized environment to save the spend there, and that itself costs money.
jiggawatts 5y ago
Over decades I've slowly convinced myself that the best approach is to split production into groups and do rolling updates. One of the smoothest run environments I've ever worked with had ten silos. A failed update would not impact more than 10% of the users, and we could quickly redirect them to the remaining 90% of the platform without a material performance impact.
- corty 5y ago
  That definitely is an option, but viability depends on what your environment is doing: "Sorry, we lost 10% of your bank transfers due to an update" just won't be acceptable, but "we delivered 10% less catpics today" might be. And I wonder how many platforms can be made to work like that.
bmm6o 5y ago
Sure, they will be different in scale. But can't you keep them synched in all the ways that matter for TFA? Like your clusters will be different sizes but will have the same resource definitions, etc.