3 ms·
While I do not agree with everything presented in this article (especially item #2), I definitely share the overall sentiment. For some of our customers, we op
by bob1029 7y ago
While I do not agree with everything presented in this article (especially item #2), I definitely share the overall sentiment.
For some of our customers, we operate 2 environments which are both effectively production. The only real difference between these is the users who have access. Normal production allows all expected users. "Pre" production allows only 2-3 specific users who understand the intent of this environment and the potential damage they might cause. In these ideal cases, we go: local development -> internal QA -> pre production -> production actual. These customers do not actually have a dedicated testing or staging environment. Everyone loves this process who has seen it in action. The level of confidence in an update going from pre production to production is pretty much absolute at this point.
The amount of frustration this has eliminated is staggering. At least in cases where we were allowed by our customers the ability to practice it. For many there is still that ancient fear that if we haven't tested for a few hours in staging that the world will end. For others, weeks of bullshit ceremony can be summarily dismissed in favor of actually meeting the business needs directly and with courage. Hiding in staging is ultimately cowardice. You don't want to deal with the bugs you know will be found in production, so you keep it there as long as possible. And then, when it does finally go to production, it's inevitably a complete shitshow because you've been making months worth of changes built upon layers of assumptions that have never been validated against reality.
This all said, there are definitely specific ecosystems in which the traditional model of test/staging/prod works out well, but I find these to be rare in practice. Most of the time, production is hooked up to real-world consequences that can never be fully replicated in a staging or test environment. We've built some incredibly elaborate simulators and still cannot 100% prove that code passing on these will succeed in production against the real deal.
- perlgeek 7y agoI've worked with a customer who also had a post-production environment. They used it for the sole purpose of being able to replicate problems and do root-cause analysis in case things went horribly wrong. Then they took a snapshot of prod, synced it to post-prod, hotfixed prod as fast as possible, and then did their detailed analysis in post-prod. This wasn't cheap; they payed Oracle somewhere between 50k€ and 200k€ a year just for the database for this environment, but they considered it worth it. (They were also in a pretty tightly regulated vertical). My main takeaway is that I don't think there is a one-size-fits-all answer to the question of how many and what environments you need. IME having at least one "buffer" between dev and prod is a good thing, but I'm not sure to which extend my experience generalizes.