6 ms·

Production-Oriented Development

115 points by jgrodziski 7y ago

wgerard 7y ago
Cool article, enjoy the summary of relevant knowledge that's been passed around various circles. I do disagree with: > Environments like staging or pre-prod are a fucking lie. You need an environment that runs on production settings but isn't production. Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME. There's just so many classes of bugs that are easily caught by some sort of pre-prod environment, including stupid things like "I marked this dependency as development-only but actually it's needed in production as well". Development environments are frequently so incredibly far removed from production environments that some sort of intermediary between production is almost always so helpful to me that the extra work involved in maintaining that staging environment is well worth it. It's not the same as production obviously, but it's a LOT closer than development.
- drewcoo 7y ago
  > You need an environment that runs on production settings but isn't production. Why? > Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME. That's an anecdote, not a reason. Also, just because you've done it that way doesn't mean it has to be done that way, like you asserted. > There's just so many classes of bugs that are easily caught by some sort of pre-prod environment Also does not support the claim that you need a pre-prod env. > Development environments Whoa, there! You're sneaking yet another kind of environment into the conversation? Maybe not. This is unclear, given the many different ways that people do work. > not the same as production obviously, but it's a LOT closer You seem to want something like production. There is nothing more like production than production. If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.
  - jayd16 7y ago
    The obvious answer is so you can test infrastructure changes and data migrations without impacting users.
    - drewcoo 7y ago
      In ye olde times that was the intent of "staging." Deploys were more expensive then. That's back when we talked about having some number of 9s instead of MTTR.
  - derefr 7y ago
    > If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way. You seem to be assuming 1. some sort of large-horizontal-scale production system with multiple customers, where the impact of a failure can be minimized by minimizing the number of users exposed to new features, and where 2. there's no type of bug in the code that would potentially take down production as a whole. What if your production system is, say, a bank's ACH reconciliation logic? A medical device? A car? The live server for a popular MMORPG? A telephone backbone switch? A television or radio broadcast station? In these cases, your software isn't a service with multiple distinct customers that each make requests to it, where you can test your new code on one customer in a thousand; your software is just running and doing something—one, unified something per instance of the system (though that process may track multiple customers)—and if the code is wrong, then the whole system the software operates will fail. How do you test software for such systems? Usually by having a "production simulation" whose failure won't kill people or cost a million dollars in lost revenue.
    - Roboprog 7y ago
      Thank you for contrasting life building the latest social media platform from what many of the rest of us do. Currently I work on systems to prepare and validate birth and death certificates for the state, counties, hospitals, et al, and this whole “throw it against the wall and see what sticks” methodology doesn’t fly. Nor would it have worked when preparing and presenting investment account information 5 years ago, nor the job 10 years ago processing lawsuit and insurance claim cases and legal bills. Nor any place that I at least have ever worked in the last 30 years.
cjfd 7y ago
I think I disagree with this about 100%. Sure, production is what it is all about in the end. But how do you know the letters you just typed are going to be any good in production? They might just crash and burn there. That is why we need all those quality gates. The sooner and the farther removed from production that you discover a problem the easier it is to fix. Regarding the 'buy vs build' I think buying software is one of the most risky things that you can do. Since it cost money you cannot then say 'o well, i guess it just did not pan out, let us just not use it'. Now you are kind of married to the software. And some of the worst software out there is paid for. E.g., jira vs. redmine. This is actually a bit ironic considering the fact that I actually am writing software in my job that is sold.... O well, it actually is sold as a part of a piece of hardware, so it is not really sofware as such..... Regarding the last point, failure can be made uncommon if a relatively safe route to production is available, starting with a language that verifies the correct use of types, automated tests that verify the correctness of code, a testing environment that one attempts to keep close to what production is like and so on. Getting a call that production is not working is the event that I am trying to prevent by all means possible, and I think research would be able to show that people who get fewer calls, not just because production is failing, but in general, fewer calls regarding whatever subject, will live longer and happier.
- wpietri 7y ago
  I think you're missing a couple things here. One is the difference between optimizing for MTBF and MTTR (respectively, mean time between failures and mean time to repair). Quality gates improve the former but make the latter worse. I think optimizing for MTTR (and also minimizing blast radius) is much more effective in the long term even in preventing bugs. For many reasons, but big among them is that quality gates can only ever catch the bugs you expect; it isn't until you ship to real people that you catch the bugs that you didn't expect. But the value of optimizing for fast turnaround isn't just avoiding bugs. It's increasing value delivery and organizational learning ability. The other is that I think this grows out of an important cultural difference: the balance between blame for failure and reward for improvement. Organizations that are blame-focused are much less effective at innovation and value delivery. But they're also less effective at actual safety. [1] To me, the attitude in, "Getting a call that production is not working is the event that I am trying to prevent by all means possible," sounds like it's adaptive in a blame-avoidance environment, but not in actual improvement. Yes, we should definitely use lots of automated tests and all sorts of other quality-improvement practices. And let's definitely work to minimize the impact of bugs. But we must not be afraid of production issues, because those are how we learn what we've missed. [1] For those unfamiliar, I recommend Dekker's "Field Guide to Human Error": https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648257 https://www.amazon.com/Field-Guide-Understanding-Human-Error...
bob1029 7y ago
While I do not agree with everything presented in this article (especially item #2), I definitely share the overall sentiment. For some of our customers, we operate 2 environments which are both effectively production. The only real difference between these is the users who have access. Normal production allows all expected users. "Pre" production allows only 2-3 specific users who understand the intent of this environment and the potential damage they might cause. In these ideal cases, we go: local development -> internal QA -> pre production -> production actual. These customers do not actually have a dedicated testing or staging environment. Everyone loves this process who has seen it in action. The level of confidence in an update going from pre production to production is pretty much absolute at this point. The amount of frustration this has eliminated is staggering. At least in cases where we were allowed by our customers the ability to practice it. For many there is still that ancient fear that if we haven't tested for a few hours in staging that the world will end. For others, weeks of bullshit ceremony can be summarily dismissed in favor of actually meeting the business needs directly and with courage. Hiding in staging is ultimately cowardice. You don't want to deal with the bugs you know will be found in production, so you keep it there as long as possible. And then, when it does finally go to production, it's inevitably a complete shitshow because you've been making months worth of changes built upon layers of assumptions that have never been validated against reality. This all said, there are definitely specific ecosystems in which the traditional model of test/staging/prod works out well, but I find these to be rare in practice. Most of the time, production is hooked up to real-world consequences that can never be fully replicated in a staging or test environment. We've built some incredibly elaborate simulators and still cannot 100% prove that code passing on these will succeed in production against the real deal.
- perlgeek 7y ago
  I've worked with a customer who also had a post-production environment. They used it for the sole purpose of being able to replicate problems and do root-cause analysis in case things went horribly wrong. Then they took a snapshot of prod, synced it to post-prod, hotfixed prod as fast as possible, and then did their detailed analysis in post-prod. This wasn't cheap; they payed Oracle somewhere between 50k€ and 200k€ a year just for the database for this environment, but they considered it worth it. (They were also in a pretty tightly regulated vertical). My main takeaway is that I don't think there is a one-size-fits-all answer to the question of how many and what environments you need. IME having at least one "buffer" between dev and prod is a good thing, but I'm not sure to which extend my experience generalizes.
alkonaut 7y ago
This article makes a lot of assumptions that only hold true in a very specific set of circumstances: - that it’s possible for the team developing the product to deploy or monitor it (example cases where it isn’t: most things that aren’t web based such as desktop, most things embedded into hardware that might not yet exist etc.) - that if you can deliver continuously, customers actually accept that you do. Customers may want Big Bang releases every two years and reject the idea of the software changing the slightest in between. - not validating a deployment for a long time before it meets customers is also only ok if the impact of mistakes is that you deploy a fix. If the next release window is a year away and/or if people are harmed by a faulty product then you probably want it painstakingly manually tested. My point is: if you are a team developing and operating a product that is as web site/app/service and you are free to choose if and when to deploy, then most of the article is indeed good advice. But then you are also facing the simple edge case among software deployment scenarios.
- inetknght 7y ago
  I generally agree with your points, with one exception: > If the next release window is a year away and/or if people are harmed by a faulty product then you probably want it painstakingly manually tested. No, you manual testing should only be for the things which are difficult to automatically test. But I think you should _always_ strive for extensive automatic testing. Even with hardware which doesn't yet exist, mocks are perfect for that.
  - alkonaut 7y ago
    Agreed. Whether something is manually or automatically tested is really an implementation detail, but it’s economically insane to not have computers do the testing to the largest extent possible. It’s also possible to extend the argument to the relation between “compiler verification” and “test verification”. That is: don’t spend time writing tests for things a compiler could catch.
- derefr 7y ago
  > that it’s possible for the team developing the product to deploy or monitor it (example cases where it isn’t: most things that aren’t web based such as desktop, most things embedded into hardware that might not yet exist etc.) In these cases, you can have a pre-production embedded (in the sense of "embedded journalism") field test, where the developers come out to the production line and/or testing field to iterate on the software together with other departments + the final customers. IIRC this is done often in military weapons testing—you'll often find the software engineer of a new UAV autonavigation system at the testing range for that system, doing live pair-debugging with the field operator.
polote 7y ago
> 2. Buy Almost Always Beats Build Strongly disagree with that, well maybe it is a good idea when you are over founded by VC where cost of money is equal to zero and you don't want to master what you are working on but in all other cases this is wrong, you shouldn't rebuild everything from scratch but creating a company is not the same as playing with LEGO And this is the same argument as saying you should have everything in AWS because if you self host you will have to hire devops engineer
- simonw 7y ago
  Could you expand more on why you disagree with this? Do you believe the opposite - that "Build Almost Always Beats Buy"? I've made the build-vs-buy decision many times in my career. I don't necessarily regret /all/ of those times, but the general lesson I've learned time and time again is that you're going to end up investing WAY too much time maintaining your special version of X when you should have spent that time solving problems unique to your business model.
- dtech 7y ago
  If you're building something that already reasonably exists, you better be sure you can do it 3x better (for some economic metric of better, e.g. cheaper, bringing in more revenue). If not, you're wasting your money in a different way, by not focusing on the things that really bring in revenue or by paying salary to people to maintain it.
zentropia 7y ago
I'm sorry but I'm tired of Medium paywall to the point I don't want to read anything there.
- GordonS 7y ago
  Also published at: https://paulosman.me/2019/12/30/production-oriented-development.html https://paulosman.me/2019/12/30/production-oriented-developm...
- diddid 7y ago
  I couldn’t agree more.
jayd16 7y ago
Some good points but some controversial ones. I think a manual QA team is very valuable. Sure the tests pass but what if the UI is confusing or disorienting. QA can be user advocates in a way a unit test can't be. I work in games so maybe it's just a squishier design philosophy but you can't unit test fun. I also don't understand the worry about other environments. If you're automating deployments how is another environment added work? Shouldn't it be just as easy to deploy to?
- dodobirdlord 7y ago
  I think the valuable purpose you are describing of QA is better achieved by having a UX team earlier in the pipeline.
  - davedx 7y ago
    We have both, and I'm really glad we do. A good QA team tests more than just "funtionality" and "usability" -- they test the product in its entirety, and notice all sorts of things that non-QA people would miss. Our QA people also often poke the database and look at REST request/responses too. I think going without this kind of full spectrum testing is just shooting yourself in the foot. You can totally do continuous delivery but still use a QA team.
  - jedieaston 7y ago
    I always liked Basecamp's approach on this, which was that every team working on a widget (or part of a widget), had to be composed of 2 developers for 1 designer, so the designer was continually involved in whatever they were working on to give a UX perspective. https://basecamp.com/shapeup/2.2-chapter-08#team-and-project-sizes https://basecamp.com/shapeup/2.2-chapter-08#team-and-project...
  - jayd16 7y ago
    I do support constant deployments to the QA environment (also a no-no apparently). That can keep the QA team involved at all times. I wouldn't suggest waiting on large changes before having QA do a pass.
tluyben2 7y ago
Right. Commenting on a very specific part of this with an anecdote. I did some integration with a ‘neo bank’ a few years ago where the CTO said testing and staging envs are a lie. I vehemently disagree(d) but they were paying to only test on production. I guess you can guess what happened (and not because of us as we spent 10k of our own money to build a simulator of their env; I have some kind of pride); testing was extremely expensive (it is a bank so production testing and having bugs is actually losing money... also they could not really test millions of transactions because of that so there were bugs, many bugs, in their system...), violated rules and the company died and got bought for scraps. I understand the sentiment but I agree with 2, 3 and 6 of this article, the rest is, imho, actually dangerous in many non startup cases. Example; Simple is always better IF you can apply it; a lot of companies and people you work with do not do so simple. A lot of companies still have SOAP, CORBA or in house protocols and you will have to work with it. So you can shout from the rafters that simple wins; you will not get the project. That can be a decision but I do not see many people who finally got into a bank/insurer/manufacturer/... go ‘well, your tech is not simple in my definition so I will look elsewhere’. It is a nice utopia and maybe it will happen when all legacy gets phased out in 50-100 years.
- veeralpatel979 7y ago
  Thanks for your comment! But I don't think all legacy will ever get phased out. Today's code will become tomorrow's legacy.
  - tluyben2 7y ago
    I agree, I meant 50-100 yrs with a bit of wink as no-one can reason about that period of time in software obviously.
- deleted 7y ago
  [deleted]
VHRanger 7y ago
Why would you ever share the medium version with all the added crud when the author has a version of the post on his personal blog?
ivan_ah 7y ago
What exactly is would be the disadvantage of running something in stating environment before running it "for real" in production? I'm assuming the staging environment is an exact clone of production (except reduced size: fewer app servers + smaller DB instance)? I understand the deploy-often-and-rollback-if-there-is-a-problem strategy, but certain things like DB migrations and config changes are difficult to rollback, so doing a dry run in a staging environment seems like a good thing...
capkutay 7y ago
It's funny in my career I've observed similar development styles. But I always just thought of this as great/good developers verses average/mediocre developers. the A+ coders would always make their code very easy to access, deploy from a user standpoint, debug, read etc. The mediocre guys would wait for someone else to hit a landmine beefore fixing something that was obviously wrong.
maxwellg 7y ago
Non-production environments are useful for more than testing application code. Changing underlying infrastructure (Upgrading a database, networking shenanigans, messing around with ELB or Nginx settings) requires testing too. Having the same traffic / data shape in pre-prod is not as important.
0x445442 7y ago
I agree with most everything said in the article but with a big condition. If I as an engineer am responsible for everything the author says I should be responsible for then I want total control of the tech stack and runtime environment.
gfodor 7y ago
I agree with most of this but the point about QA gating deploys should be amended. A 5 minute integration test on a pre-flight box in the production environment by the deploying engineer is a form of QA, and can catch a lot of issues. It shouldn’t be considered anti pattern. Manually verifying critical paths in production before putting them live is about the best thing you can do to ensure no push results in catastrophic breakage. Without such a preflight box, or automated incremental rollouts, you are kind of doing a Hail Mary, since you are exposing all users immediately to a system that has not been verified in production before going live.
nojvek 7y ago
He lost me at “non production environments are bullshit”. In dev you can break almost anything, no biggie. In stage if you break something, great just don’t deploy it to prod. If you break something in prod, well ... you may end up going below SLA and may legit lose money and your customers trust. Don’t YOLO into prod. Build reliable shit.
banq 7y ago
this is another Agile or DevOps
luord 7y ago
I agree with most of the points, but I have serious caveats on the first two. 1. No, the engineers should not by default be on call; the owners of the product are the first call line. If they're not engineers or if they're engineers but don't have enough time to deal with all incidents–in short, if they need to delegate–they better be willing to pay very generously for the extra hours of on call duty. 2. No, hosted is not better than open source, both for philosophical and operational reasons: mostly, you become subject to the whims of the provider. A good compromise is hosted open source solutions, which at least takes you half way to a migration, if the need for one comes up. That aside, I very much agree on everything else.
sadness2 7y ago
Radical! Some counter-points, though. - Infrastructure as code and schemas as code make it easier to keep environmental parity, because everything can be rolled back/forwards/reset with easy source control and CD operations. Visual environment diffing and drift detection can make this even easier. - Make your stage and prod into a blue-green situation, where if stage is ready to go, you flip users onto it. I can guarantee your stage and prod will both be respected as prod then. Failing that, just add load/stress tests to stage to make it more prod-like. - Non-prod environments and attention are not necessarily debt, but they are expensive insurance premiums. You should only pay those premiums if you need the insurance. It's about risk management. - As time passes, the people who wrote a specific part of a system don't know it anymore, so having them babysit 'their' code in production has diminishing returns. On the other hand, having a systems quality team who have a broad mandate to bugfix, put in preventative measure, reduce technical debt, improve observibility and establish good patterns for developers to do these things, can enabled these things to actually happen, when just telling devs who are busy making features that they should happen often doesn't make them happen. Also there are devs who enjoy creating new things, and others who love trouble-shooting and metrics.

115 points by jgrodziski 7y ago

Cool article, enjoy the summary of relevant knowledge that's been passed around various circles. I do disagree with: > Environments like staging or pre-prod are a fucking lie. You need an environment that runs on production settings but isn't production. Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME. There's just so many classes of bugs that are easily caught by some sort of pre-prod environment, including stupid things like "I marked this dependency as development-only but actually it's needed in production as well". Development environments are frequently so incredibly far removed from production environments that some sort of intermediary between production is almost always so helpful to me that the extra work involved in maintaining that staging environment is well worth it. It's not the same as production obviously, but it's a LOT closer than development.

drewcoo 7y ago
> You need an environment that runs on production settings but isn't production. Why? > Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME. That's an anecdote, not a reason. Also, just because you've done it that way doesn't mean it has to be done that way, like you asserted. > There's just so many classes of bugs that are easily caught by some sort of pre-prod environment Also does not support the claim that you need a pre-prod env. > Development environments Whoa, there! You're sneaking yet another kind of environment into the conversation? Maybe not. This is unclear, given the many different ways that people do work. > not the same as production obviously, but it's a LOT closer You seem to want something like production. There is nothing more like production than production. If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.
- jayd16 7y ago
  The obvious answer is so you can test infrastructure changes and data migrations without impacting users.
  - drewcoo 7y ago
    In ye olde times that was the intent of "staging." Deploys were more expensive then. That's back when we talked about having some number of 9s instead of MTTR.
- derefr 7y ago
  > If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way. You seem to be assuming 1. some sort of large-horizontal-scale production system with multiple customers, where the impact of a failure can be minimized by minimizing the number of users exposed to new features, and where 2. there's no type of bug in the code that would potentially take down production as a whole. What if your production system is, say, a bank's ACH reconciliation logic? A medical device? A car? The live server for a popular MMORPG? A telephone backbone switch? A television or radio broadcast station? In these cases, your software isn't a service with multiple distinct customers that each make requests to it, where you can test your new code on one customer in a thousand; your software is just running and doing something—one, unified something per instance of the system (though that process may track multiple customers)—and if the code is wrong, then the whole system the software operates will fail. How do you test software for such systems? Usually by having a "production simulation" whose failure won't kill people or cost a million dollars in lost revenue.
  - Roboprog 7y ago
    Thank you for contrasting life building the latest social media platform from what many of the rest of us do. Currently I work on systems to prepare and validate birth and death certificates for the state, counties, hospitals, et al, and this whole “throw it against the wall and see what sticks” methodology doesn’t fly. Nor would it have worked when preparing and presenting investment account information 5 years ago, nor the job 10 years ago processing lawsuit and insurance claim cases and legal bills. Nor any place that I at least have ever worked in the last 30 years.

cjfd 7y ago

I think I disagree with this about 100%. Sure, production is what it is all about in the end. But how do you know the letters you just typed are going to be any good in production? They might just crash and burn there. That is why we need all those quality gates. The sooner and the farther removed from production that you discover a problem the easier it is to fix. Regarding the 'buy vs build' I think buying software is one of the most risky things that you can do. Since it cost money you cannot then say 'o well, i guess it just did not pan out, let us just not use it'. Now you are kind of married to the software. And some of the worst software out there is paid for. E.g., jira vs. redmine. This is actually a bit ironic considering the fact that I actually am writing software in my job that is sold.... O well, it actually is sold as a part of a piece of hardware, so it is not really sofware as such..... Regarding the last point, failure can be made uncommon if a relatively safe route to production is available, starting with a language that verifies the correct use of types, automated tests that verify the correctness of code, a testing environment that one attempts to keep close to what production is like and so on. Getting a call that production is not working is the event that I am trying to prevent by all means possible, and I think research would be able to show that people who get fewer calls, not just because production is failing, but in general, fewer calls regarding whatever subject, will live longer and happier.

wpietri 7y ago
I think you're missing a couple things here. One is the difference between optimizing for MTBF and MTTR (respectively, mean time between failures and mean time to repair). Quality gates improve the former but make the latter worse. I think optimizing for MTTR (and also minimizing blast radius) is much more effective in the long term even in preventing bugs. For many reasons, but big among them is that quality gates can only ever catch the bugs you expect; it isn't until you ship to real people that you catch the bugs that you didn't expect. But the value of optimizing for fast turnaround isn't just avoiding bugs. It's increasing value delivery and organizational learning ability. The other is that I think this grows out of an important cultural difference: the balance between blame for failure and reward for improvement. Organizations that are blame-focused are much less effective at innovation and value delivery. But they're also less effective at actual safety. [1] To me, the attitude in, "Getting a call that production is not working is the event that I am trying to prevent by all means possible," sounds like it's adaptive in a blame-avoidance environment, but not in actual improvement. Yes, we should definitely use lots of automated tests and all sorts of other quality-improvement practices. And let's definitely work to minimize the impact of bugs. But we must not be afraid of production issues, because those are how we learn what we've missed. [1] For those unfamiliar, I recommend Dekker's "Field Guide to Human Error": https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/0754648257 https://www.amazon.com/Field-Guide-Understanding-Human-Error...

bob1029 7y ago

While I do not agree with everything presented in this article (especially item #2), I definitely share the overall sentiment. For some of our customers, we operate 2 environments which are both effectively production. The only real difference between these is the users who have access. Normal production allows all expected users. "Pre" production allows only 2-3 specific users who understand the intent of this environment and the potential damage they might cause. In these ideal cases, we go: local development -> internal QA -> pre production -> production actual. These customers do not actually have a dedicated testing or staging environment. Everyone loves this process who has seen it in action. The level of confidence in an update going from pre production to production is pretty much absolute at this point. The amount of frustration this has eliminated is staggering. At least in cases where we were allowed by our customers the ability to practice it. For many there is still that ancient fear that if we haven't tested for a few hours in staging that the world will end. For others, weeks of bullshit ceremony can be summarily dismissed in favor of actually meeting the business needs directly and with courage. Hiding in staging is ultimately cowardice. You don't want to deal with the bugs you know will be found in production, so you keep it there as long as possible. And then, when it does finally go to production, it's inevitably a complete shitshow because you've been making months worth of changes built upon layers of assumptions that have never been validated against reality. This all said, there are definitely specific ecosystems in which the traditional model of test/staging/prod works out well, but I find these to be rare in practice. Most of the time, production is hooked up to real-world consequences that can never be fully replicated in a staging or test environment. We've built some incredibly elaborate simulators and still cannot 100% prove that code passing on these will succeed in production against the real deal.

perlgeek 7y ago
I've worked with a customer who also had a post-production environment. They used it for the sole purpose of being able to replicate problems and do root-cause analysis in case things went horribly wrong. Then they took a snapshot of prod, synced it to post-prod, hotfixed prod as fast as possible, and then did their detailed analysis in post-prod. This wasn't cheap; they payed Oracle somewhere between 50k€ and 200k€ a year just for the database for this environment, but they considered it worth it. (They were also in a pretty tightly regulated vertical). My main takeaway is that I don't think there is a one-size-fits-all answer to the question of how many and what environments you need. IME having at least one "buffer" between dev and prod is a good thing, but I'm not sure to which extend my experience generalizes.

alkonaut 7y ago

This article makes a lot of assumptions that only hold true in a very specific set of circumstances: - that it’s possible for the team developing the product to deploy or monitor it (example cases where it isn’t: most things that aren’t web based such as desktop, most things embedded into hardware that might not yet exist etc.) - that if you can deliver continuously, customers actually accept that you do. Customers may want Big Bang releases every two years and reject the idea of the software changing the slightest in between. - not validating a deployment for a long time before it meets customers is also only ok if the impact of mistakes is that you deploy a fix. If the next release window is a year away and/or if people are harmed by a faulty product then you probably want it painstakingly manually tested. My point is: if you are a team developing and operating a product that is as web site/app/service and you are free to choose if and when to deploy, then most of the article is indeed good advice. But then you are also facing the simple edge case among software deployment scenarios.

inetknght 7y ago
I generally agree with your points, with one exception: > If the next release window is a year away and/or if people are harmed by a faulty product then you probably want it painstakingly manually tested. No, you manual testing should only be for the things which are difficult to automatically test. But I think you should _always_ strive for extensive automatic testing. Even with hardware which doesn't yet exist, mocks are perfect for that.
- alkonaut 7y ago
  Agreed. Whether something is manually or automatically tested is really an implementation detail, but it’s economically insane to not have computers do the testing to the largest extent possible. It’s also possible to extend the argument to the relation between “compiler verification” and “test verification”. That is: don’t spend time writing tests for things a compiler could catch.
derefr 7y ago
> that it’s possible for the team developing the product to deploy or monitor it (example cases where it isn’t: most things that aren’t web based such as desktop, most things embedded into hardware that might not yet exist etc.) In these cases, you can have a pre-production embedded (in the sense of "embedded journalism") field test, where the developers come out to the production line and/or testing field to iterate on the software together with other departments + the final customers. IIRC this is done often in military weapons testing—you'll often find the software engineer of a new UAV autonavigation system at the testing range for that system, doing live pair-debugging with the field operator.

polote 7y ago

> 2. Buy Almost Always Beats Build Strongly disagree with that, well maybe it is a good idea when you are over founded by VC where cost of money is equal to zero and you don't want to master what you are working on but in all other cases this is wrong, you shouldn't rebuild everything from scratch but creating a company is not the same as playing with LEGO And this is the same argument as saying you should have everything in AWS because if you self host you will have to hire devops engineer

simonw 7y ago
Could you expand more on why you disagree with this? Do you believe the opposite - that "Build Almost Always Beats Buy"? I've made the build-vs-buy decision many times in my career. I don't necessarily regret /all/ of those times, but the general lesson I've learned time and time again is that you're going to end up investing WAY too much time maintaining your special version of X when you should have spent that time solving problems unique to your business model.
dtech 7y ago
If you're building something that already reasonably exists, you better be sure you can do it 3x better (for some economic metric of better, e.g. cheaper, bringing in more revenue). If not, you're wasting your money in a different way, by not focusing on the things that really bring in revenue or by paying salary to people to maintain it.

zentropia 7y ago

I'm sorry but I'm tired of Medium paywall to the point I don't want to read anything there.

GordonS 7y ago
Also published at: https://paulosman.me/2019/12/30/production-oriented-development.html https://paulosman.me/2019/12/30/production-oriented-developm...
diddid 7y ago
I couldn’t agree more.

jayd16 7y ago

Some good points but some controversial ones. I think a manual QA team is very valuable. Sure the tests pass but what if the UI is confusing or disorienting. QA can be user advocates in a way a unit test can't be. I work in games so maybe it's just a squishier design philosophy but you can't unit test fun. I also don't understand the worry about other environments. If you're automating deployments how is another environment added work? Shouldn't it be just as easy to deploy to?

dodobirdlord 7y ago
I think the valuable purpose you are describing of QA is better achieved by having a UX team earlier in the pipeline.
- davedx 7y ago
  We have both, and I'm really glad we do. A good QA team tests more than just "funtionality" and "usability" -- they test the product in its entirety, and notice all sorts of things that non-QA people would miss. Our QA people also often poke the database and look at REST request/responses too. I think going without this kind of full spectrum testing is just shooting yourself in the foot. You can totally do continuous delivery but still use a QA team.
- jedieaston 7y ago
  I always liked Basecamp's approach on this, which was that every team working on a widget (or part of a widget), had to be composed of 2 developers for 1 designer, so the designer was continually involved in whatever they were working on to give a UX perspective. https://basecamp.com/shapeup/2.2-chapter-08#team-and-project-sizes https://basecamp.com/shapeup/2.2-chapter-08#team-and-project...
- jayd16 7y ago
  I do support constant deployments to the QA environment (also a no-no apparently). That can keep the QA team involved at all times. I wouldn't suggest waiting on large changes before having QA do a pass.

tluyben2 7y ago

Right. Commenting on a very specific part of this with an anecdote. I did some integration with a ‘neo bank’ a few years ago where the CTO said testing and staging envs are a lie. I vehemently disagree(d) but they were paying to only test on production. I guess you can guess what happened (and not because of us as we spent 10k of our own money to build a simulator of their env; I have some kind of pride); testing was extremely expensive (it is a bank so production testing and having bugs is actually losing money... also they could not really test millions of transactions because of that so there were bugs, many bugs, in their system...), violated rules and the company died and got bought for scraps. I understand the sentiment but I agree with 2, 3 and 6 of this article, the rest is, imho, actually dangerous in many non startup cases. Example; Simple is always better IF you can apply it; a lot of companies and people you work with do not do so simple. A lot of companies still have SOAP, CORBA or in house protocols and you will have to work with it. So you can shout from the rafters that simple wins; you will not get the project. That can be a decision but I do not see many people who finally got into a bank/insurer/manufacturer/... go ‘well, your tech is not simple in my definition so I will look elsewhere’. It is a nice utopia and maybe it will happen when all legacy gets phased out in 50-100 years.

veeralpatel979 7y ago
Thanks for your comment! But I don't think all legacy will ever get phased out. Today's code will become tomorrow's legacy.
- tluyben2 7y ago
  I agree, I meant 50-100 yrs with a bit of wink as no-one can reason about that period of time in software obviously.
deleted 7y ago
[deleted]

VHRanger 7y ago

Why would you ever share the medium version with all the added crud when the author has a version of the post on his personal blog?

ivan_ah 7y ago

What exactly is would be the disadvantage of running something in stating environment before running it "for real" in production? I'm assuming the staging environment is an exact clone of production (except reduced size: fewer app servers + smaller DB instance)? I understand the deploy-often-and-rollback-if-there-is-a-problem strategy, but certain things like DB migrations and config changes are difficult to rollback, so doing a dry run in a staging environment seems like a good thing...

capkutay 7y ago

It's funny in my career I've observed similar development styles. But I always just thought of this as great/good developers verses average/mediocre developers. the A+ coders would always make their code very easy to access, deploy from a user standpoint, debug, read etc. The mediocre guys would wait for someone else to hit a landmine beefore fixing something that was obviously wrong.

maxwellg 7y ago

Non-production environments are useful for more than testing application code. Changing underlying infrastructure (Upgrading a database, networking shenanigans, messing around with ELB or Nginx settings) requires testing too. Having the same traffic / data shape in pre-prod is not as important.

0x445442 7y ago

I agree with most everything said in the article but with a big condition. If I as an engineer am responsible for everything the author says I should be responsible for then I want total control of the tech stack and runtime environment.

gfodor 7y ago

I agree with most of this but the point about QA gating deploys should be amended. A 5 minute integration test on a pre-flight box in the production environment by the deploying engineer is a form of QA, and can catch a lot of issues. It shouldn’t be considered anti pattern. Manually verifying critical paths in production before putting them live is about the best thing you can do to ensure no push results in catastrophic breakage. Without such a preflight box, or automated incremental rollouts, you are kind of doing a Hail Mary, since you are exposing all users immediately to a system that has not been verified in production before going live.

nojvek 7y ago

He lost me at “non production environments are bullshit”. In dev you can break almost anything, no biggie. In stage if you break something, great just don’t deploy it to prod. If you break something in prod, well ... you may end up going below SLA and may legit lose money and your customers trust. Don’t YOLO into prod. Build reliable shit.

banq 7y ago

this is another Agile or DevOps

luord 7y ago

I agree with most of the points, but I have serious caveats on the first two. 1. No, the engineers should not by default be on call; the owners of the product are the first call line. If they're not engineers or if they're engineers but don't have enough time to deal with all incidents–in short, if they need to delegate–they better be willing to pay very generously for the extra hours of on call duty. 2. No, hosted is not better than open source, both for philosophical and operational reasons: mostly, you become subject to the whims of the provider. A good compromise is hosted open source solutions, which at least takes you half way to a migration, if the need for one comes up. That aside, I very much agree on everything else.

sadness2 7y ago

Radical! Some counter-points, though. - Infrastructure as code and schemas as code make it easier to keep environmental parity, because everything can be rolled back/forwards/reset with easy source control and CD operations. Visual environment diffing and drift detection can make this even easier. - Make your stage and prod into a blue-green situation, where if stage is ready to go, you flip users onto it. I can guarantee your stage and prod will both be respected as prod then. Failing that, just add load/stress tests to stage to make it more prod-like. - Non-prod environments and attention are not necessarily debt, but they are expensive insurance premiums. You should only pay those premiums if you need the insurance. It's about risk management. - As time passes, the people who wrote a specific part of a system don't know it anymore, so having them babysit 'their' code in production has diminishing returns. On the other hand, having a systems quality team who have a broad mandate to bugfix, put in preventative measure, reduce technical debt, improve observibility and establish good patterns for developers to do these things, can enabled these things to actually happen, when just telling devs who are busy making features that they should happen often doesn't make them happen. Also there are devs who enjoy creating new things, and others who love trouble-shooting and metrics.