15 ms·
All of us test in production all the time (2019)
- hn_throwaway_99 6y agoThis article is inane to me. Someone decided to write a lengthy blog post on the deliberate and obvious misinterpretation of a meme. DevOps and Observability are well described disciplines. I don't know anyone who has uttered the meme "I don't always test, but when I do, I test in production" who meant that monitoring, observing and reviewing prod behavior is bad. They invariably meant that they were rushed to release things in prod before it was adequately verified to ensure your customers don't have crappy experiences.
- igetspam 6y agoCouldn't agree more. Of course you use your product and do service quality measurements. What you don't do is toss "untested" code at your customers (possible exception here is hotfixes), which is what the meme means. Bad post written by someone either intentionally misinterpreting the meaning or dangerously unqualified to understand it.
- afarrell 6y ago> willfully misinterpreting Often, when someone looks like they are “willfully misinterpreting” something, they are actually presenting uncomfortable implications or experiences of how a phrase is put into practice. Right or wrong, it is irritating to listen to. Is it worth enduring your irritation in order to see if someone is meaningfully correct? ————— > Don’t toss “untested” code. “tested” is not a binary state. Code could always be better-tested —- otherwise why would you have hotfixes?
- timwilder 6y agoAnother read is that it’s just a hook to get into a meaningful subject - observability can be considered part of the test process. The inflammatory title gets clicks! The author’s product is based on folks accepting this premise and seeking tools to do better. I do think the setup to get to the substance is a bit long and indirect - but it’s not a given, especially at series B/C startups that folks think about production monitoring as part of the QA process so I do like to see more pieces on that.
- dnautics 6y agoThe opinion of the article is that the catchphrase has become an excuse to build poor observability and in-prod testing tools, and we should be better about that.
- tutfbhuf 6y agoThe problem is that this article is misusing the catchphrase and now people are discussing the misuse of the meme instead of the article's content.
- hn_throwaway_99 6y ago> The opinion of the article is that the catchphrase has become an excuse to build poor observability and in-prod testing tools, and we should be better about that. Is that a problem that actually exists though? I've certainly never encountered it, as the types of teams the are strict about good pre-release testing are also the types of teams that are big on observability.
- dnautics 6y agoHaha that's why I said "opinion". If you're curious about my opinion. I think there are some good points, that observability tools could be better. I think "throw it over the fence to ops" is a slow-boiling problem coming down the pike. Are observability tool abjectly poor though? I don't think so.
- bdcravens 6y agoI didn't get that. It seems to me the point was there's too many developers feel their responsibility ends when they hit deploy.
- classified 6y agoThe author apparently didn’t get the sarcasm in the "I test in prod" meme.
- lifeisstillgood 6y agoI think that's unfair - my reading is the article was that no matter how well you test before prod, there are emergent factors that cannot be tested for - and I agree. Test what you can pre release and have good tools to handle failures and be aware of the risk landscape.
- afarrell 6y agoIs that meme sarcastic? I thought it was an expression of dismay.
- deleted 6y ago[deleted]
- tutfbhuf 6y agoOr he actually did, which makes it even worse. Either way this went wrong.
- sagichmal 6y agoThe author originated the meme. It is not meant to be interpreted as sarcastic.
- classified 6y agoSo much the worse.
- sagichmal 6y agoOn the contrary, it's the best strategy for reducing risk in modern web service architectures.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- csours 6y agoWell no, but actually yes. Kind of like how you don't get requirements from talking to customers or "the business". You get requirements from demoing. If your first demo is your prod deploy, then guess what, you've just deployed a proof of concept. It's a de facto thing that just happens even if you don't plan on it. In fact it happens BECAUSE you don't plan on it. More to this point: there are many different kinds of tests and what we are most commonly referring to is "regression tests". There are lots of bugs that are not due to regressions.
- simonw 6y ago"You get requirements from demoing." I really like that - what an insightful observation, neatly condensed into a few words.
- m1117 6y agoTest in prod is terrible. Write unit and integration tests, test on staging, and then monitor Prod.
- site-packages1 6y agoSorry to make the assumption that you didn't read the article, but it really sounds like you didn't and are just making a comment about the headline. I would recommend reading the article.
- m1117 6y agoYes, I think the headline must reflect the content of the article, otherwise it confuses me and make me write false comments and wastes my time.
- Nbox9 6y agoThat’s really close to what the article says. It also suggests flexing the feature in Prod, which is a totally valuable thing. The article is mostly about the “Chuck it over the fence” mentality some developers and engineering cultures have about code that has shipped into production.
- throwaway894345 6y agoAdvice like this is good for applications that place a high value on stability and quality, which aren't deployed frequently, and which are monolithic (or rather, which are deployed monolithically--microservices that are deployed all at once are less a problem as it pertains to testing). However, the further your application deviates from these invariants, the less good this advice becomes. And there are good reasons to deviate--many applications exist in a business context in which production bugs are often not a very big deal (e.g., social networks); however, the ability to iterate rapidly (especially the ability to test new features, etc rapidly) is paramount. In these contexts, it's entirely appropriate to trade off the ability to test in advance in exchange for the ability to iterate more rapidly--this includes things like moving to a microservice deployment cadence (different components deployed at different times to elide expensive cross-team deployment coordination moments). Microservices are harder to test to the same confidence interval because it's hard to fix the versions of all of the different components in order to assure reproducible tests, and the value of doing so is diminished because in practice any given combination of versions is likely short-lived since one or more components will be redeployed in a matter of hours or even minutes. Of course, unit testing and integration testing are still valuable in these contexts, but they aren't as valuable as they are in other contexts, so testing in prod (coupled with the ability to quickly fix bugs) is necessary to make up the difference. So like everything in software development, "it depends".
- vinceguidry 6y agoA lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the team finished the sprint's work in a few days. I knew where the bottlenecks were should the project need to scale, and scaling issues never took down the site completely. Only to throw all that expertise away when it was just decided to replatform one day. Because I guess stable just wasn't good enough. After it happened twice, well, might as well go into devops. Y'all are gonna need it.
- burntoutfire 6y ago> After it happened twice, well, might as well go into devops. My observation of the devops world is that the rotation of technologies and frameworks there is faster than in backend development.
- cagenut 6y agoTrue but imho its somewhat of a fan-out or at the very least knock-on effect. The explosion in devops complexity is a necessary byproduct of the explosion of projects that adopt new languages, runtimes, datastores, etc. No one ever fully deletes and retires the old system, so the overall environmental complexity grows exponentially. The tools it takes to solve those exponentially larger problems (k8s) look ridiculous when you consider them in the context of "a rails app with some node stuff". But they make perfect sense when you say "every quarter or two, for a decade, a dev team has adopted a new component or two into the stack. less than three of which have ever been deleted."
- moreaccountspls 6y agoYes, but there at least is some genuine innovation going on. The backend frameworks are just churn for solving the same boring and lucrative business problems in new, fashionable ways.
- 6y ago
- simonw 6y agoIt seems likely that some of the comments here are reacting to the headline as opposed to reading the article - which is one of my favourite pieces of writing on the subject of observability and responsible maintenance of large scale systems. Lots of great quotable bits in here. Here's just one for people who didn't make it to the bottom: "There’s a lot of daylight between just throwing your code over the wall and waiting to get paged and having alert eyes on your code as it’s shipped, watching your instrumentation, and actively flexing the new features."
- dang 6y agoWhat would be a more accurate and neutral title?
- contingencies 6y agoWe all test in prod
- dang 6y agoOk, there's actually a phrase from the article that says that, so we'll put that up there and see what happens.
- inopinatus 6y agoFor us curious latecomers, what was the original title?
- LukeShu 6y ago"I test in prod (2019)"
- Shoop 6y agoMaybe "Once you deploy, you aren’t testing code anymore, you’re testing systems"? Trying to lift something out of the article instead of writing a new title.
- tehlike 6y agoMost large companies encourage testing in prod. It's called split test. Clearly precursor to that is staging, and precursor to that is integration tests and whatnot. But none of that catches what you can with a/b test, and great monitoring.
- bszupnick 6y agoExactly what I was thinking, but I think that a/b test "feature" is mostly framed (and thus used?) for BI purposes as opposed to "let's see if this breaks". Don't get me wrong, where I work they definitely are open to making mistakes in production. We roll out "risky" new features with split tests and I think we use that logic well so this notion obviously exists, but I'm not sure how widespread it is.
- tehlike 6y agoI have seen google and Facebook first hand. The practice is widespread, but the acceptance of "let's test it in prod" as a real stage in software development varies. Some people look at you like you are crazy person that should not be trusted, some people acknowledge and know it to be true. Feature testing is one thing, but you can go a/b testing a whole binary between new and old version to catch interesting bugs too.
- deleted 6y ago[deleted]
- throwawaw 6y agoHeadline is unfortunate, and then the first part of the essay is spent justifying/walking back the title. The good part starts a few paragraphs down, at "We conduct experiments in risk management every single day".
- aunty_helen 6y ago>Nobody invests in their “test in prod” tooling. Firstly, what is logging then? How is this not tooling to ensure things are running smoothly? `less +F` anyone? Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing.
- mgkimsal 6y ago"Firstly, what is logging then?" "Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing." This presume that you, the person expected to fix problem X actually have access to logs and the servers where problems are happening. I've been at multiple projects/companies where this simply Isn't Allowed(tm). "You wrote this code, you need to fix it! It's broken!" "Let me get on the server and take a look at the logs to see what's going on". "That violates our security policy! You can't do that!" I was tasked with 'finding a problem' and was told to look in the logs. We knew what day the problem was, but I couldn't get anyone to confirm if the log files were in UTC or something else (turns out it was something else). HOWEVER, I never actually got the actual log files. I didn't have access directly (that takes about a week to go through the chain of command to approve), so someone just sent me small snippets from where they thought the problem might be. So even orgs that tick all their checkboxes of stats/analytics/logging... sometimes seem to forget that access to the collected info is a requirement too.
- aunty_helen 6y agoFeel your pain <3 You can tell it's been a few years now but I have worked at Big Corp Inc. before. I know what it's like to ask someone in charge of 8k people to personally approve my server permission escalation even though he's never met me, doesn't know anything about the IT systems, has never visited the site I work at or probably even been in the country I reside. Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever google call their data storage). Then you only need that permission. As for the non-UTC servers... -_- but if it's a problem, you can always append the tz info to the log date format.
- 29athrowaway 6y agoI usually try to make all my development environments the same. Make them differ only in terms of infrastructure, not code. The more code differences you have between environments, the less valuable pre-production testing becomes.
- 0xdeadbeefbabe 6y ago> We’re a startup. Startups don’t tend to fail because they moved too fast. They tend to fail because they obsess over trivialities that don’t actually provide business value. It was important that we reach a reasonable level of confidence, handle errors, and have multiple levels of fail-safes (i.e., backups)." Do established companies, or stopdown, fail because they moved too fast? Can we dismiss Charity's astonishingly good advice over perceived startupiness? That's dumb.
- danpozmanter 6y ago"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.
- julianlam 6y agoI don't imagine Charity (article author) was implying that the Amazon method of being on-call for your code was ideal. I took it to mean that code ownership is important and you should be responsible for fixing things when your code blows up
- yladiz 6y agoI would argue that using the right words is important here. “On call” typically refers to a specific thing, where if something breaks that person on call is, well, called (or paged, etc.) to fix it when it happens, even at 2am. This is how I took the meaning of that paragraph as well. If the author meant that code ownership is important, or that the engineer(s) who wrote the code are responsible, that message could have been conveyed by saying, “the persons who wrote the code should test it once it’s deployed and are responsible for fixing if it breaks or is broken when deployed.” This is much clearer and doesn’t use terms that could be understood incorrectly.
- mxz3000 6y agoWhen you're Oncall you aren't Oncall 24/7 forever. It usually rotates amongst the engineers in the team. I'm on a team with about 10 engineers in it so you're on call about a week every two months. I call that manageable. Engineers should 100% be responsible for owning their code, and fixing any issues that arises from it. After all they're the ones that wrote it, aren't they the best people to fix it when it breaks?
- jsmith12673 6y agoWant to jump in here - I have worked at a company where engineers are not on call for their code, and it was a living nightmare. _You_ might not be on call for your code, but _somebody_ will be. Often some poor SRE/ops person that has absolutely no idea what the app is doing/or why it's failing in production. Not being on-call makes engineers complicit. I've seen it all, known memory leaks shipped into production, apps where half the endpoints couldn't even be compiled, code dumping the production redis at 1AM ... and every time the pain just felt on deaf ears. If your code is what wakes you up in the middle of the night, you have: - Incentive to fix/mitigate as soon as possible. - No blame game to play. Either the error was made by you, or someone on your team. It doesn't have to go up 3 rungs on the ladder then back down again. I don't think the author was suggesting that everyone should always be on call, just that you _must_ be responsible for your own code in production
- bluedino 6y agoMost places that I've worked, that "test in prod", do so for one reason. It all comes down to laziness. The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. The second scenario is that they do have a testing environment-however, for some reason, it's broken. Some change was made, usually in a hurry to meet a deadline, and it's been broken ever since. This one is usually a result of the fix being too difficult, because the test environment was haphazardly thrown together in the first place.
- sagichmal 6y ago> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similarity to prod to be useful.
- UweSchmidt 6y agoAs QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is often so complicated (banks, insurances...) that it should have been considered early during design. So, I wouldn't use the word "lazy", but we could do better.
- virtue3 6y agoIt's not the architecture, it's the representing all the states in prod in testing. Not saying it's impossible, just not always feasible to have it actively "perfect". I've found that an "ok" testing envrionment and solid unit testing / monitoring tends to do better. And then b/g deploys and canaries :/
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- siliconc0w 6y agoSo it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-prod environments and release processes I've seen they usually have basic problems and would catch a lot more issues with additional engineering investment. I.e Releases aren't automated, self-contained, or idempotent. Unit testing coverage is bad, integration tests are non-existent, data used isn't reflective of reality, there is no performance or load testing done, downstream or upstream systems have poorly defined interfaces and are excessively mocked or just not considered.
- smoe 6y agoAgreed I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not working. They are a fairly cheap way to tell me during development that this ain't going to work no matter what. And if everything compiles and the tests and reviews pass, I think this has a good shot at working out in production, but there still might well be cases and circumstances I haven't or coulnd't have considered. No setup will cover 100%, but with not that much work you can already get quite far. The tricky thing in my opinions is more on deciding which kind of tests, processes and automations are actually worth implementing and maintaining at a given stage of a project.
- table-delete 6y agoGlenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors." The preface to this is even more enlightening: "When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors. Therefore, don’t test a program to show that it works; rather, you should start with the assumption that the program contains errors (a valid assumption for almost any program) and then test the program to find as many of the errors as possible."
- ianamartin 6y agoI love articles like this because it's so easy to just add that company to a list of places to never ever work. I did read the whole article, btw. It's an absolute clickbait title that the author doesn't really mean, and after the article spends a lot of time diffusing the clickbait title it really boils down to, "This is hard, so I give up." It's true that many--if not most--companies operate this way without ever acknowledging it. And that's bad. It's also true that systems are harder to test than code. But it's not deep fucking magic. Look at the work aphyr does. Look at the testing work that the FoundationDB team did to prove their system's guarantees. Look at the work that security and devops people do every day. She is right that it is hard to test systems. So what? We don't get paid as much as we do because it's easy. In a certain environment, it is truly impossible to test a system. That's when you have a dev culture that refuses to actually design knowable systems. A much better approach for the article would be to address exactly why systems are so hard to test rather than just saying fuck it. Everything she cites in her list of things that are hard to test are absolutely testable, if you have a knowable system. The real problem here is that agile/scrum/Xtreme programming practices inevitably and by principle do not result in knowable, testable systems. When you have 30+ agile teams on their own sprint cycles and product managers leaning on them to ship features and figure the rest out later, there can be no other result than fragile, broken, unknowable, untestable system. But the answer to that isn't "Everybody else is doing it so why can't I." The answer isn't to "embrace it." The answer isn't "This is hard, fuck it." The answer is most definitely not to make individual engineers pay the price of being on call because a company's culture and process are totally and completely hosed. The answer is to address the problems in your company that caused this situation in the first place. The answer is to get your head out of the feature cult and the velocity war and reset your priorities. Systems aren't hard because your engineers suck. They're hard because companies suck. Systems are hard because in most places, no one is allowed to spend more than a couple minutes thinking about the systems. Agile culture after your early startup cycle is a lot like being a 40 year old guy who's 30 lbs over weight. How did this happen? How did I get here? I was just taking life one thing at a time and getting shit done. Now nothing works quite as well as it used to, it's harder to find dates, and everything just sort of hurts. Would anyone in their right mind just say, "Embrace it! Most 40 year old tech dudes look about like you and are in the same situation! It's fine!" No. Of course not. You have to realize that your priorities have been totally broken for the last 15-20 years of your life, that you really weren't getting shit done, and you have to take some responsibility for your diet and get off your ass and exercise. That's what companies have to do. They won't, of course. But they have to, otherwise they'll die young deaths. This article is totally correct when she recognizes a terrible symptom of unhealthy companies. But her treatment is hopelessly and tragically wrong.
- inopinatus 6y agoApropos of this in the context of hiring, one of the biggest green flags I'll attach to someone's CV is any combination of infrastructure/operations, and application development, at the same job. Since the 1990s I've worked around ISP, hosting, and cloud firms. Many have a core of general purpose people that can't help themselves but have one foot in both graves (we call this DevOps or SRE now, but those are new labels for a long-standing viewpoint). This often correlates to someone who embodies the mindset in the article, viz. that they they will gladly and actively defend any ditch we dig together. They always get on my interview list (the green flag). Very often the interviews are a wide-ranging, free-flowing, and in-depth discussion of multilateral & cross-functional technology/process interactions. Many of the people I've met with this combination will progress, either immediately or eventually, to become very effective CTOs, tech co-founders, or the highest levels of IC at larger tech firms. Interestingly, and I say this purely anecdotally because I am not actually qualified to make the diagnosis, some of them also appear to me to have an attentional difference, or present from an unprivileged background, and may not have followed a standard educational path as a result of either. Which is to say that I usually delete "must have bachelors degree" from any JD that HR ask me to authorize.
- ficklepickle 6y agoWow, you just described me. Except I'm about 4 years into my career. Glad to hear that I have a type. It is oddly reassuring in a manner that I can't currently articulate. Most places I apply I just don't hear back from. Places I have worked had no idea how to use me effectively. I've applied for very few jobs because of this (on the spectrum, have trouble with rejection). I do a lot of free or underpaid work or just do something that fascinates me. The poverty sucks but I love that I get to pursue so many interesting things. I have faith that I will find somewhere that is a good fit one day. In the meantime, I get to explore my interests and grow. Thanks for this post, it gave my hope a much needed boost.
- xelxebar 6y agoYour self-description aligns very much with my own. Feel like we might get along well. Would love to chat. Shoot me a PM if you're willing. Cheers, mate. There definitely is a place for people like us in the world.
- t-writescode 6y agoWould a better word be "validating" in production? I try to validate all of my changes in production, see them run, see them output expected values, etc. I try very hard to test everything I can prior to production; but, then I read loglines, watch graphs or run the code myself, all to validate that all the interconnected pieces are working together.
- mickotron 6y agoOnce upon a time I was a tester. Now I am a sys admin/DevOps engineer (what is a name, really?). You would not believe the amount of bugs I would find from both vendor code and internal code. Bugs that require dynamic execution and context-spexific scenarios to reveal themselves. Then you had misinterpretations of requirements, which is not something the developers would find, considering they wrote the code and believed it met the requirements. I worked for the police force of my region as a software tester for all their critical systems. I once developed a test tool that created packets to mimic mobile phone calls to police dispatch. I tested every combination of the data spec, including the very last one, where there was a critical bug. The vendor code had implemented the earlier version of the spec, not the latest. The vendor's test tool created the wrong packets as it targeted the wrong spec. Mine caught the bug cause it creates the correct packets. As much as I agree with the author's ideas of familiarity with Production, proper unbiased and independent testing is still something this industry needs, even if it's unfashionable.