9 ms·

Dev-owned testing: Why it fails in practice and succeeds in theory

161 points by rbanffy 10mo ago

mrits 10mo ago
Author-Owned proof reading is next
__alexs 10mo ago
This just seems to be basically a blog post that somehow got published in ACM?
- pjdesno 10mo ago
  No, they got it published in ACM SIGSOFT Software Engineering Notes. That's one of the things that publication is for. The paper is a well-supported (if not well-proofread) position paper, synthesizing the author's thoughts and others' prior work but not reporting any new experimental results or artifacts. The author isn't an academic, but someone at Amazon who has written nearly 20 articles like this, many reporting on the intersection of academic theory and the real world, all published in Software Engineering Notes. As an academic (in systems, not software engineering) who spent 15 years in industry before grad school, I think this perspective is valuable. In addition academics don't get much credit for this sort of article, so there are a lot fewer of them than there ought to be.
kayo_20211030 10mo ago
A nice piece that outlines all the challenges, the opportunities, and the cultural and social adjustments that need to be made within organizations to maximize the chance of left-shifted testing being successful. IMPO, as a developer, I see QA's role as being "auditors" with a mandate to set the guidelines, understand the process, and assess the outcomes. I'm wary of the foxes being completely responsible for guarding the hen-house unless the processes are structured and audited in a fundamentally different way. That takes fundamental organizational change.
brap 10mo ago
Do people actually send PRs with no tests? That is so bizarre to me
- pjdesno 10mo ago
  If your review was based on features shipped, and your bosses let you send PRs with no tests, would you? And before you say "no" - would you still do that if your company used stack ranking, and you were worried about being at the bottom of the stack? Developers may understand that "XYZ is better", but if management provides enough incentives for "not XYZ", they're going to get "not XYZ".
  - hamdingers 10mo ago
    Breaking prod repeatedly probably impacts your stack ranking too.
    - Jtsummers 10mo ago
      Depends on how easily the failure is connected back to you personally. If you introduce a flaw this year and it breaks the system in two years, it won't fall back on you but the poor sap that triggered your bug.
    - actionfromafar 10mo ago
      So can "heroically" save prod ... anti patterns.
  - wccrawford 10mo ago
    That actually wasn't why I didn't write tests a lot of the time. What stopped me was that after a year of writing tests, I was moved to a higher priority project, and the person who followed me didn't write tests. So when I came back, many of the tests were broken. I had to fix all those in order to get new ones to not be a bother. Repeat again, but this time I came back and the unit testing suite had fundamentally altered its nature. None of the tests worked and they all needed to be rewritten for a new paradigm. I gave up on tests for that system at that point. It simply wasn't worthwhile. Management didn't care at all, despite how many times I told them how much more reliable it made that system, and it was the only system that survived the first giant penetration test with no problems. That doesn't mean I quite testing. I still wrote tests whenever I thought it would help me with what I was currently working on. And that was quite often. But I absolutely didn't worry about old tests, and I didn't worry about making sure others could use my tests. They were never going to try. The final straw, less than a year before I was laid off, was when they decided my "storybook" tests weren't worth keeping in the repo and deleted them. That made me realized exactly how much they valued unit tests. That isn't to say they had no tests. There was a suite of tests written by the boss that we were required to run. They were all run against live or dev servers with a browser-control framework, and they were shaky for years. But they were required, so they were actually kept working. Nobody wrote new tests for it until something failed and caused a problem, though. tl;dr - There are a lot of reasons that people choose not to write tests, and not just for job security.
weinzierl 10mo ago
The article argues that Dev-Owned testing isn't wrong but all the arguments it presents support that it is. I always understood shift-left as doing more tests earlier. That is pretty uncontroversial and where the article is still on the right track. It derails at the moment it equates shift-left with dev-owned testing - a common mistake. You can have quality owned by QA specialists in every development cycle and it is something that consistently works.
- TheSoftwareGuy 10mo ago
  I'm interested, as I've never been in an org with QA specialists. What does that look like?
  - weinzierl 10mo ago
    The are breakers. Good devs are makers. You can be both but I have yet to meet someone who is equally good in both mindsets.
  - bluGill 10mo ago
    You do everything the same as today. Then you turn it over to QA who keep finding weird things that you never thought of. QA finds more than half your written bugs (of course I don't write a bug everytime a unit test fails when doing TDD, but sometimes I find a bug in code I wrote a few weeks ago and I write that up so I can focus on the story I'm doing today and not forget about the bug) QA should not be replacing anything a developer does, it should be a supplement because you can't think of everything. We also use QA because we are making multi-million dollar embedded machines. One QA can put the code of 10 different developers on the machine and verify it works as well in the real world as it does in software simulation.
  - philk10 10mo ago
    They find all the things the devs and their automated tests missed, then they mentor the devs in how to test for these and they work out how the bug could have been found earlier. Rinse and repeat until the tester is struggling to find issues and has coached the devs out of his job
thmpp 10mo ago
If as a developer you want to be seen as someone advancing and taking ownership and responsibility, testing must be part of the process. Sending an untested product or a product that you as a software engineer do not monitor, essentially means you can never be sure you created an actual correct product. That is no engineering. If the org guidelines prevent it, some cultural piece prevents it. Adding QA outside, which tests software regularly using different approaches, finding intersections etc. is a different topic. Both are necessary.
- sigmoid10 10mo ago
  The problem in big companies is that as a developer, you are usually several layers of people removed from the people actually using the product. Yes you can take ownership and implement unit tests and integration tests and e2e tests in your pipeline, to ensure the product works exactly as you intended. But that doesn't mean it works as management or marketing or the actual user intended.
sethammons 10mo ago
While good points are made, I worry this gives the wrong impression. The paper doesn't say it is impossible, just hard. I have, very successfully, worked with dev owned testing. Why it worked: the team set the timelines for delivery of software, the team built their acceptance and integration tests based on system inputs and outputs based on the edges of their systems, the team owned being on-call, and the team automated as much as possible (no repeatable manual testing aside from sanity checks on first release). There was no QA person or team, but there was a quality focused dev on the team whose role was to ensure others kept the testing bar high. They ensured logs, metrics, and tests met the team bar. This role rotated. There was a ci/cd team. They made sure the test system worked, but teams maintained their own ci configuration. We used buildkite, so each project had its own buildkite.yml. The team was expected by eng leaders to set up basic testing before development. In one case, our team had to spend several sprints setting up generators to make the expected inputs and sinks to capture output. This was a flagship project and lots of future development was expected. It very much paid off. Our test approach was very much "slow is smooth and smooth is fast." We would deploy multiple times a day. Tests were 10 or so minutes and very comprehensive. If a bug got out, tests were updated. The tests were very reliable because the team prioritized them. Eventually people stopped even manually verifying their code because if the tests were green, you _knew_ it worked. Beyond our team, into the wider system, there was a light weight acceptance test setup and the team registered tests there, usually one per feature. This was the most brittle part because a failed test could be because another team or a system failure. But guess what? That is the same as production if not more noisy. So we had the same level of logging, metrics, and alerts (limited to business hours). Good logs would tell you immediately what was wrong. Automated alerts generally alerted the right team, and that team was responsible for a quick response. If a team was dropping the ball on system stability, that reflected bad on the team and they were to prioritize stability. It worked. Hands down the best dev org I have part of.
- regularfry 10mo ago
  I've worked in a strong dev-owned testing team too. The culture was a sort of positive can-I-catch-you-out competitiveness that can be quite hard to replicate, and there was no concept of any one person taking point on quality.
boxed 10mo ago
I think they've got that the other way around.
OptionOfT 10mo ago
The conversation is usually: devs can write their own tests. We don't need QA. And the first part is true. We can. But that's not why we have (had) QA. First: it's not the best use of our time. I believe dev and QA are separate skillset. Of course there is overlap. Second, and most important: it's a separate person, an additional person who can question the ticket, and who can question my translation of the ticket into software. And lastly: they don't suffer from the curse of knowledge on how I implemented the ticket. I miss my QA colleagues. When I joined my current employer there were 8 or so. Initially I was afraid to give them my work, afraid of bad feedback. Never have I met such graceful people who took the time in understanding something, and talking to me to figure out where there was a mismatch. And then they were deemed not needed.
- torginus 10mo ago
  I very rarely worked with good QA. In my mind a good QA understands the feature we're working on, deploys the correct version, thoroughly tests the feature understanding what it's supposed and not supposed to do, and if they happen to find a bug, they create a bug ticket where they describe the environment in full and what steps are necessary to reproduce it. For automation tests, very few are capable of writing tests that test the spec, not implementation, contain sound technical practices, and properly address flakiness. For example it's very common to see a test that clicks the login button and instead of waiting for the login, the wait 20 seconds. Which is both too much, and 1% of the time too little. Whenever I worked with devs, they almost always managed to do all this, sometimes they needed a bit of guidance, but that's it. Very very few QA ever did (not that they seemed to bothered by that). A lot of QA have expressed that devs 'look down' on them. I can't comment on that, but the signal-to-noise ratio of bug tickets is so low, that often it's you have to do their job and repeat everything as well. This has been a repeated experience for me with multiple companies and a lot of places don't have proper feedback loops, so it doesn't even bother them as they're not affected by the poor quality of bug reports, but devs have to spend the extra time.
  - bbayles 10mo ago
    I used to work with a QA person who really drove me nuts. They would misunderstand the point of a feature, and then write pages and pages of misguided commentary about what they saw when trying to test it. We'd repeat this a few times for every release. This forced me to start making my feature proposals as small as possible. I would defensively document everything, and sprinkle in little summaries to make things as clear as possible. I started writing scripts to help isolate the new behavior during testing. ...eventually I realized that this person was somehow the best QA person I'd ever worked with.
KingOfCoders 10mo ago
Developers want things to work. QA wants things to break. What worked for me, devs write ALL the tests, QA does selective code reviews of those tests making devs write better tests. I also wrote the failure of Dev-Owned Testing: "Tests are bad for developers" https://www.amazingcto.com/tests-are-bad-for-developers/ https://www.amazingcto.com/tests-are-bad-for-developers/
darkwater 10mo ago
First they came with the NoOps movement, and you were happy cause those damned ops people were always complaining and slowing you down. I can manage my infra! Then, they came with the dev-owned testing and fired all the QAs, and you were happy because they were always breaking your app and slowing you down. I can write my tests! Now, they are coming with LLM agents and you don't own the product...
- BigTTYGothGF 10mo ago
  > you were happy ... you were happy Like heck I was.
  - bluGill 10mo ago
    I have worked with bad ops people who didn't let anything get done, and good ops people who knew how to do tricky things and kept the system working so I didn't have to care. I have worked with good and bad QA testers. Guess who I'm glad are gone.
terribleidea 10mo ago
Most orgs I've worked for are so growth and product-focused that if you try adjusting your estimates to include proper testing, you get push back, and you have to ARGUE your case as to why a feature will take two weeks instead of one. This is the thing I hate the most about work, having to ARGUE with PMs because they can't accept an estimate, there's often some back-and-forth. "What if you do X instead?" "Team Y (always uses hacks and adds technical debt with every single feature they touch) did something similar in two days." But we're just communicating and adding transparency so that's good and it certainly doesn't matter that it starts taking up 4+ hours of your time in Slack conversations and meetings of people 'level setting' 'getting on the same page' trying to help you 'figure out' how to 'reduce scope' etc. etc. Also, I think testing via unit or integration tests should be standard regardless, and that isn't what I am thinking about here. I'm thinking about QA, the way QA does it. You hammer your feature with a bunch of weird bullshit like false and unexpected inputs, what happens if I refresh the page in strange ways, what happens if i make an update and force the cache to NOT clear, what happens if I drop my laptop in a dumpster while making the request from firefox and safari at the same time logged in as the same user, what happens if I turn off my internet in the middle of a file upload, and so on. When devs say that devs should be responsible for testing, they usually mean the former (unit and integration tests), and not this separate skillset of coming up with a bunch of weird edge cases for your code. And yes, unit tests SHOULD hit the edge cases, but QA is just better at it. You usually don't have engineers testing what happens when you try sending in Mandarin characters as input (unless they live in China, I guess). All of that effort should bring up your estimates because it is non-trivial. This is what getting rid of QA means, not happy path end-to-end testing plus some unit and integration tests.
- 2OEH8eoCRo0 10mo ago
  The stupid fast tempo of our industry grinds my gears. When I worked defense we moved slowly and methodically. It almost felt too slow. Now in the private sector I move like triple the speed but we often need to go back and redo and refactor things. I think it averages out to a similar rate of progress but in defense at least I had my sanity.
- JohnMakin 10mo ago
deleted 10mo ago
[deleted]
donatj 10mo ago
> The problem is not that dev-owned testing is a flawed idea, but that it is usually poorly planned In our case there was zero plan. One day they just let our entire QA team go. Literally no direction at all on how to deal with not having QA. It's been close to a year and we're still trying to figure out how to keep things from going on fire. For a while we were all testing each other's work. They're mad that this is slowing down our "velocity", and now they're pushing us to test our own work instead... Testing your own work is the kind of thing an imbecile recommends. I tested it while I wrote it. I thought it was good. Done. I have all the blind spots I had when I wrote it "testing it" after the fact.
- nlunbeck 10mo ago
  Frustrating to see. Dev-led testing is too fundamentally different from a QA function, just as any amount of E2E tests can't replace manual testing. Each tries to solve for a different type of problem. Is it possible to do effective dev peer "QA" without essentially duplicating the QA role? And forget about testing one's own work..
tracerbulletx 10mo ago
My experience with this was great. It went really well. We also did our own ops with in a small boundary of systems organized based on domain. I felt total ownership for it, could fix anything in it, deploy anything with any release strategy, monitor anything, and because of that had very little anxiety about being on call for it, best environment I ever worked in.
dasil003 10mo ago
Oh man do I have opinions. First of all, I've seen all type of teams be successful, ranging from zero QA at all, to massive QA teams with incredible power (eg. Format QA at Sony in Europe). I have absolutely seen teams with no QA deliver high quality full stop, the title is nonsense. My firm belief is that QA can raise the ceiling of quality significantly if you know what you're doing, but there is also huge moral hazard of engineers dropping the ball on quality at implementation time and creating a situation where adding more QA resources doesn't actually improve quality, just communication churn and ticket activity. By the way the same phenomenon can happen with product people as well (and I've also seen teams without product managers do better than teams with them in certain circumstantes). The most important anchor point for me is that engineering must fundamentally own quality. This is because we are closer to the implementation and can anticipate more failure modes than anyone else. That doesn't mean other roles don't contribute significantly to quality (product, design, QA, ops absolutely do), but it means we can't abdicate our responsibility to deliver high quality code and systems by leaning on some other function and getting lazy about how we ensure we are building right. What level of testing is appropriate for engineers to do is quite project and product specific, but it is definitely greater than zero. This goes double in the age of AI.
- travisjungroth 10mo ago
  > The most important anchor point for me is that engineering must fundamentally own quality. This is huge. I was selling software to help QA. I saw a CEO demand a Head of QA guarantee their super buggy app be free of bugs by a certain date. This is terrible. She didn’t write the thing. Total responsibility without authority trap. She was, not at all to my surprise, fired. I think the deal fell through and I don’t know how else things ended up with them. QA’s job is signal. If you’re getting clear signal, they’re doing their job.
  - philk10 10mo ago
    yes - testing gives information, what is done with that information is a business decision
  - hobs 10mo ago
cratermoon 10mo ago
The paper highlights the problem in two words of the first sentence of the abstract: "shrink QA". Corporations do it to save money, and accept the loss of quality as the cost of doing business. Therein lies part of the reason for the sad state of software today.
- gwbas1c 10mo ago
  I suspect they underestimate the loss of quality; or underestimate the consequences.
  - cratermoon 10mo ago
    I don't think they underestimate the loss of quality. As for the consequences, "No one in this world, so far as I know ... has ever lost money by underestimating the intelligence of the great masses of the plain people" - H. L. Mencken
  - cratermoon 10mo ago
    Also this list of Windows bugs, just from this week. - https://learn.microsoft.com/en-us/answers/questions/5709441/windows-11-unable-to-install-2026-01-security-upda https://learn.microsoft.com/en-us/answers/questions/5709441/... - https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-25h2 https://learn.microsoft.com/en-us/windows/release-health/sta... - https://www.techspot.com/news/110848-onedrive-backup-feature-making-users-local-files-seemingly.html https://www.techspot.com/news/110848-onedrive-backup-feature...
tomtoday 10mo ago
Note: the references in the article seem to be incorrect [4] through [7] are all the same article. I do not think that was intentional.
gwbas1c 10mo ago
As a developer, I frequently tell higher ups that "I have a conflict of interest" when it comes to testing. Even though I fully attempt to make perfect software, often I have blind spots or assumptions that an independent tester finds. That being said: Depending on what you're making and what platform(s) you target, developer-owned testing either is feasible or not. For example, if you're making a cross-platform product, it's not feasible for a developer to regression test on Windows 10, 11, MacOS, 10 distros of Linux. In contrast, if you're targeting a web API, it's feasible for a developer to write tests at the HTTP layer against a real database.
time4tea 10mo ago
The abstract says it really: "It was clearly a top-down decision" Many many things that are imposed like this will fail. Its not willful non-compliance even, its just that its hard for people to do things differently, while still being the same people in the same teams, making the same products, with the same timelines... Context is key here, lots of people see a thing that works well and think they can copy the activities of the successful team, without realising they need to align the mindset.. and the activities will follow. The activities might be different, and thats OK! In a different context, you'd expect that. I'd argue that in most contexts you don't need a QA team at all, and if you do have one, then it will look a lot different to what you might think. For example, it would be put after a release, not before it.. QA teams are too slow to deal with 2000+ releases a year - not their fault, they are human.. need to reframe the value statement.
MoreQARespect 10mo ago
This paper has 7 references and 4 of them are to a single google blog post that treats test flakiness as an unavoidable fact of life rather than a class of bug which can and should be fixed. Aside from the red flag of one blog post being >50% of all citations it is also the saddest blog post google ever put their name to. There is very little of interest in this paper.
monster_truck 10mo ago
Something I've always believed, and my experience with shipping multinational software on a schedule that has severe drop-dead dates confirmed: If you are contractually obligated to deliver a product that does x, y, and z correctly? QA is the only way to do that seriously. If you don't have QA, you don't care about testing full stop. This only compounds when you have to comply with safety regulations in every country, completely setting aside the strong moral obligation you should feel to ensure you go far above & beyond mere compliance given the potential for harm. This compounds again when you are reliant upon deliverables from multiple tiers of hardware and software suppliers, each contract with its own drop-dead dates you must enforce. When one of them misfires, and that is a "when, not if", they are going to lie through their teeth and you will need hard proof. These are not small fines, they are company-killing amounts of money. Nobody profits in this situation. I've been through it twice, both times it was a herculean effort to break even. Hell, even a single near-miss handled poorly is enough to lose out on millions in potential future work. The upsides are quite nice, though. I didn't know it was possible to get more than 100% of your salary as a bonus until then. Don't take my word for it, though. Ask your insurance agent about the premiums for contractual liability insurance with and without a QA team. If you can provide metrics on their performance, -10-15% is not uncommon, this discount increases over time. Without one? +15-50% depending.
gwbas1c 10mo ago
> At Google, for example, 16% of tests exhibited flakiness This really surprised me. In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. In the past year, we had a flaky unit test that caused about 1-2% of builds to fail. Upon fixing it, we learned it was what caused a deadlock in a production service every 5-6 months. As a result of fixing this one "flaky" test, we eliminated our biggest cause of manual intervention in our production environments.
- inetknght 10mo ago
  > > At Google, for example, 16% of tests exhibited flakiness > This really surprised me. It doesn't surprise me. Close to 80% of Google code is actually quite terrible. > In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. Yup, that's my experience too. I flip shit when I start seeing flaky tests because it means a lot of actual work. Luckily there's generally a lot of tools available (sanitizers are a godsend) to fix it, and a lot of experience I rely on to look for smells when those tools come up empty. It makes me really sad when I work on a project with few (or, gasp, zero) tests. I've spent the last 5 months just adding tests to my current project at $employer and... those tests have revealed all kinds of problems in dependencies created by other teams at $employer. Ugh.
- zmj 10mo ago
  At scale, every test is flaky.
  - gwbas1c 10mo ago
    Oh no, a well-written unit test is 100% deterministic. It's once you start doing tests in real environments with real databases and real networks that the unpredictability of the real world creeps in!
  - m3047 10mo ago
    Well, strictly anecdata but there was a time when I worked on code which for $reasons had no dedicated test environment, and externalities were not our problem according to management. So we were reaching out to the big bad world for testing, but we didn't own the network between it and us. I ended up writing tests for the network. Caught a lot of problems with the network. Didn't make any friends. DISCLAIMER: I've been around IT for probably the majority of y'all's lifetimes, so I'm not saying this happens often. But just because something is fundamentally wrong, doesn't mean that all fundamentally wrong things are the same. In my experience they differ more from each other than the possible good ways of doing the same thing. Don't conflate things without a good reason.
gwbas1c 10mo ago
(Joke) Can't AI just replace QA?
- jayd16 10mo ago
  Honestly very pleased that AI is not dominating the discussion. Nature is healing.
physicsguy 10mo ago
I think devs owning testing only works where they’re consumers of the product. So a developer productivity tool - perfect. A fully fledged engineering application targeting monitoring of assets? Not so much
wpollock 10mo ago
One purpose of QA testing is compliance assurances, including with applicable policies, industry regulations, and laws. While devs are (usually) good at functioal testing, QA (usually) does non-functional testing better. I have not known any devs that test for GDPR compliance for example. (I am certain many devs do test for that, just stating my personal experience.)
inetknght 10mo ago
Dev-owned testing is great when it succeeds. It's definitely not the only way. But you really do need the whole team to be on-board with the concept, or willing to train the team to do it. If that doesn't line up then you're not going to have a good time. I've been on teams that own the whole pipeline, and have lead teams where I made it mandatory that the engineer writing a feature must also write tests to exercise that feature, and must also write sad-path tests too. That gets enforced during review of the pull request. It works. But it takes a lot of effort to teach the whole team how to write testable code, how to write good tests, how to write sad-path tests, and how to even identify what sad paths might exist that they didn't think about. I can tell you from experience that, when it does succeed and the whole team has high collaboration, then individual developer's work output is high, and new features can be introduced very rapidly with far fewer bugs making it to production than relying on a whole QA team to find all the problems. It fails in practice because most (not all!) devs don't want to "waste time" doing that, and instead rely on QA cycles to tell them that something is wrong. Alas, QA cycles are a hell of a lot slower than the developer writing the tests. QA teams often don't have access to (or perhaps don't understand) the source code, and so they're left trying to find bugs through a user interfaces. That's valuable, but takes a completely different skillset and is a poor time to find a lot of the basic bugs that can show up. On the other hand, the teams I've been on that failed (especially hard) often had huge (!) QA teams and budgets. Despite the size of team and budget, multiple projects would fall over from inertia and bickering between teams about who owns which bug, or which bug needs priority fixing.
- nlunbeck 10mo ago
  (Disclaimer: I work for shipyard.build so i am biased here..) Dev-owned testing (or even dev-involved testing) is much more realistic when devs have shorter feedback loops between their code and its deployment. So often i've seen momentum get lost when devs have a wait period before they can run tests/do basic manual testing. Then the test aspect becomes the thing that "slows down" devs before they ship a feature, so they might tend towards shortcuts
- m3047 10mo ago
  For shiggles I searched the entire first page of comments for "testable" (looking for "how to write testable code" as a proxy for devs write testable code if they write tests) and this is the only mention. So +1 to you!
threethirtytwo 10mo ago
Simple. Because all theories are approximations. And our most high resolution theory of reality assumes reality is intrinsically random making it so that even the most accurate theory of reality can't predict anything.
johnea 10mo ago
I feel the need to point out a phrase that was very popular among my dev peers: The difference between theory and reality, is that in theory they're the same, but in reality they're not... While any new feature, or bug fix, introduced by a dev should certainly be tested at that dev's desk to confirm to themselves that it's correct; it should also (of course) be tested by a product test group (call it QA if you must) to insure that all functional features of the product are still fully and correctly implemented. I would aim a big fat finger at "agile", "scrum", "standup" culture for encouraging the violation of this, very obvious, testing requirement. "What have you accomplished in the last 4 hours", type of management interface to development, fully and completely misses the primacy of confirming the functionality of updates before release. This is really due to management, especially C-suite management of startups, living in a make believe world of deadlines and feature requirements pulled arbitrarily out of their ass, while refusing (or not having the capacity) to understand the technical issues involved.
theptip 10mo ago
> Why does dev-owned testing look so compelling in theory, yet fall short in almost every real-world implementation Seems like a weird assertion. Plenty of startups do “dev owned testing” ie not hiring dedicated QA. Lots of big-tech does too. Indeed I’d venture it’s by far the most common approach on a market valuation-weighted basis.
- array_key_first 10mo ago
  It's very common and most software is extremely buggy, so that adds up. I think most consumer software just would not stand the scrutiny of enterprise or professional applications. It just does not work a lot of the time, which is truly insane to think about. I mean, imagine if your compiler just 10% of the time did not work and output a wrong program. But when I log into website XYZ, that's about the rate of failure.
flambojones 10mo ago
Having been at Microsoft when we had SDETs for everything (and I miss it greatly, though the way we could write a feature and then just toss it to test was ridiculous), I think things have swung too far away. On one hand, engineers needed to take more ownership of writing things other than low-level unit tests. On the other, the SDETs added immense value a ton of ways, like writing thorough test specs based off of the feature spec (rather than the design spec), testing without blind spots due to knowledge of the implementation, implementation of proper test libraries and frameworks to make tests better and easier to write, and an adversarial approach to trying to break things that makes things more robust. I've also worked with manual QA for product facing flows, and while they added value with some of their approaches to ensuring quality - poking at our scenarios and tests, and looking more closely at subjective things - they often seemed to work as a crutch for the parts of code paths that engineers had made too difficult to test. I've never seen anywhere attempt to replace the value that SDETs delivered with what engineers were tasked with. I'd argue it's not necessarily possible to fully replicate that when you're testing your own things. But with services now, it also seems like product/management are more willing to have slightly few assurances around quality and just counting on catching some in production in favor of velocity. I've never seen places that got rid of QA
zbentley 10mo ago
Another vital quality of good QA teams is that they often serve as one of the last/main repositories of tribal knowledge about how an org's entire software system actually behaves/works together. As businesses grow and products get more complex and teams get more siloed, this is really important.
__MatrixMan__ 10mo ago
It pretty much comes down to whether QA is just doing what dev tells them--in which case they're not applying any scrutiny to dev's decisions, or if they're deciding for themselves what constitutes an appropriate validation for the dev work at hand.
dgunay 10mo ago
I have limited experience working in orgs with a QA apparatus. Just my anecdotes: The one time I got to work with a QA person, he was worse than useless. He was not technical enough to even use cURL, much less do anything like automated e2e testing, so he'd have to manually test every single thing we wanted to deploy. I had to write up extremely detailed test plans to help him understand exactly what buttons he had to press in the app to test a feature. Sometimes he'd modify the code to try and make testing it easier, break the feature in doing so, and then report that it didn't work. In nearly all cases it would have been faster for me to just test the code myself. The majority of the time I've worked in orgs where there is no QA team, the devs are expected to own the quality of their output. This works okay when you're in a group of conscientious and talented engineers, but you very quickly find out who really cares about quality and who either doesn't know any better or doesn't care. You will constantly battle management to have enough time to adequately test anything. Every bit of test automation you want to build has to be smuggled in with a new feature or a bugfix. So really, they both suck, pick your poison. I prefer the latter, but I'm open to experiencing what good looks like in terms of dedicated QA.
- philk10 10mo ago
  that sucks, hope you get to work with a good tester/QA person at some point so you get to see what they can do
wesselbindt 10mo ago
Conflict of interest that the author fails to mention: he's a QA manager at Amazon, and has a vested interest in QA being seen as a necessary role. It may well be, but this is definitely a conflict of interest. Aside from that, this article is incredibly heavy on theory and very light on empirical fact. Its bibliography consists of a very narrow selection of blogs (4 of the articles he quotes are one and the same blog somehow), which talk about a very narrow subset of the industry. This article not referencing the serious and well-established research that has been done on the effectivity of dev-owned tests by for example the DORA folks, almost seems dishonest. The clickbait title, when compared to the content of the article, is outright dishonest. The author theorizes about some warts dev-owned testing may have at some specific companies, but this is a very far cry from it failing in practice, especially when you compare them to the warts of offloading quality to a different team. It's probably a bit harsh, but I feel like, as an industry, we should have a higher standard of empiricism when it comes to evaluating our ways of working.
shuntress 10mo ago
I'm surprised that with this many comments about the relationship between testing, development, and QA there is so little mention of environment and deploy process. The usability of your test environment (and associated tooling) has a massive impact on quality assurance. Every small difference between Production and Production-Plus-Feature creates friction and, even in systems of only moderate complexity, that friction adds up fast.
pftg 10mo ago
I cannot believe the excuse for why shift-left QA is “not working” is that Amazon hires developers who can’t learn basic testing skills that QA engineers picked up in three months. If developers can’t write valid code for tests, that’s on the organization, not on the practice. The author forgot to mention the costs of handoffs, which paid off all those tiny learning investments. Shift-left has over 30 years of proof as one of the most effective ways to build reliable software. P.S. This isn’t an ACM article; it’s a strongly opinionated post based on personal experience. P.P.S. I'm not against QA, but make them as bug/quality hunters, instead of toil.
somewhereoutth 10mo ago
If your product interface is with humans, you test it firstly with devs, then QA, then your customers. Devs are a bit leaky for bugs/non-conformance, so if you skip the QA, then your customer is exposed. For some industries this is fine - for others, not so much.
tom_m 10mo ago
Writing tests cases is part of development. Programmers simply need to do it. It helps them immensely. Product acceptance and user acceptance testing is entirely different and often conflated. That's where people go wrong and are too simple minded (or cheap) to understand or invest in both.
tom_m 10mo ago
This paper is really good and accurate. Many people would benefit from reading it.
squirrellous 10mo ago
I work with an amazing QA team right now. They are good programmers that don’t mind doing lots of grunt work, understand the goal of the software deeply, collaborates extremely well with other teams, and are just all around great people. I expect they would kill it as a non-QA dev team as well, so I’m always grateful to have such partners.
solatic 10mo ago
Devs need to write the 1% of automated tests needed just to prove that what they wrote works in the ideal case. QA is valuable for writing the 99% of automated tests that prove that the software works in the edge cases, with DevOps occasionally dropping in to make sure that the test suite runs quickly. The way you solve Product and QA being at odds is very simple: QA loses, until they don't. When trying to find product-market-fit, it doesn't make sense to delay delivery to prove that an experiment works in exceptional circumstances. Eventually you do have product-market-fit, and you want to harden the features you already shipped, which is where QA comes in - better internal QA finds the bugs rather than your (future) customers. Eventually you start launching features to a massive audience on day 1, and you need QA to reduce reputational risk before you ship. The right time for QA to intercede and get a veto on delivery changes over the lifetime of the product, and part of whether or not QA is a net-add is whether your organization (leadership) is flexible enough to accept and implement that flexibility.
polotics 10mo ago
Having gone through the whole cycle from 1 dev... to 2 dev-lead... to 3 large-ish team qa-lead for offshore devs... to 4 qa-lead for offshore qa... to 5 actual qa ("a" is for assurance as opposed to the quality control that passes as qa) in an industry that needs it... to 6 kind-of principal engineer... I would advise you that generalities about qa are useless, environments differ. Still always true: some mgmt will either want more for less, or even something for nothing, and in the long run we are all dead, and some folks' horizon is surprisingly short!
fizlebit 10mo ago
I think that vibe coding now with anthropic tools and the latest model means that the cost of writing integration tests is significantly reduced. When the company ships a large product that has components from many teams, there is still a role for QA engineers who run nightly tests and chase teams to help diagnose the issue when there is an issue found. If you don't have such a central team publishing golden versions, then everybody is chasing the same bug. Ideally the integration tests are part of the change acceptance flow, but low frequency bugs (occur maybe 1 in 100 test runs) can still sneak through.
DiskoHexyl 10mo ago
All the great QAs with whom I've worked would have made good developers (and they actually WERE good developers, only with a QA name and salary). The problem is that a great QA earns less than a mediocre developer within the same company. And has a much lower status. And also fewer career opportunities elsewhere. No wonder most of those guys switched at one point or another
isquaredr 10mo ago
Economically devs are the only ones that can do it. If I am a good QA engineer that cares enough about the product to do a good job, I could probably be making more money as a product manager. These days when every iota of attention can be leveraged to the extreme by AI you can’t afford to tie up that attention testing a phone number input across 100 countries and interpreting the results.

161 points by rbanffy 10mo ago

mrits 10mo ago

Author-Owned proof reading is next

__alexs 10mo ago

This just seems to be basically a blog post that somehow got published in ACM?

pjdesno 10mo ago
No, they got it published in ACM SIGSOFT Software Engineering Notes. That's one of the things that publication is for. The paper is a well-supported (if not well-proofread) position paper, synthesizing the author's thoughts and others' prior work but not reporting any new experimental results or artifacts. The author isn't an academic, but someone at Amazon who has written nearly 20 articles like this, many reporting on the intersection of academic theory and the real world, all published in Software Engineering Notes. As an academic (in systems, not software engineering) who spent 15 years in industry before grad school, I think this perspective is valuable. In addition academics don't get much credit for this sort of article, so there are a lot fewer of them than there ought to be.

kayo_20211030 10mo ago

A nice piece that outlines all the challenges, the opportunities, and the cultural and social adjustments that need to be made within organizations to maximize the chance of left-shifted testing being successful. IMPO, as a developer, I see QA's role as being "auditors" with a mandate to set the guidelines, understand the process, and assess the outcomes. I'm wary of the foxes being completely responsible for guarding the hen-house unless the processes are structured and audited in a fundamentally different way. That takes fundamental organizational change.

brap 10mo ago

Do people actually send PRs with no tests? That is so bizarre to me

pjdesno 10mo ago
If your review was based on features shipped, and your bosses let you send PRs with no tests, would you? And before you say "no" - would you still do that if your company used stack ranking, and you were worried about being at the bottom of the stack? Developers may understand that "XYZ is better", but if management provides enough incentives for "not XYZ", they're going to get "not XYZ".
- hamdingers 10mo ago
  Breaking prod repeatedly probably impacts your stack ranking too.
  - Jtsummers 10mo ago
    Depends on how easily the failure is connected back to you personally. If you introduce a flaw this year and it breaks the system in two years, it won't fall back on you but the poor sap that triggered your bug.
  - actionfromafar 10mo ago
    So can "heroically" save prod ... anti patterns.
- wccrawford 10mo ago
  That actually wasn't why I didn't write tests a lot of the time. What stopped me was that after a year of writing tests, I was moved to a higher priority project, and the person who followed me didn't write tests. So when I came back, many of the tests were broken. I had to fix all those in order to get new ones to not be a bother. Repeat again, but this time I came back and the unit testing suite had fundamentally altered its nature. None of the tests worked and they all needed to be rewritten for a new paradigm. I gave up on tests for that system at that point. It simply wasn't worthwhile. Management didn't care at all, despite how many times I told them how much more reliable it made that system, and it was the only system that survived the first giant penetration test with no problems. That doesn't mean I quite testing. I still wrote tests whenever I thought it would help me with what I was currently working on. And that was quite often. But I absolutely didn't worry about old tests, and I didn't worry about making sure others could use my tests. They were never going to try. The final straw, less than a year before I was laid off, was when they decided my "storybook" tests weren't worth keeping in the repo and deleted them. That made me realized exactly how much they valued unit tests. That isn't to say they had no tests. There was a suite of tests written by the boss that we were required to run. They were all run against live or dev servers with a browser-control framework, and they were shaky for years. But they were required, so they were actually kept working. Nobody wrote new tests for it until something failed and caused a problem, though. tl;dr - There are a lot of reasons that people choose not to write tests, and not just for job security.

weinzierl 10mo ago

The article argues that Dev-Owned testing isn't wrong but all the arguments it presents support that it is. I always understood shift-left as doing more tests earlier. That is pretty uncontroversial and where the article is still on the right track. It derails at the moment it equates shift-left with dev-owned testing - a common mistake. You can have quality owned by QA specialists in every development cycle and it is something that consistently works.

TheSoftwareGuy 10mo ago
I'm interested, as I've never been in an org with QA specialists. What does that look like?
- weinzierl 10mo ago
  The are breakers. Good devs are makers. You can be both but I have yet to meet someone who is equally good in both mindsets.
- bluGill 10mo ago
  You do everything the same as today. Then you turn it over to QA who keep finding weird things that you never thought of. QA finds more than half your written bugs (of course I don't write a bug everytime a unit test fails when doing TDD, but sometimes I find a bug in code I wrote a few weeks ago and I write that up so I can focus on the story I'm doing today and not forget about the bug) QA should not be replacing anything a developer does, it should be a supplement because you can't think of everything. We also use QA because we are making multi-million dollar embedded machines. One QA can put the code of 10 different developers on the machine and verify it works as well in the real world as it does in software simulation.
- philk10 10mo ago
  They find all the things the devs and their automated tests missed, then they mentor the devs in how to test for these and they work out how the bug could have been found earlier. Rinse and repeat until the tester is struggling to find issues and has coached the devs out of his job

thmpp 10mo ago

If as a developer you want to be seen as someone advancing and taking ownership and responsibility, testing must be part of the process. Sending an untested product or a product that you as a software engineer do not monitor, essentially means you can never be sure you created an actual correct product. That is no engineering. If the org guidelines prevent it, some cultural piece prevents it. Adding QA outside, which tests software regularly using different approaches, finding intersections etc. is a different topic. Both are necessary.

sigmoid10 10mo ago
The problem in big companies is that as a developer, you are usually several layers of people removed from the people actually using the product. Yes you can take ownership and implement unit tests and integration tests and e2e tests in your pipeline, to ensure the product works exactly as you intended. But that doesn't mean it works as management or marketing or the actual user intended.

sethammons 10mo ago

While good points are made, I worry this gives the wrong impression. The paper doesn't say it is impossible, just hard. I have, very successfully, worked with dev owned testing. Why it worked: the team set the timelines for delivery of software, the team built their acceptance and integration tests based on system inputs and outputs based on the edges of their systems, the team owned being on-call, and the team automated as much as possible (no repeatable manual testing aside from sanity checks on first release). There was no QA person or team, but there was a quality focused dev on the team whose role was to ensure others kept the testing bar high. They ensured logs, metrics, and tests met the team bar. This role rotated. There was a ci/cd team. They made sure the test system worked, but teams maintained their own ci configuration. We used buildkite, so each project had its own buildkite.yml. The team was expected by eng leaders to set up basic testing before development. In one case, our team had to spend several sprints setting up generators to make the expected inputs and sinks to capture output. This was a flagship project and lots of future development was expected. It very much paid off. Our test approach was very much "slow is smooth and smooth is fast." We would deploy multiple times a day. Tests were 10 or so minutes and very comprehensive. If a bug got out, tests were updated. The tests were very reliable because the team prioritized them. Eventually people stopped even manually verifying their code because if the tests were green, you _knew_ it worked. Beyond our team, into the wider system, there was a light weight acceptance test setup and the team registered tests there, usually one per feature. This was the most brittle part because a failed test could be because another team or a system failure. But guess what? That is the same as production if not more noisy. So we had the same level of logging, metrics, and alerts (limited to business hours). Good logs would tell you immediately what was wrong. Automated alerts generally alerted the right team, and that team was responsible for a quick response. If a team was dropping the ball on system stability, that reflected bad on the team and they were to prioritize stability. It worked. Hands down the best dev org I have part of.

regularfry 10mo ago
I've worked in a strong dev-owned testing team too. The culture was a sort of positive can-I-catch-you-out competitiveness that can be quite hard to replicate, and there was no concept of any one person taking point on quality.

boxed 10mo ago

I think they've got that the other way around.

OptionOfT 10mo ago

The conversation is usually: devs can write their own tests. We don't need QA. And the first part is true. We can. But that's not why we have (had) QA. First: it's not the best use of our time. I believe dev and QA are separate skillset. Of course there is overlap. Second, and most important: it's a separate person, an additional person who can question the ticket, and who can question my translation of the ticket into software. And lastly: they don't suffer from the curse of knowledge on how I implemented the ticket. I miss my QA colleagues. When I joined my current employer there were 8 or so. Initially I was afraid to give them my work, afraid of bad feedback. Never have I met such graceful people who took the time in understanding something, and talking to me to figure out where there was a mismatch. And then they were deemed not needed.

torginus 10mo ago
I very rarely worked with good QA. In my mind a good QA understands the feature we're working on, deploys the correct version, thoroughly tests the feature understanding what it's supposed and not supposed to do, and if they happen to find a bug, they create a bug ticket where they describe the environment in full and what steps are necessary to reproduce it. For automation tests, very few are capable of writing tests that test the spec, not implementation, contain sound technical practices, and properly address flakiness. For example it's very common to see a test that clicks the login button and instead of waiting for the login, the wait 20 seconds. Which is both too much, and 1% of the time too little. Whenever I worked with devs, they almost always managed to do all this, sometimes they needed a bit of guidance, but that's it. Very very few QA ever did (not that they seemed to bothered by that). A lot of QA have expressed that devs 'look down' on them. I can't comment on that, but the signal-to-noise ratio of bug tickets is so low, that often it's you have to do their job and repeat everything as well. This has been a repeated experience for me with multiple companies and a lot of places don't have proper feedback loops, so it doesn't even bother them as they're not affected by the poor quality of bug reports, but devs have to spend the extra time.
- bbayles 10mo ago
  I used to work with a QA person who really drove me nuts. They would misunderstand the point of a feature, and then write pages and pages of misguided commentary about what they saw when trying to test it. We'd repeat this a few times for every release. This forced me to start making my feature proposals as small as possible. I would defensively document everything, and sprinkle in little summaries to make things as clear as possible. I started writing scripts to help isolate the new behavior during testing. ...eventually I realized that this person was somehow the best QA person I'd ever worked with.

KingOfCoders 10mo ago

Developers want things to work. QA wants things to break. What worked for me, devs write ALL the tests, QA does selective code reviews of those tests making devs write better tests. I also wrote the failure of Dev-Owned Testing: "Tests are bad for developers" https://www.amazingcto.com/tests-are-bad-for-developers/ https://www.amazingcto.com/tests-are-bad-for-developers/

darkwater 10mo ago

First they came with the NoOps movement, and you were happy cause those damned ops people were always complaining and slowing you down. I can manage my infra! Then, they came with the dev-owned testing and fired all the QAs, and you were happy because they were always breaking your app and slowing you down. I can write my tests! Now, they are coming with LLM agents and you don't own the product...

BigTTYGothGF 10mo ago
> you were happy ... you were happy Like heck I was.
- bluGill 10mo ago
  I have worked with bad ops people who didn't let anything get done, and good ops people who knew how to do tricky things and kept the system working so I didn't have to care. I have worked with good and bad QA testers. Guess who I'm glad are gone.

terribleidea 10mo ago

Most orgs I've worked for are so growth and product-focused that if you try adjusting your estimates to include proper testing, you get push back, and you have to ARGUE your case as to why a feature will take two weeks instead of one. This is the thing I hate the most about work, having to ARGUE with PMs because they can't accept an estimate, there's often some back-and-forth. "What if you do X instead?" "Team Y (always uses hacks and adds technical debt with every single feature they touch) did something similar in two days." But we're just communicating and adding transparency so that's good and it certainly doesn't matter that it starts taking up 4+ hours of your time in Slack conversations and meetings of people 'level setting' 'getting on the same page' trying to help you 'figure out' how to 'reduce scope' etc. etc. Also, I think testing via unit or integration tests should be standard regardless, and that isn't what I am thinking about here. I'm thinking about QA, the way QA does it. You hammer your feature with a bunch of weird bullshit like false and unexpected inputs, what happens if I refresh the page in strange ways, what happens if i make an update and force the cache to NOT clear, what happens if I drop my laptop in a dumpster while making the request from firefox and safari at the same time logged in as the same user, what happens if I turn off my internet in the middle of a file upload, and so on. When devs say that devs should be responsible for testing, they usually mean the former (unit and integration tests), and not this separate skillset of coming up with a bunch of weird edge cases for your code. And yes, unit tests SHOULD hit the edge cases, but QA is just better at it. You usually don't have engineers testing what happens when you try sending in Mandarin characters as input (unless they live in China, I guess). All of that effort should bring up your estimates because it is non-trivial. This is what getting rid of QA means, not happy path end-to-end testing plus some unit and integration tests.

2OEH8eoCRo0 10mo ago
The stupid fast tempo of our industry grinds my gears. When I worked defense we moved slowly and methodically. It almost felt too slow. Now in the private sector I move like triple the speed but we often need to go back and redo and refactor things. I think it averages out to a similar rate of progress but in defense at least I had my sanity.
JohnMakin 10mo ago

deleted 10mo ago

[deleted]

donatj 10mo ago

> The problem is not that dev-owned testing is a flawed idea, but that it is usually poorly planned In our case there was zero plan. One day they just let our entire QA team go. Literally no direction at all on how to deal with not having QA. It's been close to a year and we're still trying to figure out how to keep things from going on fire. For a while we were all testing each other's work. They're mad that this is slowing down our "velocity", and now they're pushing us to test our own work instead... Testing your own work is the kind of thing an imbecile recommends. I tested it while I wrote it. I thought it was good. Done. I have all the blind spots I had when I wrote it "testing it" after the fact.

nlunbeck 10mo ago
Frustrating to see. Dev-led testing is too fundamentally different from a QA function, just as any amount of E2E tests can't replace manual testing. Each tries to solve for a different type of problem. Is it possible to do effective dev peer "QA" without essentially duplicating the QA role? And forget about testing one's own work..

tracerbulletx 10mo ago

My experience with this was great. It went really well. We also did our own ops with in a small boundary of systems organized based on domain. I felt total ownership for it, could fix anything in it, deploy anything with any release strategy, monitor anything, and because of that had very little anxiety about being on call for it, best environment I ever worked in.

dasil003 10mo ago

Oh man do I have opinions. First of all, I've seen all type of teams be successful, ranging from zero QA at all, to massive QA teams with incredible power (eg. Format QA at Sony in Europe). I have absolutely seen teams with no QA deliver high quality full stop, the title is nonsense. My firm belief is that QA can raise the ceiling of quality significantly if you know what you're doing, but there is also huge moral hazard of engineers dropping the ball on quality at implementation time and creating a situation where adding more QA resources doesn't actually improve quality, just communication churn and ticket activity. By the way the same phenomenon can happen with product people as well (and I've also seen teams without product managers do better than teams with them in certain circumstantes). The most important anchor point for me is that engineering must fundamentally own quality. This is because we are closer to the implementation and can anticipate more failure modes than anyone else. That doesn't mean other roles don't contribute significantly to quality (product, design, QA, ops absolutely do), but it means we can't abdicate our responsibility to deliver high quality code and systems by leaning on some other function and getting lazy about how we ensure we are building right. What level of testing is appropriate for engineers to do is quite project and product specific, but it is definitely greater than zero. This goes double in the age of AI.

travisjungroth 10mo ago
> The most important anchor point for me is that engineering must fundamentally own quality. This is huge. I was selling software to help QA. I saw a CEO demand a Head of QA guarantee their super buggy app be free of bugs by a certain date. This is terrible. She didn’t write the thing. Total responsibility without authority trap. She was, not at all to my surprise, fired. I think the deal fell through and I don’t know how else things ended up with them. QA’s job is signal. If you’re getting clear signal, they’re doing their job.
- philk10 10mo ago
  yes - testing gives information, what is done with that information is a business decision
- hobs 10mo ago

cratermoon 10mo ago

The paper highlights the problem in two words of the first sentence of the abstract: "shrink QA". Corporations do it to save money, and accept the loss of quality as the cost of doing business. Therein lies part of the reason for the sad state of software today.

gwbas1c 10mo ago
I suspect they underestimate the loss of quality; or underestimate the consequences.
- cratermoon 10mo ago
  I don't think they underestimate the loss of quality. As for the consequences, "No one in this world, so far as I know ... has ever lost money by underestimating the intelligence of the great masses of the plain people" - H. L. Mencken
- cratermoon 10mo ago
  Also this list of Windows bugs, just from this week. - https://learn.microsoft.com/en-us/answers/questions/5709441/windows-11-unable-to-install-2026-01-security-upda https://learn.microsoft.com/en-us/answers/questions/5709441/... - https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-25h2 https://learn.microsoft.com/en-us/windows/release-health/sta... - https://www.techspot.com/news/110848-onedrive-backup-feature-making-users-local-files-seemingly.html https://www.techspot.com/news/110848-onedrive-backup-feature...

tomtoday 10mo ago

Note: the references in the article seem to be incorrect [4] through [7] are all the same article. I do not think that was intentional.

gwbas1c 10mo ago

As a developer, I frequently tell higher ups that "I have a conflict of interest" when it comes to testing. Even though I fully attempt to make perfect software, often I have blind spots or assumptions that an independent tester finds. That being said: Depending on what you're making and what platform(s) you target, developer-owned testing either is feasible or not. For example, if you're making a cross-platform product, it's not feasible for a developer to regression test on Windows 10, 11, MacOS, 10 distros of Linux. In contrast, if you're targeting a web API, it's feasible for a developer to write tests at the HTTP layer against a real database.

time4tea 10mo ago

The abstract says it really: "It was clearly a top-down decision" Many many things that are imposed like this will fail. Its not willful non-compliance even, its just that its hard for people to do things differently, while still being the same people in the same teams, making the same products, with the same timelines... Context is key here, lots of people see a thing that works well and think they can copy the activities of the successful team, without realising they need to align the mindset.. and the activities will follow. The activities might be different, and thats OK! In a different context, you'd expect that. I'd argue that in most contexts you don't need a QA team at all, and if you do have one, then it will look a lot different to what you might think. For example, it would be put after a release, not before it.. QA teams are too slow to deal with 2000+ releases a year - not their fault, they are human.. need to reframe the value statement.

MoreQARespect 10mo ago

This paper has 7 references and 4 of them are to a single google blog post that treats test flakiness as an unavoidable fact of life rather than a class of bug which can and should be fixed. Aside from the red flag of one blog post being >50% of all citations it is also the saddest blog post google ever put their name to. There is very little of interest in this paper.

monster_truck 10mo ago

Something I've always believed, and my experience with shipping multinational software on a schedule that has severe drop-dead dates confirmed: If you are contractually obligated to deliver a product that does x, y, and z correctly? QA is the only way to do that seriously. If you don't have QA, you don't care about testing full stop. This only compounds when you have to comply with safety regulations in every country, completely setting aside the strong moral obligation you should feel to ensure you go far above & beyond mere compliance given the potential for harm. This compounds again when you are reliant upon deliverables from multiple tiers of hardware and software suppliers, each contract with its own drop-dead dates you must enforce. When one of them misfires, and that is a "when, not if", they are going to lie through their teeth and you will need hard proof. These are not small fines, they are company-killing amounts of money. Nobody profits in this situation. I've been through it twice, both times it was a herculean effort to break even. Hell, even a single near-miss handled poorly is enough to lose out on millions in potential future work. The upsides are quite nice, though. I didn't know it was possible to get more than 100% of your salary as a bonus until then. Don't take my word for it, though. Ask your insurance agent about the premiums for contractual liability insurance with and without a QA team. If you can provide metrics on their performance, -10-15% is not uncommon, this discount increases over time. Without one? +15-50% depending.

gwbas1c 10mo ago

> At Google, for example, 16% of tests exhibited flakiness This really surprised me. In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. In the past year, we had a flaky unit test that caused about 1-2% of builds to fail. Upon fixing it, we learned it was what caused a deadlock in a production service every 5-6 months. As a result of fixing this one "flaky" test, we eliminated our biggest cause of manual intervention in our production environments.

inetknght 10mo ago
> > At Google, for example, 16% of tests exhibited flakiness > This really surprised me. It doesn't surprise me. Close to 80% of Google code is actually quite terrible. > In my experience, usually a flaky test indicates some kind of race condition, and often a difficult-to-reproduce bug. Yup, that's my experience too. I flip shit when I start seeing flaky tests because it means a lot of actual work. Luckily there's generally a lot of tools available (sanitizers are a godsend) to fix it, and a lot of experience I rely on to look for smells when those tools come up empty. It makes me really sad when I work on a project with few (or, gasp, zero) tests. I've spent the last 5 months just adding tests to my current project at $employer and... those tests have revealed all kinds of problems in dependencies created by other teams at $employer. Ugh.
zmj 10mo ago
At scale, every test is flaky.
- gwbas1c 10mo ago
  Oh no, a well-written unit test is 100% deterministic. It's once you start doing tests in real environments with real databases and real networks that the unpredictability of the real world creeps in!
- m3047 10mo ago
  Well, strictly anecdata but there was a time when I worked on code which for $reasons had no dedicated test environment, and externalities were not our problem according to management. So we were reaching out to the big bad world for testing, but we didn't own the network between it and us. I ended up writing tests for the network. Caught a lot of problems with the network. Didn't make any friends. DISCLAIMER: I've been around IT for probably the majority of y'all's lifetimes, so I'm not saying this happens often. But just because something is fundamentally wrong, doesn't mean that all fundamentally wrong things are the same. In my experience they differ more from each other than the possible good ways of doing the same thing. Don't conflate things without a good reason.

gwbas1c 10mo ago

(Joke) Can't AI just replace QA?

jayd16 10mo ago
Honestly very pleased that AI is not dominating the discussion. Nature is healing.

physicsguy 10mo ago

I think devs owning testing only works where they’re consumers of the product. So a developer productivity tool - perfect. A fully fledged engineering application targeting monitoring of assets? Not so much

wpollock 10mo ago

One purpose of QA testing is compliance assurances, including with applicable policies, industry regulations, and laws. While devs are (usually) good at functioal testing, QA (usually) does non-functional testing better. I have not known any devs that test for GDPR compliance for example. (I am certain many devs do test for that, just stating my personal experience.)

inetknght 10mo ago

Dev-owned testing is great when it succeeds. It's definitely not the only way. But you really do need the whole team to be on-board with the concept, or willing to train the team to do it. If that doesn't line up then you're not going to have a good time. I've been on teams that own the whole pipeline, and have lead teams where I made it mandatory that the engineer writing a feature must also write tests to exercise that feature, and must also write sad-path tests too. That gets enforced during review of the pull request. It works. But it takes a lot of effort to teach the whole team how to write testable code, how to write good tests, how to write sad-path tests, and how to even identify what sad paths might exist that they didn't think about. I can tell you from experience that, when it does succeed and the whole team has high collaboration, then individual developer's work output is high, and new features can be introduced very rapidly with far fewer bugs making it to production than relying on a whole QA team to find all the problems. It fails in practice because most (not all!) devs don't want to "waste time" doing that, and instead rely on QA cycles to tell them that something is wrong. Alas, QA cycles are a hell of a lot slower than the developer writing the tests. QA teams often don't have access to (or perhaps don't understand) the source code, and so they're left trying to find bugs through a user interfaces. That's valuable, but takes a completely different skillset and is a poor time to find a lot of the basic bugs that can show up. On the other hand, the teams I've been on that failed (especially hard) often had huge (!) QA teams and budgets. Despite the size of team and budget, multiple projects would fall over from inertia and bickering between teams about who owns which bug, or which bug needs priority fixing.

nlunbeck 10mo ago
(Disclaimer: I work for shipyard.build so i am biased here..) Dev-owned testing (or even dev-involved testing) is much more realistic when devs have shorter feedback loops between their code and its deployment. So often i've seen momentum get lost when devs have a wait period before they can run tests/do basic manual testing. Then the test aspect becomes the thing that "slows down" devs before they ship a feature, so they might tend towards shortcuts
m3047 10mo ago
For shiggles I searched the entire first page of comments for "testable" (looking for "how to write testable code" as a proxy for devs write testable code if they write tests) and this is the only mention. So +1 to you!

threethirtytwo 10mo ago

Simple. Because all theories are approximations. And our most high resolution theory of reality assumes reality is intrinsically random making it so that even the most accurate theory of reality can't predict anything.

johnea 10mo ago

I feel the need to point out a phrase that was very popular among my dev peers: The difference between theory and reality, is that in theory they're the same, but in reality they're not... While any new feature, or bug fix, introduced by a dev should certainly be tested at that dev's desk to confirm to themselves that it's correct; it should also (of course) be tested by a product test group (call it QA if you must) to insure that all functional features of the product are still fully and correctly implemented. I would aim a big fat finger at "agile", "scrum", "standup" culture for encouraging the violation of this, very obvious, testing requirement. "What have you accomplished in the last 4 hours", type of management interface to development, fully and completely misses the primacy of confirming the functionality of updates before release. This is really due to management, especially C-suite management of startups, living in a make believe world of deadlines and feature requirements pulled arbitrarily out of their ass, while refusing (or not having the capacity) to understand the technical issues involved.

theptip 10mo ago

> Why does dev-owned testing look so compelling in theory, yet fall short in almost every real-world implementation Seems like a weird assertion. Plenty of startups do “dev owned testing” ie not hiring dedicated QA. Lots of big-tech does too. Indeed I’d venture it’s by far the most common approach on a market valuation-weighted basis.

array_key_first 10mo ago
It's very common and most software is extremely buggy, so that adds up. I think most consumer software just would not stand the scrutiny of enterprise or professional applications. It just does not work a lot of the time, which is truly insane to think about. I mean, imagine if your compiler just 10% of the time did not work and output a wrong program. But when I log into website XYZ, that's about the rate of failure.

flambojones 10mo ago

Having been at Microsoft when we had SDETs for everything (and I miss it greatly, though the way we could write a feature and then just toss it to test was ridiculous), I think things have swung too far away. On one hand, engineers needed to take more ownership of writing things other than low-level unit tests. On the other, the SDETs added immense value a ton of ways, like writing thorough test specs based off of the feature spec (rather than the design spec), testing without blind spots due to knowledge of the implementation, implementation of proper test libraries and frameworks to make tests better and easier to write, and an adversarial approach to trying to break things that makes things more robust. I've also worked with manual QA for product facing flows, and while they added value with some of their approaches to ensuring quality - poking at our scenarios and tests, and looking more closely at subjective things - they often seemed to work as a crutch for the parts of code paths that engineers had made too difficult to test. I've never seen anywhere attempt to replace the value that SDETs delivered with what engineers were tasked with. I'd argue it's not necessarily possible to fully replicate that when you're testing your own things. But with services now, it also seems like product/management are more willing to have slightly few assurances around quality and just counting on catching some in production in favor of velocity. I've never seen places that got rid of QA

zbentley 10mo ago

Another vital quality of good QA teams is that they often serve as one of the last/main repositories of tribal knowledge about how an org's entire software system actually behaves/works together. As businesses grow and products get more complex and teams get more siloed, this is really important.

__MatrixMan__ 10mo ago

It pretty much comes down to whether QA is just doing what dev tells them--in which case they're not applying any scrutiny to dev's decisions, or if they're deciding for themselves what constitutes an appropriate validation for the dev work at hand.

dgunay 10mo ago

I have limited experience working in orgs with a QA apparatus. Just my anecdotes: The one time I got to work with a QA person, he was worse than useless. He was not technical enough to even use cURL, much less do anything like automated e2e testing, so he'd have to manually test every single thing we wanted to deploy. I had to write up extremely detailed test plans to help him understand exactly what buttons he had to press in the app to test a feature. Sometimes he'd modify the code to try and make testing it easier, break the feature in doing so, and then report that it didn't work. In nearly all cases it would have been faster for me to just test the code myself. The majority of the time I've worked in orgs where there is no QA team, the devs are expected to own the quality of their output. This works okay when you're in a group of conscientious and talented engineers, but you very quickly find out who really cares about quality and who either doesn't know any better or doesn't care. You will constantly battle management to have enough time to adequately test anything. Every bit of test automation you want to build has to be smuggled in with a new feature or a bugfix. So really, they both suck, pick your poison. I prefer the latter, but I'm open to experiencing what good looks like in terms of dedicated QA.

philk10 10mo ago
that sucks, hope you get to work with a good tester/QA person at some point so you get to see what they can do

wesselbindt 10mo ago

Conflict of interest that the author fails to mention: he's a QA manager at Amazon, and has a vested interest in QA being seen as a necessary role. It may well be, but this is definitely a conflict of interest. Aside from that, this article is incredibly heavy on theory and very light on empirical fact. Its bibliography consists of a very narrow selection of blogs (4 of the articles he quotes are one and the same blog somehow), which talk about a very narrow subset of the industry. This article not referencing the serious and well-established research that has been done on the effectivity of dev-owned tests by for example the DORA folks, almost seems dishonest. The clickbait title, when compared to the content of the article, is outright dishonest. The author theorizes about some warts dev-owned testing may have at some specific companies, but this is a very far cry from it failing in practice, especially when you compare them to the warts of offloading quality to a different team. It's probably a bit harsh, but I feel like, as an industry, we should have a higher standard of empiricism when it comes to evaluating our ways of working.

shuntress 10mo ago

I'm surprised that with this many comments about the relationship between testing, development, and QA there is so little mention of environment and deploy process. The usability of your test environment (and associated tooling) has a massive impact on quality assurance. Every small difference between Production and Production-Plus-Feature creates friction and, even in systems of only moderate complexity, that friction adds up fast.

pftg 10mo ago

I cannot believe the excuse for why shift-left QA is “not working” is that Amazon hires developers who can’t learn basic testing skills that QA engineers picked up in three months. If developers can’t write valid code for tests, that’s on the organization, not on the practice. The author forgot to mention the costs of handoffs, which paid off all those tiny learning investments. Shift-left has over 30 years of proof as one of the most effective ways to build reliable software. P.S. This isn’t an ACM article; it’s a strongly opinionated post based on personal experience. P.P.S. I'm not against QA, but make them as bug/quality hunters, instead of toil.

somewhereoutth 10mo ago

If your product interface is with humans, you test it firstly with devs, then QA, then your customers. Devs are a bit leaky for bugs/non-conformance, so if you skip the QA, then your customer is exposed. For some industries this is fine - for others, not so much.

tom_m 10mo ago

Writing tests cases is part of development. Programmers simply need to do it. It helps them immensely. Product acceptance and user acceptance testing is entirely different and often conflated. That's where people go wrong and are too simple minded (or cheap) to understand or invest in both.

tom_m 10mo ago

This paper is really good and accurate. Many people would benefit from reading it.

squirrellous 10mo ago

I work with an amazing QA team right now. They are good programmers that don’t mind doing lots of grunt work, understand the goal of the software deeply, collaborates extremely well with other teams, and are just all around great people. I expect they would kill it as a non-QA dev team as well, so I’m always grateful to have such partners.

solatic 10mo ago

Devs need to write the 1% of automated tests needed just to prove that what they wrote works in the ideal case. QA is valuable for writing the 99% of automated tests that prove that the software works in the edge cases, with DevOps occasionally dropping in to make sure that the test suite runs quickly. The way you solve Product and QA being at odds is very simple: QA loses, until they don't. When trying to find product-market-fit, it doesn't make sense to delay delivery to prove that an experiment works in exceptional circumstances. Eventually you do have product-market-fit, and you want to harden the features you already shipped, which is where QA comes in - better internal QA finds the bugs rather than your (future) customers. Eventually you start launching features to a massive audience on day 1, and you need QA to reduce reputational risk before you ship. The right time for QA to intercede and get a veto on delivery changes over the lifetime of the product, and part of whether or not QA is a net-add is whether your organization (leadership) is flexible enough to accept and implement that flexibility.

polotics 10mo ago

Having gone through the whole cycle from 1 dev... to 2 dev-lead... to 3 large-ish team qa-lead for offshore devs... to 4 qa-lead for offshore qa... to 5 actual qa ("a" is for assurance as opposed to the quality control that passes as qa) in an industry that needs it... to 6 kind-of principal engineer... I would advise you that generalities about qa are useless, environments differ. Still always true: some mgmt will either want more for less, or even something for nothing, and in the long run we are all dead, and some folks' horizon is surprisingly short!

fizlebit 10mo ago

I think that vibe coding now with anthropic tools and the latest model means that the cost of writing integration tests is significantly reduced. When the company ships a large product that has components from many teams, there is still a role for QA engineers who run nightly tests and chase teams to help diagnose the issue when there is an issue found. If you don't have such a central team publishing golden versions, then everybody is chasing the same bug. Ideally the integration tests are part of the change acceptance flow, but low frequency bugs (occur maybe 1 in 100 test runs) can still sneak through.

DiskoHexyl 10mo ago

All the great QAs with whom I've worked would have made good developers (and they actually WERE good developers, only with a QA name and salary). The problem is that a great QA earns less than a mediocre developer within the same company. And has a much lower status. And also fewer career opportunities elsewhere. No wonder most of those guys switched at one point or another

isquaredr 10mo ago

Economically devs are the only ones that can do it. If I am a good QA engineer that cares enough about the product to do a good job, I could probably be making more money as a product manager. These days when every iota of attention can be leveraged to the extreme by AI you can’t afford to tie up that attention testing a phone number input across 100 countries and interpreting the results.