9 ms·

>developers now own the test writing and CI infrastructure runs and reports the tests. It has really made software more reliable I switched from medical, with

by diydsp 6y ago

>developers now own the test writing and CI infrastructure runs and reports the tests. It has really made software more reliable I switched from medical, with fierce QA, to more general SW dev over the last 3 years. Developers waste time writing braindead unit tests and don't stress their software like QA teams. Major bugs run undetected for weeks and cause chaos. Managers grin believing they're net-ahead.

bigtimegames 6y ago
Hmm i think the pace of changes has made software break more, but unit tests and integration tests are an improvement. But I agree writing unit tests takes a hella lotta time, not sure if a sr eng should be doing that 1/2 the time
01100011 6y ago
I'm at a FANG with some of the smartest people I have ever encountered in my life, yet no one seems to agree with me that devs should not write their own tests. There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. We also seem to be obsessed with testing that is half-way between unit level testing and system level testing. We test APIs and features more or less in isolation. Tests should be written by antagonistic developers who are not at all familiar with the code. They should stress the capabilities promised in the interface documentation. Hiring more people who are dedicated to testing costs a lot of money though. I don't see things changing anytime soon.
- robbyt 6y ago
  The problem is, when devs don't write their own tests, the code they write tends to be untestable.
  - onion2k 6y ago
    I suspect you think the developer writes the code and then the QA engineer writes tests for that code. That's the wrong way round. The tests are written first, and they all fail, then the dev has to write code to make them pass. By definition it's literally impossible to write untestable code that way because then the tests won't pass. It's the principle of TDD but separated in to different roles. It works well and produces good results but you need exceptionally good planning upfront to actually do it.
    - eliasbagley 6y ago
      Who designs the interfaces that the tests call?
      - onion2k 6y ago
        That's the planning part. Usually it's done as a collaboration between the systems architect and the QA team. The process is essentially "design the system architecture -> document the APIs -> write the tests -> implement the code -> QA". It takes a level of rigour and planning that most teams don't want to do, and it relegates developers to a pretty minor role in the whole development process. They end up just filling in the blank bits in someone else's design. Most changes in team structure over the past couple of decades have been about developers taking on more of the responsibilities rather than less. It's actually one of the aspects of TDD that makes it so robust. If you properly design your code and really think about its APIs you get a better app at the end. It does take longer to get something a user can see though, but often the process gets to a finished iteration of an app quicker because there's fewer cycles of debugging and fixing needed.
        realusername 6y ago
        > The process is essentially "design the system architecture -> document the APIs -> write the tests -> implement the code -> QA". It takes a level of rigour and planning that most teams don't want to do That's a very charitable way to say it, on my case I would call this kind of process very convoluted and similar to early 2000s style development. I obviously would not want to work in such a bureaucratic environment where every small tech decision has to go through multiple boards.
        onion2k 6y ago
        I obviously would not want to work in such a bureaucratic environment where every small tech decision has to go through multiple boards. Everyone who works on the app is making those decisions no matter what you do. The difference is that in the "old style" the decisions were made before the code is written, and in the agile "new style" the decisions are made after the code is written. Often that's fine, because developers are generally good at their work and they make decent decisions, but sometimes they get it wrong and that's when code design issues arise. It's also what leads to automated tests failing to test for a lot of cases that a good QA engineer would have written tests for. Those things have an impact on the user. A huge amount of the code in apps we use every day was never designed. No one thought it through. No one considered the edge cases. Features are thrown together in a week and 'sort of' work. Every time you see a shitty broken website, or a bug in production, or some crappy slow thing that should be fast the reason behind the problem is that the developers who made it didn't take the time (or even have the time in really bad companies) to think about what they were building. A lot of developers like that working environment because they can hack on things and move on to the next challenge quickly. That's fun. I argue it's also bad for users, and I care more about users than I care about developers (and I say that as a developer whose been making web stuff for almost 25 years.) I'm not arguing that we go back to Prince2 and waterfall. There's a limit to my tolerance of bureaucracy too. I'm saying that things have gone a bit too far the other way, and many developers need to spend more time planning what code is written before leaping in and coding up the first solution they think will pass the acceptance criteria someone from product wrote.
- deleted 6y ago
  [deleted]
- stickfigure 6y ago
  That sounds like an incredibly slow feedback cycle. How would that work? Write some code, check it in, wait for someone else to write tests... and come back in hours/days/weeks to fix any issues? Getting constantly dragged back to fix last week's bugs sounds like a chapter from Dante's Inferno. I'd rather discover them while the code is fresh in my mind. Without tests, how do I know when I'm done?
  - quiescant_dodo 6y ago
    In the past, I had pseudo-pair programmed in order to test. We'd agree on the external-facing API and pair-program a slow in-memory implementation. Once that was completed, we would split off and one of us would write test cases against the API (fixing up the in-memory implementation as needed) while the other would work on a proper database-backed implementation. I haven't done this in probably 7 years, though. Mostly because I haven't cultivated this kind of working relationship with my coworkers.
  - fossuser 6y ago
    I've seen both types of org layouts where I've worked (we moved from one to the other). Initially separate QA teams that would write tests, a subset of which might do manual testing in the UI. The feedback loop for this was brutal as you suggest and caused a lot of problems, with delayed breaks that become hard to fix and debug. It's also hard for QA to often know what the code is doing or interact closely with the dev teams. You can try to get them to be as close as possible, but it doesn't work super well. Now devs are responsible for writing tests and teams think about potential failures when designing things/discussing the plan with their team. They were supposed to consider these things before too, but the incentives weren't aligned - now they are. We have infrastructure teams to make automated testing as easy as possible and spin up a test environment where you can see your change, etc. I think this model is better, testing well is hard and reasoning about failures and how things can go wrong is an important skill to develop. Offloading this to a different team feels like a variant of the dev that 'just writes features' and doesn't consider actual deployment. This is often made doubly worse by companies that have QA teams often treating them as second class citizens both in status and in pay. I think it's better to have it be part of the job of the person writing the feature also write the tests, it doesn't help that writing tests is something few people know how to do coming out of school.
- cbryan 6y ago
  You're getting heavily downvoted and I'd love to know why. I've seen similar attitudes toward testing among the groups I've worked with, nobody really wants to do it. Are folks downvoting this because they don't like testing? Is this idea that prevalent?
- irjustin 6y ago
  Count me in as one of those who disagree with you. I 100% side with the grandparent comment. When you've worked with a great QA, it's night and day vs a dev writing their own tests, but a dev not writing its own tests is a nightmare. Ideally there's a place for both and they work very closely with one another to make the software stable at all levels. It's arguable the SRE is the new QA engineer.
- void_mint 6y ago
  > yet no one seems to agree with me that devs should not write their own tests. I'll die on this hill. The developer that wrote the code by definition cannot write an appropriate test suite for it. It is entirely possible/probable that details missed in implementation will be missed entirely in test, as the developer missed them. < controversial take> Manual QA promotes bad habits and is not a great thing to introduce to an org. SDETs/QA Engineers/Developers whose role is explicitly to create tests for an org are worth their salary 10x. </ controversial take >
  - kqr 6y ago
    > are worth their salary 10x. Are you saying it's a bad role to get into because it's underpaid? ;)
    - EdwardDiego 6y ago
      Yes, sadly for some reason in most orgs, QA/QE whatever you're calling it, pays lower than dev work. Even though, IMO, they're worth far more.
      - hvidgaard 6y ago
        They are worth more in the same sense a salesman is worth more than the dev. Without the dev neither have anything to do, yet they all provide equally valuable work. Perhaps it's not a matter of value, but supply and demand.
      - NicoJuicy 6y ago
        I don't agree. You can't finish a project without developers and you can finish it without testers. I've seen a tester changing requirements on the go, delaying features for things that aren't even the case. Those change requests then cause bugs, because... Dev. The most important thing for a dev is domain knowledge and letting them care about the product. Usually, when the original Dev leaves ( monolith), it's enough to call the project abandoned, but maintained. Leaving the devs wanting to spend a minimal amount of work with it. And that's going to trigger sloppyness.
- nixpulvis 6y ago
  I feel like this isn't that hard. 1. Write (some) tests as you develop to help you Writing down your program's test cases should not add much time (assuming you have test infrastructure setup like you should), since you are already having to consider these things to simply figure out what the right logic is. With a little practice, you find a good rhythm for how far to take it for yourself. In my experience, I find this actually leads to faster and more correct solutions. 2. Collect tests while in review to cover the requirements Once you've written it though, it's always important to try and get others to look over your code and try and find flaws. Ideally, they can even write tests for you to add to the feature. This is just a standard part of any good peer review. 3. REGRESSION TEST Finally, though I'm not sure it matters very much who writes it, regression tests are something I never see stressed enough. I feel like if you are serious about testing, every bug should be required to add at least one test to the suite before it can be closed. Just my 2-cents.
- kqr 6y ago
  > There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. This is a very common -- but inevitable -- misunderstanding. The real reason the same person should write tests and code is if you find a person who is competent enough in both technical domain and problem domain to write the tests -- you definitely want that person to write the code as well. See how it's not "developers should write tests"? It's "domain experts should write the tests... and domain experts should write the implementation." The two tasks happen to need overlapping skills.
- ohazi 6y ago
  Black box test - Testing a unit without any information about its internals. You provide inputs, you record outputs, and you have a spec. By definition, the author of a unit cannot write a black box test for it. Any test they write is going to be a white box (clear box?) test, because they know how the unit works. If you interpreted the spec incorrectly when writing the unit, you're going to interpret it incorrectly in exactly the same way when writing the test. White box testing has its place, but you need a second person if you want to do black box testing.
- tablespoon 6y ago
  > I'm at a FANG with some of the smartest people I have ever encountered in my life, yet no one seems to agree with me that devs should not write their own tests. > There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. We also seem to be obsessed with testing that is half-way between unit level testing and system level testing. We test APIs and features more or less in isolation. You're both right. Devs should write tests for their own code, but they shouldn't be the only ones writing them. One of the problems (at least at my employer) is QA is considered a place for second-rate developers. Some of them are so bad that they rely on the devs to basically write their tests for them, by requiring extremely detailed and specific acceptance criteria. Only once have I worked with a tester that was engaged enough to really understand the requirements on their own and call out weird behavior or corner cases that weren't spelled out in advance. It was great.
  - xupybd 6y ago
    I worked with a tester turned dev. He was amazing, his attention to detail and ability to grok specifications was second to none
  - sumedh 6y ago
    > Some of them are so bad So why is your company hiring bad QAs?
    - cassianoleal 6y ago
      The answer seems to be in the same message: > QA is considered a place for second-rate developers Obviously, if you look for second-rate developers to be your testers, you will end up with second-rate testers or worse!
      - tablespoon 6y ago
        > Obviously, if you look for second-rate developers to be your testers, you will end up with second-rate testers or worse! Yeah, and to make my thoughts a little clearer: if your organization does that, it's no wonder some people come to the (wrong) conclusion that only devs should test their own code. To come to the right conclusion in such an org, you either need an outlier experience (e.g. a talented tester who chose the role against organizational incentives) or the ability to see past your own direct experiences (which is really hard).
- seer 6y ago
  ¿Porque no los dos? QA teams and devs write totally different tests. Devs go about it to help future refactoring, e.g. codify what the feature is supposed to do, so that if we change it in the future, we will not inadvertently break something. QAs write tests that look from the point of the user. QA tests are arguably more valuable, but writing testable software itself can only be done if the devs too are invested in the task. Maybe we can form DevQA teams? the way we did for DevOps and DevSecOps? But alas, I still think having everyone though out the whole stack invested in writing tests makes the the final product better.
- friendzis 6y ago
  > yet no one seems to agree with me that devs should not write their own tests. Change Request (or whatever you call it in your flavor of agile) is the source informal description of behavior, while Specification, Code and Tests are three distinct formal descriptions of the same. If any of those three are written down by the same person, assumptions and thinking errors translate to the other largely defeating the very purpose of writing down another formal description in the first place. Unit tests became mainstream with proliferation of weak-dynamic languages (JS, Python) and concepts like TDD largely became mainstream due to lack of formal interface specification capabilities in the languages themselves. Stricter languages (Java, C++) cover huge portions of the testing in the type system. Tests written by the developer are okay if they are used as a substitute for a formal type system. However, tests intended to catch logic errors will contain the very same logic errors found in the code and such tests will only give you false confidence. It does not mean that tests must be written by a dedicated QA, but rather that tests should be written by a different person for them to serve the purpose they are intended to serve.
- killtimeatwork 6y ago
  > Hiring more people who are dedicated to testing costs a lot of money though. I don't see things changing anytime soon. Doesnt Google have a Software Developer in Test role? I work in a bank and every project has these people - their only job is to write automated tests (usually in Python robot framework).
  - dekhn 6y ago
    Yes, I was hired at Google to do dedicated testing (Test Engineer was my title). Note that at Google, many test engineers were "supposed to" build test infra, not write tests or do testing. I was brought on to test some systems that were critically important, but built by a team that focused more on operations than coding. I set up a continuous build, fixed all the broken tests, and prevented team members from checking in broken code. I don't think developers can be entirely trusted to QA their code.
- b0afc375b5 6y ago
  I think it's more appropriate to say, the quality of (enterprise?) software should not rely SOLELY on tests written by a software developer. Is it not common knowledge already that code review is the single most valuable tool in getting rid of bugs? I remember reading it on Code Complete, maybe The Pragmatic Programmer. Can't remember. I think there's still value in writing your own tests, at least with regard to refactoring (and sleeping better at night after deploying to production).
  - BlargMcLarg 6y ago
    > Is it not common knowledge already that code review is the single most valuable tool in getting rid of bugs? No, it isn't. Because there isn't that much evidence it is. It is a good tool, but "most valuable tool"? That's a very bold claim. I would still place code analyzing tools, linters and fast feedback cycles (= fast compilation and ability to check changes) over code reviews, if only for the fact many code reviewers tend to divert into "I like this way better" arguments.
    - b0afc375b5 6y ago
      Yeah definitely poor choice of words there. I misremembered the book. Anyway here's a cherry picked excerpt from Code Complete, 2nd edition: > Glenford Myers points out that human processes (inspections and walk-throughs, for instance) tend to be better than computer-based testing at finding certain kinds of errors and that the opposite is true for other kinds of errors (1979). > This result was confirmed in a later study, which found that code reading detected more interface defects and functional testing detected more control defects (Basili, Selby, and Hutchens 1986). Test guru Boris Beizer reports that informal test approaches typically achieve only 50–60 percent test coverage unless you're using a coverage analyzer (Johnson 1994). So I would conclude that the best approach is to use both code reviews/inspections and automated tests/linters/analyzers.
      - BlargMcLarg 6y ago
        I'm not debating automatic vs human approaches as much as how code reviews fit and why blind focus on them misses the forest for the trees in regards to other human approaches. When it takes 20 minutes from starting compilation to getting to the point of change, the Dev/QA/Review trifecta looks a lot different than when this takes 3 minutes. (Ideally,) developers are a lot less likely to make errors out of fatigue. They are more likely to pay attention to detail (when it takes 20 minutes, it's easy to go "whatever QA/reviewer will verify"). Reviewers start checking out the code and testing the change manually, rather than just looking at the code and pawning it off to QA. QA gets more time on their hands to proactively eliminate issues by being part of the design, instead of being the guys pointing out mistakes after the fact. Having shorter compilation cycles paves the way to increase the quality of all three segments of this cycle, yet it is often overlooked the most despite seeming so obvious. The part about QA is perhaps the most infuriating. Most business logic is so simple it can be presented as test/result matrices, which generally translate cleanly into requirements and automated tests, are easy to reason with, implicitly have three parties with different perspectives, and should be blind to technical details giving devs more freedom. Yet instead, this piece gets relegated to "software requirement engineers" writing walls of text, who tend to forget even the simplest cases (null, empty list/string, etc.), and any automated tests are then designed by the same developers making the implementation and only caught after the fact by manual QA a few months later.
- rkangel 6y ago
  > We also seem to be obsessed with testing that is half-way between unit level testing and system level testing I think that's where most testing should live. Testing single modules means you don't test how they plug together and makes it hard to refactor responsibilities between modules, such that it often doesn't happen. Full system level testing is painful to write as you often have to mock out the world, and painful to run as it it usually slow. The middle ground where you test a block of modules together through the top interface is a great middle ground. It does not replace the others- you need some full system level testing to check that your system integrates properly, and most modules will need some unit testing (and some will need a lot), but it's the best "bang for your buck" in terms of invested effort.
- ganafagol 6y ago
  I'll give an example for why the original author of the code should write the test. Say you write a sort and for performance reasons you use a different algorithm beyond collection size 100. This is prone to bugs around collection size 100. Your test needs to shed light on this dark corner. A QA generalist can't know that. They write some generic test cases. Size 0, 1 and 2 since they are inherently dark corners. Then some medium size and some large collections. They'll likely miss the 100. And the bug that's there for size 99. You need both and ideally the original author writes the test and somebody else reviews the test thoroughly. That reviewer should have an intimate understanding of the spec you are implementing. The more formal that spec, the less need for that external reviewer since the less ambiguity there is leaving room for missunderstanding. In a perfect world, the spec is completely formalized and instead of a test that shoots in the dark, you'd write a formal machine-verified correctness proof, automatically covering all cases. Then you would not need that external reviewer and could just have the programmer do it by themselves.
rotbart 6y ago
That's where red-green TDD and pairing comes in. Unit tests != class level tests.
ithkuil 6y ago
There are tests and there are tests. Software engineers benefit from having tests that encode invariants they cannot otherwise encode in the type system, so that they have a machine assisted refactoring aid. It's just an extension of writing the code; if you think you know what you want your code to do, you just add more of that But it's very hard for software engineers to think about what happens outside of that realm, whether because the software encounters edge cases naturally or due to "creative" ways the end users find to (ab)use the software. Some of those cases can be explored with a fuzzer, some of them are best found with actual human QA testers. Obligatory: https://youtu.be/baY3SaIhfl0 https://youtu.be/baY3SaIhfl0 As always, things stop working when you just go through the motions and become cargo cults.
ozim 6y ago
That is my main beef with developer testing "braindead unit tests" I see loads of tests if `int a = b;` really happened. But everyone feels good because they are doing unit testing as "gods of agile and TDD" lined out... Try to say that such test is useless and even harmful to long term value. You get "we all are people and make mistakes, we want to prevent that", if someone is trying to prevent wrong assignment in strongly typed language well I will call them "braindead".
Cthulhu_ 6y ago
Yeah I don't believe tests will replace testers; tests are a safety net for regressions, a safety net to allow for refactoring without changing the in- and outputs, but they are NOT a replacement for testers who look at and work with things from a user's or a business' point of view. I mean by all means, if a tester finds an issue, write a test to replicate it before fixing it so that it can't recur. But also, tests are code and code rots. Changing that implementation entirely voids the test. And at some point your test codebase size far exceeds that of your implementation.
dx034 6y ago
And no unit tests can replace an actual person testing the software. Unit tests can catch bugs earlier but getting rid of a QA department (or even one person) manually testing features is worrisome. The developer will always be blind to their own work. But it's not just software development, online news articles also tend to get worse with more journalists publishing without a proof-reader involved.
sidlls 6y ago
Absolutely my experience as well (not medical, but aviation/simulation). What's worse is when the same developers and managers pat themselves on the back for writing new unit tests to cover the discovered bugs.

by diydsp 6y ago

bigtimegames 6y ago

Hmm i think the pace of changes has made software break more, but unit tests and integration tests are an improvement. But I agree writing unit tests takes a hella lotta time, not sure if a sr eng should be doing that 1/2 the time

01100011 6y ago

I'm at a FANG with some of the smartest people I have ever encountered in my life, yet no one seems to agree with me that devs should not write their own tests. There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. We also seem to be obsessed with testing that is half-way between unit level testing and system level testing. We test APIs and features more or less in isolation. Tests should be written by antagonistic developers who are not at all familiar with the code. They should stress the capabilities promised in the interface documentation. Hiring more people who are dedicated to testing costs a lot of money though. I don't see things changing anytime soon.

robbyt 6y ago
The problem is, when devs don't write their own tests, the code they write tends to be untestable.
- onion2k 6y ago
  I suspect you think the developer writes the code and then the QA engineer writes tests for that code. That's the wrong way round. The tests are written first, and they all fail, then the dev has to write code to make them pass. By definition it's literally impossible to write untestable code that way because then the tests won't pass. It's the principle of TDD but separated in to different roles. It works well and produces good results but you need exceptionally good planning upfront to actually do it.
  - eliasbagley 6y ago
    Who designs the interfaces that the tests call?
    - onion2k 6y ago
      That's the planning part. Usually it's done as a collaboration between the systems architect and the QA team. The process is essentially "design the system architecture -> document the APIs -> write the tests -> implement the code -> QA". It takes a level of rigour and planning that most teams don't want to do, and it relegates developers to a pretty minor role in the whole development process. They end up just filling in the blank bits in someone else's design. Most changes in team structure over the past couple of decades have been about developers taking on more of the responsibilities rather than less. It's actually one of the aspects of TDD that makes it so robust. If you properly design your code and really think about its APIs you get a better app at the end. It does take longer to get something a user can see though, but often the process gets to a finished iteration of an app quicker because there's fewer cycles of debugging and fixing needed.
      - realusername 6y ago
        > The process is essentially "design the system architecture -> document the APIs -> write the tests -> implement the code -> QA". It takes a level of rigour and planning that most teams don't want to do That's a very charitable way to say it, on my case I would call this kind of process very convoluted and similar to early 2000s style development. I obviously would not want to work in such a bureaucratic environment where every small tech decision has to go through multiple boards.
        onion2k 6y ago
        I obviously would not want to work in such a bureaucratic environment where every small tech decision has to go through multiple boards. Everyone who works on the app is making those decisions no matter what you do. The difference is that in the "old style" the decisions were made before the code is written, and in the agile "new style" the decisions are made after the code is written. Often that's fine, because developers are generally good at their work and they make decent decisions, but sometimes they get it wrong and that's when code design issues arise. It's also what leads to automated tests failing to test for a lot of cases that a good QA engineer would have written tests for. Those things have an impact on the user. A huge amount of the code in apps we use every day was never designed. No one thought it through. No one considered the edge cases. Features are thrown together in a week and 'sort of' work. Every time you see a shitty broken website, or a bug in production, or some crappy slow thing that should be fast the reason behind the problem is that the developers who made it didn't take the time (or even have the time in really bad companies) to think about what they were building. A lot of developers like that working environment because they can hack on things and move on to the next challenge quickly. That's fun. I argue it's also bad for users, and I care more about users than I care about developers (and I say that as a developer whose been making web stuff for almost 25 years.) I'm not arguing that we go back to Prince2 and waterfall. There's a limit to my tolerance of bureaucracy too. I'm saying that things have gone a bit too far the other way, and many developers need to spend more time planning what code is written before leaping in and coding up the first solution they think will pass the acceptance criteria someone from product wrote.
deleted 6y ago
[deleted]
stickfigure 6y ago
That sounds like an incredibly slow feedback cycle. How would that work? Write some code, check it in, wait for someone else to write tests... and come back in hours/days/weeks to fix any issues? Getting constantly dragged back to fix last week's bugs sounds like a chapter from Dante's Inferno. I'd rather discover them while the code is fresh in my mind. Without tests, how do I know when I'm done?
- quiescant_dodo 6y ago
  In the past, I had pseudo-pair programmed in order to test. We'd agree on the external-facing API and pair-program a slow in-memory implementation. Once that was completed, we would split off and one of us would write test cases against the API (fixing up the in-memory implementation as needed) while the other would work on a proper database-backed implementation. I haven't done this in probably 7 years, though. Mostly because I haven't cultivated this kind of working relationship with my coworkers.
- fossuser 6y ago
  I've seen both types of org layouts where I've worked (we moved from one to the other). Initially separate QA teams that would write tests, a subset of which might do manual testing in the UI. The feedback loop for this was brutal as you suggest and caused a lot of problems, with delayed breaks that become hard to fix and debug. It's also hard for QA to often know what the code is doing or interact closely with the dev teams. You can try to get them to be as close as possible, but it doesn't work super well. Now devs are responsible for writing tests and teams think about potential failures when designing things/discussing the plan with their team. They were supposed to consider these things before too, but the incentives weren't aligned - now they are. We have infrastructure teams to make automated testing as easy as possible and spin up a test environment where you can see your change, etc. I think this model is better, testing well is hard and reasoning about failures and how things can go wrong is an important skill to develop. Offloading this to a different team feels like a variant of the dev that 'just writes features' and doesn't consider actual deployment. This is often made doubly worse by companies that have QA teams often treating them as second class citizens both in status and in pay. I think it's better to have it be part of the job of the person writing the feature also write the tests, it doesn't help that writing tests is something few people know how to do coming out of school.
cbryan 6y ago
You're getting heavily downvoted and I'd love to know why. I've seen similar attitudes toward testing among the groups I've worked with, nobody really wants to do it. Are folks downvoting this because they don't like testing? Is this idea that prevalent?
irjustin 6y ago
Count me in as one of those who disagree with you. I 100% side with the grandparent comment. When you've worked with a great QA, it's night and day vs a dev writing their own tests, but a dev not writing its own tests is a nightmare. Ideally there's a place for both and they work very closely with one another to make the software stable at all levels. It's arguable the SRE is the new QA engineer.
void_mint 6y ago
> yet no one seems to agree with me that devs should not write their own tests. I'll die on this hill. The developer that wrote the code by definition cannot write an appropriate test suite for it. It is entirely possible/probable that details missed in implementation will be missed entirely in test, as the developer missed them. < controversial take> Manual QA promotes bad habits and is not a great thing to introduce to an org. SDETs/QA Engineers/Developers whose role is explicitly to create tests for an org are worth their salary 10x. </ controversial take >
- kqr 6y ago
  > are worth their salary 10x. Are you saying it's a bad role to get into because it's underpaid? ;)
  - EdwardDiego 6y ago
    Yes, sadly for some reason in most orgs, QA/QE whatever you're calling it, pays lower than dev work. Even though, IMO, they're worth far more.
    - hvidgaard 6y ago
      They are worth more in the same sense a salesman is worth more than the dev. Without the dev neither have anything to do, yet they all provide equally valuable work. Perhaps it's not a matter of value, but supply and demand.
    - NicoJuicy 6y ago
      I don't agree. You can't finish a project without developers and you can finish it without testers. I've seen a tester changing requirements on the go, delaying features for things that aren't even the case. Those change requests then cause bugs, because... Dev. The most important thing for a dev is domain knowledge and letting them care about the product. Usually, when the original Dev leaves ( monolith), it's enough to call the project abandoned, but maintained. Leaving the devs wanting to spend a minimal amount of work with it. And that's going to trigger sloppyness.
nixpulvis 6y ago
I feel like this isn't that hard. 1. Write (some) tests as you develop to help you Writing down your program's test cases should not add much time (assuming you have test infrastructure setup like you should), since you are already having to consider these things to simply figure out what the right logic is. With a little practice, you find a good rhythm for how far to take it for yourself. In my experience, I find this actually leads to faster and more correct solutions. 2. Collect tests while in review to cover the requirements Once you've written it though, it's always important to try and get others to look over your code and try and find flaws. Ideally, they can even write tests for you to add to the feature. This is just a standard part of any good peer review. 3. REGRESSION TEST Finally, though I'm not sure it matters very much who writes it, regression tests are something I never see stressed enough. I feel like if you are serious about testing, every bug should be required to add at least one test to the suite before it can be closed. Just my 2-cents.
kqr 6y ago
> There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. This is a very common -- but inevitable -- misunderstanding. The real reason the same person should write tests and code is if you find a person who is competent enough in both technical domain and problem domain to write the tests -- you definitely want that person to write the code as well. See how it's not "developers should write tests"? It's "domain experts should write the tests... and domain experts should write the implementation." The two tasks happen to need overlapping skills.
ohazi 6y ago
Black box test - Testing a unit without any information about its internals. You provide inputs, you record outputs, and you have a spec. By definition, the author of a unit cannot write a black box test for it. Any test they write is going to be a white box (clear box?) test, because they know how the unit works. If you interpreted the spec incorrectly when writing the unit, you're going to interpret it incorrectly in exactly the same way when writing the test. White box testing has its place, but you need a second person if you want to do black box testing.
tablespoon 6y ago
> I'm at a FANG with some of the smartest people I have ever encountered in my life, yet no one seems to agree with me that devs should not write their own tests. > There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. We also seem to be obsessed with testing that is half-way between unit level testing and system level testing. We test APIs and features more or less in isolation. You're both right. Devs should write tests for their own code, but they shouldn't be the only ones writing them. One of the problems (at least at my employer) is QA is considered a place for second-rate developers. Some of them are so bad that they rely on the devs to basically write their tests for them, by requiring extremely detailed and specific acceptance criteria. Only once have I worked with a tester that was engaged enough to really understand the requirements on their own and call out weird behavior or corner cases that weren't spelled out in advance. It was great.
- xupybd 6y ago
  I worked with a tester turned dev. He was amazing, his attention to detail and ability to grok specifications was second to none
- sumedh 6y ago
  > Some of them are so bad So why is your company hiring bad QAs?
  - cassianoleal 6y ago
    The answer seems to be in the same message: > QA is considered a place for second-rate developers Obviously, if you look for second-rate developers to be your testers, you will end up with second-rate testers or worse!
    - tablespoon 6y ago
      > Obviously, if you look for second-rate developers to be your testers, you will end up with second-rate testers or worse! Yeah, and to make my thoughts a little clearer: if your organization does that, it's no wonder some people come to the (wrong) conclusion that only devs should test their own code. To come to the right conclusion in such an org, you either need an outlier experience (e.g. a talented tester who chose the role against organizational incentives) or the ability to see past your own direct experiences (which is really hard).
seer 6y ago
¿Porque no los dos? QA teams and devs write totally different tests. Devs go about it to help future refactoring, e.g. codify what the feature is supposed to do, so that if we change it in the future, we will not inadvertently break something. QAs write tests that look from the point of the user. QA tests are arguably more valuable, but writing testable software itself can only be done if the devs too are invested in the task. Maybe we can form DevQA teams? the way we did for DevOps and DevSecOps? But alas, I still think having everyone though out the whole stack invested in writing tests makes the the final product better.
friendzis 6y ago
> yet no one seems to agree with me that devs should not write their own tests. Change Request (or whatever you call it in your flavor of agile) is the source informal description of behavior, while Specification, Code and Tests are three distinct formal descriptions of the same. If any of those three are written down by the same person, assumptions and thinking errors translate to the other largely defeating the very purpose of writing down another formal description in the first place. Unit tests became mainstream with proliferation of weak-dynamic languages (JS, Python) and concepts like TDD largely became mainstream due to lack of formal interface specification capabilities in the languages themselves. Stricter languages (Java, C++) cover huge portions of the testing in the type system. Tests written by the developer are okay if they are used as a substitute for a formal type system. However, tests intended to catch logic errors will contain the very same logic errors found in the code and such tests will only give you false confidence. It does not mean that tests must be written by a dedicated QA, but rather that tests should be written by a different person for them to serve the purpose they are intended to serve.
killtimeatwork 6y ago
> Hiring more people who are dedicated to testing costs a lot of money though. I don't see things changing anytime soon. Doesnt Google have a Software Developer in Test role? I work in a bank and every project has these people - their only job is to write automated tests (usually in Python robot framework).
- dekhn 6y ago
  Yes, I was hired at Google to do dedicated testing (Test Engineer was my title). Note that at Google, many test engineers were "supposed to" build test infra, not write tests or do testing. I was brought on to test some systems that were critically important, but built by a team that focused more on operations than coding. I set up a continuous build, fixed all the broken tests, and prevented team members from checking in broken code. I don't think developers can be entirely trusted to QA their code.
b0afc375b5 6y ago
I think it's more appropriate to say, the quality of (enterprise?) software should not rely SOLELY on tests written by a software developer. Is it not common knowledge already that code review is the single most valuable tool in getting rid of bugs? I remember reading it on Code Complete, maybe The Pragmatic Programmer. Can't remember. I think there's still value in writing your own tests, at least with regard to refactoring (and sleeping better at night after deploying to production).
- BlargMcLarg 6y ago
  > Is it not common knowledge already that code review is the single most valuable tool in getting rid of bugs? No, it isn't. Because there isn't that much evidence it is. It is a good tool, but "most valuable tool"? That's a very bold claim. I would still place code analyzing tools, linters and fast feedback cycles (= fast compilation and ability to check changes) over code reviews, if only for the fact many code reviewers tend to divert into "I like this way better" arguments.
  - b0afc375b5 6y ago
    Yeah definitely poor choice of words there. I misremembered the book. Anyway here's a cherry picked excerpt from Code Complete, 2nd edition: > Glenford Myers points out that human processes (inspections and walk-throughs, for instance) tend to be better than computer-based testing at finding certain kinds of errors and that the opposite is true for other kinds of errors (1979). > This result was confirmed in a later study, which found that code reading detected more interface defects and functional testing detected more control defects (Basili, Selby, and Hutchens 1986). Test guru Boris Beizer reports that informal test approaches typically achieve only 50–60 percent test coverage unless you're using a coverage analyzer (Johnson 1994). So I would conclude that the best approach is to use both code reviews/inspections and automated tests/linters/analyzers.
    - BlargMcLarg 6y ago
      I'm not debating automatic vs human approaches as much as how code reviews fit and why blind focus on them misses the forest for the trees in regards to other human approaches. When it takes 20 minutes from starting compilation to getting to the point of change, the Dev/QA/Review trifecta looks a lot different than when this takes 3 minutes. (Ideally,) developers are a lot less likely to make errors out of fatigue. They are more likely to pay attention to detail (when it takes 20 minutes, it's easy to go "whatever QA/reviewer will verify"). Reviewers start checking out the code and testing the change manually, rather than just looking at the code and pawning it off to QA. QA gets more time on their hands to proactively eliminate issues by being part of the design, instead of being the guys pointing out mistakes after the fact. Having shorter compilation cycles paves the way to increase the quality of all three segments of this cycle, yet it is often overlooked the most despite seeming so obvious. The part about QA is perhaps the most infuriating. Most business logic is so simple it can be presented as test/result matrices, which generally translate cleanly into requirements and automated tests, are easy to reason with, implicitly have three parties with different perspectives, and should be blind to technical details giving devs more freedom. Yet instead, this piece gets relegated to "software requirement engineers" writing walls of text, who tend to forget even the simplest cases (null, empty list/string, etc.), and any automated tests are then designed by the same developers making the implementation and only caught after the fact by manual QA a few months later.
rkangel 6y ago
> We also seem to be obsessed with testing that is half-way between unit level testing and system level testing I think that's where most testing should live. Testing single modules means you don't test how they plug together and makes it hard to refactor responsibilities between modules, such that it often doesn't happen. Full system level testing is painful to write as you often have to mock out the world, and painful to run as it it usually slow. The middle ground where you test a block of modules together through the top interface is a great middle ground. It does not replace the others- you need some full system level testing to check that your system integrates properly, and most modules will need some unit testing (and some will need a lot), but it's the best "bang for your buck" in terms of invested effort.
ganafagol 6y ago
I'll give an example for why the original author of the code should write the test. Say you write a sort and for performance reasons you use a different algorithm beyond collection size 100. This is prone to bugs around collection size 100. Your test needs to shed light on this dark corner. A QA generalist can't know that. They write some generic test cases. Size 0, 1 and 2 since they are inherently dark corners. Then some medium size and some large collections. They'll likely miss the 100. And the bug that's there for size 99. You need both and ideally the original author writes the test and somebody else reviews the test thoroughly. That reviewer should have an intimate understanding of the spec you are implementing. The more formal that spec, the less need for that external reviewer since the less ambiguity there is leaving room for missunderstanding. In a perfect world, the spec is completely formalized and instead of a test that shoots in the dark, you'd write a formal machine-verified correctness proof, automatically covering all cases. Then you would not need that external reviewer and could just have the programmer do it by themselves.

rotbart 6y ago

That's where red-green TDD and pairing comes in. Unit tests != class level tests.

ithkuil 6y ago

There are tests and there are tests. Software engineers benefit from having tests that encode invariants they cannot otherwise encode in the type system, so that they have a machine assisted refactoring aid. It's just an extension of writing the code; if you think you know what you want your code to do, you just add more of that But it's very hard for software engineers to think about what happens outside of that realm, whether because the software encounters edge cases naturally or due to "creative" ways the end users find to (ab)use the software. Some of those cases can be explored with a fuzzer, some of them are best found with actual human QA testers. Obligatory: https://youtu.be/baY3SaIhfl0 https://youtu.be/baY3SaIhfl0 As always, things stop working when you just go through the motions and become cargo cults.

ozim 6y ago

That is my main beef with developer testing "braindead unit tests" I see loads of tests if `int a = b;` really happened. But everyone feels good because they are doing unit testing as "gods of agile and TDD" lined out... Try to say that such test is useless and even harmful to long term value. You get "we all are people and make mistakes, we want to prevent that", if someone is trying to prevent wrong assignment in strongly typed language well I will call them "braindead".

Cthulhu_ 6y ago

Yeah I don't believe tests will replace testers; tests are a safety net for regressions, a safety net to allow for refactoring without changing the in- and outputs, but they are NOT a replacement for testers who look at and work with things from a user's or a business' point of view. I mean by all means, if a tester finds an issue, write a test to replicate it before fixing it so that it can't recur. But also, tests are code and code rots. Changing that implementation entirely voids the test. And at some point your test codebase size far exceeds that of your implementation.

dx034 6y ago

And no unit tests can replace an actual person testing the software. Unit tests can catch bugs earlier but getting rid of a QA department (or even one person) manually testing features is worrisome. The developer will always be blind to their own work. But it's not just software development, online news articles also tend to get worse with more journalists publishing without a proof-reader involved.

sidlls 6y ago

Absolutely my experience as well (not medical, but aviation/simulation). What's worse is when the same developers and managers pat themselves on the back for writing new unit tests to cover the discovered bugs.