5 ms·

Jetbrains have a really solid testing team. Lots of places have a two tier system where the real developers write the code and those who don't make the cut tes

Jetbrains have a really solid testing team. Lots of places have a two tier system where the real developers write the code and those who don't make the cut test the code, with pay delta and an aspiration of being promoted out of testing. Other places have a mandatory stint in testing for new developments as a way to get some headcount on the task. Jetbrains don't do that. Or at least they didn't sometime before covid when I met a bunch of their test devs at a conference. The developers mostly doing testing were equals to those mostly doing product work. Possibly with a more extreme bias towards case analysis. I don't think it's a coincidence that jetbrains treat their test team as peers to the others and that their software seems to mostly not fall over in the field.

gleenn 2y ago
I feel like developers who don't write tests while writing their production code aren't doing a real job in 2024. Sure you might have a QA team or something, but that should be a vast minority of the tests being written. Test driven development and Agile and all that had some people upset and whiny 20 years ago, but if I see code without tests, I assume it's broken. The only way anyone knows anything works both now AND when the next 10, 100, 1000 commits roll in, your manually-tested edge cases have long been forgotten and I doubt you even got it right at all. Let's keep normalizing writing tests and thinking, up front, how to ensure our code works. That's the best time to think about all those pesky edge cases, and it's the only practical way to keep code working. Don't hoist this on to some poor chum because you are lazy or don't know how. (Spelling)
- stouset 2y ago
  Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems. Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly. These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point. The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.
  - kortilla 2y ago
    It sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater
    - SatvikBeri 2y ago
      It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
      - tetha 2y ago
        This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right. Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such. That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.
    - stouset 2y ago
      Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable. I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.
- BobbyTables2 2y ago
  I fully agree. For most of my career, I hated the idea of writing (unit) tests. They were almost always a waste of time. At one point, I ended up on a team that had virtually no QA and only did automated end-to-end tests on a complex product. After working through a lot of tech debt on the testing (took a few months), it was solid and caught many defects introduced by junior/new developers (easily missed in code review). I now have the fortune to be in a different group with a similar mindset. I’m motivated to write tests for my own code just to prove to myself that it works properly in odd error cases too. My team prefers testing instead of “run fast and break things.” (And with multiple decades of experience, I humbly can state I cause significantly fewer bugs than in the past.) This also makes me write code in an easily testable manner. Trying to test already written code is usually a lengthy and painful exercise in frustration. It’s also too easy to write error handlers that are NEVER actually exercised once, even manually — and surprise, they may not even work! People who never write tests do this quite often.
yreg 2y ago
A class system with developers writing production code looking down on developers writing tests doesn't seem healthy to me.
- BobbyTables2 2y ago
  It never is!
kabdib 2y ago
I've had "quality-team-as-peer" situations only a few times in my career, and it's been great every time. Getting meaningful feedback on specs and early builds from people who knew what they were doing (and were happy doing it) was like the first time I had a real editor (the human kind) marking up my writing, just a wonderful feeling that someone had your back and that you all had the same goal: Happy customers.
BobbyTables2 2y ago
It is indeed shocking what happens when a company doesn’t treat its (offshore) QA team as third class peons relative to the developers. I’ve seen both and have actually seen much higher product quality when the QA team is smaller (or even almost non existent). Turns out white-box testing usually fails to catch things only SMEs would know.
- HideousKojima 2y ago
  I frankly think my company's offshore devs and QA (largely India-based) are a net negative. The existence of a decent sized team to do QA manually has made adoption of automated testing (and writing code that it easily testable) almost impossible because no one wants to change.
- spc476 2y ago
  At my previous job, prior to new management (we got bought out), my team worked closely with QA during the development process---they would get daily, often times, more than daily, builds of our project and so by the time it got to the staging environment, it pretty much worked [1]. New management comes in, forces a hard wall between us and QA and pretty much, we went from "favored vendor" of our customer [2], to "least favorite vendor" in less than a year. One deployment took four attempts to get right. Bugs found left and right. All because the new management didn't want to "pollute" QA's testing mindset with implementation details. Sigh. [1] During the 11 years I was there, there were only two deployments that were reversed, prior to new management. And the bugs were found during the immediate testing after deployment and we were able to reverse quickly. [2] The Oligarchic Cell Phone Company.
rtpg 2y ago
I'm shocked that there are places in 2024 that have a separate set of devs writing test code. What decade is this?
- JonChesterfield 2y ago
  The (unit) tests developers write themselves check the developer did what they intended to, and then that things don't change. Whatever case analysis the developer missed in the implementation tends to be missing from their tests as well, as they didn't think of it. Testing your own work also does nothing to protect against correctly implementing the wrong thing.
  - rtpg 2y ago
    This is a fair point. I imagine in a stricter environment you could (for example) have tests before implementations even to avoid huge feedback loops
- adrianN 2y ago
  Having a separate test team is pretty much mandatory in safety critical development. You really want people who haven’t written the code go through the specification independently and write test cases for every requirement.
- deredede 2y ago
  Having one set of developers write the code and another set of developers write tests from the same specification without reading the code is probably the most effective ways to use tests for high software quality and is typically required by certification bodies e.g. for avionics.
  - rtpg 2y ago
    That’s a good point for things with certain kinds of specs, but it seems a hell of a lot more straightforward for the person who is writing an implementation to also be in a testing mindset (if only to be writing testable code) Perhaps people in these environments are made of different stuff but touching both sides feels almost essential to me.
high_na_euv 2y ago
Ive worked in hw industry and arch team created specs, dev was developing product and its own tests And val team was creating fake product which was used to help to write tests without product (so tests were available faster, so deploy was faster), or just tests - all of this basing on spec It worked well when it comes to bug/issues/inconsistencies catching

gleenn 2y ago

I feel like developers who don't write tests while writing their production code aren't doing a real job in 2024. Sure you might have a QA team or something, but that should be a vast minority of the tests being written. Test driven development and Agile and all that had some people upset and whiny 20 years ago, but if I see code without tests, I assume it's broken. The only way anyone knows anything works both now AND when the next 10, 100, 1000 commits roll in, your manually-tested edge cases have long been forgotten and I doubt you even got it right at all. Let's keep normalizing writing tests and thinking, up front, how to ensure our code works. That's the best time to think about all those pesky edge cases, and it's the only practical way to keep code working. Don't hoist this on to some poor chum because you are lazy or don't know how. (Spelling)

stouset 2y ago
Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems. Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly. These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point. The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.
- kortilla 2y ago
  It sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater
  - SatvikBeri 2y ago
    It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
    - tetha 2y ago
      This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right. Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such. That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.
  - stouset 2y ago
    Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable. I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.
BobbyTables2 2y ago
I fully agree. For most of my career, I hated the idea of writing (unit) tests. They were almost always a waste of time. At one point, I ended up on a team that had virtually no QA and only did automated end-to-end tests on a complex product. After working through a lot of tech debt on the testing (took a few months), it was solid and caught many defects introduced by junior/new developers (easily missed in code review). I now have the fortune to be in a different group with a similar mindset. I’m motivated to write tests for my own code just to prove to myself that it works properly in odd error cases too. My team prefers testing instead of “run fast and break things.” (And with multiple decades of experience, I humbly can state I cause significantly fewer bugs than in the past.) This also makes me write code in an easily testable manner. Trying to test already written code is usually a lengthy and painful exercise in frustration. It’s also too easy to write error handlers that are NEVER actually exercised once, even manually — and surprise, they may not even work! People who never write tests do this quite often.

yreg 2y ago

A class system with developers writing production code looking down on developers writing tests doesn't seem healthy to me.

BobbyTables2 2y ago
It never is!

kabdib 2y ago

I've had "quality-team-as-peer" situations only a few times in my career, and it's been great every time. Getting meaningful feedback on specs and early builds from people who knew what they were doing (and were happy doing it) was like the first time I had a real editor (the human kind) marking up my writing, just a wonderful feeling that someone had your back and that you all had the same goal: Happy customers.

BobbyTables2 2y ago

It is indeed shocking what happens when a company doesn’t treat its (offshore) QA team as third class peons relative to the developers. I’ve seen both and have actually seen much higher product quality when the QA team is smaller (or even almost non existent). Turns out white-box testing usually fails to catch things only SMEs would know.

HideousKojima 2y ago
I frankly think my company's offshore devs and QA (largely India-based) are a net negative. The existence of a decent sized team to do QA manually has made adoption of automated testing (and writing code that it easily testable) almost impossible because no one wants to change.
spc476 2y ago
At my previous job, prior to new management (we got bought out), my team worked closely with QA during the development process---they would get daily, often times, more than daily, builds of our project and so by the time it got to the staging environment, it pretty much worked [1]. New management comes in, forces a hard wall between us and QA and pretty much, we went from "favored vendor" of our customer [2], to "least favorite vendor" in less than a year. One deployment took four attempts to get right. Bugs found left and right. All because the new management didn't want to "pollute" QA's testing mindset with implementation details. Sigh. [1] During the 11 years I was there, there were only two deployments that were reversed, prior to new management. And the bugs were found during the immediate testing after deployment and we were able to reverse quickly. [2] The Oligarchic Cell Phone Company.

rtpg 2y ago

I'm shocked that there are places in 2024 that have a separate set of devs writing test code. What decade is this?

JonChesterfield 2y ago
The (unit) tests developers write themselves check the developer did what they intended to, and then that things don't change. Whatever case analysis the developer missed in the implementation tends to be missing from their tests as well, as they didn't think of it. Testing your own work also does nothing to protect against correctly implementing the wrong thing.
- rtpg 2y ago
  This is a fair point. I imagine in a stricter environment you could (for example) have tests before implementations even to avoid huge feedback loops
adrianN 2y ago
Having a separate test team is pretty much mandatory in safety critical development. You really want people who haven’t written the code go through the specification independently and write test cases for every requirement.
deredede 2y ago
Having one set of developers write the code and another set of developers write tests from the same specification without reading the code is probably the most effective ways to use tests for high software quality and is typically required by certification bodies e.g. for avionics.
- rtpg 2y ago
  That’s a good point for things with certain kinds of specs, but it seems a hell of a lot more straightforward for the person who is writing an implementation to also be in a testing mindset (if only to be writing testable code) Perhaps people in these environments are made of different stuff but touching both sides feels almost essential to me.

high_na_euv 2y ago

Ive worked in hw industry and arch team created specs, dev was developing product and its own tests And val team was creating fake product which was used to help to write tests without product (so tests were available faster, so deploy was faster), or just tests - all of this basing on spec It worked well when it comes to bug/issues/inconsistencies catching