5 ms·

Totally agree with all your points except >there's not a lot of empirical evidence to support claims that automated tests will improve developer productivity o

Totally agree with all your points except >there's not a lot of empirical evidence to support claims that automated tests will improve developer productivity or code quality. If you'd have said "there aren't any published papers _proving_ this" I would've agreed simply because I do not know if such studies exist or not (they might); but I'd argue that automated tests obviously improve code quality and productivity. And also ramp up time, which is important in fast growing teams (my case).

kqr 5y ago
The question that matters is not whether automated tests improve code quality and productivity. The question that needs to be answered is "if I took the time I previously spent on automated testing, and instead spent it on something else like code review or design review, would that result in greater code quality and productivity than automated testing did?" And phrased that way, you see it's not a dichotomy. It depends on what you would spend the time on instead of automated testing. It depends on whether you already had peer review in your process. It depends on what sort of automated tests you spend time on. It's very hard to say something general without those details. ---- Note that I'm not taking a stance against automated testing. I prefer working in a test-first style myself. I'm just saying it's not that simple.
hoten 5y ago
It's so obviously self evident that automated tests improve productivity, especially for developers new to a codebase. A developer makes a non trivial change. How do they know nothing broke? If there are no tests, they must spend an inordinate amount of time learning the entire system / product to gain any confidence they didn't break something, only to likely miss something that a senior team member will point out. The back/forth will take multiple developers time. At worst, it gets punted all the way to a QA team. Could be days. Or, with tests, that happens far less often and the feedback loop can be more like 10 minutes. Of course, tests don't solve onboarding, but they provide nice guard rails.
- dboreham 5y ago
  This assumes the tests are mostly valid. My experience in recent years has been that they won't be.
  - hoten 5y ago
    Of course I'm assuming useful tests, why wouldn't I? Note I don't claim tests completely solve all productivity or onboarding problems. It's just odd to claim it's somehow ~unclear that automated tests aren't good for productivity. Forget the extremes (over testing, too many unit tests vs integration, potential rigidness): tests done right are productivity multipliers. And that's not a "No True Scotsman" retort because the original claim had no qualifiers, and was suggesting that any version of automated tests are orthogonal to developer productivity. All you must do to refute that is think of any codebase you were able to make progress on faster because a test told you what was wrong faster than it'd take you to blame source code and send an email to a developer you maybe have never met yet.
    - toast0 5y ago
      > Of course I'm assuming useful tests, why wouldn't I? Because useful tests seem pretty hard to have in practice. Especially when starting testing for a project without tests and a team without testing experience. Early testing reduces productivity, and the proposed gains may never develop.
    - gregjor 5y ago
      > It's just odd to claim it's somehow ~unclear that automated tests aren't good for productivity. No it's not, because it is unclear. Where's the actual evidence? How do you even define and measure "productivity" in software development? My customers define productivity as "Delivered according to spec on time and within budget." That's the only measurement they care about. As hard as it is for programmers to understand beautiful "clean" code and automated tests are generally not stakeholder priorities. Yes, bad code and no testing has a cost that may have to get paid down the road. I think stakeholders understand that software will fall short of perfection. For a long time the standard practice was unit testing, integration testing, and (if you were lucky) rigorous QA. Plenty of usable software got written that way. There's nothing inherently wrong with TDD and automated testing, and when done right those techniques can reduce the need for more complex testing (integration and QA) and help the team understand what the code is supposed to do. That's not the only way to write good code that works, but it's one way. The problems of defining, measuring, and improving programmer productivity have been known and discussed since the 1960s. We have a set of fairly vague best practices (also known since the 60s/70s) but there's no silver bullet. The problems with developing non-trivial software in team environments mainly come from communication and team dynamics, not from lack of style guides or automated tests. Every little bit can help, of course, but let's not kid ourselves that TDD and automated testing are magic fairy dust.
jiggawatts 5y ago
I'm replying to this and also some of the sibling comments. No, automated development-time tests do not necessarily equate to higher quality, given that they: - Take time away from other activities - May provide a false positive, lulling developers into a false sense of confidence - Increase the time taken for certain CI/CD operations, sometimes by hours or days! - The presence of a large volume of "test" code means that certain refactoring operations take a lot longer, which is a real cost. People go into some work environments where tests DID help, and they generalise that to ALL work environments. Automated build-time tests are less important, and entire categories may be potentially unnecessary when: - The product is written in a strongly-typed language that catches most errors at compile time. - If high-level "linting" tools are used to further enhance the code quality before the executable ever runs. - If feedback from the production environment is more relevant. E.g.: for web applications with performance issues being monitored by an APM, but no critical code stability concerns at build time. Think typical web apps with no real consequences to a crash, not finance applications where a small error could mean Real Money. - Where time-to-market is more important than chasing 100% robustness at the expense of meeting release dates. You often see people from a PHP or Python background go on and on about the importance of tests, but those are dynamically typed languages with lots of unexpected pitfalls that only testing will uncover. Meanwhile entire teams do just fine without any automated tests when using languages like C#, Java, or Rust.
- cweagans 5y ago
  Types do a great job at assuring you’re getting generally the right kind of data in and out of different functions/methods, but they don’t assure that the program is actually doing the right thing. Strong typing just helps to assure that the program isn’t doing specific categories of the wrong thing (in the same way that Rust and Go make it difficult to write memory-unsafe code). I mean, Go and Rust both have testing capabilities built in. If it wasn’t necessary, why would they do that? Beyond the most trivial of projects, I can’t think of a single project that I’ve worked on (or even looked at on GitHub) that has weighed the costs and benefits of automated testing and decided it just wasn’t worth it.
  - jiggawatts 5y ago
itsdrewmiller 5y ago
Generally speaking "empirical evidence" means "published papers", with a little bit of flex for preprints or trustworthy-but-not-peer-reviewed experimental findings.
- gregjor 5y ago
  I was using it to mean actual data rather than anecdotes. The subject of programmer productivity is both much-studied and discussed and very hard to define, measure, and compare. The studies that do have some empirical evidence seem to show that team dynamics and individual programmer personalities have the most dramatic effects, compared to things like programming language, tests, style guides, etc. It's not that consistent styles and automated testing are bad ideas, it's that they don't clearly make a big dent in team productivity in cases where one or a few incompetent (or merely slow) programmers, or one jerk, can derail everyone's productivity. This I why I emphasized demonstrating a commitment to the team and showing some contributions before criticizing the environment and trying to propose changes. If you want the project to grind to a halt tell your new co-workers they are doing everything wrong. No automated test suite is going to fix that problem.
gregjor 5y ago
I think you are agreeing with me. There's anecdotal evidence like you offer. It's not necessarily obvious to everyone that automated tests improve code quality or it would be a settled issue (it's not). In my own career (40+ years) I have worked with automated testing and TDD, and without, and I can't say that TDD "obviously" improves code quality. I know other professional programmers who have that experience. Like everything else in software development it's probably situational and depends on the individuals and the team.

kqr 5y ago

The question that matters is not whether automated tests improve code quality and productivity. The question that needs to be answered is "if I took the time I previously spent on automated testing, and instead spent it on something else like code review or design review, would that result in greater code quality and productivity than automated testing did?" And phrased that way, you see it's not a dichotomy. It depends on what you would spend the time on instead of automated testing. It depends on whether you already had peer review in your process. It depends on what sort of automated tests you spend time on. It's very hard to say something general without those details. ---- Note that I'm not taking a stance against automated testing. I prefer working in a test-first style myself. I'm just saying it's not that simple.

hoten 5y ago

It's so obviously self evident that automated tests improve productivity, especially for developers new to a codebase. A developer makes a non trivial change. How do they know nothing broke? If there are no tests, they must spend an inordinate amount of time learning the entire system / product to gain any confidence they didn't break something, only to likely miss something that a senior team member will point out. The back/forth will take multiple developers time. At worst, it gets punted all the way to a QA team. Could be days. Or, with tests, that happens far less often and the feedback loop can be more like 10 minutes. Of course, tests don't solve onboarding, but they provide nice guard rails.

dboreham 5y ago
This assumes the tests are mostly valid. My experience in recent years has been that they won't be.
- hoten 5y ago
  Of course I'm assuming useful tests, why wouldn't I? Note I don't claim tests completely solve all productivity or onboarding problems. It's just odd to claim it's somehow ~unclear that automated tests aren't good for productivity. Forget the extremes (over testing, too many unit tests vs integration, potential rigidness): tests done right are productivity multipliers. And that's not a "No True Scotsman" retort because the original claim had no qualifiers, and was suggesting that any version of automated tests are orthogonal to developer productivity. All you must do to refute that is think of any codebase you were able to make progress on faster because a test told you what was wrong faster than it'd take you to blame source code and send an email to a developer you maybe have never met yet.
  - toast0 5y ago
    > Of course I'm assuming useful tests, why wouldn't I? Because useful tests seem pretty hard to have in practice. Especially when starting testing for a project without tests and a team without testing experience. Early testing reduces productivity, and the proposed gains may never develop.
  - gregjor 5y ago
    > It's just odd to claim it's somehow ~unclear that automated tests aren't good for productivity. No it's not, because it is unclear. Where's the actual evidence? How do you even define and measure "productivity" in software development? My customers define productivity as "Delivered according to spec on time and within budget." That's the only measurement they care about. As hard as it is for programmers to understand beautiful "clean" code and automated tests are generally not stakeholder priorities. Yes, bad code and no testing has a cost that may have to get paid down the road. I think stakeholders understand that software will fall short of perfection. For a long time the standard practice was unit testing, integration testing, and (if you were lucky) rigorous QA. Plenty of usable software got written that way. There's nothing inherently wrong with TDD and automated testing, and when done right those techniques can reduce the need for more complex testing (integration and QA) and help the team understand what the code is supposed to do. That's not the only way to write good code that works, but it's one way. The problems of defining, measuring, and improving programmer productivity have been known and discussed since the 1960s. We have a set of fairly vague best practices (also known since the 60s/70s) but there's no silver bullet. The problems with developing non-trivial software in team environments mainly come from communication and team dynamics, not from lack of style guides or automated tests. Every little bit can help, of course, but let's not kid ourselves that TDD and automated testing are magic fairy dust.

jiggawatts 5y ago

I'm replying to this and also some of the sibling comments. No, automated development-time tests do not necessarily equate to higher quality, given that they: - Take time away from other activities - May provide a false positive, lulling developers into a false sense of confidence - Increase the time taken for certain CI/CD operations, sometimes by hours or days! - The presence of a large volume of "test" code means that certain refactoring operations take a lot longer, which is a real cost. People go into some work environments where tests DID help, and they generalise that to ALL work environments. Automated build-time tests are less important, and entire categories may be potentially unnecessary when: - The product is written in a strongly-typed language that catches most errors at compile time. - If high-level "linting" tools are used to further enhance the code quality before the executable ever runs. - If feedback from the production environment is more relevant. E.g.: for web applications with performance issues being monitored by an APM, but no critical code stability concerns at build time. Think typical web apps with no real consequences to a crash, not finance applications where a small error could mean Real Money. - Where time-to-market is more important than chasing 100% robustness at the expense of meeting release dates. You often see people from a PHP or Python background go on and on about the importance of tests, but those are dynamically typed languages with lots of unexpected pitfalls that only testing will uncover. Meanwhile entire teams do just fine without any automated tests when using languages like C#, Java, or Rust.

cweagans 5y ago
Types do a great job at assuring you’re getting generally the right kind of data in and out of different functions/methods, but they don’t assure that the program is actually doing the right thing. Strong typing just helps to assure that the program isn’t doing specific categories of the wrong thing (in the same way that Rust and Go make it difficult to write memory-unsafe code). I mean, Go and Rust both have testing capabilities built in. If it wasn’t necessary, why would they do that? Beyond the most trivial of projects, I can’t think of a single project that I’ve worked on (or even looked at on GitHub) that has weighed the costs and benefits of automated testing and decided it just wasn’t worth it.
- jiggawatts 5y ago

itsdrewmiller 5y ago

Generally speaking "empirical evidence" means "published papers", with a little bit of flex for preprints or trustworthy-but-not-peer-reviewed experimental findings.

gregjor 5y ago
I was using it to mean actual data rather than anecdotes. The subject of programmer productivity is both much-studied and discussed and very hard to define, measure, and compare. The studies that do have some empirical evidence seem to show that team dynamics and individual programmer personalities have the most dramatic effects, compared to things like programming language, tests, style guides, etc. It's not that consistent styles and automated testing are bad ideas, it's that they don't clearly make a big dent in team productivity in cases where one or a few incompetent (or merely slow) programmers, or one jerk, can derail everyone's productivity. This I why I emphasized demonstrating a commitment to the team and showing some contributions before criticizing the environment and trying to propose changes. If you want the project to grind to a halt tell your new co-workers they are doing everything wrong. No automated test suite is going to fix that problem.

gregjor 5y ago

I think you are agreeing with me. There's anecdotal evidence like you offer. It's not necessarily obvious to everyone that automated tests improve code quality or it would be a settled issue (it's not). In my own career (40+ years) I have worked with automated testing and TDD, and without, and I can't say that TDD "obviously" improves code quality. I know other professional programmers who have that experience. Like everything else in software development it's probably situational and depends on the individuals and the team.