15 ms·

Software engineering is programming over time. It's not so hard to write code that is correct today, if that's all that tests did then they wouldn't be worth th

by rictic 6y ago

Software engineering is programming over time. It's not so hard to write code that is correct today, if that's all that tests did then they wouldn't be worth the effort. We write tests so that we know whether future changes have broken the system or not. This article sounds like a reaction to the practice of writing many small, hermetic unit tests, which do little more than recapitulate the code under test. The weakest parts of a system are often in the joints, so the most important tests to write and to run are the integration tests, the ones that tell you with the most confidence whether the code works in the real world or not. See, for example, this puzzling line from the article: > Other tests grow obsolete because the target platform is retired, but not the test. There's even less pressure to remove stale tests than useless code. Watch as the workaround for Windows XP is removed from the code, but not the test checking it still works. How do you remove the code for a feature, but not the test for that feature unless either you aren't running your tests or your tests aren't testing the things that you care about?

RobinL 6y ago
Absolutely agree. My biggest lightbulb moments with testing have come when needing to do significant refactoring. Without tests, I feel aimless because it's hard to get feedback about whether the refactoring has 'worked'. With tests there's a nice, tight feedback loop. Not saying it gives any guarantees, in my experience it has dramatically increased how quickly I could improve the code - to the extent where in some cases I wouldn't have even tried in the absence of tests because I wouldn't have been confident I could improve it.
- ajeet_dhaliwal 6y ago
  Agree with this and the parent comment. For the most part automated tests are for testing for regression. The highest ROI tests for complex projects are end-to-end in my experience - where the gateway interface is the API or UI - because they test an entire flow and can let you know if there is an issue somewhere, ideally with an error that can narrow it down to a specific function or file/module. It does seem like the author is talking about unit tests based on what he/she is saying which can imo have less ROI but they're still not useless. If you have limited resources (don't we all) then start with a developer working on end-to-end tests, it's further away from the code being tested but it's great 'bang for buck', and run them on an automatic schedule - each commit may be overkill because end-to-end tests can take a long time. The second part of the article mentions that stale tests are often not removed. However if running tests using either CI or on a regular schedule and viewing the results everyday, this can't happen, they would be removed or modified quickly. This does raise a valid point that within our industry test results data is often not handled well and is an afterthought. My hypothesis is that this is actually the reason some developers get frustrated with tests or do not see the value or why you may get stale tests not being removed. The problem of poor management of test results data itself (the output from testing) rather than testing itself is the problem. A small plug here but I started tesults.com to address this very problem., integration takes a few minutes if you use a popular test framework. There's a forever use free tier so try it out, and if you need more but don't have or can't get budget but love what it does then send me an email and I'll expand your free tier. The important thing is to try to see if it makes thing better for you and you use it as part of your release management.
- EdwardDiego 6y ago
  Exactly. I picked up some legacy invoicing code that's 13 years old, and people are terrified of breaking it, so it's just quietly rotted away and half of it isn't even used anymore, but no-one can quite tell which half. First thing I did was add tests so that I can refactor with confidence.
harryf 6y ago
Think the question "What's the return on investment on writing tests?" doesn't get asked enough. There's a blind assumption that writing lots of tests is always good and code coverage tools tend to push the idea that you're not done until you reach 100% coverage. Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working, so you can get it in the hands of some customers and begin validating a business case. How many tests do you need in this case? And how much time do you allow for it? Most of the code you're going to produce will be thrown away and the more compelling the prototype is in terms of functionality, features and design the stronger the signal you'll get back from potential customers. To me there's still too much "testing is my religion" amongst developers and not enough "here's the quantified value of me spending 3 days writing tests and here's how that fits in the context of where our business is right now"
- kqr 6y ago
  Maybe. Maybe a feature-rich but rough and unpolished prototype is what will delight customers the most. Or maybe focusing on an absolutely minimal core, but executing it really well with high performance and having it run rock-solidly will attract the potential customers more. I think this depends a lot on demographics, but in broad strokes, I think the former is overvalued compared to the latter. I.e. contrary to your experience!
  - ralphstodomingo 6y ago
    I would be inclined to agree except people generally have low standards of performance (ofc depending on the industry!). People also rarely care about the hand of the puppeteer, what keeps the lights on parts especially if we're talking about a prototype.
    - kqr 6y ago
      I think people care a great deal more than they have words to express. We're not very good at educating our customers in ways to discuss and analyse technical performance. We're getting better at it when it comes to uptime, but latency discussions still lag (hah!) behind. (This is from purely personal experience. I have only had one customer ever, when asked about performance requirements, who could list some. Everyone else has been "I guess I want it to be... good?")
collyw 6y ago
I agree with you on integration tests being the most useful ones. The problem is that they are generally the most time consuming to write.
- exdsq 6y ago
  You can always start with black box testing so you check that bit of the program matches some expected input/output pairs, and then add more white box integration tests if the code is important/fiddly. It’s an easy way to get some validation on a particular subset of the system for a small amount of effort.
Tainnor 6y ago
Small unit tests are valuable iff your small units are well designed and valuable. If they are brittle, temporary components they don't have much value, I agree, but writing mostly integration tests suffers from the combinatorial explosion problem and from making it hard to express succinctly what the important bit about a piece of code is (and besides, almost no test is 100% integrated, you always isolate something, even in full end-to-end tests). If you have many functional, stateless components, the unit tests can be particularly valuable. But I think many tests end up being badly written because people don't necessarily ask the fundamental question: "does this increase my confidence in the code?" I think that's the most important consideration when writing tests.
UK-Al05 6y ago
Are you defining unit tests are per class tests? This is the biggest mistake. You should be testing a unit of behaviour, like a business rule or an effect to be expected. If this involves multiple objects collaborating then so be it. But they should not cross architectural boundaries like http calls, or database calls. If you can only make public api classes accessible, and make all helper classes inaccessible to users. That also forces you test things from only a public api standpoint. Tests that just check an object calls another object, but exhibits no desired behaviour in its self are pointless, and just couple everything to the implementation. I.e Certain methods, are called in a certain order with specific parameters. When you do that, you make it really hard to change things(just design changes, not desired behaviour changes) without breaking every test.
- chriswarbo 6y ago
  Absolutely. I've worked in places which enforced this decouple-everything-from-everything approach, and it just grinds work to a halt, makes test failures commonplace and uninformative. I think there's a definite problem with terminology when it comes to testing. I've written about this before; the latest example being http://chriswarbo.net/blog/2020-07-07-more_testing_terminology.html http://chriswarbo.net/blog/2020-07-07-more_testing_terminolo... (just submitted to HN https://news.ycombinator.com/item?id=23757346 https://news.ycombinator.com/item?id=23757346 )
- exdsq 6y ago
  You’re talking about integration tests, not unit tests. Unit tests should focus on a single method. Integration tests can combine methods/objects/APIs/etc for business rules. Your best bet is a combination of unit tests, integration tests, and e2e tests (there’s some old advice about a 70/20/10 split but this is pretty arbitrary).
  - UK-Al05 6y ago
    Yes this what has spread around and what people think of as unit tests. But it an awful way of splitting tests. Code bases following this end up with brittle tests, that break at slightest design change with no functional change. Here's a good article kent beck retweeted on the subject. https://twitter.com/KevlinHenney/status/1266383805520084993 https://twitter.com/KevlinHenney/status/1266383805520084993 I define unit tests as fast, and don't cross architectural boundaries, and operate on a well defined public interface. And test some kind actual property you care about. I don't follow "strict" rules like unit test per class, per function etc. A lot classes just extract to some to some helper class, and they operate well together and the helper is not likely to be used anywhere else. Just make the helper class private and test as a unit against some actual desired property you want to test. Integration tests is when you bring external things into the mix like databases, or http calls.
pmontra 6y ago
I agree that tests are an insurance against changes. They find regressions and are also lights in the darkness that points to where we have to work on. I'm moving 3 db fields (let's call them a, b, c) from 3 tables (A, B, C) to a fourth one (D) right now. Adding a, b, c to D is easy, the data migration is also easy (a single update from a CTE coalescing a, b, c from the 3 tables -- the value in a wins over b and over c). And now let's see if anything breaks (it has to.) I run the test suite. Only a dozen errors. Some are where I expected them to happen, some are totally unexpected. There are parts of the code base I forgot about in the last year or never heard about (it's a team of about 5 developers.) Without tests, no matter the language, typing system, etc any change like that would be a nightmare.
- Eridrus 6y ago
  Not to downplay tests, but this would be caught by a typing system if all access to DB fields was through an ORM such that all access could be type checked. Not to say that everyone should use an ORM, but this would be a very simple property for a type checker to catch if you let it.
  - pmontra 6y ago
    Possibly, but (simplifying a lot) the actual refactoring involves JSONB fields that store serializations of objects which include those fields plus a yet unknown amount of code that might be using them, etc. And we're ending up redesigning part of the UI because 1) it makes sense and 2) it makes the migration to the new backend simpler. Luckily we're pretty well covered by tests.

by rictic 6y ago

RobinL 6y ago

Absolutely agree. My biggest lightbulb moments with testing have come when needing to do significant refactoring. Without tests, I feel aimless because it's hard to get feedback about whether the refactoring has 'worked'. With tests there's a nice, tight feedback loop. Not saying it gives any guarantees, in my experience it has dramatically increased how quickly I could improve the code - to the extent where in some cases I wouldn't have even tried in the absence of tests because I wouldn't have been confident I could improve it.

ajeet_dhaliwal 6y ago
Agree with this and the parent comment. For the most part automated tests are for testing for regression. The highest ROI tests for complex projects are end-to-end in my experience - where the gateway interface is the API or UI - because they test an entire flow and can let you know if there is an issue somewhere, ideally with an error that can narrow it down to a specific function or file/module. It does seem like the author is talking about unit tests based on what he/she is saying which can imo have less ROI but they're still not useless. If you have limited resources (don't we all) then start with a developer working on end-to-end tests, it's further away from the code being tested but it's great 'bang for buck', and run them on an automatic schedule - each commit may be overkill because end-to-end tests can take a long time. The second part of the article mentions that stale tests are often not removed. However if running tests using either CI or on a regular schedule and viewing the results everyday, this can't happen, they would be removed or modified quickly. This does raise a valid point that within our industry test results data is often not handled well and is an afterthought. My hypothesis is that this is actually the reason some developers get frustrated with tests or do not see the value or why you may get stale tests not being removed. The problem of poor management of test results data itself (the output from testing) rather than testing itself is the problem. A small plug here but I started tesults.com to address this very problem., integration takes a few minutes if you use a popular test framework. There's a forever use free tier so try it out, and if you need more but don't have or can't get budget but love what it does then send me an email and I'll expand your free tier. The important thing is to try to see if it makes thing better for you and you use it as part of your release management.
EdwardDiego 6y ago
Exactly. I picked up some legacy invoicing code that's 13 years old, and people are terrified of breaking it, so it's just quietly rotted away and half of it isn't even used anymore, but no-one can quite tell which half. First thing I did was add tests so that I can refactor with confidence.

harryf 6y ago

Think the question "What's the return on investment on writing tests?" doesn't get asked enough. There's a blind assumption that writing lots of tests is always good and code coverage tools tend to push the idea that you're not done until you reach 100% coverage. Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working, so you can get it in the hands of some customers and begin validating a business case. How many tests do you need in this case? And how much time do you allow for it? Most of the code you're going to produce will be thrown away and the more compelling the prototype is in terms of functionality, features and design the stronger the signal you'll get back from potential customers. To me there's still too much "testing is my religion" amongst developers and not enough "here's the quantified value of me spending 3 days writing tests and here's how that fits in the context of where our business is right now"

kqr 6y ago
Maybe. Maybe a feature-rich but rough and unpolished prototype is what will delight customers the most. Or maybe focusing on an absolutely minimal core, but executing it really well with high performance and having it run rock-solidly will attract the potential customers more. I think this depends a lot on demographics, but in broad strokes, I think the former is overvalued compared to the latter. I.e. contrary to your experience!
- ralphstodomingo 6y ago
  I would be inclined to agree except people generally have low standards of performance (ofc depending on the industry!). People also rarely care about the hand of the puppeteer, what keeps the lights on parts especially if we're talking about a prototype.
  - kqr 6y ago
    I think people care a great deal more than they have words to express. We're not very good at educating our customers in ways to discuss and analyse technical performance. We're getting better at it when it comes to uptime, but latency discussions still lag (hah!) behind. (This is from purely personal experience. I have only had one customer ever, when asked about performance requirements, who could list some. Everyone else has been "I guess I want it to be... good?")

collyw 6y ago

I agree with you on integration tests being the most useful ones. The problem is that they are generally the most time consuming to write.

exdsq 6y ago
You can always start with black box testing so you check that bit of the program matches some expected input/output pairs, and then add more white box integration tests if the code is important/fiddly. It’s an easy way to get some validation on a particular subset of the system for a small amount of effort.

Tainnor 6y ago

Small unit tests are valuable iff your small units are well designed and valuable. If they are brittle, temporary components they don't have much value, I agree, but writing mostly integration tests suffers from the combinatorial explosion problem and from making it hard to express succinctly what the important bit about a piece of code is (and besides, almost no test is 100% integrated, you always isolate something, even in full end-to-end tests). If you have many functional, stateless components, the unit tests can be particularly valuable. But I think many tests end up being badly written because people don't necessarily ask the fundamental question: "does this increase my confidence in the code?" I think that's the most important consideration when writing tests.

UK-Al05 6y ago

Are you defining unit tests are per class tests? This is the biggest mistake. You should be testing a unit of behaviour, like a business rule or an effect to be expected. If this involves multiple objects collaborating then so be it. But they should not cross architectural boundaries like http calls, or database calls. If you can only make public api classes accessible, and make all helper classes inaccessible to users. That also forces you test things from only a public api standpoint. Tests that just check an object calls another object, but exhibits no desired behaviour in its self are pointless, and just couple everything to the implementation. I.e Certain methods, are called in a certain order with specific parameters. When you do that, you make it really hard to change things(just design changes, not desired behaviour changes) without breaking every test.

chriswarbo 6y ago
Absolutely. I've worked in places which enforced this decouple-everything-from-everything approach, and it just grinds work to a halt, makes test failures commonplace and uninformative. I think there's a definite problem with terminology when it comes to testing. I've written about this before; the latest example being http://chriswarbo.net/blog/2020-07-07-more_testing_terminology.html http://chriswarbo.net/blog/2020-07-07-more_testing_terminolo... (just submitted to HN https://news.ycombinator.com/item?id=23757346 https://news.ycombinator.com/item?id=23757346 )
exdsq 6y ago
You’re talking about integration tests, not unit tests. Unit tests should focus on a single method. Integration tests can combine methods/objects/APIs/etc for business rules. Your best bet is a combination of unit tests, integration tests, and e2e tests (there’s some old advice about a 70/20/10 split but this is pretty arbitrary).
- UK-Al05 6y ago
  Yes this what has spread around and what people think of as unit tests. But it an awful way of splitting tests. Code bases following this end up with brittle tests, that break at slightest design change with no functional change. Here's a good article kent beck retweeted on the subject. https://twitter.com/KevlinHenney/status/1266383805520084993 https://twitter.com/KevlinHenney/status/1266383805520084993 I define unit tests as fast, and don't cross architectural boundaries, and operate on a well defined public interface. And test some kind actual property you care about. I don't follow "strict" rules like unit test per class, per function etc. A lot classes just extract to some to some helper class, and they operate well together and the helper is not likely to be used anywhere else. Just make the helper class private and test as a unit against some actual desired property you want to test. Integration tests is when you bring external things into the mix like databases, or http calls.

pmontra 6y ago

I agree that tests are an insurance against changes. They find regressions and are also lights in the darkness that points to where we have to work on. I'm moving 3 db fields (let's call them a, b, c) from 3 tables (A, B, C) to a fourth one (D) right now. Adding a, b, c to D is easy, the data migration is also easy (a single update from a CTE coalescing a, b, c from the 3 tables -- the value in a wins over b and over c). And now let's see if anything breaks (it has to.) I run the test suite. Only a dozen errors. Some are where I expected them to happen, some are totally unexpected. There are parts of the code base I forgot about in the last year or never heard about (it's a team of about 5 developers.) Without tests, no matter the language, typing system, etc any change like that would be a nightmare.

Eridrus 6y ago
Not to downplay tests, but this would be caught by a typing system if all access to DB fields was through an ORM such that all access could be type checked. Not to say that everyone should use an ORM, but this would be a very simple property for a type checker to catch if you let it.
- pmontra 6y ago
  Possibly, but (simplifying a lot) the actual refactoring involves JSONB fields that store serializations of objects which include those fields plus a yet unknown amount of code that might be using them, etc. And we're ending up redesigning part of the UI because 1) it makes sense and 2) it makes the migration to the new backend simpler. Luckily we're pretty well covered by tests.