3 ms·

I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, ass

I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological. I have a weakly-held preference (that I will not fight for in code reviews) for not unit testing imperative code at all because I think it has very limited usefulness and is just more code to maintain. Instead, I prefer to rely on integration or end-to-end testing to validate the behavior of those parts of the system. But I think unit tests are extremely useful for functions. It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic. I think it is always useful to have edge case tests - when is it an error, what happens right on the boundary between error and non-error, etc. - and to have at least one illustrative happy-path example. (I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification.) Where this breaks down is when the vast majority of code is imperative, which is very common in practice. My usual approach there is to lean heavily on higher-level testing, and refactor out little chunks that can be more reasonably unit tested, when practical. And I also write and maintain lots of these not-very-useful imperative tests, because people get itchy about low unit test coverage and I'd rather write the tests than spend time convincing people that they aren't very useful.

PartiallyTyped 3y ago
> I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification. I am that person! I am a big fan of property based testing and try to add such tests whenever possible. They serve as "edge-case hunters" and are usually part of my programming loop. As soon as something has been modified, a test suite executes. Writing property tests first and the code later allows you think about the invariants and behaviour of what you are building in a way that regular unit tests just don't.
- sanderjd 3y ago
  Yeah I get it. But if I have a function where the domain is signed 64-bit integers and the specification is some f(x) between zero and a billion and an error otherwise, I don't want to iterate through all the legal inputs and make sure they do the right thing. If I'm using a language that can do that with its type system or other kind of capability, then I'm happy to use that! But I'm also not grumpy about using python or rust or all the other normal languages just because they can't do this. I'm happy to just write a few cases mostly concentrated around zero and a billion.
  - PartiallyTyped 3y ago
    Technically you can do that with property based testing just fine, e.g. for all values outside boundaries ensure a throw. The goal of property based testing is to help you identify invariants in your code and then verify them.
    - sanderjd 3y ago
      I just looked up Hypothesis in python and it does indeed look pretty cool for this kind of thing. I should try this more!
      - PartiallyTyped 3y ago
        The best part imho is the shrinking.
MoreQARespect 3y ago
Clean functions with limited dependencies which encode non-trivial domain logic are indeed well served by unit tests - even if theyre a bit imperative. But, outside of contrived interview scenarios this type of code is actually somewhat niche and outside of that niche unit tests suck. Ive worked on tons of projects where it accounted for 1-2% of the code base. Unit tests not only werent capable of catching bugs they werent even capable of replicating 98% of them. Unit test theater. Meanwhile, the other day I did TDD with hermetic end to end tests using playwright, mitmproxy, mailpit and screenshot/snapshot testing. It felt like I was living in the future.
- perrygeo 3y ago
  In most software, the units themselves are not that interesting. Basic business logic and data transforms, surrounded by a larger codebase that glues all the integration points together into a working stateful system. I've met a few unit-testing extremists who insisted on pristine-mocked unit tests and only unit tests. Their tests were incomprehensible (a good test should serve as documentation and acceptance criteria - but their code was 90% of concerned with mocking service apis) and importantly, their unit tests repeatedly failed to catch serious regressions that should have been apparent to anyone who simply executed "main" without mocks. We had production outages on the regular. The funny thing is, the unit test extremists doubled down, rejecting the idea of integration or e2e testing entirely (yes, I'm serious) and claiming that prod failures indicate we need to unit testing EVERYTHING. They even went so far as to claim developers need not actually run their software, unit testing is fully sufficient. Not surprisingly, their code had perfectly constructed units but no overall design... the units composed into an unmaintainable dumpster fire as an actual system.
  - RussianCow 3y ago
    I would generally argue that if your unit tests have mocks, they're not actually unit tests. What you're describing sounds like integration tests that completely missed the point (i.e. the "integration" but).
    - 3y ago
JamesLeonis 3y ago
> I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological. I agree. I usually see the I/O layer weaved with the data transform layer. Often in the same function call. And, usually, the answer is to pull those two layers apart, put tests around the transforms, and push the I/O to the edges. Usually... > It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic. (de)serialization functions are a good general case purely-functional example. Data is transformed and you need to verify the transform was correctly performed, has all kinds of unique edge-cases, and should gracefully inform the caller when something is wrong.
- sanderjd 3y ago
  Nothing to add, just: yep, all of this.

PartiallyTyped 3y ago

> I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification. I am that person! I am a big fan of property based testing and try to add such tests whenever possible. They serve as "edge-case hunters" and are usually part of my programming loop. As soon as something has been modified, a test suite executes. Writing property tests first and the code later allows you think about the invariants and behaviour of what you are building in a way that regular unit tests just don't.

sanderjd 3y ago
Yeah I get it. But if I have a function where the domain is signed 64-bit integers and the specification is some f(x) between zero and a billion and an error otherwise, I don't want to iterate through all the legal inputs and make sure they do the right thing. If I'm using a language that can do that with its type system or other kind of capability, then I'm happy to use that! But I'm also not grumpy about using python or rust or all the other normal languages just because they can't do this. I'm happy to just write a few cases mostly concentrated around zero and a billion.
- PartiallyTyped 3y ago
  Technically you can do that with property based testing just fine, e.g. for all values outside boundaries ensure a throw. The goal of property based testing is to help you identify invariants in your code and then verify them.
  - sanderjd 3y ago
    I just looked up Hypothesis in python and it does indeed look pretty cool for this kind of thing. I should try this more!
    - PartiallyTyped 3y ago
      The best part imho is the shrinking.

MoreQARespect 3y ago

Clean functions with limited dependencies which encode non-trivial domain logic are indeed well served by unit tests - even if theyre a bit imperative. But, outside of contrived interview scenarios this type of code is actually somewhat niche and outside of that niche unit tests suck. Ive worked on tons of projects where it accounted for 1-2% of the code base. Unit tests not only werent capable of catching bugs they werent even capable of replicating 98% of them. Unit test theater. Meanwhile, the other day I did TDD with hermetic end to end tests using playwright, mitmproxy, mailpit and screenshot/snapshot testing. It felt like I was living in the future.

perrygeo 3y ago
In most software, the units themselves are not that interesting. Basic business logic and data transforms, surrounded by a larger codebase that glues all the integration points together into a working stateful system. I've met a few unit-testing extremists who insisted on pristine-mocked unit tests and only unit tests. Their tests were incomprehensible (a good test should serve as documentation and acceptance criteria - but their code was 90% of concerned with mocking service apis) and importantly, their unit tests repeatedly failed to catch serious regressions that should have been apparent to anyone who simply executed "main" without mocks. We had production outages on the regular. The funny thing is, the unit test extremists doubled down, rejecting the idea of integration or e2e testing entirely (yes, I'm serious) and claiming that prod failures indicate we need to unit testing EVERYTHING. They even went so far as to claim developers need not actually run their software, unit testing is fully sufficient. Not surprisingly, their code had perfectly constructed units but no overall design... the units composed into an unmaintainable dumpster fire as an actual system.
- RussianCow 3y ago
  I would generally argue that if your unit tests have mocks, they're not actually unit tests. What you're describing sounds like integration tests that completely missed the point (i.e. the "integration" but).
  - 3y ago

JamesLeonis 3y ago

> I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological. I agree. I usually see the I/O layer weaved with the data transform layer. Often in the same function call. And, usually, the answer is to pull those two layers apart, put tests around the transforms, and push the I/O to the edges. Usually... > It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic. (de)serialization functions are a good general case purely-functional example. Data is transformed and you need to verify the transform was correctly performed, has all kinds of unique edge-cases, and should gracefully inform the caller when something is wrong.

sanderjd 3y ago
Nothing to add, just: yep, all of this.