4 ms·

There is a lot of testing that is waste. If you can unit test something as a unit then do it... the moment you put a MOCK in that code is waste. That means to

There is a lot of testing that is waste. If you can unit test something as a unit then do it... the moment you put a MOCK in that code is waste. That means to test your system you need to go end to end. API -> processing -> storage -> response.... Write the records and read them back. That also means your UI needs to do it to! JS keeps adding features in the browser, and none of them result in reliable testing. We need to do better. But end to end testing is hard, our environment is complicated, it takes too long. Fix it (its fixable)! Bound your test system to run from nothing to complete in under 20 mins. Treat failed tests and long runs like production outages. Write your end to end testing as stand alone. That means if you move your code base to a new language, your tests should still be valid and running. Does this mean dont write unit tests? NO you should have some unit tests. Things that matrix out well (think email validation), are the softballs that should have some unit tests. Highly contentious code that business "logic" that is a dense ugly switch... unit test that, build the mocks, because if you have to touch that you want every confidence that it will pass your E2E tests. Watch where you duplicate: Its the job of the front end to fully exercise its code. Its up to API devs to fully test their API's, to hit all their boundary conditions. Will there be duplication, sure, but its going to make clear where breakages are happening between layers.

valenterry 3y ago
I would almost agree, but what do you do about external systems? Take postgres for example - yes, you can easily set one up. But how do you test concurrency issues due to timing? This can be necessary but very tricky to achieve without mocks/fakes or whatever it's called.
- mrkeen 3y ago
  > But how do you test concurrency issues due to timing? You don't. 1) The outcome of the tests will be: 0.1% There is an issue. 99.9% Maybe there's an issue. 2) Finding concurrency issues needs fine-grained targeting. You can't just write a "does anything race?" test. Testing won't reveal the location of the races. You already need to know about the race condition to be able to write a test for it. And once you know it's there, what are you testing? Are you making sure that the system does the wrong thing?
  - valenterry 3y ago
    I meant more concrete cases. Say there is some weird behaviour and you suspect that it's due to a race condition. Maybe someone forgot to do "select for update" and you want to write a test to confirm. How to do that? Even if you know 100% that it's a race condition, you still might want to add a test to prevent it from happening again no?
  - IanCal 3y ago
    > And once you know it's there, what are you testing? That you've fixed it.
- RamblingCTO 3y ago
  I've had great success with using testcontainers and a self-contained module interfacing with that. As we use hexagonal architecture, these are adapters.
RamblingCTO 3y ago
> the moment you put a MOCK in that code is waste. I disagree. I mock the interfaces to other components that have clearly defined output in a clean architecture to properly unit test (i.e. a use case). Everything else causes components to be mixed and things to be tested again and again, causing complex and glued together tests. It all depends on proper architecture, which is hard to get right though. We're using kotlin (sealed classes ftw), hexagonal arch + ddd (use cases) and graphql/rest on top. Everything split up by gradle multi modules with architecture tests to verify that modules have not been mixed. We also have a domain module with plain kotlin that implements business logic free of implementation details. With this high separation of concerns unit testing is the only thing you need to do (automated).
leghifla 3y ago
Just a fun anecdote on end-to-end testing I did 20 years ago while working on the compression of images. Some parts of the algorithm were easy to test, others very hard. On the easy part was loading and saving the file. I tried to test as best as I could, but the lossy part was not really tested. Then time to benchmark the algo on some images. The results were kind of OK, but some images compressed much better than others at low bit-rate, while the content did not look much different. On square images, everything was fine. On rectangular ones, at high-bit rates it was somewhat fine, but strange patterns like ghosts of another part on the image started to appear for lower and lower bit rates. Of course, I searched on the mostly algorithmic parts what was going on, and failed. Only after a holiday break and fresh eyes did I found the root cause: inversion of height and width in some part of the loading AND saving. I had "end to end" test of the loading and saving, but the error was symmetric, so undetected. Young engineer error...
whstl 3y ago
IMO/IME, mocks in unit tests can be extremely dangerous when used alone, in non-static-typed languages. Changes in interfaces or behaviour of a module won't break the unit tests of modules that depend on it. So you must rely on either typechecking (not available in all languages) or integration tests (which will then trigger a change that will cause the unit-test-with-mocks to break). If there is neither to help, you code just breaks in production. Not only that, but this can cause some design ossification: classes that have too many dependents become very difficult to change, since you now also have to modify all mocks of it.
- bvrmn 3y ago
  In Python one could use auto specs to be sure test would be broken on calling mocks with changed API[1]. I don't think it's hard to implement (if already not) similar feature for other dynamic languages. [1]: https://docs.python.org/3/library/unittest.mock.html#auto-speccing https://docs.python.org/3/library/unittest.mock.html#auto-sp...

valenterry 3y ago

I would almost agree, but what do you do about external systems? Take postgres for example - yes, you can easily set one up. But how do you test concurrency issues due to timing? This can be necessary but very tricky to achieve without mocks/fakes or whatever it's called.

mrkeen 3y ago
> But how do you test concurrency issues due to timing? You don't. 1) The outcome of the tests will be: 0.1% There is an issue. 99.9% Maybe there's an issue. 2) Finding concurrency issues needs fine-grained targeting. You can't just write a "does anything race?" test. Testing won't reveal the location of the races. You already need to know about the race condition to be able to write a test for it. And once you know it's there, what are you testing? Are you making sure that the system does the wrong thing?
- valenterry 3y ago
  I meant more concrete cases. Say there is some weird behaviour and you suspect that it's due to a race condition. Maybe someone forgot to do "select for update" and you want to write a test to confirm. How to do that? Even if you know 100% that it's a race condition, you still might want to add a test to prevent it from happening again no?
- IanCal 3y ago
  > And once you know it's there, what are you testing? That you've fixed it.
RamblingCTO 3y ago
I've had great success with using testcontainers and a self-contained module interfacing with that. As we use hexagonal architecture, these are adapters.

RamblingCTO 3y ago

> the moment you put a MOCK in that code is waste. I disagree. I mock the interfaces to other components that have clearly defined output in a clean architecture to properly unit test (i.e. a use case). Everything else causes components to be mixed and things to be tested again and again, causing complex and glued together tests. It all depends on proper architecture, which is hard to get right though. We're using kotlin (sealed classes ftw), hexagonal arch + ddd (use cases) and graphql/rest on top. Everything split up by gradle multi modules with architecture tests to verify that modules have not been mixed. We also have a domain module with plain kotlin that implements business logic free of implementation details. With this high separation of concerns unit testing is the only thing you need to do (automated).

leghifla 3y ago

Just a fun anecdote on end-to-end testing I did 20 years ago while working on the compression of images. Some parts of the algorithm were easy to test, others very hard. On the easy part was loading and saving the file. I tried to test as best as I could, but the lossy part was not really tested. Then time to benchmark the algo on some images. The results were kind of OK, but some images compressed much better than others at low bit-rate, while the content did not look much different. On square images, everything was fine. On rectangular ones, at high-bit rates it was somewhat fine, but strange patterns like ghosts of another part on the image started to appear for lower and lower bit rates. Of course, I searched on the mostly algorithmic parts what was going on, and failed. Only after a holiday break and fresh eyes did I found the root cause: inversion of height and width in some part of the loading AND saving. I had "end to end" test of the loading and saving, but the error was symmetric, so undetected. Young engineer error...

whstl 3y ago

IMO/IME, mocks in unit tests can be extremely dangerous when used alone, in non-static-typed languages. Changes in interfaces or behaviour of a module won't break the unit tests of modules that depend on it. So you must rely on either typechecking (not available in all languages) or integration tests (which will then trigger a change that will cause the unit-test-with-mocks to break). If there is neither to help, you code just breaks in production. Not only that, but this can cause some design ossification: classes that have too many dependents become very difficult to change, since you now also have to modify all mocks of it.

bvrmn 3y ago
In Python one could use auto specs to be sure test would be broken on calling mocks with changed API[1]. I don't think it's hard to implement (if already not) similar feature for other dynamic languages. [1]: https://docs.python.org/3/library/unittest.mock.html#auto-speccing https://docs.python.org/3/library/unittest.mock.html#auto-sp...