4 ms·

It's always a lie? You're basically arguing that code with bad test coverage never works. Production code is copping constant manual end-to-end tests. Those te

It's always a lie? You're basically arguing that code with bad test coverage never works. Production code is copping constant manual end-to-end tests. Those tests are more valuable than most unit testing, and some automated black box testing. I'll take stable production system over the unit tested new code any day of the week.

peterkelly 7y ago
That's the kind of thinking that leads to situations like this: https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultrasonic-fingerprint-scanner-fooled-by-screen-protector/ https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultr... A lot of the time it's just as important to make sure certain things can't work as making sure that certain other things will.
jiggawatts 7y ago
Here's the an important piece of real-world statistics for you, which I've read about in research papers and personally verified in large-scale systems: For every 1 complaint or help desk call you receive, there are between 100 and 1,000 users that experience the same issue but didn't bother to complain or call. Production testing is literally 0.1% to 1% as effective as actual testing.
- toast0 7y ago
  Depending on your user numbers, this works out pretty well. If you've got 100k users, you're likely to get reports on anything that hits at least 1% of them. In my experience, you can write much simpler code that only needs to work in production than code that also needs to be unit testable. Adding hooks and separating bits and pieces so they can be tested can often make the code much less concise and harder to follow. I would rather spend engineering time making sure problem reports from users get read and escalated and addressed than on testing things that are better left untested. Unit tests, and even integration tests are unlikely to let you know when the feature works as designed, but not as users expect.
  - rob74 7y ago
    My favorite example of "things that are better left untested" so far is a method that returned a hard-coded array and a test which checked whether the method really did return that hard-coded array. So, if you wanted to add something to that array, you now had to add it in two places. Sure, that's an easy way to increase test coverage, but maybe you should focus on areas where unit testing makes more sense first?
  - oselhn 7y ago
    I prefer to fix bugs before they hit production. It is almost impossible to do that without tests. Also testable code is usually much more readable than untestable god objects, tightly coupled modules and spagetti code.
    - odshoifsdhfs 7y ago
      without tests doesn't mean bad architecture. I can go the other way and say that code that ends up being 'modified' to be testable, with 43 mocks to just be able to setup a class is as much crap as code without tests. I would say I never achieved more than 20% or so of test coverage in my products and production code. It doesn't mean it is a mess of tightly coupled modules in any way. But TDD and any other dogmas in development are like this. Solution A prevents (maybe) a problem, thus if you aren't using it all other code has that problem and defendants will just keep trying to make it so.
      - nostrebored 7y ago
        If you need 43 mocks to unit test a class, your TDD has already found a bad smell. Why are there so many classes tied to a single component?
  - doctorpangloss 7y ago
    Video games frequently succeed despite colossal bugginess. Pokémon Go’s launch is a great example. Engineers really undermine their authority when they talk about testing and quality in abstract and isolation to the user’s experience.
- DrScump 7y ago
  between 100 and 1,000 users that experience the same issue but didn't bother to complain or call Which is often because users have no (established, straightforward) process for reporting, or recognizing, a defect. I found an ugly 2FA bug in a major ride-sharing app's onboarding and was shocked to find that they have no informal mechanism for reporting any defect. (I thought it important enough that I went through the hassle of reporting it through HackerOne, not seeking compensation but simply attention to the problem. It was closed within minutes without a glance or meaningful comment.[0] I'm not shocked anymore, as this seems to be increasingly common. Another growing trend is meaningless and even juvenile error messages. "Oops!" (with no context that could aid diagnosis). "Something went wrong." (ditto, and I see this out of the leading ticket site 300 times a year). Etc. [0] it's possible that it was closed because that particular onboarding was later obsoleted, but still, that's just a bad look.
Aeolun 7y ago
> It's always a lie? You're basically arguing that code with bad test coverage never works. It may work, or it may not work. Without the tests you won’t ever actually know. Maybe your customers are bouncing on bugs, and while you are earning enough, you could have been earning 3 times as much without those bounces.

peterkelly 7y ago

That's the kind of thinking that leads to situations like this: https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultrasonic-fingerprint-scanner-fooled-by-screen-protector/ https://www.digitaltrends.com/mobile/samsung-galaxy-s10-ultr... A lot of the time it's just as important to make sure certain things can't work as making sure that certain other things will.

jiggawatts 7y ago

Here's the an important piece of real-world statistics for you, which I've read about in research papers and personally verified in large-scale systems: For every 1 complaint or help desk call you receive, there are between 100 and 1,000 users that experience the same issue but didn't bother to complain or call. Production testing is literally 0.1% to 1% as effective as actual testing.

toast0 7y ago
Depending on your user numbers, this works out pretty well. If you've got 100k users, you're likely to get reports on anything that hits at least 1% of them. In my experience, you can write much simpler code that only needs to work in production than code that also needs to be unit testable. Adding hooks and separating bits and pieces so they can be tested can often make the code much less concise and harder to follow. I would rather spend engineering time making sure problem reports from users get read and escalated and addressed than on testing things that are better left untested. Unit tests, and even integration tests are unlikely to let you know when the feature works as designed, but not as users expect.
- rob74 7y ago
  My favorite example of "things that are better left untested" so far is a method that returned a hard-coded array and a test which checked whether the method really did return that hard-coded array. So, if you wanted to add something to that array, you now had to add it in two places. Sure, that's an easy way to increase test coverage, but maybe you should focus on areas where unit testing makes more sense first?
- oselhn 7y ago
  I prefer to fix bugs before they hit production. It is almost impossible to do that without tests. Also testable code is usually much more readable than untestable god objects, tightly coupled modules and spagetti code.
  - odshoifsdhfs 7y ago
    without tests doesn't mean bad architecture. I can go the other way and say that code that ends up being 'modified' to be testable, with 43 mocks to just be able to setup a class is as much crap as code without tests. I would say I never achieved more than 20% or so of test coverage in my products and production code. It doesn't mean it is a mess of tightly coupled modules in any way. But TDD and any other dogmas in development are like this. Solution A prevents (maybe) a problem, thus if you aren't using it all other code has that problem and defendants will just keep trying to make it so.
    - nostrebored 7y ago
      If you need 43 mocks to unit test a class, your TDD has already found a bad smell. Why are there so many classes tied to a single component?
- doctorpangloss 7y ago
  Video games frequently succeed despite colossal bugginess. Pokémon Go’s launch is a great example. Engineers really undermine their authority when they talk about testing and quality in abstract and isolation to the user’s experience.
DrScump 7y ago
between 100 and 1,000 users that experience the same issue but didn't bother to complain or call Which is often because users have no (established, straightforward) process for reporting, or recognizing, a defect. I found an ugly 2FA bug in a major ride-sharing app's onboarding and was shocked to find that they have no informal mechanism for reporting any defect. (I thought it important enough that I went through the hassle of reporting it through HackerOne, not seeking compensation but simply attention to the problem. It was closed within minutes without a glance or meaningful comment.[0] I'm not shocked anymore, as this seems to be increasingly common. Another growing trend is meaningless and even juvenile error messages. "Oops!" (with no context that could aid diagnosis). "Something went wrong." (ditto, and I see this out of the leading ticket site 300 times a year). Etc. [0] it's possible that it was closed because that particular onboarding was later obsoleted, but still, that's just a bad look.

Aeolun 7y ago

> It's always a lie? You're basically arguing that code with bad test coverage never works. It may work, or it may not work. Without the tests you won’t ever actually know. Maybe your customers are bouncing on bugs, and while you are earning enough, you could have been earning 3 times as much without those bounces.