6 ms·
Here's the an important piece of real-world statistics for you, which I've read about in research papers and personally verified in large-scale systems: For ev
by jiggawatts 7y ago
Here's the an important piece of real-world statistics for you, which I've read about in research papers and personally verified in large-scale systems:
For every 1 complaint or help desk call you receive, there are between 100 and 1,000 users that experience the same issue but didn't bother to complain or call.
Production testing is literally 0.1% to 1% as effective as actual testing.
- toast0 7y agoDepending on your user numbers, this works out pretty well. If you've got 100k users, you're likely to get reports on anything that hits at least 1% of them. In my experience, you can write much simpler code that only needs to work in production than code that also needs to be unit testable. Adding hooks and separating bits and pieces so they can be tested can often make the code much less concise and harder to follow. I would rather spend engineering time making sure problem reports from users get read and escalated and addressed than on testing things that are better left untested. Unit tests, and even integration tests are unlikely to let you know when the feature works as designed, but not as users expect.
- rob74 7y agoMy favorite example of "things that are better left untested" so far is a method that returned a hard-coded array and a test which checked whether the method really did return that hard-coded array. So, if you wanted to add something to that array, you now had to add it in two places. Sure, that's an easy way to increase test coverage, but maybe you should focus on areas where unit testing makes more sense first?
- oselhn 7y agoI prefer to fix bugs before they hit production. It is almost impossible to do that without tests. Also testable code is usually much more readable than untestable god objects, tightly coupled modules and spagetti code.
- odshoifsdhfs 7y agowithout tests doesn't mean bad architecture. I can go the other way and say that code that ends up being 'modified' to be testable, with 43 mocks to just be able to setup a class is as much crap as code without tests. I would say I never achieved more than 20% or so of test coverage in my products and production code. It doesn't mean it is a mess of tightly coupled modules in any way. But TDD and any other dogmas in development are like this. Solution A prevents (maybe) a problem, thus if you aren't using it all other code has that problem and defendants will just keep trying to make it so.
- nostrebored 7y agoIf you need 43 mocks to unit test a class, your TDD has already found a bad smell. Why are there so many classes tied to a single component?
- doctorpangloss 7y agoVideo games frequently succeed despite colossal bugginess. Pokémon Go’s launch is a great example. Engineers really undermine their authority when they talk about testing and quality in abstract and isolation to the user’s experience.
- samus 7y ago> Unit tests, and even integration tests are unlikely to let you know when the feature works as designed, but not as users expect. This is really missing the point of automated tests. Unit tests are for developers and ensure that low-level functionality doesn't break when changes happen. Integration tests verify that components fit together and that it stays that way. End-to-end-tests ensure that user functionality doesn't break. Together, the above turn refactoring and rewriting of components from a business risk into an activity that can happen simultaneously when adding new features. This can ensure that there would never be such a thing as legacy code.
- toast0 7y agoLegacy code is just code that has proven itself to be useful to the business. If you're telling me spending all day fiddling with testing is going to mean I have nothing to show for it, I guess I agree. I think there is room for testing, but you need to really restrict the scope to those parts of the code where the requirements don't change often, and where the natural interface is amiable to testing, and often, you need someone whose job is testing and who didn't write the code to actually do it. Incidentally, those parts of the code where the requirements don't change often get to a point where the code doesn't change either, which should mean the tests won't fail, because the inputs stayed the same.
- DrScump 7y agobetween 100 and 1,000 users that experience the same issue but didn't bother to complain or call Which is often because users have no (established, straightforward) process for reporting, or recognizing, a defect. I found an ugly 2FA bug in a major ride-sharing app's onboarding and was shocked to find that they have no informal mechanism for reporting any defect. (I thought it important enough that I went through the hassle of reporting it through HackerOne, not seeking compensation but simply attention to the problem. It was closed within minutes without a glance or meaningful comment.[0] I'm not shocked anymore, as this seems to be increasingly common. Another growing trend is meaningless and even juvenile error messages. "Oops!" (with no context that could aid diagnosis). "Something went wrong." (ditto, and I see this out of the leading ticket site 300 times a year). Etc. [0] it's possible that it was closed because that particular onboarding was later obsoleted, but still, that's just a bad look.