4 ms·
Edit: Some spacing, and grammar, and some sentence structure. +1. Putting too much faith on the test coverage is a sign of too much faith in humans. Some of m
by tehlike 10y ago
Edit: Some spacing, and grammar, and some sentence structure.
+1. Putting too much faith on the test coverage is a sign of too much faith in humans.
Some of my time goes to mobile development/android these days, and here are facts:
1. On android, there is occasionally this one vendor that crashes randomly for some piece of code. Because they modified the source code for android when they shouldn't. And it's just one model, among thousands of others. And it's different model for different code pieces.
2. Android API is not exactly one would assume. They return nulls for cases when they should return empty lists, and vice versa. They do this occasionally. So you don't really know when an api returns null vs empty list, and some apis (like get running processes) should never return empty, except that it does in some bizarre cases. So your code would need to test both empty list and null, which adds to your test scenarios. Aside from null/empty list, there are many other examples.
3. When you release, all your tests work fine, at least for what they test. There is this one client that uses things a bit differently than others. You also fail 10% of the time for that client because of how they use an API. Especially in Android, how they use an API does matter. A simple example is using Application Context vs Activity Context vs Custom Context implementations. You'd assume they should all work fine, because Context is an abstraction and
Liskov substitution principle should apply, but it doesn't because it's leaky abstraction. Thus, somethings works with some context while others work with other contexts, and you don't really know which.
I came to the conclusion that for most cases, testing is a must to verify correctness of known scenarios, and known regressions. However, for automated upgrades/updates, testing will not work. Not at all. Doing automated updates/upgrades, it should be a controlled release. it makes more sense to make the release A/B test way, if possible, where small population (say 1% of the users) get control/previous release, and another population of the same size but disjoint gets the experiment/new release. Then you compare your baseline metrics. Could be # of http connections, number of user messages sent, number of crashes, etc.
Testing can only cover what humans can think of. Unfortunately, we don't know what we don't know. That's the main problem.
- ZenoArrow 10y agoEven if you do automated testing you should still do exploratory testing, it's not an either/or situation. Automated testing allows you to cover the bulk of your system functionality, exploratory testing allows you to pick up on the unexpected quirks that crop up. Automated testing frees up the time of testers to do exploratory testing as they don't have to run through all tests manually.
- beat 10y agoTrue, testing won't always catch unknown unknowns. BUT. When the bug that slips through the testing is discovered, the first thing to do is write a test that exercises the bug and makes it repeatable. This is sometimes difficult in the case of intermittent or highly condition-dependent bugs, but simply learning how to repeat the behavior is an important learning experience. The bug should not be fixed until there's an automatic test that induces the bug. That way, you can test to know that your fix actually works.
- rabidrat 10y agoThat's a great ideal. But often the code has to be substantially rearchitected to allow for automated testing like this (especially for those difficult bugs, as you mention), and when the fix only requires changing a couple lines of code, it's not worth the risk. Although it would be great if we went through the exercise of "how would we write an automated test for this?" at a high level, even if it doesn't get implemented. That might get people thinking about how to architect their code so that it's more testable from the get-go.
- beat 10y agoIf the code is written with a proper testing suite from the start, it's not hard. If you're dealing with a legacy code base that doesn't have a testing harness (I've done this a lot), then the first thing you should be doing before making changes is writing tests for them! Every time I do the "I don't need a test for this, it's just an obvious little bug" thing, and I implement a test-free fix, it fails! Every. Single. Time. Maybe it's because I'm stupid, but I like to think I'm above average. So if above-average programmers have a high risk of releasing "bug fixes" that don't actually fix correctly (or cause other problems) because they didn't write good tests, that really turns the "not worth the risk" response on its head. It's not that it's not worth the risk to write tests. It's not worth the risk to not write tests!
- tehlike 10y agoThat's the part I meant when i said "known regressions". If there is a bug, you write the test, it doesn't happen again... Until next refactor, because most often you'll find that the tests were written at a lower level, instead of a functional test (that tests the e2e functionality) it was written to test individual pieces of SUT such as individual methods in a class. This happens quite a lot. Devs will prefer unit tests over more complex e2e tests which takes a lot more effort to write (because they are slow, flaky, more complex). Any friction in this process will make the dev prefer the easier way to test (which does not mean future proof).