3 ms·
Well of course they could have done that. To be honest, though, if someone on my team suggested we implement an automated test that tries sending every Unicode
by jmull 9y ago
Well of course they could have done that.
To be honest, though, if someone on my team suggested we implement an automated test that tries sending every Unicode character (and it would be applied to every interface of every app and API, right?) I would have objected that this was an over-complicated, over-engineered solution that will probably be too slow.
I'd argue that a set of test data selected to cover a range of patterns, especially ones that are considered risky (either is know to have caused problems in the past or appear complicated or tricky) would have almost as good coverage and be an order of magnitude more useful.
The problem with "run-every-case" tests is that they start off slow and get exponentially slower if you try to go deep you very quickly end up with tests that take too long to run to be useful. (e.g., {every Unicode character} is one thing... {Every Unicode character} X {every interface and app} is a LOT more. {Every Unicode character} X {every interface and app} X {every build} X {every device model} X etc. is impossible). So you end up with very broad but very shallow tests. And that means less coverage ultimately, not more.
Generally, I think you'll get more efficient, effective, useful tests if you tailor the test data sets to the problems you see rather than going for blanket coverage.
- mitchty 9y ago> Generally, I think you'll get more efficient, effective, useful tests if you tailor the test data sets to the problems you see rather than going for blanket coverage. Sure, but fuzzing and "throw everything and the kitchen sink at it" type testing will generally expose logic bugs you weren't aware existed. Aka the unknown unknowns. Its not like this is an either/or proposition, you could run the fuzzing type gauntlet tests every week or so.
- wvenable 9y agoSince this affects a huge number of applications and even desktop applications, it's a low-level library that is causing the problem. I'm not sure it's unreasonable to test every unicode codepoint against shared library if that library is responsible for the rendering of unicode text. But I agree it would be unreasonable to do that for every app that uses the library.
- jmull 9y agoYou have a point. It's probably not unreasonable to test every Unicode codepoint against a low-level shared library, at least for APIs that process a single codepoint. I don't think it's a no-brainer though. And it is unreasonable to expect a testing technique to be applied in every case where it is reasonable to do so. There are a lot of techniques that are reasonable to use in any give case but it would be no sense at all to apply every one of them. To focus on this one, now, amounts to Monday morning quarterbacking. Once you know the bug it's easy to see how it could have been caught. Of course if you somehow knew of a bug ahead of time you wouldn't need any test at all!
- wvenable 9y agoYeah, I don't disagree. Arguing this would caught by better testing is close to implying that all software bugs are avoidable; we simply have to test "better" until there are no bugs ever.
- alexbeloi 9y agoI'm no expert but this seems like a good case for doing random subsampling during the test. IMO fast probabilistic total coverage test is better than no test or partial coverage.