4 ms·
> Suppose your developers write one bug a year which will slip through test and crash the users' machine. Suppose you know this, so in an effort to catch the bu
by gwern 10y ago
> Suppose your developers write one bug a year which will slip through test and crash the users' machine. Suppose you know this, so in an effort to catch the bug you do more testing. In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking. That actually makes the testing harder but… Suppose you do one release a year. That release blue screens the machine. The user now sees every release you do crashes his machine. 100% of your releases screw up. If instead you release weekly, one release a year still crashes the machine but the user sees 51 releases a year which don’t. Less than 2% of your releases screw up.
This example makes no sense. The user in either case still gets 1 crash per year. They're actually worse off in the many-releases case because in the annual update scenario, they can at least block off a few days to cope with the upgrade scenario (in the way that sysadmins schedule upgrades and downtime for the least costly times of years), but in the weekly release, they could be screwed over at any of 52 times a year at random, and knowing Murphy, it'll be at the worst time. '% of releases which crash' is irrelevant to anything.
- ZenoArrow 10y agoI agree, it makes no sense, not only from a planning perspective from an implementation perspective, specifically... "In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking." If you wanted to keep testing costs low, you wouldn't do bigger releases, you'd create automated tests. You may spend more effort up front on building the tests, but as long as you make the test components modular and target the tests at the right level of your application the 'cost' of testing will decrease over time.
- AstralStorm 10y agoYeah, sure, everything can be tested in an automated way. I can name a few projects that tried it. They were released with a hilarious amount of bugs. Also writing good automated tests requires a great test developer. The thing is, anyone with such credentials would be a great developer and as such, not a tester. Even if you go fully test-driven, which makes it much cheaper, the cost of test a lot development model is surprisingly high for any application of useful size. Just imagine trying to write even something a simple as MS Paint with good test coverage.
- rco8786 10y agoI'm not sure what would be so special or difficult about writing unit tests for Ms Paint.
- Retric 10y agoMS paint includes user selected faunts. That's an endless rabbit hole of complexity.
- ZenoArrow 10y agoValidation of new fonts is handled by Windows rather than Paint, I don't see why a new font that meets the criteria set out for Windows would cause any massive test burden for Paint.
- Retric 10y agoYou're suggestion is to not test a massive chunk of functionality. That may be reasonable choice, but it's a long way from total test coverage.
- ZenoArrow 10y agoYou target the functionality that resides the application, and its integration with other software. However, you don't need to test Windows functionality, that's for Windows developers to do. 'Total test coverage' is about testing the application under test, not every component of the OS it builds upon.
- ktRolster 10y agoUsers don't care if a bug is due to your code or the code in a library you depend on. It's up to you to make sure things work either way.
- 10y ago
- AstralStorm 10y agoYou mean they will block the one big release for a year? Or are you implying there will be a working hotfix soon after the release? What about security updates then? See, in a weekly model you can probably safely skip a cycle. Not so with a big release. If the release has some critical functionality broken, you might be waiting a long time for it to get fixed.