4 ms·
> We had built a system that generated thousands of lines of logs for every test, with lots of “failures” recorded in them. Things like “tried to initialize FOO
by aasasd 8y ago
> We had built a system that generated thousands of lines of logs for every test, with lots of “failures” recorded in them. Things like “tried to initialize FOOBAR_CONTROLLER: FAILED!!!,” but we just ran that code on all machines, even ones without the FOOBAR_CONTROLLER hardware. So no one noticed when another 5 lines of errors popped up in a 2000-line log file.
This right there is a big red flag. The whole bendy business is bad enough, but here you're actively training people to ignore the wolf cries.
Don't allow false failures in tests. The entire test suite needs to be binary: either everything works, or it fails.
- taneq 8y agoLikewise, this is why it's important not to let compiler warnings slide. If your build process spews out a bazillion lines of cruft then that one new warning which indicates a serious error will get lost in the noise. I don't go so far as to enable warnings-as-errors or anything but I do build with -Wall and generally won't check in code which builds with warnings.
- munk-a 8y agoAs a company you can decide where to fall on this scale, I applaud you for -Wall'ing but for most it's too much, the important thing is to make sure that everyone has agreed which warnings are acceptable and you refuse code that violates that. We're using PHP (so runtime warnings, instead of compiler issues) but we have a zero-error policy, if anything raises a warning or error we refuse to merge (or ticketize fixing since a bunch of stuff is legacy and we're still paying it down).
- stdclass 8y agoThe standard in PHP-frameworks like Laravel is to automatically throw Exceptions when a Notice happens, which at first I thought was overly aggressive but I would never go back to writing PHP where a Notice is thrown and ignored.
- munk-a 8y agoIt depends on the era of PHP to be honest, pre-PHP 5.2 (IMO) notices were mostly for pretty small stuff that was accepted as being necessary for sanity, i.e. trying to read a value out of $arr[$dem1][$dem2][$dem3] would require three separate isset checks to do safely, I always leaned toward building a user-space recursive isset function but it came with a penalty to performance. Starting in 5.4 and getting far more prominent in 5.6, PHP started breaking bad backwards compatibility and failing to respect notices could easily lead to errors on version bumping up to 7.
- aasasd 8y agoAs I mentioned in the sister comment, by ignoring notices you're setting yourself up for garbage values being passed through your code and into user-facing functionality, including handling of money. That doesn't depend on the PHP version.
- aasasd 8y agoThere's an extra point with notices: they are likely to lead to logical errors instead of technical ones. The algorithm works, but it receives a garbage value (null aka zero, usually). So it gives some garbage as a result. In nine cases, resulting garbage goes further into various barely meaningful code, but in one it will go into some user-facing functionality, including handling of money.
- detaro 8y agoIsn't that the point of the article pretty much? You easily get used to ignorable errors, making it harder to catch the non-ignorable, if you don't pay attention to not do that.
- aasasd 8y agoSoftware has a luxury of being discrete most of the time. So we can actually do discrete―and automated―checks on whether something works as we think it should. Humans only read the end result, and even that isn't always necessary. In fact, maybe NASA should have similarly ‘unbendable’ (cough) software checklists. The temperature is 39ºF? Red color says no launch today.