2 ms·

I've had this feeling for a while that unit testing tools are ripe for 'disruption'. The main problem I see is relating the test cases "back" to features and br

I've had this feeling for a while that unit testing tools are ripe for 'disruption'. The main problem I see is relating the test cases "back" to features and broken/flaky tests. For procrastination, the thing that works for me is timeboxing (setting a timer for as little as 10/20 minutes and just getting started). See if that works for you.

MaulingMonkey 10y ago
> I've had this feeling for a while that unit testing tools are ripe for 'disruption'. The main problem I see is relating the test cases "back" to features and broken/flaky tests. This is part of my interest. For one of the nastiest debugging sessions in recent memory, I eventually resorted to performing a manual perforce bisect of history (no p4 bisect), and manual testing to invoke system UI. This revealed that spamming some call to check internet connectivity would cause a "crash" (via exit(3) with no useful debug spam and an equally useless callstack) if you opened the windows 8 charm bar for more than 10 seconds in a certain way. I spent a good week or two trying to track that down before resorting to brute force. So there's two tools right there that would've saved me a ton of time: 1) p4 bisect. git bisect exists - writing tools to iterate over arbitrary VCSes shouldn't be that hard. 2) System level input capture and replay. This already exists in the form of tools like AutoHotKey, and there's similar tools for e.g. Xbox 360 gamepad input - but these are scattered tools, each which must be separately learned, configured, and integrated, and usually not focused on creating 'unit' tests. . There's also a lot of low hanging fruit for basic test types. Most unit test frameworks offer little more than a fancy assert(). Some build systems will have some basic correlation of test failures to VCS versions. Meanwhile, I want to test things like...: 1) Did change X impact this performance-critical bit of software? What about on other machines than mine? I shouldn't have to manually profile every change to see performance impacts, good or bad. I have a very basic prototype of this displaying the results of my latest checkin @ http://test.maulingmonkey.com/ http://test.maulingmonkey.com/ 2) I have a GLSL shader that works on my device (tm). I want automatic test failures when it turns out this shader crashes the driver when linking the shader on phones with a specific GPU vendor. This boils down to little more than having better tools to manage the running of unit tests against multiple machines/devices, and correlating the results in a sane fashion. Running database unit tests against every phone is usually a waste of time, but running graphics unit tests against only the phones connected to the build machines isn't enough - it should test against every phone connected to a workstation in the studio overnight. It should also test against the AWS Device Farm. 3) I have a cross-platform renderer that works on PS4, XB1, Windows, OS X, iOS, and Android. If I render this scene, do I get approximately the same result? Or is XB1 rendering a black screen instead? Do certain Android phones have major precision issues? Which GPU Driver versions misrender? 4) I want to fuzz-test user profile parsing. "Forever" - by which I mean, whenever a build machine or workstation is idle. So just script your CI server to run SDL MiniFuzz! But I also want to fuzz things which don't have a file based interface - such as network protocols, containers, user control input, ...

MaulingMonkey 10y ago

> I've had this feeling for a while that unit testing tools are ripe for 'disruption'. The main problem I see is relating the test cases "back" to features and broken/flaky tests. This is part of my interest. For one of the nastiest debugging sessions in recent memory, I eventually resorted to performing a manual perforce bisect of history (no p4 bisect), and manual testing to invoke system UI. This revealed that spamming some call to check internet connectivity would cause a "crash" (via exit(3) with no useful debug spam and an equally useless callstack) if you opened the windows 8 charm bar for more than 10 seconds in a certain way. I spent a good week or two trying to track that down before resorting to brute force. So there's two tools right there that would've saved me a ton of time: 1) p4 bisect. git bisect exists - writing tools to iterate over arbitrary VCSes shouldn't be that hard. 2) System level input capture and replay. This already exists in the form of tools like AutoHotKey, and there's similar tools for e.g. Xbox 360 gamepad input - but these are scattered tools, each which must be separately learned, configured, and integrated, and usually not focused on creating 'unit' tests. . There's also a lot of low hanging fruit for basic test types. Most unit test frameworks offer little more than a fancy assert(). Some build systems will have some basic correlation of test failures to VCS versions. Meanwhile, I want to test things like...: 1) Did change X impact this performance-critical bit of software? What about on other machines than mine? I shouldn't have to manually profile every change to see performance impacts, good or bad. I have a very basic prototype of this displaying the results of my latest checkin @ http://test.maulingmonkey.com/ http://test.maulingmonkey.com/ 2) I have a GLSL shader that works on my device (tm). I want automatic test failures when it turns out this shader crashes the driver when linking the shader on phones with a specific GPU vendor. This boils down to little more than having better tools to manage the running of unit tests against multiple machines/devices, and correlating the results in a sane fashion. Running database unit tests against every phone is usually a waste of time, but running graphics unit tests against only the phones connected to the build machines isn't enough - it should test against every phone connected to a workstation in the studio overnight. It should also test against the AWS Device Farm. 3) I have a cross-platform renderer that works on PS4, XB1, Windows, OS X, iOS, and Android. If I render this scene, do I get approximately the same result? Or is XB1 rendering a black screen instead? Do certain Android phones have major precision issues? Which GPU Driver versions misrender? 4) I want to fuzz-test user profile parsing. "Forever" - by which I mean, whenever a build machine or workstation is idle. So just script your CI server to run SDL MiniFuzz! But I also want to fuzz things which don't have a file based interface - such as network protocols, containers, user control input, ...