3 ms·

IMO, this post takes a left turn off the freeway when it segues into fuzz testing. Those are super effective, no doubt, but I've come to the conclusion that fin

IMO, this post takes a left turn off the freeway when it segues into fuzz testing. Those are super effective, no doubt, but I've come to the conclusion that finding bugs isn't the hard part. Dan even admits fixing them is the hard part: > why don't you fix the bugs yourself? I do fix some bugs, but there literally aren't enough hours in a week for me to debug and fix every bug I run into.

ChrisMarshallNY 5y ago
My experience, is that finding bugs is the time-consuming part. Fixing them, is often relatively fast. Crashes are good. If I can reproduce a crash, I can usually enact a fix fairly quickly. The issue is that "reproduce" part. For example, threading issues can be a bear. They might crash, once a week, and the crash dumps are near-worthless. I can often do better with a Ouija board. I've always been a very good bug-fixer and problem-solver, but I've learned that the best way to solve problems, is to not have them, in the first place; so good design, and high-quality development methodology, pays dividends. I write about my approach here: https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...
- jldugger 5y ago
  In my professional SRE experience, you are an outlier. I analyze canaries on a regular basis for bugs. Like, the only thing lazier than fuzzing is using your customers to fuzz. One category of bug I file coming out of that is NullPointerExceptions. NPEs are always a bug, and they come with a stack trace and messages about which variable was null. Yet only about 10 percent of the bugs I've filed came back fixed. Often months later. The rest are either still open or closed without fixing in an effort to purge the backlog of old bugs.
  - ChrisMarshallNY 5y ago
    > closed without fixing in an effort to purge the backlog of old bugs. Ugh. :( I like to run on a "constant beta" basis. I fix every bug that I can find -even the small ones- immediately; halting work going forward, until they are fixed. As you can imagine, this was not popular with my employers, so I was not allowed to work that way. Since leaving my last company, that's exactly how I've worked, and the results are nothing short of miraculous. As a result, I actually had to dismantle a JIRA reporting system that I set up, because I get so few reports. A sticky note is usually all I need.

ChrisMarshallNY 5y ago

My experience, is that finding bugs is the time-consuming part. Fixing them, is often relatively fast. Crashes are good. If I can reproduce a crash, I can usually enact a fix fairly quickly. The issue is that "reproduce" part. For example, threading issues can be a bear. They might crash, once a week, and the crash dumps are near-worthless. I can often do better with a Ouija board. I've always been a very good bug-fixer and problem-solver, but I've learned that the best way to solve problems, is to not have them, in the first place; so good design, and high-quality development methodology, pays dividends. I write about my approach here: https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...

jldugger 5y ago
In my professional SRE experience, you are an outlier. I analyze canaries on a regular basis for bugs. Like, the only thing lazier than fuzzing is using your customers to fuzz. One category of bug I file coming out of that is NullPointerExceptions. NPEs are always a bug, and they come with a stack trace and messages about which variable was null. Yet only about 10 percent of the bugs I've filed came back fixed. Often months later. The rest are either still open or closed without fixing in an effort to purge the backlog of old bugs.
- ChrisMarshallNY 5y ago
  > closed without fixing in an effort to purge the backlog of old bugs. Ugh. :( I like to run on a "constant beta" basis. I fix every bug that I can find -even the small ones- immediately; halting work going forward, until they are fixed. As you can imagine, this was not popular with my employers, so I was not allowed to work that way. Since leaving my last company, that's exactly how I've worked, and the results are nothing short of miraculous. As a result, I actually had to dismantle a JIRA reporting system that I set up, because I get so few reports. A sticky note is usually all I need.