3 ms·

I've always been impressed by hardware QA test teams I've worked with. On Google Fiber, they had an elaborate lab with every possible piece of consumer electro

I've always been impressed by hardware QA test teams I've worked with. On Google Fiber, they had an elaborate lab with every possible piece of consumer electronics equipment in there, and would evaluate every release against a (controlled) unfriendly RF environment. ("In version 1.2.3.4, the download from this MacBook Pro while the microwave was running was 123.4Mbps, but in version 1.2.4.5, it's 96.8Mbps." We actually had a lot of complexity beyond this that they tested, like bandsteering, roaming, etc.) I was always extremely impressed because they came up with test cases I wouldn't have thought of, and the feedback to the development team was always valuable to act on. If they're finding this issue, we get pages of charts and graphs and an invite to the lab. If a customer finds this issue, it just eats away at our customer satisfaction while we guess what could possibly have changed. Best to find the issue in QA or development. As for software engineers handling QA, I'm very much in favor of development teams doing as much as possible. I often see tests bolted on to the very end of projects, which isn't going to lead to good tests. I think that software engineers are missing good training on what to be suspicious of, and what best practices are. There are tons of books written on things like "how to write baby's first test", but honestly, as an industry, we're past that. We need resources on what you should look out for while reviewing designs, what you should look out for while reviewing code, what should trigger alarm bells in your head while you're writing code. I'm always surprised how I'll write some code that's weird, say to myself "this is weird", and then immediately write a test to watch it change from failing to passing. Like times when you're iterating over something where normally the exit condition is "i < max", but this one time, it's different, it actually has to be "i <= max". I get paranoid and write a lot of tests to check my work. Building that paranoia is key. > I like putting "" into every field and pressing submit. Going deeper into the training aspect, something I find very useful are fuzz tests. I have written a bunch of them and they have always found a few easy-to-fix but very-annoying-to-users bugs. I would never make a policy like "every PR must include a fuzz test", but I think it would be valuable to tell new hires how to write them, and why they might help find bugs. No need to have a human come up with weird inputs when your idle CI supercomputer can do it every night! (Of course, building that infrastructure is a pain. I run them on my workstation when I remember and it interests me. Great system.) At the end of the day, I'm somewhat disappointed in the standards that people set for software. To me, if I make something for you and it blows up in your hands... I feel really shitty. So I try to avoid that in the software world by trying to break things as I make them, and ensure that if you're going to spend time using something, you don't have a bad experience. I think it's rare, and it shouldn't be, it should be something the organization values from the top to the bottom. I suppose the market doesn't incentive quality as much as it should, and as a result, organizations don't value it as much as they should. But wouldn't it be nice to be the one software company that just makes good stuff that always works and doesn't require you to have 2 week calls with the support team? I'd buy it. And I like making it. But I'm just a weirdo, I guess.

ryan-duve 3y ago
> Going deeper into the training aspect, something I find very useful are fuzz tests. Could you share some details of fuzz tests that you've found useful? I tend to work with backend systems and am trying to figure out whether they will still be useful in addition to unit and integration tests.
- avidiax 3y ago
  Fuzz tests are most useful if they are run continuously/concurrently with development. Doing it that way, a change or decision that causes a fuzz test failure hasn't been built upon yet. Imagine building an earthquake resistant house on a shaker table vs. putting the 100% completed house on the same shaker table. Doing fuzz testing at the end leads to a lot of low priority but high cost bugs being filed (and many low-cost bugs as well). The utility of it is quite clear for security bugs. It requires low effort to find lots of crashes or errors that might be exploitable. For development in general, it tends to identify small errors or faulty architectural or synchronization decisions very early, while they are still easy to repair.
- jrockway 3y ago
  Sure! I wrote my first fuzz test with a go1.18 prerelease just to get a feel for the new framework. It found a crash instantly: https://github.com/jrockway/json-logs/commit/77c87854914d756283211d077f482a1b4dce594d https://github.com/jrockway/json-logs/commit/77c87854914d756.... This commit contains the fuzz test and the fix. (3 cases could be nil; I only ever thought about 2 of them. The fuzzer found the 3rd almost immediately.) For my project at work, we have an (g)RPC server with a ton of methods. The protos have evolved since 2015 while trying to be backwards compatible with older clients. As a result, many request parameters are optional but not expressed as oneof, so it's hard to figure out what a valid message actually is, and different parts of the code do it differently. (As an aside, oneof is totally backwards compatible... but the protos evolved without knowing that.) I wrote a fuzz test that generates random protos and calls every single RPC with them: https://github.com/pachyderm/pachyderm/blob/master/src/server/rpc_fuzz_test.go https://github.com/pachyderm/pachyderm/blob/master/src/serve... This test is imperfect in many ways; it knows nothing about protos, and just generates random bytes and sends it to every method. Most of the time, our protoc-gen-validate validators reject the message as being nonsense, but that's totally fine. The test found about 50 crashes pretty immediately (almost null messages, basically a non-null message where the first tag is null), and one really interesting one after about 16 hours of runtime on my 32 core Threadripper workstation. Writing this and fixing the code took about a day of work. I just randomly did it one day after writing the call-GRPC-methods-with-introspection code for a generic CLI interface. I did this concurrent with a professional security review as part of our company's acquisition and... they didn't find any of these crashes. So, I think I added some value. It's worth mentioning that most people wrap gRPC calls with a defer func() { ... recover() } block so that bad RPCs don't crash the server. We didn't at the time, but I implemented the fuzz tests with such a wrapper. (So the test doesn't crash when it finds a panic, I just return a special response code when there is a panic, and the test is marked as failing. Then the next test can proceed immediately in the presence of --keep-fuzzing or whatever.) Either way, I prefer an error message like "field foo in message BarRequest is required" to "Aborted: invalid memory reference or nil pointer dereference." and now we have that for all the cases the fuzzer noticed. The user is sad either way, but at least they can maybe self-help in the first case. All in all, I haven't spent a ton of time writing fuzz tests, but each one has caught some bugs, so I'm happy. A good tool to have in your toolkit.

ryan-duve 3y ago

> Going deeper into the training aspect, something I find very useful are fuzz tests. Could you share some details of fuzz tests that you've found useful? I tend to work with backend systems and am trying to figure out whether they will still be useful in addition to unit and integration tests.

avidiax 3y ago
Fuzz tests are most useful if they are run continuously/concurrently with development. Doing it that way, a change or decision that causes a fuzz test failure hasn't been built upon yet. Imagine building an earthquake resistant house on a shaker table vs. putting the 100% completed house on the same shaker table. Doing fuzz testing at the end leads to a lot of low priority but high cost bugs being filed (and many low-cost bugs as well). The utility of it is quite clear for security bugs. It requires low effort to find lots of crashes or errors that might be exploitable. For development in general, it tends to identify small errors or faulty architectural or synchronization decisions very early, while they are still easy to repair.
jrockway 3y ago
Sure! I wrote my first fuzz test with a go1.18 prerelease just to get a feel for the new framework. It found a crash instantly: https://github.com/jrockway/json-logs/commit/77c87854914d756283211d077f482a1b4dce594d https://github.com/jrockway/json-logs/commit/77c87854914d756.... This commit contains the fuzz test and the fix. (3 cases could be nil; I only ever thought about 2 of them. The fuzzer found the 3rd almost immediately.) For my project at work, we have an (g)RPC server with a ton of methods. The protos have evolved since 2015 while trying to be backwards compatible with older clients. As a result, many request parameters are optional but not expressed as oneof, so it's hard to figure out what a valid message actually is, and different parts of the code do it differently. (As an aside, oneof is totally backwards compatible... but the protos evolved without knowing that.) I wrote a fuzz test that generates random protos and calls every single RPC with them: https://github.com/pachyderm/pachyderm/blob/master/src/server/rpc_fuzz_test.go https://github.com/pachyderm/pachyderm/blob/master/src/serve... This test is imperfect in many ways; it knows nothing about protos, and just generates random bytes and sends it to every method. Most of the time, our protoc-gen-validate validators reject the message as being nonsense, but that's totally fine. The test found about 50 crashes pretty immediately (almost null messages, basically a non-null message where the first tag is null), and one really interesting one after about 16 hours of runtime on my 32 core Threadripper workstation. Writing this and fixing the code took about a day of work. I just randomly did it one day after writing the call-GRPC-methods-with-introspection code for a generic CLI interface. I did this concurrent with a professional security review as part of our company's acquisition and... they didn't find any of these crashes. So, I think I added some value. It's worth mentioning that most people wrap gRPC calls with a defer func() { ... recover() } block so that bad RPCs don't crash the server. We didn't at the time, but I implemented the fuzz tests with such a wrapper. (So the test doesn't crash when it finds a panic, I just return a special response code when there is a panic, and the test is marked as failing. Then the next test can proceed immediately in the presence of --keep-fuzzing or whatever.) Either way, I prefer an error message like "field foo in message BarRequest is required" to "Aborted: invalid memory reference or nil pointer dereference." and now we have that for all the cases the fuzzer noticed. The user is sad either way, but at least they can maybe self-help in the first case. All in all, I haven't spent a ton of time writing fuzz tests, but each one has caught some bugs, so I'm happy. A good tool to have in your toolkit.