5 ms·

As a user of software, I get similarly frustrated as the author. ("user" here includes use of third-party libraries to build on) However, developing system-leve

As a user of software, I get similarly frustrated as the author. ("user" here includes use of third-party libraries to build on) However, developing system-level software, I've come to realise that even if you really, really care about the quality of your software, you can still be bitten by statistics. Basically, developing error-free software is comparatively easy if your software effectively performs no I/O, that is, it behaves like a program in a computer science paper: read in some data on launch, grind through some computation, emit output, terminate. Barring catastrophic hardware failure of CPU or memory, this is a nicely deterministic programming model. You stand a chance writing correct code. Throw "real" I/O into the mix, and almost anything can fail in weird ways, and your code has to be prepared for it. Network I/O is guaranteed to fail sooner or later while the developer is using the software. So it usually gets taken into account in some way, usually only distinguishing between "there is no connection" and "there is a connection". There are a myriad of other cases in between that are usually not even considered. Disk I/O can fail for a variety of reasons. Not just hardware failure; file systems aren't perfect, especially when confronted with power failure, kernel panics, etc. Randomly flipped bits happen. (yes, really) Not only are there are bugs in the GUI framework you're using, other GUI programs are running at the same time and they can inadvertently interact with your program due to the shared GUI framework use. Other programs can inadvertently interact with yours in other ways: locked files, claimed sockets, contention for any kind of resource, race conditions, thread/task scheduling - you name it. Timing bugs are ubiquitous. Everything you do in your program takes >0 time. Maybe on your system, with your data set, it looks like 0. Maybe because it takes slightly less than one video frame's worth of time. On your customer's system, it takes longer. If they click something before your operation has completed, and you haven't anticipated this, your program will fail in weird ways. Where I live, I can't get an internet connection with less than about 80ms latency even to the nearest servers, let alone to North America, where most servers sit (more like 200ms). You wouldn't believe how much software handles this terribly. The problem is complexity - in many cases, unavoidable complexity, not the accidental complexity us developers keep railing against. Most of these error cases are extremely rare. The thing is, with thousands or millions of people using your software, extremely rare bugs suddenly become a very frequent occurrence! Yet the tools for dealing with this kind of thing are somewhere between terrible and non-existent. There are some tools for simulating difficult network conditions; those are comparatively easy to make. I'm not aware of similar software that simulates OS API call failures. Or a "file system from hell" that wreaks havoc with your file I/O. Fuzzing a program in such a way would likely uncover countless bugs. valgrind and its myriad of plugins are great, but as developers we almost certainly under-use it. Developing such tools is obviously expensive, and even they won't catch all bugs. But I'm pretty sure they could reduce the probability of running into bugs by a few orders of magnitude. Don't even get me started on how programming languages don't help you handle error conditions or timing problems even if you try.

evincarofautumn 14y ago
As someone who works on compilers for a living, even “read in some data on launch, grind through some computation, emit output, terminate” is not an easy problem. Regardless of what tools you use, there are no substitutes for testing, care, and rigour. And testing.
- pmjordan 14y ago
  I would not be surprised if compilers were the most complex examples of that kind of program in the wild. Still, I would be surprised if automated testing of a compiler was as difficult as, say, test cases kernel-space driver that simulate hardware failure. (this is one example of something I'm currently dealing with) This is probably why compilers tend to have comprehensive test suites, and why drivers (and other I/O-heavy software) usually don't. To be clear: I'm not saying writing compilers is easy; I'm saying automated testing of compilers is conceptually not that hard to imagine, so it gets done. Whereas testing I/O and other external interaction involves ridiculous levels of yak shaving as it has almost no existing tool support, so it doesn't happen. Yet, most of the software we as humans interact with directly is exactly that I/O-heavy stuff.
nitrogen 14y ago
...Or a "file system from hell" that wreaks havoc with your file I/O. Fuzzing a program in such a way would likely uncover countless bugs.... Though you are probably familiar with them, there are I/O fuzzers like zzuf[0]. I bet it'd be relatively straightforward to write a similar tool that uses LD_PRELOAD to intercept and randomly delay various library calls. Sometimes I'll test my own network software by piping /dev/urandom over netcat: dd if=/dev/urandom bs=1K count=1K | nc [hostname] [port] [0] http://caca.zoy.org/wiki/zzuf http://caca.zoy.org/wiki/zzuf
- pmjordan 14y ago
  Yeah, fuzzing stream inputs is relatively straightforward. It's everything else that affects program outcome that's the issue. Trying to implement something like this via LD_PRELOAD manually is going to be a pretty enormous task. Realistically, you'd have to automatically detect the calls made, for system calls you have ptrace (most commonly used by strace), though I'm not sure how you reroute them. I guess there must be some kernel-side hook. Still, that's likely to be too low-level for GUIs for example. Preloading dynamic libraries like that also only works well on ELF-based systems, other OSes behave rather differently, and if you're trying to test kernel code, it's a whole other problem. Once you're simulating errors, you have to specify correct behaviour. For stream input fuzzing for security purposes, you're usually just probing for crashes. In the general case, the test framework will need to know the intended outcome of errors. For either part, I'm not aware of anything existing like this. I'm slowly custom-building little islands of functionality like it, but it's a far cry from some kind of general solution.

evincarofautumn 14y ago

As someone who works on compilers for a living, even “read in some data on launch, grind through some computation, emit output, terminate” is not an easy problem. Regardless of what tools you use, there are no substitutes for testing, care, and rigour. And testing.

pmjordan 14y ago
I would not be surprised if compilers were the most complex examples of that kind of program in the wild. Still, I would be surprised if automated testing of a compiler was as difficult as, say, test cases kernel-space driver that simulate hardware failure. (this is one example of something I'm currently dealing with) This is probably why compilers tend to have comprehensive test suites, and why drivers (and other I/O-heavy software) usually don't. To be clear: I'm not saying writing compilers is easy; I'm saying automated testing of compilers is conceptually not that hard to imagine, so it gets done. Whereas testing I/O and other external interaction involves ridiculous levels of yak shaving as it has almost no existing tool support, so it doesn't happen. Yet, most of the software we as humans interact with directly is exactly that I/O-heavy stuff.

nitrogen 14y ago

...Or a "file system from hell" that wreaks havoc with your file I/O. Fuzzing a program in such a way would likely uncover countless bugs.... Though you are probably familiar with them, there are I/O fuzzers like zzuf[0]. I bet it'd be relatively straightforward to write a similar tool that uses LD_PRELOAD to intercept and randomly delay various library calls. Sometimes I'll test my own network software by piping /dev/urandom over netcat: dd if=/dev/urandom bs=1K count=1K | nc [hostname] [port] [0] http://caca.zoy.org/wiki/zzuf http://caca.zoy.org/wiki/zzuf

pmjordan 14y ago
Yeah, fuzzing stream inputs is relatively straightforward. It's everything else that affects program outcome that's the issue. Trying to implement something like this via LD_PRELOAD manually is going to be a pretty enormous task. Realistically, you'd have to automatically detect the calls made, for system calls you have ptrace (most commonly used by strace), though I'm not sure how you reroute them. I guess there must be some kernel-side hook. Still, that's likely to be too low-level for GUIs for example. Preloading dynamic libraries like that also only works well on ELF-based systems, other OSes behave rather differently, and if you're trying to test kernel code, it's a whole other problem. Once you're simulating errors, you have to specify correct behaviour. For stream input fuzzing for security purposes, you're usually just probing for crashes. In the general case, the test framework will need to know the intended outcome of errors. For either part, I'm not aware of anything existing like this. I'm slowly custom-building little islands of functionality like it, but it's a far cry from some kind of general solution.