3 ms·

Google has a pretty mature TCP testing framework, called packetdrill: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41848.pdf h

Google has a pretty mature TCP testing framework, called packetdrill: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41848.pdf http://static.googleusercontent.com/media/research.google.co... It supports all the features you need, from packet structure to timing: https://code.google.com/p/packetdrill/ https://code.google.com/p/packetdrill/

acveilleux 11y ago
It still treats the whole stack as a blackbox. Necessarily the effort to cover important boundary cases throughout the TCP stack can get overwhelming.
- ori_b 11y ago
  Honestly, I've found black box testing at public boundaries of a system gives the best payoff vs development time, especially if you can make them run quickly. Most issues seem to be catchable with this, and way too many unit tests end up becoming 'change detector' tests, where you test tautologies about implementation details.
  - gsnedders 11y ago
    Black-box testing as a first move is often a good move, IMO — but it's important to accept that at some point you're going to stop hitting so many edge-cases in the implementation that ought be tested. Some of those edge-cases should probably be tested with black-box tests, and some should probably be tested with unit tests. One has to consider the performance cost of black-box tests, though. It will almost always be higher than that of unit tests, simply because more code will get run per test (this is especially true of network stuff, as even going through the loopback device is comparatively insanely expensive). That said, in some cases, it may well still be the right choice.
  - golergka 11y ago
    > black box testing at public boundaries of a system But that sounds more like integration testing, not unit testing, doesn't it?
jsnell 11y ago
I watched the Usenix talk on packetdrill, and while it's a really neat tool it seems to me that it doesn't really support timing very well. The tests are run in real time - running 650 tests take almost half hour, which would normally be considered completely unacceptable for any kind of TDD. The timing can also vary from one run to another, so all timestamps have a few milliseconds of wiggle room automatically, and a small proportion of runs (1/2500) still fail randomly due to timer inaccuracies. And unit tests that fail non-deterministically are really unpalatable. Is the talk outdated re: these issues? Great hack despite that, and I would certainly have loved to write a scripting language for expressing tests rather than use C++. And the packet syntax they've chosen is probably superior to what I ended up with. But what I describe in the post dates back to 2011, so packetdrill wasn't yet around for us to take inspiration from :-/

acveilleux 11y ago

It still treats the whole stack as a blackbox. Necessarily the effort to cover important boundary cases throughout the TCP stack can get overwhelming.

ori_b 11y ago
Honestly, I've found black box testing at public boundaries of a system gives the best payoff vs development time, especially if you can make them run quickly. Most issues seem to be catchable with this, and way too many unit tests end up becoming 'change detector' tests, where you test tautologies about implementation details.
- gsnedders 11y ago
  Black-box testing as a first move is often a good move, IMO — but it's important to accept that at some point you're going to stop hitting so many edge-cases in the implementation that ought be tested. Some of those edge-cases should probably be tested with black-box tests, and some should probably be tested with unit tests. One has to consider the performance cost of black-box tests, though. It will almost always be higher than that of unit tests, simply because more code will get run per test (this is especially true of network stuff, as even going through the loopback device is comparatively insanely expensive). That said, in some cases, it may well still be the right choice.
- golergka 11y ago
  > black box testing at public boundaries of a system But that sounds more like integration testing, not unit testing, doesn't it?

jsnell 11y ago

I watched the Usenix talk on packetdrill, and while it's a really neat tool it seems to me that it doesn't really support timing very well. The tests are run in real time - running 650 tests take almost half hour, which would normally be considered completely unacceptable for any kind of TDD. The timing can also vary from one run to another, so all timestamps have a few milliseconds of wiggle room automatically, and a small proportion of runs (1/2500) still fail randomly due to timer inaccuracies. And unit tests that fail non-deterministically are really unpalatable. Is the talk outdated re: these issues? Great hack despite that, and I would certainly have loved to write a scripting language for expressing tests rather than use C++. And the packet syntax they've chosen is probably superior to what I ended up with. But what I describe in the post dates back to 2011, so packetdrill wasn't yet around for us to take inspiration from :-/