7 ms·

Unit testing a TCP stack (2015)

135 points by PuercoPop 5y ago

cjfd 5y ago
Yes, a TCP stack certainly is complex enough to warrant serious automated testing and/or TDD. The idea of putting the TCP stack in user space is interesting. If one actually could map the memory of the whole device into user space one could maybe have fewer system calls and therefore have better performance. Also, what I find somewhat irritating about using a linux system is how often one needs to run commands as root (sudo) for common administrative tasks like mounting a disk or stuff like that. Having a user space TCP stack could also decrease the need for that as far as setting up the network is concerned. If the linux machine is single user, as most of them are nowadays, it makes more sense that way, I think.
- rmetzler 5y ago
  > one needs to run commands as root (sudo) for common administrative tasks like mounting a disk I would think if you don't do this, an attacker who is able to execute code but is non-root yet could easily elevate permissions by shadowing legitimate pathes and trick root into executing untrusted code. I'm not a security engineer and just find it interesting, so if my thinking is off, please correct me.
- boomlinde 5y ago
  There's nothing inherent about Linux which prevents you from running everything as uid 0. If you're fine with every process you run having the same full privileges and shared ownership of everything, you should. Most machines, at least outside embedded devices, are not like this. They are multi-user systems even when there's only ever one breathing thing at the desk because it offers a degree of separation between the privileges of your daemons, your pid 1, your web browser etc.
  - heurisko 5y ago
    I think the point was that root shouldn’t be required for “common administrative tasks”. The nuclear option of running everything as root doesn’t address this.
    - stingraycharles 5y ago
      What are the common administrative tasks related to networking that require root for networking? All I can think about is stuff like route tables and dhcp, both of which live at the IP/Ethernet level rather than TCP.
      - adrian_b 5y ago
        Starting any network server process that uses ports under 1000, like most standard protocols (https, http, ssh, smtp, dns, ntp, dhcp etc.), requires root rights on any UNIX-like operating system. Most personal computers do not need server processes (unless you want to connect remotely to them), but your question was not restricted to them.
        boomlinde 5y ago
        From a practical point of view, regardless of the scope of the original question, this is the kind of scenario where you'd really want the restriction. More than a simple administrative task it's a dangerous attack vector to allow any user to launch your httpd or DNS. That being said, check out capabilities(7) in Linux. You can grant an executable the privilege of binding to a low port when run by non-0 uid through setcap. This is a good compromise.
        convolvatron 5y ago
        this whole 'privledged ports' nonsense is left over from a time where some process on another machine running on a low port was somehow to be trusted - because the person running that process was another administrator, and you can generally trust those guys (as opposed to unwashed users). that world didn't last very long, and I wish we could vent some of these designs that didn't pass the test of time.
- stingraycharles 5y ago
  > Having a user space TCP stack could also decrease the need for [root privileges] as far as setting up the network is concerned. I think it’s important to distinguish between the protocol (TCP) and the hardware device. You would still absolutely need to talk to the device, it’s just that moving a lot of the logic to user space means much less context switching for system calls for the application. I can imagine on Linux you can talk directly to /dev/eth0 if you would want to (in the same way that you can talk to /dev/sda), and then you would be back at square one regarding root privileges.
  - monocasa 5y ago
    > I can imagine on Linux you can talk directly to /dev/eth0 if you would want to (in the same way that you can talk to /dev/sda), and then you would be back at square one regarding root privileges. It's a AF_PACKET, SOCK_RAW socket rather than a device file, but yes.
- stefan_ 5y ago
  The whole "map the PCIE device into userspace process memory" thing is called DPDK (https://www.dpdk.org/ https://www.dpdk.org/)
  - ay 5y ago
    And you can combine the two: https://fd.io/docs/vpp/master/whatisvpp/hoststack.html https://fd.io/docs/vpp/master/whatisvpp/hoststack.html And there is a sister project using this tech to get noticeable speed-ups: https://wiki.fd.io/view/VSAP https://wiki.fd.io/view/VSAP Disclaimer: I am involved with the VPP project.
- avinassh 5y ago
  > The idea of putting the TCP stack in user space is interesting. Indeed! Julia Evans wrote a really nice post explaining the usecases and benefits - https://jvns.ca/blog/2016/06/30/why-do-we-use-the-linux-kernels-tcp-stack/ https://jvns.ca/blog/2016/06/30/why-do-we-use-the-linux-kern...
- yakubin 5y ago
  You don't need to be root to mount disks, when you have udisks installed (which would be almost all distros by default). See udisksctl(1): <https://manpages.debian.org/buster/udisks2/udisksctl.1.en.html https://manpages.debian.org/buster/udisks2/udisksctl.1.en.ht...>
eatonphil 5y ago
I couldn't make it past serialization/deserialization logic in my own hobbiest TCP/IP stack. Even that part was super buggy. Next time around I'm definitely going to be unit testing more parts otherwise it's too hard for a beginner to get the easy parts right let alone the harder parts. Also, take a look at gvisor's network stack. It's definitely unit tested. https://github.com/google/gvisor/tree/master/pkg/tcpip/link/ethernet https://github.com/google/gvisor/tree/master/pkg/tcpip/link/... (an example)
- amscanne 5y ago
  This is perhaps a better example: https://github.com/google/gvisor/blob/master/pkg/tcpip/transport/tcp/tcp_test.go https://github.com/google/gvisor/blob/master/pkg/tcpip/trans... Also, some networking tests use separate frameworks (which look more like the setup the original post is describing, since those are needed also), e.g.: https://github.com/google/gvisor/tree/master/test/packetimpact/tests https://github.com/google/gvisor/tree/master/test/packetimpa...
gravypod 5y ago
There's a lot of push back from engineers - especially people at lower levels of the stack - against testing infrastructure. One particularly famous example is Linux. Rather than testing before merging in code, they merge in code and then test the release candidate as a whole. It also seems game developers are extremely against automated testing frameworks as a whole. I've heard many times that it would be impossible to develop an enemy AI in a test-driven way (I did this for a senior project in college - finished the AI before the game was able to even start testing it [0]). I wonder what would need to happen to convince people that: 1. Even if you do something extremely low level, you can draw a distinction between your hardware and the interface that 99% of your software runs at. 2. You can develop complex behaviors iteratively with automated testing just like you can develop complex programs iteratively (tests are just programs). [0] - https://github.com/gravypod/it491-disabler-ai https://github.com/gravypod/it491-disabler-ai
- lloydatkinson 5y ago
  I agree - it seems it's only excuses that keep certain devs away from testing. Even Factorio manages testing, though its more of an integration test. I'm sure it could be done with unit tests too.
  - gravypod 5y ago
    I think even integration tests would be a huge improvement for most teams and software. As long as some automated checks are run before code is merged you're going to save yourself a lot of heart ache.
- PezzaDev 5y ago
  Automated testing is hard to justify for games because you are not simply finishing features and moving on. You are constantly experimenting. Throwing out features and ideas and reworking them is part of the creative process. Automated testing adds overhead to iteration and isn't free, so you have to be very selective. At the end of the day it is more important for the game to be fun than it is stable.
  - 5y ago
jsnell 5y ago
One benefit we discovered with this test framework after the blog post was written was that it made it much more convenient to do fuzzing and differential testing of the TCP stack. The core problem with fuzzing TCP is that there's a lot of incrementally built up state, and everything is extremely timing-dependent. You basically need the fuzzer to have a model of TCP state so that it can effectively explore the state space, which is quite complicated and not something you can do with off-the shelf tools. But once you have a bunch of unit tests designed to put the TCP stack into a specific state + a way of saving and restoring that state, it's really easy to just have snapshot of interesting situations where you can run a fuzzer on the next packet to be transmitted and see what happens.
10000truths 5y ago
It would be nice to have a bring-your-own-I/O TCP stack library that *doesn’t* rely on custom callbacks - something like BearSSL but for TCP, where the stack is just a pure state machine object and the user is responsible for explicitly shunting packets to and from the state machine, retaining control over when and how the I/O is done. Instead of having to define callbacks for retrieving time and consuming packets, why not explicitly pass the timestamp and packet data to a state machine object via a direct function call?
- Slix 5y ago
  Is Cloudflare's Quiche QUIC library https://github.com/cloudflare/quiche https://github.com/cloudflare/quiche similar to what you're looking for? All I/O must be done by the caller.
  - 10000truths 5y ago
    Yeah, that’s the general idea. Essentially a state machine with a send queue and receive queue, and four operations: - Input received raw data - Output received application data - Input application data to send - Output raw data to send Obviously, since TCP connection state is time sensitive, the “raw data” wouldn’t just be the IP packet and headers, but also a time stamp telling the state machine when that packet was received/sent. If you want the state machine to keep track of time even when no packets are being received or sent, there could be an additional operation just to input a timestamp without additional packets. In effect, time is just another input that the user is responsible for feeding to the state machine at sufficiently fine intervals. In practice, you could emulate this pattern with a callback-oriented protocol stack by populating an in-memory send/receive queue in your callback function, but that design can be somewhat inflexible because it forces potentially undesirable constraints, e.g. an extra memory copy that could otherwise be elided.

135 points by PuercoPop 5y ago

cjfd 5y ago

Yes, a TCP stack certainly is complex enough to warrant serious automated testing and/or TDD. The idea of putting the TCP stack in user space is interesting. If one actually could map the memory of the whole device into user space one could maybe have fewer system calls and therefore have better performance. Also, what I find somewhat irritating about using a linux system is how often one needs to run commands as root (sudo) for common administrative tasks like mounting a disk or stuff like that. Having a user space TCP stack could also decrease the need for that as far as setting up the network is concerned. If the linux machine is single user, as most of them are nowadays, it makes more sense that way, I think.

rmetzler 5y ago
> one needs to run commands as root (sudo) for common administrative tasks like mounting a disk I would think if you don't do this, an attacker who is able to execute code but is non-root yet could easily elevate permissions by shadowing legitimate pathes and trick root into executing untrusted code. I'm not a security engineer and just find it interesting, so if my thinking is off, please correct me.
boomlinde 5y ago
There's nothing inherent about Linux which prevents you from running everything as uid 0. If you're fine with every process you run having the same full privileges and shared ownership of everything, you should. Most machines, at least outside embedded devices, are not like this. They are multi-user systems even when there's only ever one breathing thing at the desk because it offers a degree of separation between the privileges of your daemons, your pid 1, your web browser etc.
- heurisko 5y ago
  I think the point was that root shouldn’t be required for “common administrative tasks”. The nuclear option of running everything as root doesn’t address this.
  - stingraycharles 5y ago
    What are the common administrative tasks related to networking that require root for networking? All I can think about is stuff like route tables and dhcp, both of which live at the IP/Ethernet level rather than TCP.
    - adrian_b 5y ago
      Starting any network server process that uses ports under 1000, like most standard protocols (https, http, ssh, smtp, dns, ntp, dhcp etc.), requires root rights on any UNIX-like operating system. Most personal computers do not need server processes (unless you want to connect remotely to them), but your question was not restricted to them.
      - boomlinde 5y ago
        From a practical point of view, regardless of the scope of the original question, this is the kind of scenario where you'd really want the restriction. More than a simple administrative task it's a dangerous attack vector to allow any user to launch your httpd or DNS. That being said, check out capabilities(7) in Linux. You can grant an executable the privilege of binding to a low port when run by non-0 uid through setcap. This is a good compromise.
        convolvatron 5y ago
        this whole 'privledged ports' nonsense is left over from a time where some process on another machine running on a low port was somehow to be trusted - because the person running that process was another administrator, and you can generally trust those guys (as opposed to unwashed users). that world didn't last very long, and I wish we could vent some of these designs that didn't pass the test of time.
stingraycharles 5y ago
> Having a user space TCP stack could also decrease the need for [root privileges] as far as setting up the network is concerned. I think it’s important to distinguish between the protocol (TCP) and the hardware device. You would still absolutely need to talk to the device, it’s just that moving a lot of the logic to user space means much less context switching for system calls for the application. I can imagine on Linux you can talk directly to /dev/eth0 if you would want to (in the same way that you can talk to /dev/sda), and then you would be back at square one regarding root privileges.
- monocasa 5y ago
  > I can imagine on Linux you can talk directly to /dev/eth0 if you would want to (in the same way that you can talk to /dev/sda), and then you would be back at square one regarding root privileges. It's a AF_PACKET, SOCK_RAW socket rather than a device file, but yes.
stefan_ 5y ago
The whole "map the PCIE device into userspace process memory" thing is called DPDK (https://www.dpdk.org/ https://www.dpdk.org/)
- ay 5y ago
  And you can combine the two: https://fd.io/docs/vpp/master/whatisvpp/hoststack.html https://fd.io/docs/vpp/master/whatisvpp/hoststack.html And there is a sister project using this tech to get noticeable speed-ups: https://wiki.fd.io/view/VSAP https://wiki.fd.io/view/VSAP Disclaimer: I am involved with the VPP project.
avinassh 5y ago
> The idea of putting the TCP stack in user space is interesting. Indeed! Julia Evans wrote a really nice post explaining the usecases and benefits - https://jvns.ca/blog/2016/06/30/why-do-we-use-the-linux-kernels-tcp-stack/ https://jvns.ca/blog/2016/06/30/why-do-we-use-the-linux-kern...
yakubin 5y ago
You don't need to be root to mount disks, when you have udisks installed (which would be almost all distros by default). See udisksctl(1): <https://manpages.debian.org/buster/udisks2/udisksctl.1.en.html https://manpages.debian.org/buster/udisks2/udisksctl.1.en.ht...>

eatonphil 5y ago

I couldn't make it past serialization/deserialization logic in my own hobbiest TCP/IP stack. Even that part was super buggy. Next time around I'm definitely going to be unit testing more parts otherwise it's too hard for a beginner to get the easy parts right let alone the harder parts. Also, take a look at gvisor's network stack. It's definitely unit tested. https://github.com/google/gvisor/tree/master/pkg/tcpip/link/ethernet https://github.com/google/gvisor/tree/master/pkg/tcpip/link/... (an example)

amscanne 5y ago
This is perhaps a better example: https://github.com/google/gvisor/blob/master/pkg/tcpip/transport/tcp/tcp_test.go https://github.com/google/gvisor/blob/master/pkg/tcpip/trans... Also, some networking tests use separate frameworks (which look more like the setup the original post is describing, since those are needed also), e.g.: https://github.com/google/gvisor/tree/master/test/packetimpact/tests https://github.com/google/gvisor/tree/master/test/packetimpa...

gravypod 5y ago

There's a lot of push back from engineers - especially people at lower levels of the stack - against testing infrastructure. One particularly famous example is Linux. Rather than testing before merging in code, they merge in code and then test the release candidate as a whole. It also seems game developers are extremely against automated testing frameworks as a whole. I've heard many times that it would be impossible to develop an enemy AI in a test-driven way (I did this for a senior project in college - finished the AI before the game was able to even start testing it [0]). I wonder what would need to happen to convince people that: 1. Even if you do something extremely low level, you can draw a distinction between your hardware and the interface that 99% of your software runs at. 2. You can develop complex behaviors iteratively with automated testing just like you can develop complex programs iteratively (tests are just programs). [0] - https://github.com/gravypod/it491-disabler-ai https://github.com/gravypod/it491-disabler-ai

lloydatkinson 5y ago
I agree - it seems it's only excuses that keep certain devs away from testing. Even Factorio manages testing, though its more of an integration test. I'm sure it could be done with unit tests too.
- gravypod 5y ago
  I think even integration tests would be a huge improvement for most teams and software. As long as some automated checks are run before code is merged you're going to save yourself a lot of heart ache.
PezzaDev 5y ago
Automated testing is hard to justify for games because you are not simply finishing features and moving on. You are constantly experimenting. Throwing out features and ideas and reworking them is part of the creative process. Automated testing adds overhead to iteration and isn't free, so you have to be very selective. At the end of the day it is more important for the game to be fun than it is stable.
- 5y ago

jsnell 5y ago

One benefit we discovered with this test framework after the blog post was written was that it made it much more convenient to do fuzzing and differential testing of the TCP stack. The core problem with fuzzing TCP is that there's a lot of incrementally built up state, and everything is extremely timing-dependent. You basically need the fuzzer to have a model of TCP state so that it can effectively explore the state space, which is quite complicated and not something you can do with off-the shelf tools. But once you have a bunch of unit tests designed to put the TCP stack into a specific state + a way of saving and restoring that state, it's really easy to just have snapshot of interesting situations where you can run a fuzzer on the next packet to be transmitted and see what happens.

10000truths 5y ago

It would be nice to have a bring-your-own-I/O TCP stack library that *doesn’t* rely on custom callbacks - something like BearSSL but for TCP, where the stack is just a pure state machine object and the user is responsible for explicitly shunting packets to and from the state machine, retaining control over when and how the I/O is done. Instead of having to define callbacks for retrieving time and consuming packets, why not explicitly pass the timestamp and packet data to a state machine object via a direct function call?

Slix 5y ago
Is Cloudflare's Quiche QUIC library https://github.com/cloudflare/quiche https://github.com/cloudflare/quiche similar to what you're looking for? All I/O must be done by the caller.
- 10000truths 5y ago
  Yeah, that’s the general idea. Essentially a state machine with a send queue and receive queue, and four operations: - Input received raw data - Output received application data - Input application data to send - Output raw data to send Obviously, since TCP connection state is time sensitive, the “raw data” wouldn’t just be the IP packet and headers, but also a time stamp telling the state machine when that packet was received/sent. If you want the state machine to keep track of time even when no packets are being received or sent, there could be an additional operation just to input a timestamp without additional packets. In effect, time is just another input that the user is responsible for feeding to the state machine at sufficiently fine intervals. In practice, you could emulate this pattern with a callback-oriented protocol stack by populating an in-memory send/receive queue in your callback function, but that design can be somewhat inflexible because it forces potentially undesirable constraints, e.g. an extra memory copy that could otherwise be elided.