10 ms·

Turtles on the Wire: Understanding How the OS Uses the Modern NIC (2016)

300 points by nz 10y ago

bluetech 10y ago
Interesting article. Another related article I found interesting: https://www.coverfire.com/articles/queueing-in-the-linux-network-stack/ https://www.coverfire.com/articles/queueing-in-the-linux-net... Discusses some of the queues in the Linux network stack.
tedunangst 10y ago
Buggy firmware with edge cases is putting it mildly. I suppose a checksum of 0000 or ffff is technically an edge case, but not all that uncommon, and a pretty popular thing to get wrong.
- gonzo 10y ago
  Even the standards got it wrong. https://tools.ietf.org/html/rfc1624 https://tools.ietf.org/html/rfc1624
pthreads 10y ago
This is a very useful write-up. I thoroughly enjoyed it i.e. found it informative. Thank you.
drewg123 10y ago
The biggest innovation that I've seen in quite some time in terms of making creative use of hardware offloads is Hans Petter Selasky's RSS assisted LRO in FreeBSD. On our workloads (~100K connections, 16 core / 32 HTT FreeBSD based 100GbE CDN server) LRO was rather ineffective because there were roughly 3K connections / rx queue. Even with large interrupt coalescing parameters and large ring sizes, the odds of encountering 2 packets from the same connection within a few packets of each other, even in a group of 1000 or more, are rather small. The first idea we had was to use a hash table to aggregate flows. This helped, but had the draw back of a much higher cache footprint. Hps had the idea that we could sort packets by RSS hash ID before passing them to LRO. This would put packets from the same connection adjacent to each other, thereby allowing the LRO without a hash table to work. Our LRO aggregation rate went from ~1.1:1 to well over 2:1, and we reduce CPU use by roughly 10%. This code is in FreeBSD-current right now (see tcp_lro_queue_mbuf())
- pebblexe 10y ago
  FreeBSD also has netmap https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
ams6110 10y ago
As an aside, is anyone using the Joyent cloud stuff in production? Any good comparisons to Openstack? Looking for something easier to manage.
policedemil 10y ago
Great article! A lot of this is way beyond me, but I'm generally interested in the process of how a NIC filters based on MAC addresses. I'm in the humanities and certain scholars working with culture and technology love to make a huge deal about data leakage and how intertwined we all are precisely because you can put a NIC in promiscuous mode and cap packets that weren't meant for you. The whole point is that because your NIC is constantly receiving data meant for others (i.e. because it's filtering the MAC addresses), something like privacy on networks is always problematic. I've always found the whole point somewhat overstated. So, could anyone explain real quick the process of how a NIC decides whether a packet/frame is actually bound for it or link some good resources? For example, does the NIC automatically store the frame/packet in a buffer, then read the header, and then decide to discard? Or can it read the header before storing the rest of the frame? How much has been read at the point the NIC decides to drop it or move it up the stack? Reading all of every packet seems improbable to me because if it were the case, laptop 1 (awake but not downloading anything) would experience significant battery drain due to constantly filtering network traffic that was meant for laptop 2. I'm not sure that really maps to my experience. Also, I assume there are also differences for LAN vs WiFi? Any help on the matter would be greatly appreciated! I've tried google diving on this question many times before and it's really hard to find much on it.
- eru 10y ago
  To answer the first point you brought up: privacy does not depend on who reads your network packages---it depend on encryption.
- javajosh 10y ago
  The serial nature of the physical connection cannot be overstated. Bits flow to your NIC one at a time. A 1Gb/s NIC is detecting a billion wiggles in voltage per second. Structure is imposed on these wiggles in stages: first, the A/D conversion happens, making the voltage wiggles 1's and 0's, then ethernet framing, then IP packet parsing, then TCP packet/ordering, then the application handles IO (and can, and often does, define even more structure, such as HTTP). You might look up the OSI network layering model (or the OSI 7-layer burrito, as I call it). My understanding is that MAC filtering happens after ethernet framing, and before putting into the ring via DMA, and packets failing that test do not generate interrupts. Your NIC hardware is choosing to ignore packets not addressed to it because, generally, it's pretty useless to listen in on other people's packets. Especially these days when your most likely to capture HTTPS encrypted data.
en4bz 10y ago
This is why I'm really hoping RDMA [1] will catch on soon. It would be great if there was a cloud provider that would enable this feature on some of their offerings. Amazon has done something similar by allowing kernel bypass via DPDK [2] with their ENA offering but Kernel bypass is inferior to RDMA in so many ways IMO. At this point we have 200Gbit/s NICs being provided by Mellanox [3]. CPUs aren't getting any faster and the scale out approach is extremely difficult to get right without going across NUMA domains [4]. Based on the progression of CPUs lately there just isn't going to be enough time to process all these packets AND have time left over to actually run your application. There's a lot of work focusing on data locality at the moment but at this point it's still not fool proof and the work that has been done is woefully under documented. As the article mentioned we've already added a bunch of hardware offloads. RDMA is just a continuation of these offloads but unfortunately it requires some minor changes on the application side to take advantage of which is why it's probably been slow to be adopted. RDMA has so many great applications for data-transfer for backend services. Whether it's queries between a web-server and a DB, replication/clustering of DBs, or micro-service fabric with micro seconds latency. Overall there's a lot of low hanging fruit that could be optimized with RDMA. [1] https://en.wikipedia.org/wiki/Remote_direct_memory_access https://en.wikipedia.org/wiki/Remote_direct_memory_access [2] http://dpdk.org/doc/nics http://dpdk.org/doc/nics [3] http://www.mellanox.com/page/products_dyn?product_family=266&mtag=connectx_6_en_card http://www.mellanox.com/page/products_dyn?product_family=266... [4] http://rhelblog.redhat.com/2015/09/29/pushing-the-limits-of-kernel-networking/ http://rhelblog.redhat.com/2015/09/29/pushing-the-limits-of-...
- kev009 10y ago
  This has been ongoing debate since the beginning. RDMA means running a large amount of protocol in silicon and firmware. Can you imagine what might go wrong? In the FreeBSD community it is somewhat common knowledge that Chelsio is the best Ethernet vendor, mainly because their driver isn't a trainwreck like the dominant Intel (which share BSD HAL code on BSD/Linux/Illumos and are quite similar) If you are alright with the trade offs of a firmware stack, Chelsio do have some pretty slick TCP full offloads that require no substantial application changes: http://www.chelsio.com/wp-content/uploads/resources/T6-100G-DDP-FreeBSD.pdf http://www.chelsio.com/wp-content/uploads/resources/T6-100G-...

300 points by nz 10y ago

Interesting article. Another related article I found interesting: https://www.coverfire.com/articles/queueing-in-the-linux-network-stack/ https://www.coverfire.com/articles/queueing-in-the-linux-net... Discusses some of the queues in the Linux network stack.

tedunangst 10y ago

Buggy firmware with edge cases is putting it mildly. I suppose a checksum of 0000 or ffff is technically an edge case, but not all that uncommon, and a pretty popular thing to get wrong.

gonzo 10y ago
Even the standards got it wrong. https://tools.ietf.org/html/rfc1624 https://tools.ietf.org/html/rfc1624

pthreads 10y ago

This is a very useful write-up. I thoroughly enjoyed it i.e. found it informative. Thank you.

drewg123 10y ago

The biggest innovation that I've seen in quite some time in terms of making creative use of hardware offloads is Hans Petter Selasky's RSS assisted LRO in FreeBSD. On our workloads (~100K connections, 16 core / 32 HTT FreeBSD based 100GbE CDN server) LRO was rather ineffective because there were roughly 3K connections / rx queue. Even with large interrupt coalescing parameters and large ring sizes, the odds of encountering 2 packets from the same connection within a few packets of each other, even in a group of 1000 or more, are rather small. The first idea we had was to use a hash table to aggregate flows. This helped, but had the draw back of a much higher cache footprint. Hps had the idea that we could sort packets by RSS hash ID before passing them to LRO. This would put packets from the same connection adjacent to each other, thereby allowing the LRO without a hash table to work. Our LRO aggregation rate went from ~1.1:1 to well over 2:1, and we reduce CPU use by roughly 10%. This code is in FreeBSD-current right now (see tcp_lro_queue_mbuf())

pebblexe 10y ago
FreeBSD also has netmap https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4 https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4

ams6110 10y ago

As an aside, is anyone using the Joyent cloud stuff in production? Any good comparisons to Openstack? Looking for something easier to manage.

policedemil 10y ago

Great article! A lot of this is way beyond me, but I'm generally interested in the process of how a NIC filters based on MAC addresses. I'm in the humanities and certain scholars working with culture and technology love to make a huge deal about data leakage and how intertwined we all are precisely because you can put a NIC in promiscuous mode and cap packets that weren't meant for you. The whole point is that because your NIC is constantly receiving data meant for others (i.e. because it's filtering the MAC addresses), something like privacy on networks is always problematic. I've always found the whole point somewhat overstated. So, could anyone explain real quick the process of how a NIC decides whether a packet/frame is actually bound for it or link some good resources? For example, does the NIC automatically store the frame/packet in a buffer, then read the header, and then decide to discard? Or can it read the header before storing the rest of the frame? How much has been read at the point the NIC decides to drop it or move it up the stack? Reading all of every packet seems improbable to me because if it were the case, laptop 1 (awake but not downloading anything) would experience significant battery drain due to constantly filtering network traffic that was meant for laptop 2. I'm not sure that really maps to my experience. Also, I assume there are also differences for LAN vs WiFi? Any help on the matter would be greatly appreciated! I've tried google diving on this question many times before and it's really hard to find much on it.

eru 10y ago
To answer the first point you brought up: privacy does not depend on who reads your network packages---it depend on encryption.
javajosh 10y ago
The serial nature of the physical connection cannot be overstated. Bits flow to your NIC one at a time. A 1Gb/s NIC is detecting a billion wiggles in voltage per second. Structure is imposed on these wiggles in stages: first, the A/D conversion happens, making the voltage wiggles 1's and 0's, then ethernet framing, then IP packet parsing, then TCP packet/ordering, then the application handles IO (and can, and often does, define even more structure, such as HTTP). You might look up the OSI network layering model (or the OSI 7-layer burrito, as I call it). My understanding is that MAC filtering happens after ethernet framing, and before putting into the ring via DMA, and packets failing that test do not generate interrupts. Your NIC hardware is choosing to ignore packets not addressed to it because, generally, it's pretty useless to listen in on other people's packets. Especially these days when your most likely to capture HTTPS encrypted data.

en4bz 10y ago

This is why I'm really hoping RDMA [1] will catch on soon. It would be great if there was a cloud provider that would enable this feature on some of their offerings. Amazon has done something similar by allowing kernel bypass via DPDK [2] with their ENA offering but Kernel bypass is inferior to RDMA in so many ways IMO. At this point we have 200Gbit/s NICs being provided by Mellanox [3]. CPUs aren't getting any faster and the scale out approach is extremely difficult to get right without going across NUMA domains [4]. Based on the progression of CPUs lately there just isn't going to be enough time to process all these packets AND have time left over to actually run your application. There's a lot of work focusing on data locality at the moment but at this point it's still not fool proof and the work that has been done is woefully under documented. As the article mentioned we've already added a bunch of hardware offloads. RDMA is just a continuation of these offloads but unfortunately it requires some minor changes on the application side to take advantage of which is why it's probably been slow to be adopted. RDMA has so many great applications for data-transfer for backend services. Whether it's queries between a web-server and a DB, replication/clustering of DBs, or micro-service fabric with micro seconds latency. Overall there's a lot of low hanging fruit that could be optimized with RDMA. [1] https://en.wikipedia.org/wiki/Remote_direct_memory_access https://en.wikipedia.org/wiki/Remote_direct_memory_access [2] http://dpdk.org/doc/nics http://dpdk.org/doc/nics [3] http://www.mellanox.com/page/products_dyn?product_family=266&mtag=connectx_6_en_card http://www.mellanox.com/page/products_dyn?product_family=266... [4] http://rhelblog.redhat.com/2015/09/29/pushing-the-limits-of-kernel-networking/ http://rhelblog.redhat.com/2015/09/29/pushing-the-limits-of-...

kev009 10y ago
This has been ongoing debate since the beginning. RDMA means running a large amount of protocol in silicon and firmware. Can you imagine what might go wrong? In the FreeBSD community it is somewhat common knowledge that Chelsio is the best Ethernet vendor, mainly because their driver isn't a trainwreck like the dominant Intel (which share BSD HAL code on BSD/Linux/Illumos and are quite similar) If you are alright with the trade offs of a firmware stack, Chelsio do have some pretty slick TCP full offloads that require no substantial application changes: http://www.chelsio.com/wp-content/uploads/resources/T6-100G-DDP-FreeBSD.pdf http://www.chelsio.com/wp-content/uploads/resources/T6-100G-...