6 ms·

I love the opinionated writing style. I wish I knew more about this so I could relate to the comments about the "compiler braindamage", "mbuf chain stupidity",

I love the opinionated writing style. I wish I knew more about this so I could relate to the comments about the "compiler braindamage", "mbuf chain stupidity", and "netipl software interrupt bs".

delinka 13y ago
I read those things as sarcasm (e.g. "none of that buffer-overrun-checking garbage") but perhaps there's more to it.
minimax 13y ago
The "compiler braindamage" is that it is generating instructions to load individual registers 4 bytes at a time where it could instead generate instructions that load eight (properly aligned) bytes at a time into two four-byte registers at once. The bytes in question are some fields in the tcp header and some fields in the pcb (which is like all the state related to the TCP connection). These instructions here are loading the header into registers: ld [%i0+4],%l3 ! load packet tcp header fields ld [%i0+8],%l4 ld [%i0+12],%l2 ld [%i0+16],%l0 That's the assembler for this: u_long seq = ((u_long*)ti)[1]; u_long ack = ((u_long*)ti)[2]; u_long flg = ((u_long*)ti)[3]; u_long sum = ((u_long*)ti)[4]; ld loads 4 bytes at a time into a register but there is a sparc instruction ldd that will load 8 bytes into two registers at a time. If the compiler used ldd these four instructions turn into two. That gets us from 33 to 31. I'm not 100% clear which two fields from the pcb can be loaded simultaneously but there is this line ld [%i1+72],%o0 ! compute header checksum and then further down ld [%i1+68],%o0 which I think are: u_long cksum = tp->ph_sum; and if ((flg & FMASK) == tp->pred_flags && seq == tp->rcv_nxt) { Obviously that line is multiple instructions but what I meant is the part where tp->rcv_nxt is loaded into a register. So probably tp->ph_sum and tp->rcv_nxt are adjacent in his version of struct tcpcb and he thinks the compiler should do use the same parallel load instruction (loading two registers from 8 bytes at once) for those fields.
darbelo 13y ago
The "mbuf chain stupidity" refers to the way memory buffers are handled in the kernel. His implementation uses 'pbufs' to store a single packet contiguously in kernel memory. This is in contrast to BSD[1], where a packet could span multiple 'mbufs' which were 'chained' into a linked list. You can "man 9 mbuf" on your nearest BSD derivative to glean a bit more information about what he is opposing. Today's mbufs are not quite the same as they were back then, but the parts that he hated are still there :) [1] "my kernel looks nothing at all like any version of BSD"
- laichzeit0 13y ago
  For a walk through of the BSD TCP/IP source code there's also Steven's TCP/IP illustrated volume 2. The first few chapters painstakingly go through mbufs and exactly how the data structure is implemented and used. Much of this won't apply to current kernels but it's probably close enough for what Van Jacobson is talking about. http://www.amazon.com/TCP-IP-Illustrated-Implementation-Vol/dp/020163354X http://www.amazon.com/TCP-IP-Illustrated-Implementation-Vol/...
chetanahuja 13y ago
mbuf chains [1] (or something like them) are the canonical way of receiving data from the network. You basically "tag" every incoming chunk of data as an mbuf, and keep the data as linked lists of said mbufs (hence the mubf "chain"). Some key properties of incoming network data are that it's: 1) Arriving asynchronously, at arbitrary time points (actually the OS code gets to handle the incoming data in interrupt processing routines) 2) Arriving in arbitrary quantities in each new chunk(as opposed to, say, reading nice, aligned, blocks from the disk) 3) Possibly arriving out of order. Now if you consider the requirements that you have this uncertain sized chunks of data that you (the network stack code) need to parse into (and possibly reorder into) one of many possible protocols then deliver it in a nicely packaged form to a user process (which may or may not be ready to receive this data at this moment) --- and, additionally, there's a lot of pressure to avoid unnecessary memory copies -- you'll be inevitably led towards an architecture that looks somewhat like mbuf chains. The cited writeup and accompanying code has some crucial details hiding in the following snippet "the Packets go in 'pbufs' which are, in general, the property of a particular device." The point I think is that it's special purpose code written for a narrow use/demo case. Not meant to be taken seriously for an actual, general purpose OS. tl;dr mbuf chains are inevitable for general purpose networking. Resistance is futile. [1] https://developer.apple.com/library/mac/documentation/darwin/conceptual/NKEConceptual/SocketManipulation/SocketManipulation.html https://developer.apple.com/library/mac/documentation/darwin...

I love the opinionated writing style. I wish I knew more about this so I could relate to the comments about the "compiler braindamage", "mbuf chain stupidity", and "netipl software interrupt bs".

delinka 13y ago

I read those things as sarcasm (e.g. "none of that buffer-overrun-checking garbage") but perhaps there's more to it.

minimax 13y ago

The "compiler braindamage" is that it is generating instructions to load individual registers 4 bytes at a time where it could instead generate instructions that load eight (properly aligned) bytes at a time into two four-byte registers at once. The bytes in question are some fields in the tcp header and some fields in the pcb (which is like all the state related to the TCP connection). These instructions here are loading the header into registers: ld [%i0+4],%l3 ! load packet tcp header fields ld [%i0+8],%l4 ld [%i0+12],%l2 ld [%i0+16],%l0 That's the assembler for this: u_long seq = ((u_long*)ti)[1]; u_long ack = ((u_long*)ti)[2]; u_long flg = ((u_long*)ti)[3]; u_long sum = ((u_long*)ti)[4]; ld loads 4 bytes at a time into a register but there is a sparc instruction ldd that will load 8 bytes into two registers at a time. If the compiler used ldd these four instructions turn into two. That gets us from 33 to 31. I'm not 100% clear which two fields from the pcb can be loaded simultaneously but there is this line ld [%i1+72],%o0 ! compute header checksum and then further down ld [%i1+68],%o0 which I think are: u_long cksum = tp->ph_sum; and if ((flg & FMASK) == tp->pred_flags && seq == tp->rcv_nxt) { Obviously that line is multiple instructions but what I meant is the part where tp->rcv_nxt is loaded into a register. So probably tp->ph_sum and tp->rcv_nxt are adjacent in his version of struct tcpcb and he thinks the compiler should do use the same parallel load instruction (loading two registers from 8 bytes at once) for those fields.

darbelo 13y ago

The "mbuf chain stupidity" refers to the way memory buffers are handled in the kernel. His implementation uses 'pbufs' to store a single packet contiguously in kernel memory. This is in contrast to BSD[1], where a packet could span multiple 'mbufs' which were 'chained' into a linked list. You can "man 9 mbuf" on your nearest BSD derivative to glean a bit more information about what he is opposing. Today's mbufs are not quite the same as they were back then, but the parts that he hated are still there :) [1] "my kernel looks nothing at all like any version of BSD"

laichzeit0 13y ago
For a walk through of the BSD TCP/IP source code there's also Steven's TCP/IP illustrated volume 2. The first few chapters painstakingly go through mbufs and exactly how the data structure is implemented and used. Much of this won't apply to current kernels but it's probably close enough for what Van Jacobson is talking about. http://www.amazon.com/TCP-IP-Illustrated-Implementation-Vol/dp/020163354X http://www.amazon.com/TCP-IP-Illustrated-Implementation-Vol/...

chetanahuja 13y ago

mbuf chains [1] (or something like them) are the canonical way of receiving data from the network. You basically "tag" every incoming chunk of data as an mbuf, and keep the data as linked lists of said mbufs (hence the mubf "chain"). Some key properties of incoming network data are that it's: 1) Arriving asynchronously, at arbitrary time points (actually the OS code gets to handle the incoming data in interrupt processing routines) 2) Arriving in arbitrary quantities in each new chunk(as opposed to, say, reading nice, aligned, blocks from the disk) 3) Possibly arriving out of order. Now if you consider the requirements that you have this uncertain sized chunks of data that you (the network stack code) need to parse into (and possibly reorder into) one of many possible protocols then deliver it in a nicely packaged form to a user process (which may or may not be ready to receive this data at this moment) --- and, additionally, there's a lot of pressure to avoid unnecessary memory copies -- you'll be inevitably led towards an architecture that looks somewhat like mbuf chains. The cited writeup and accompanying code has some crucial details hiding in the following snippet "the Packets go in 'pbufs' which are, in general, the property of a particular device." The point I think is that it's special purpose code written for a narrow use/demo case. Not meant to be taken seriously for an actual, general purpose OS. tl;dr mbuf chains are inevitable for general purpose networking. Resistance is futile. [1] https://developer.apple.com/library/mac/documentation/darwin/conceptual/NKEConceptual/SocketManipulation/SocketManipulation.html https://developer.apple.com/library/mac/documentation/darwin...