4 ms·

There should be (is there?) an OS designed around zero copy principles. I want to get packets from the network delivered into a block of memory, and then all su

by clord 10y ago

There should be (is there?) an OS designed around zero copy principles. I want to get packets from the network delivered into a block of memory, and then all subsequent parsing and reading occurs on that memory, with OS provided framing and fine grained policy determining how things are interpreted by userspace. Something like the memory trick used in this virtual ring buffer but far more powerful.

monocasa 10y ago
I wouldn't say it's "designed around" it, but FreeBSD's netmap(4) is pretty similar to what you're talking about.
colanderman 10y ago
See Intel's DPDK and Tilera's mPIPE. You can't really do this with TCP though, without changing the hardware. Most modern network adapters support only a scatter/gather list mechanism for receiving/sending packets, but this doesn't work for receiving TCP payloads into contiguous locations, since TCP's data model is a byte stream. With UDP and SCTP you can do full zero-copy just fine.
- gpderetta 10y ago
  An option is to expose the the fragmented packets to the application and let it handle it. For some applications, application levels packets are rarely fragmented in the tcp stream so 0 copy handling of the incoming tcp stream is possible and fall back once in a while to a compacting copy is an option. A bigger problem is that on the send side you can't reuse the a 0 copy buffer until the remote ack has arrived for that tcp segment which complicates the interface between the application and the tcp stack.
viraptor 10y ago
Apart from very purpose specific OSes I don't think anything is designed around it. But there's a lot of work done to enable it on current systems. Linux for example has sendfile(), splice(), socket options for packet mmap (https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt https://www.kernel.org/doc/Documentation/networking/packet_m...), COW on a lot of filesystems (which means zero-copy file copy), etc. Sometimes you have to do a bit more work to achieve the zero-copy behaviour, but it's available is many cases.
signa11 10y ago
> I want to get packets from the network delivered into a block of memory, and then all subsequent parsing and reading occurs on that memory, with OS provided framing and fine grained policy determining how things are interpreted by userspace. well, as others have pointed out, dpdk / netmap are a couple of choices available for bypassing the kernel network stack completely, and get stuff directly into userland. once things are there, you can do all protocol specific parsing etc. this essentially implies that what you 'see' in memory are mmap'ped ring buffers, not the so called PDU's that you are looking for. the number of protocol-stacks available in userland is, however, to the best of my knowledge, quite limited. for example, there are couple of folks exploring the option of exporting linux-kernel's networking stack as a library under the N.U.S.E (networking in user space) project. details available here: http://people.netfilter.org/pablo/netdev0.1/papers/Library-Operating-System-with-Mainline-Linux-Network-Stack.pdf http://people.netfilter.org/pablo/netdev0.1/papers/Library-O... (https://github.com/libos-nuse/net-next-nuse https://github.com/libos-nuse/net-next-nuse), if you have a very specific application in mind that you want to tackle e.g. let's say acting as a transparent proxy, forwarding l2 etc. then things are kind of doable. implementing a generic very high performance stacks (where 2-3 cache-misses will wipe out your timing budgets) is kind of hard, imho...
plaguuuuuu 10y ago
What happens when two apps wanna do different things with that same packet?
- striking 10y ago
  Then a copy is made and written to instead. Or maybe the original is unchanged but both apps write to a different place. Additionally, any unchanged fields could be inherited prototypically from the original datum. You (necessarily) lose zero-copy when the dataflow isn't one-input-one-output.

by clord 10y ago

monocasa 10y ago

I wouldn't say it's "designed around" it, but FreeBSD's netmap(4) is pretty similar to what you're talking about.

colanderman 10y ago

See Intel's DPDK and Tilera's mPIPE. You can't really do this with TCP though, without changing the hardware. Most modern network adapters support only a scatter/gather list mechanism for receiving/sending packets, but this doesn't work for receiving TCP payloads into contiguous locations, since TCP's data model is a byte stream. With UDP and SCTP you can do full zero-copy just fine.

gpderetta 10y ago
An option is to expose the the fragmented packets to the application and let it handle it. For some applications, application levels packets are rarely fragmented in the tcp stream so 0 copy handling of the incoming tcp stream is possible and fall back once in a while to a compacting copy is an option. A bigger problem is that on the send side you can't reuse the a 0 copy buffer until the remote ack has arrived for that tcp segment which complicates the interface between the application and the tcp stack.

viraptor 10y ago

Apart from very purpose specific OSes I don't think anything is designed around it. But there's a lot of work done to enable it on current systems. Linux for example has sendfile(), splice(), socket options for packet mmap (https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt https://www.kernel.org/doc/Documentation/networking/packet_m...), COW on a lot of filesystems (which means zero-copy file copy), etc. Sometimes you have to do a bit more work to achieve the zero-copy behaviour, but it's available is many cases.

signa11 10y ago

> I want to get packets from the network delivered into a block of memory, and then all subsequent parsing and reading occurs on that memory, with OS provided framing and fine grained policy determining how things are interpreted by userspace. well, as others have pointed out, dpdk / netmap are a couple of choices available for bypassing the kernel network stack completely, and get stuff directly into userland. once things are there, you can do all protocol specific parsing etc. this essentially implies that what you 'see' in memory are mmap'ped ring buffers, not the so called PDU's that you are looking for. the number of protocol-stacks available in userland is, however, to the best of my knowledge, quite limited. for example, there are couple of folks exploring the option of exporting linux-kernel's networking stack as a library under the N.U.S.E (networking in user space) project. details available here: http://people.netfilter.org/pablo/netdev0.1/papers/Library-Operating-System-with-Mainline-Linux-Network-Stack.pdf http://people.netfilter.org/pablo/netdev0.1/papers/Library-O... (https://github.com/libos-nuse/net-next-nuse https://github.com/libos-nuse/net-next-nuse), if you have a very specific application in mind that you want to tackle e.g. let's say acting as a transparent proxy, forwarding l2 etc. then things are kind of doable. implementing a generic very high performance stacks (where 2-3 cache-misses will wipe out your timing budgets) is kind of hard, imho...

plaguuuuuu 10y ago

What happens when two apps wanna do different things with that same packet?

striking 10y ago
Then a copy is made and written to instead. Or maybe the original is unchanged but both apps write to a different place. Additionally, any unchanged fields could be inherited prototypically from the original datum. You (necessarily) lose zero-copy when the dataflow isn't one-input-one-output.