2 ms·
You can do zero copy I/O on Linux using vmsplice+extensions, and on FreeBSD using regular read/write calls. Both can do DMA with user space buffers with much le
by wahern 10y ago
You can do zero copy I/O on Linux using vmsplice+extensions, and on FreeBSD using regular read/write calls. Both can do DMA with user space buffers with much less disruption to traditional design patterns for implementing network daemons.
I think the real benefit of DPDK and netmap is that you're avoiding all the logic of the existing IP stacks, not to mention firewall rules, etc. At the same time you're now responsible for all of that. And IMO the amazing throughput most people claim to see with DPDK is a result of simply neglecting to implement all the hard logic which makes the Internet actually work. All the weirdness, head scratching, and hair pulling these solutions cause is an externality engineers will never care about, and in most cases probably be oblivious about. The exception to this state of affairs is when they're doing read-only packet sniffing and filtering, simply passing the packets back out another interface.
If most of what you care about is avoiding the cost of the kernel/userspace split, then you can just use NetBSD Rump or similar unikernel frameworks.
- cm3 10y agoDo you know if the most optimal FreeBSD zero copy support is only available for Tigeon NICs? I couldn't quite figure it out from reading the following pages: - http://www.freebsd.org/cgi/man.cgi?query=zero_copy http://www.freebsd.org/cgi/man.cgi?query=zero_copy - http://www.kegel.com/c10k.html#zerocopy http://www.kegel.com/c10k.html#zerocopy - http://people.freebsd.org/~ken/zero_copy/ http://people.freebsd.org/~ken/zero_copy/
- wahern 10y agoI don't actually know. I never went down the rabbit hole of zero-copy, only watched various teams crash and burn. I've had success simply reducing the number of data copies in userspace. I use tools like Ragel, lean fifo buffers in C with moving read and write windows (i.e. slices), etc. For example, for a [now defunct] startup, I implemented a real-time streaming radio transcoder which inserted targeted, per-listener, dynamically selected ad spots into live streams. So, it would take an existing radio source (Flash, ICY, MMS, etc), transcode the codec and format to suite the listener, and when it detected ad spots select and insert a targeted ad. On a _single_ E3 Haswell core (core, not chip) I was able to transcode 5,000 streams in real-time, each with dynamically selected and inserted ad spots, cycling between 30 seconds of the stream and a new 30 second ad for the stress test. All software; no hardware or even SIMD (other than what GCC could squeeze out). The Linux kernel was spending more CPU time handling interrupts and pushing out the packets then my daemon. At that point I knew I was already at the 80% solution and moved on to more feature development. I knew that after fiddling with the kernel I'd have more than enough performance. FFMpeg, GStreamer, etc couldn't even come close to that. I had written all my own stream parsers and writers, both so I had control over buffer management, but also because I needed frame-level control for splicing in ad spots. The only libraries I used were for the low-level codecs and for resampling. Notably, there was actually more copying then you'd think; enough to keep the interfaces relatively clean. The key, apparently, was maintaining data locality. Some libraries go through extraordinary lengths for zero-copy, but they end up with so much pointer indirection that it's a net loss.
- chmike 10y agoAvoiding the logic of tcp stack processing does indeed come into play. My colleague process data sent through a tcp connection in a LAN. So he basically ignores all the tcp header of the incomming data. He simply checks the source ip. He will test mTCP to get a full fledged tcp/ip stack so he will be able to measure it's overhead.