8 ms·

Streaming video on 10 Gigabit Ethernet and beyond

80 points by howsilly 11y ago

jand 11y ago
My state of knowledge leads me to think, that bypassing the kernel requires some non-blob network drivers with which you can tinker around. Am i mistaken? So right now, i am missing the information on what kind of NIC they were using. Any thoughts or comments on that HN-community? What vendor and product model would be a reasonable entry point for such endeavours? Answers very much appreciated.
- xtacy 11y ago
  In the past, some of my colleagues have used Intel's 82599 NICs for kernel bypass. Their Linux driver is quite good, they have a DPDK platform for developing user-space apps to directly access ring buffers on the NIC, and if you do a quick search, you should be able to find examples online. Cloudflare wrote a blog post recently about accelerated packet IO and their post mentions the 82599 NIC: https://blog.cloudflare.com/kernel-bypass/ https://blog.cloudflare.com/kernel-bypass/.
  - gonzo 11y ago
    via netmap, yes.
- Thaxll 11y ago
  Some Solarflare cards? http://www.openonload.org/ http://www.openonload.org/
  - gonzo 11y ago
    That's a blob driver.
- cpach 11y ago
  You might be interested in this writeup by Luke Gorrie: https://github.com/lukego/blog/issues/13 https://github.com/lukego/blog/issues/13 His project, Snabb Switch, also utilize the Intel 82599 10G NIC that 'xtacy mentioned.
- pinhead 11y ago
  The netmap project (http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/) or intel's dpdk help with this and don't require super fancy non-standard NICs.
  - gonzo 11y ago
    Just a modified driver and a NIC that supports 'rings'.
nly 11y ago
Kind of arcs back to the days when people were putting HTTP servers in kernel space. Slightly different tac though
- gonzo 11y ago
  but the same result, reduce copies.
  - nimish 11y ago
    Reducing context switches is probably just as important.
    - nly 11y ago
      Context switches in to kernel land are a lot cheaper on x86 than they used to be.
p1esk 11y ago
Is a single CPU core able to process 4k/50fps video stream? Or is there no need for any processing, other than encapsulating it into data packets for sending to the network card?
- revelation 11y ago
  Clearly not, which is exactly why kernel bypass is such a huge red flag: it's only possible (rather, useful) if you aren't doing anything sensible with the data anyway, or the kernel overhead would be tiny compared to the processing. Use the right tool for the job and don't funnel network data through your instruction pipeline. When they realized that for memory, they called it "DMA", and when graphics was scaling up, we created the GPU.
  - p1esk 11y ago
    What would be "the right tool for the job" here? If no processing is needed, why is the kernel bypass is "a huge red flag"?
  - detaro 11y ago
    They probably could have used cards/systems capable of RDMA instead, but I don't see what's wrong with their approach.
  - greglindahl 11y ago
    The networks used by supercomputers have have kernel bypass with DMA for more than a decade... and they do a lot of processing on the data, too. Check out Infiniband and Intel's Omni-Path for modern examples. Or the Cray T3E (1995), which had an excellent network with a user-level DMA engine that only did 8- or 64-byte transfers.
  - ryanlol 11y ago
    Are you kidding me? How could not bypassing the kernel POSSIBLY be better here? It's infinitely harder for kernel devs to optimize these scenarios than for userland devs.
- gcb0 11y ago
mixmastamyk 11y ago
Interesting, reminds me of a related question. I've looked recently for 10 gig ethernet on a new laptop and haven't been able to find it. I know it is overkill, its just that it has been about ten years already, isn't it cheap enough yet? Can't a modern ssd keep up with it?
- _wmd 11y ago
  Probably a combination of lack of need until recently (125MB/sec is more than fine to saturate a classic disk), parts cost and power consumption. I can't think of anything except connecting to a _fast_ SAN that would require a 10GBE port in a laptop. Maybe something specialized for a network engineer, but even then it's probably easier to buy dedicated equipment for line rate port monitoring
  - mixmastamyk 11y ago
    The use case is high-def video editing, which people could do on laptops, and there are thunderbolt connections at that speed. I found this: http://www.fastestssd.com/featured/ssd-rankings-the-fastest-solid-state-drives/#pcie http://www.fastestssd.com/featured/ssd-rankings-the-fastest-... Pushing 3000MB/s, which is 3GB/s which should be times 8 for gigabits, I think it should be viable now, no?
  - noja 11y ago
    > I can't think of anything except connecting to a _fast_ SAN that would require a 10GBE port in a laptop. An off-the-shelf consumer raid-5 nas would require more than a gigabit port (so a 10 gig port would be needed).
  - jamesblonde 11y ago
    A very cheap RAID5 setup with 4 spinning disks and a filesystem like ZFS or btrfs should get you about 3-500 MB/s, so 10 GbE is good for that setup on nodes.
    - trentnelson 11y ago
      The only way you're getting 3-500MB/s on a 4-disk ZFS RAID5/raidz is with very, very fast SSDs and a very, very fast CPU. Not exactly "very cheap". (The compute and I/O overhead for raidz is significant.)
pcunite 11y ago
Linus Tech Tips talked about the difficulties in getting 10 Gigabit working. https://youtu.be/D03t890dKTU https://youtu.be/D03t890dKTU
maxhou 11y ago
from the article: > Each core needs to generate a few thousand data packets per second, because Ethernet packets typically contain up to 1500 bytes. This gives the CPU around 100 microseconds to process each packet. No it doesn't, not when using TCP Segmentation Offload (TSO) This only works for a particular use-case: sending static data using TCP, but this is the most common use-case since a typical "video streaming server" is actually a simple HTTP server that serves static MP4/MPEG-TS data. for each connected client this is what happens - nginx/apache does sendfile(file, sock, off, <large_number>) - kernel issue large (> 10kB) DMA read to the file storage backend into a set of memory pages and wait for completion - kernel allocates/clone a small IP/TCP header (40 bytes) - kernel gives that small header + set of memory pages to network card, which will segment and create those 1500 bytes packets and send them on wire if you have a lot of RAM, the read from storage could even be skipped because the previously read data pages are kept in the page-cache with a LRU approach. (help if clients are requesting the same file). you can easily saturate a 10G link with spare CPU cycles on cheap hardware with that approach, no need to bypass anything.
kierank 11y ago
All this talk about TCP and HTTP streaming when they are actually trying to do low-latency UDP for broadcast production. (When you have a hammer (web technologies) everything looks like a nail I guess)

80 points by howsilly 11y ago

jand 11y ago

My state of knowledge leads me to think, that bypassing the kernel requires some non-blob network drivers with which you can tinker around. Am i mistaken? So right now, i am missing the information on what kind of NIC they were using. Any thoughts or comments on that HN-community? What vendor and product model would be a reasonable entry point for such endeavours? Answers very much appreciated.

xtacy 11y ago
In the past, some of my colleagues have used Intel's 82599 NICs for kernel bypass. Their Linux driver is quite good, they have a DPDK platform for developing user-space apps to directly access ring buffers on the NIC, and if you do a quick search, you should be able to find examples online. Cloudflare wrote a blog post recently about accelerated packet IO and their post mentions the 82599 NIC: https://blog.cloudflare.com/kernel-bypass/ https://blog.cloudflare.com/kernel-bypass/.
- gonzo 11y ago
  via netmap, yes.
Thaxll 11y ago
Some Solarflare cards? http://www.openonload.org/ http://www.openonload.org/
- gonzo 11y ago
  That's a blob driver.
cpach 11y ago
You might be interested in this writeup by Luke Gorrie: https://github.com/lukego/blog/issues/13 https://github.com/lukego/blog/issues/13 His project, Snabb Switch, also utilize the Intel 82599 10G NIC that 'xtacy mentioned.
pinhead 11y ago
The netmap project (http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/) or intel's dpdk help with this and don't require super fancy non-standard NICs.
- gonzo 11y ago
  Just a modified driver and a NIC that supports 'rings'.

nly 11y ago

Kind of arcs back to the days when people were putting HTTP servers in kernel space. Slightly different tac though

gonzo 11y ago
but the same result, reduce copies.
- nimish 11y ago
  Reducing context switches is probably just as important.
  - nly 11y ago
    Context switches in to kernel land are a lot cheaper on x86 than they used to be.

p1esk 11y ago

Is a single CPU core able to process 4k/50fps video stream? Or is there no need for any processing, other than encapsulating it into data packets for sending to the network card?

revelation 11y ago
Clearly not, which is exactly why kernel bypass is such a huge red flag: it's only possible (rather, useful) if you aren't doing anything sensible with the data anyway, or the kernel overhead would be tiny compared to the processing. Use the right tool for the job and don't funnel network data through your instruction pipeline. When they realized that for memory, they called it "DMA", and when graphics was scaling up, we created the GPU.
- p1esk 11y ago
  What would be "the right tool for the job" here? If no processing is needed, why is the kernel bypass is "a huge red flag"?
- detaro 11y ago
  They probably could have used cards/systems capable of RDMA instead, but I don't see what's wrong with their approach.
- greglindahl 11y ago
  The networks used by supercomputers have have kernel bypass with DMA for more than a decade... and they do a lot of processing on the data, too. Check out Infiniband and Intel's Omni-Path for modern examples. Or the Cray T3E (1995), which had an excellent network with a user-level DMA engine that only did 8- or 64-byte transfers.
- ryanlol 11y ago
  Are you kidding me? How could not bypassing the kernel POSSIBLY be better here? It's infinitely harder for kernel devs to optimize these scenarios than for userland devs.
gcb0 11y ago

mixmastamyk 11y ago

Interesting, reminds me of a related question. I've looked recently for 10 gig ethernet on a new laptop and haven't been able to find it. I know it is overkill, its just that it has been about ten years already, isn't it cheap enough yet? Can't a modern ssd keep up with it?

_wmd 11y ago
Probably a combination of lack of need until recently (125MB/sec is more than fine to saturate a classic disk), parts cost and power consumption. I can't think of anything except connecting to a _fast_ SAN that would require a 10GBE port in a laptop. Maybe something specialized for a network engineer, but even then it's probably easier to buy dedicated equipment for line rate port monitoring
- mixmastamyk 11y ago
  The use case is high-def video editing, which people could do on laptops, and there are thunderbolt connections at that speed. I found this: http://www.fastestssd.com/featured/ssd-rankings-the-fastest-solid-state-drives/#pcie http://www.fastestssd.com/featured/ssd-rankings-the-fastest-... Pushing 3000MB/s, which is 3GB/s which should be times 8 for gigabits, I think it should be viable now, no?
- noja 11y ago
  > I can't think of anything except connecting to a _fast_ SAN that would require a 10GBE port in a laptop. An off-the-shelf consumer raid-5 nas would require more than a gigabit port (so a 10 gig port would be needed).
- jamesblonde 11y ago
  A very cheap RAID5 setup with 4 spinning disks and a filesystem like ZFS or btrfs should get you about 3-500 MB/s, so 10 GbE is good for that setup on nodes.
  - trentnelson 11y ago
    The only way you're getting 3-500MB/s on a 4-disk ZFS RAID5/raidz is with very, very fast SSDs and a very, very fast CPU. Not exactly "very cheap". (The compute and I/O overhead for raidz is significant.)

pcunite 11y ago

Linus Tech Tips talked about the difficulties in getting 10 Gigabit working. https://youtu.be/D03t890dKTU https://youtu.be/D03t890dKTU

maxhou 11y ago

from the article: > Each core needs to generate a few thousand data packets per second, because Ethernet packets typically contain up to 1500 bytes. This gives the CPU around 100 microseconds to process each packet. No it doesn't, not when using TCP Segmentation Offload (TSO) This only works for a particular use-case: sending static data using TCP, but this is the most common use-case since a typical "video streaming server" is actually a simple HTTP server that serves static MP4/MPEG-TS data. for each connected client this is what happens - nginx/apache does sendfile(file, sock, off, <large_number>) - kernel issue large (> 10kB) DMA read to the file storage backend into a set of memory pages and wait for completion - kernel allocates/clone a small IP/TCP header (40 bytes) - kernel gives that small header + set of memory pages to network card, which will segment and create those 1500 bytes packets and send them on wire if you have a lot of RAM, the read from storage could even be skipped because the previously read data pages are kept in the page-cache with a LRU approach. (help if clients are requesting the same file). you can easily saturate a 10G link with spare CPU cycles on cheap hardware with that approach, no need to bypass anything.

kierank 11y ago

All this talk about TCP and HTTP streaming when they are actually trying to do low-latency UDP for broadcast production. (When you have a hammer (web technologies) everything looks like a nail I guess)