10 ms·

High Speed Networking: Open Sourcing our Kernel Bypass Work

376 points by XzetaU8 9y ago

peterwwillis 9y ago
There's a lot of orgs out there developing broadcast IP tech in house. Would be interesting to see an industry wide consortium on this
- cnorthwood 9y ago
  BBC R&D are participants in one: https://www.nmos.tv/about_NMI.html https://www.nmos.tv/about_NMI.html
- brensmith 9y ago
  There are some consortiums that are starting to form: http://www.videoservicesforum.org/activity_groups/RIST_poster_for_VidTrans2018Feb25.pdf http://www.videoservicesforum.org/activity_groups/RIST_poste...
Y_Y 9y ago
Cool.
dsr_ 9y ago
Next time management asks why the driver for your product should be open source, point out that it makes your product much more attractive to people who will improve and promote your product for free. That is, sales will improve.
- yilugurlu 9y ago
  if you meant Mellanox, this is exactly what i thought
  - ksec 9y ago
    I vaguely remember similar story for FreeBSD from Netflix or Limelight. And it was Mellanox as well.
    - drewg123 8y ago
      FreeBSD & Netflix. We use Mellanox on our 100GbE CDN nodes. It is a very different application from what the BBC is describing. Rather than 1 80Gb/stream, we have tens of thousands of several Mb/s streams. And rather than kernel bypass, we use heavily optimized kernel path, via sendfile and in-kernel TLS.
    - kev009 8y ago
      Limelight is primarily Chelsio with a long tail of Intel. I chose Chelsio over Mellanox because I like their drivers more, but both are top shelf NICs. One thing to note on topic, we developed a driver framework called 'iflib' in FreeBSD that is somewhat similar in scope to NDIS on Windows. If you write an iflib driver, you automatically get netmap support on FreeBSD. So we have support for all of Intel, and newer Broadcom as well, without potentially unfamiliar corporate devs having to understand the intricacies of soft ring management for userspace networking.
- alacombe 9y ago
  > it makes your product much more attractive to people who will improve and promote your product for free This is a huge myth. 1) even if you open-source, the company retains control over the project, and push their priorities first. 2) codebase can be huge and intricate, very few external contributors would actually put the time to learn it, leading to poor contribution (ie. might fix a given problem, but breaks other subsystems) 3) even if there is external contributors, is the company willing to put the manpower to process the changes and handle community interaction ? Ie. do you really want to handle bikeshed'ing style patches, potentially costing time to actual paid devs ? Most of the time, niche opensource contributions are merely are more of a marketing tool (or HR tool ala. "your profile looks great however we're too greedy to hire you, please contribute to our software for free").
  - dsr_ 9y ago
    It's a good thing for the BBC that Mellanox didn't agree with you, eh? #1 is trivially true for all products. #2 is essentially saying "if your code is crap, you will get crappy contributions". Well, if your code is crap, maybe you want to fix that? #3 is valid. You don't necessarily get great results without putting in a little effort.
    - alacombe 8y ago
      #1 is not true for bazaar-style (ie. rather few inertia) open-source projects, say the Linux kernel. At best, subsystem maintainers can push their own agenda, but even this level of control is rather limited. However, I do concede that it's rather true for non-profit focused cathedral-style (ie. lots of inertia) projects, say, the BSD. #2 does not mean that code is crappy at all, unless you consider that not writing down every single assertion made by a developer who wrote the whole subsystem as "crappy". In which case, I can assure you that all software is crappy. This might sounds as a huge generalization, but unless infinite amount of time and money, everything is a compromise, especially when deliveries are constrained. Devs who are working daily on the codebase tends to know these compromises way better than newcomers. That's my point.
      - 8y ago
- imtringued 8y ago
  Sales improve because the potential userbase has increased, not because of free contributions. Imagine you are google and you have created your own custom build system. You now want to hire more people that are familiar with it. Unfortunately the potential applicant pool is exactly 0 because nobody outside of google can even start to learn how to use the google specific tools because they simply don't exist outside of google.
  - dsr_ 8y ago
    The big barriers to adoption of a new product, service, or whatever are: 1. People knowing about it. 2. People needing it. 3. People trusting it. Like a fire needs heat, oxygen and fuel, new products survive only when they get all three corners of the triangle. If your product is cheap enough, some people will try it when they think that they have a need for something like that; trust comes eventually. Open source builds trust that in the worst case, people using your product will have a glide path out instead of a sudden brick wall. In the instant case, Mellanox gets a highly visible, reputable customer telling everyone in their industry that Mellanox NICs are high-performance, are trusted, and can be adapted to their needs. Anyone who reads this article and thinks about high-performance NICs will have a bit more trust in the Mellanox brand as a flexible system.
greyfox 9y ago
am i reading this right? are they coming up with a way to use IP to broadcast "television" in the sense that, if this was finished and open source that "anyone" could feasibly run their own "television channel/station" (edit: over IP) ?
- tiernano 9y ago
  I think it's from the cameras and recording gear to the backend where it's peiced together and that is what is broadcast... given they mention uncompressed hd and uhd, that's the impression I get...
- sandworm101 9y ago
  Also. live TV. Live feeds over IP have always had their issues. Slight delays, even the concept of buffering, is not acceptable. Packets have to come in and out of the system at roughly the speed of light. Any drop of anything will be seen on screens. Parity can help, but that requires processing that is very difficult in the acceptable timeframe. A typical TV studio could have 20 different live feeds coming in and four or five going out. They need to switch between and/or layer any number of these instantly and seamlessly. The bandwidth demands quickly become insane. Remember too that the feed will transit over many different IP-based systems. A seemingly acceptable delay on one system, a few microseconds here and there, adds up or multiplies as data streams between equipment.
- cjensen 9y ago
  Not exactly. This is about how to pass video between interoperable machines. Traditionally within a broadcast house, video+audio was sent between machines using HD-SDI using BNC coax[1]. For the first generation of HD, this ran at 1.5Gb/s. For 1080p/59.94Hz, a new standard was developed to run at 3Gb/s. For 4K at 59.94Hz, there is a 12Gb/s standard. HD-SDI routers are extremely expensive. Each input and output from a device has to be individually cabled to a router and some devices can have many dozens of I/Os. There has been a push to switch from using ridiculous numbers of cables to using IP solutions and off-the-shelf IP routers. SMPTE 2110 provides a framework for doing this using RTP [2]. 4K needs roughly 12Gb/s, so to carry a single video, you need at least 25Gb/s networking. If you want to carry many signals, the network bandwidth goes up fast -- that's why BBC is interested in 100Gb/s links. Also keep in mind the traffic is not "bursty". These links may be nearly saturated 24/7. Most off-the-shelf routers are not designed with enough buffering to handle fully saturated links on all connectors. [1] https://en.wikipedia.org/wiki/Serial_digital_interface https://en.wikipedia.org/wiki/Serial_digital_interface [2] https://en.wikipedia.org/wiki/SMPTE_2022 https://en.wikipedia.org/wiki/SMPTE_2022 (older IP standard)
kalleboo 9y ago
> unique challenge here involves handling IP packets (around 1500 bytes each) at data rates of between 1 and 8 Gigabits per second This reminded me of something. At this point it seems like Jumbo frames are never going to be widely adopted, are they? Otherwise this seems like the perfect application - massive datarates, controlled hardware/software, high-quality wiring...
- Osiris 9y ago
  For my home LAN the problem with Jumbo packets is that it seems there isn't a standard size that all NICs support. I have several machines that support 9000 bytes, some 9014 bytes, some only up to 4000 bytes. So I haven't been able to get the whole network on a single MTU. At least this was a case a few years ago.
- cremp 9y ago
  Working with Mellanox cards for a few weeks; the MTU matters a lot. The difference between an iperf test measuring 14 Gbits/s on MTU 1500 vs. 40 Gbits/s on MTU 9000. We got a couple of Connect-X5 cards, which allow switchless connections, akin to a ring topology. A lot of neat things, at just stupid line speeds, and latency levels I haven't seen in software, ever.
  - kevstev 8y ago
    Interesting. I bought a "real" netgear switch a few years back (geared to the small business market) and did some tests with jumbo frames enabled, and saw only a slightly under 10% difference in performance pushing 10GB files from my laptop to my NAS. Everything else was consumer level though. I did ensure that everything was flipped to jumbo frames. I/O shouldn't have been the factor since it was SSD on both sides. Mellanox has some interesting stuff, I used to work with it back in my HFT days.
  - shaklee3 8y ago
    Not sure why this would be. Mellanox can do line rate on 40G at 1500 using dpdk easily.
- 9y ago
chatmasta 9y ago
Can someone ELI5 the major challenges of delivering "broadcast quality" video over IP compared to cable? It seems crazy that pushing HD content over a standard aux cable is faster than downloading HD over IP. We have been pushing HD content to our TV's for 10+ years, but many people still have trouble streaming an HD movie from Netflix. Is this simply due to the overhead of IP transport supporting bidirectional communication? That is, a TV broadcast only needs to support a fixed set of N unidirectional flows (channels), but IP needs to support a dynamic set of N bidirectional flows?
- corndoge 9y ago
  As you said, broadcast is usually done with IP multicast, so it's not really bandwidth intensive. The netflix example is different, there you have a large group of people who all want different streams. At netflix's scale serving that many simultaneous connections from any reasonable amount of datacenters has the capability to saturate DC uplinks at peak hours. Hence why they've resorted to caching appliances in IXs.
  - cnorthwood 9y ago
    It'd be great if it could be done using IP multicast! Sadly over the Internet to consumer devices multicast isn't an option, sometimes it is within a particular ISP for their services, but from an arbitrary broadcaster to a receiver it's all unicast (and often pull-based using HLS or DASH)
    - corndoge 9y ago
      I meant at real scale where it makes sense. You're correct in that your upstream ISP needs to support PIM to the customer edge, which is hard to get, but available for certain customers and especially so in datacenter fabrics.
      - cnorthwood 9y ago
        ah yes, I thought you were talking about distribution rather than inside a production facility. At that case the bandwidth of uncompressed 4K (which you want to do so you're not introducing compression artifacts) is in the order of several Gbps, so bandwidth is pretty high, although multicast is used.
corndoge 9y ago
Wonder why they didn't use DPDK instead of rolling their own MLX specific thing.
- xjia 9y ago
  DPDK is a very recent thing...
  - shaklee3 9y ago
    It's been out for over 5 years, so not too recent.
  - corndoge 9y ago
    As another said, been out for quite a while, more or less the standard kernel stack bypass, and certainly less recent than this BBC thing. Wish they'd spent their time contributing better MLX support to DPDK.
  - lttlrck 9y ago
    Also netmap is extremely easy to get going. It’s a man page and some great examples and the deva are superbly helpful. I found DPDK daunting in comparison (it really is a different beast though, a real toolkit). One nice thing about Netmap is it can fall-back to emulation mode and work with any NIC. This can be really helpful if you want a single codepath but don’t always need the performance. DPDK might support this too now, I haven’t looked at it for quite a while.
    - shaklee3 8y ago
      DPDK has great documentation and examples. I agree that it's more of a beast compared to netmap and pf_ring, but it's not lacking good documentation. Their documentation on each example are really well-done. DPDK can fall back to emulation mode as well. See the AF_PACKET driver.
      - abainbridge 8y ago
        Hmmm, I found /lots/ of documentation, but lots of gaps. For example, I still have no idea how many rings, how many queues and how many cores I should use for my packet forwarding app. Presumably there's a mental model I need to learn here. It's among the first thing any DPDK user needs to know. It's not the first thing in the docs. In the examples, its magic numbers all the way with no explanation. eg http://dpdk.org/doc/guides/sample_app_ug/skeleton.html http://dpdk.org/doc/guides/sample_app_ug/skeleton.html Also, I can't find a hello world like "send a packet" application, and it's complement of "receive a packet", only forwarding apps. That's not great when I'm trying to test if my config works. I still don't know how many huge pages I need to reserve. The quick start guide says 64: http://dpdk.org/doc/quick-start http://dpdk.org/doc/quick-start That didn't work. The Mellanox rep said 8192, which did work. That's a big difference. I can't find an explanation. I invested about 1% as much time in Netmap to get to the same level of "my app nearly works".
jlebrech 9y ago
All paid by https://en.wikipedia.org/wiki/Television_licensing_in_the_United_Kingdom https://en.wikipedia.org/wiki/Television_licensing_in_the_Un...
- lainga 9y ago
  I never understood why they stick with this model instead of just levying a tax on citizens (like my beloved CBC). The BBC has been more than just a TV station for a long time; and I hear people in the UK are dead tired of having BBC license "collectors" coming round every 2 months to harass them into paying fees for the flatscreen TV they have hooked up to an XBox.
  - jgrahamc 9y ago
    I hear people in the UK are dead tired of having BBC license "collectors" coming round every 2 months I have never heard of this (I am in the UK). Yes, they'll send you letters from time to time (yearly?) but "coming round every two months"?
    - DanBC 9y ago
      The only people who have problems are the people who refuse to tick a box on a webform that says "I do not use iPlayer, I do not watch any tv live as it's broadcast".
      - jgrahamc 9y ago
        I didn't have a TV license for years because I didn't watch live TV etc. and they just sent me regular letters (I think every two years). True that those letters are kind of menacing but...
        jlebrech 8y ago
        had you been unemployed (or remote working) and been home most weekdays you would have had many interrupting knocks on the door.
decasia 9y ago
One of the sibling articles explains that part of the project involves trying to cut labor costs on covering massively multi-sited events: https://www.bbc.co.uk/rd/projects/nearly-live-production https://www.bbc.co.uk/rd/projects/nearly-live-production The idea is explicitly that you would run the equivalent of a TV control room on a web app that knows how to switch between video feeds (potentially from fixed cameras without human operators) and then can output video to... whoever is your audience. The "moving window" of near-live editorial decisions is pretty interesting as well, and seems geared towards giving non-professional editors a chance to fix mistakes.
- decasia 9y ago
  relevant TLDR: "Professional, live, multi-camera coverage isn’t practical for all events or venues at a large festival. For our research at Edinburgh Fringe Festival 2015, we experimented with placing three unmanned, static, ultra high definition cameras and two unmanned, static, high definition GoPro cameras around the circumference of the BBC venue. A lightweight video capture rig of this kind, delivering images to a cloud system, could allow a director to crop and cut between these shots in software, over the web and produce good quality coverage ‘nearly live’ at reduced cost."
  - deleted 9y ago
    [deleted]
- stephen_g 9y ago
  How does a web app know how to switch between video feeds, or cameras know where to point? Maybe we're talking about different kinds of events, but there's actually a lot of creativity that goes into camera framing, movement, shot selection, timing of cuts, etc. to end up with something that's actually interesting to watch...
  - decasia 9y ago
    Yeah on your second point; I used to do a bit of TV broadcast stuff and it was a marvelous technical world/system of its own. And it seems obvious that the BBC's proposal here is in some sense much more lo-fi than that. I think their proposal was that you could give up on being able to do camera framing and movement, but you would still maintain the ability to cut between shots at aesthetically pleasing moments. I could just about imagine that producing good results, with a relatively predictable performance and a decent technical director. But I think a system like this doesn't compete with professional video production. It competes with someone making a shaky cellphone video in a small venue -- a form of video which a lot of people seem to accept, these days. I could see the BBC's proposal being a step up from that (and a lot cheaper than a mobile TV studio setup).
kierank 9y ago
Lots of people missing the point in this discussion, the goal of this is to send Uncompressed video over closed IP networks in a studio for example. I talked about some of the problems at Demuxed last year (referencing BBC R&D's work): https://www.youtube.com/watch?v=A4L5xEXXlas https://www.youtube.com/watch?v=A4L5xEXXlas
- ckdarby 8y ago
  I don't think many people missing this point. I got the point from the article itself and even before this when I saw BBC I figured it was going to be related to video.
jwbensley 9y ago
BBC R&D gave a talk on this very project a couple of years ago at a UKNOF meeting, it was very interesting: https://m.youtube.com/watch?v=yLL8wl8YUwA https://m.youtube.com/watch?v=yLL8wl8YUwA
adrianmonk 8y ago
So if this lossless video over ethernet thing ever trickles down to consumers, what happens? Can my computer monitor become just another network device plugged into Ethernet like my printer already is? In my living room, can I ditch the video routing part of my AV receiver and just plug everything (streaming devices, video games, cable box) into a very fast ethernet switch?
- tzahola 8y ago
  That would be an enormous waste of bandwidth, considering that a fullHD stream (30fps @ 24bpp) would consume 1.4 gigabit/s.
- fulafel 8y ago
  Quickest Ethernet is up to 400 gbps now, you can probably get by with a pedestrian 10gbit switch :) Hard to see uncompressed consumer video catching on though since compression works so well.
  - e12e 8y ago
    Afaik thunderbolt is packet switched?
john37386 8y ago
Many people don't understand how or why the linux kernel is the bottlebeck at speed higher than 10 Gbps. The problem is that the linux kernel can't process many simultaneous small connections. Just to be clear: - linux can easily transfer at 40 Gbps - linux chokes at around 1M packet per second per cpu socket. Thats right. So linux can easily transfer at 40 Gbps one or few simultaneous flows But! It can't transfer several flows at 40 Gbps. The bottlebeck is the number of packets per seconds it can inspect. So if you are sending 100 big transfers, you will reach 40 Gbps. If you send 5M small transfers, linux will die. This is why netmap is handy. It offloads the packets from linux directly to the app Bbc is sending xM small requests per second, hence millions packets per seconds. In conclusion, netmap is good if you need a lot of small simultaneous connections. I reached 40M connections on chelsio 40 Gbps nic using netmap on FreeBSD 11.
- sengork 8y ago
  Thanks, I've always wondered under which conditions BSD networking performs better compared to Linux. It's the number of simultaneous streams as opposed to raw bandwidth for fewer streams.
  - john37386 8y ago
    Well BSD kernel will also choke at more or less the same limits as linux kernel. The key point here is that a netmap driver bypass the kernel and therefore open the door to many millions packets per seconds or millions requests per seconds. Not many Gbps
- bogomipz 8y ago
  This isn't specific to Linux. PPS is a limiting factor for network hardware vendor's gear as well. Lots of ~100 byte packets will bring pain to Juniper and Cisco products as well.
- signa11 8y ago
  > linux chokes at around 1M packet per second per cpu socket. which is why kernel bypass mechanisms f.e. dpdk/netmap/... etc. are getting some/lots of traction. fwiw, using dpdk (on x86-64 hw), i have had no problems with pushing minimal 64b sized packets for large (>250k) number of flows at line rate...
ausjke 8y ago
Last kernel hacking I had to do some work with 10Gbps NICs, so 80Gbps is just so attractive to my eyes. There are different pass-through/fastpath patch for different chips to avoid memcpy, or do zero-copys, but they are all kernel patches, kernel-only. An alternative method will be BPF/XDP and DPDK for which you will need modify the kernel drivers somehow for good performance. Wondering if Netmap does that already or is has nothing to do with them. All of them are addressing Dataplane packet move, hope I can have an environment to experience these close-to-100bps network in my next projects. In the meantime, I am wondering, why do you pass 4K uncompressed video using IP packets...
apazgo 8y ago
Wouldn't products like openvpn benefit from using this? Or am I missing something?
jacksmith21006 8y ago
I will be curious to see how well the new Google kernel, Zircon, performs compared to Linux. I have my doubts a micro kernel can come close to the Linux kernel efficiency. Zircon is the new micro kernel that is part of Fuschia. https://github.com/fuchsia-mirror/zircon https://github.com/fuchsia-mirror/zircon
- e12e 8y ago
  Isn't this a bit of an odd comment, on a story about kernel bypass for high performance networking?
  - jacksmith21006 8y ago
    Why? Think it is very relevant.
    - e12e 8y ago
      You doubt a microkernel can approach Linux kernel efficiency, yet here a typical kernel task is delegated to user-space for efficiency under Linux ; an approach that is similar to how networking might be handled by a microkernel? I agree that it'll be interesting to see how fuchsia turns out.
      - jacksmith21006 8y ago
        Yes definitely have my doubts. But hope it works out. Just love Flutter and hope Zircon works out. But we have to see it actually happen.
gnufx 8y ago
If this is for a more-or-less closed system, I wonder why it uses IP, and not an RDMA-type network (Infiniband etc.) which already is kernel-bypass and works routinely at around 100G, 1μs latency. There's nothing special about the Mellanox drivers amongst the Openfabrics ones in having free drivers in Linux, but they generally require blobs or separate firmware.

376 points by XzetaU8 9y ago

There's a lot of orgs out there developing broadcast IP tech in house. Would be interesting to see an industry wide consortium on this

cnorthwood 9y ago
BBC R&D are participants in one: https://www.nmos.tv/about_NMI.html https://www.nmos.tv/about_NMI.html
brensmith 9y ago
There are some consortiums that are starting to form: http://www.videoservicesforum.org/activity_groups/RIST_poster_for_VidTrans2018Feb25.pdf http://www.videoservicesforum.org/activity_groups/RIST_poste...

Y_Y 9y ago

Cool.

dsr_ 9y ago

Next time management asks why the driver for your product should be open source, point out that it makes your product much more attractive to people who will improve and promote your product for free. That is, sales will improve.

yilugurlu 9y ago
if you meant Mellanox, this is exactly what i thought
- ksec 9y ago
  I vaguely remember similar story for FreeBSD from Netflix or Limelight. And it was Mellanox as well.
  - drewg123 8y ago
    FreeBSD & Netflix. We use Mellanox on our 100GbE CDN nodes. It is a very different application from what the BBC is describing. Rather than 1 80Gb/stream, we have tens of thousands of several Mb/s streams. And rather than kernel bypass, we use heavily optimized kernel path, via sendfile and in-kernel TLS.
  - kev009 8y ago
    Limelight is primarily Chelsio with a long tail of Intel. I chose Chelsio over Mellanox because I like their drivers more, but both are top shelf NICs. One thing to note on topic, we developed a driver framework called 'iflib' in FreeBSD that is somewhat similar in scope to NDIS on Windows. If you write an iflib driver, you automatically get netmap support on FreeBSD. So we have support for all of Intel, and newer Broadcom as well, without potentially unfamiliar corporate devs having to understand the intricacies of soft ring management for userspace networking.
alacombe 9y ago
> it makes your product much more attractive to people who will improve and promote your product for free This is a huge myth. 1) even if you open-source, the company retains control over the project, and push their priorities first. 2) codebase can be huge and intricate, very few external contributors would actually put the time to learn it, leading to poor contribution (ie. might fix a given problem, but breaks other subsystems) 3) even if there is external contributors, is the company willing to put the manpower to process the changes and handle community interaction ? Ie. do you really want to handle bikeshed'ing style patches, potentially costing time to actual paid devs ? Most of the time, niche opensource contributions are merely are more of a marketing tool (or HR tool ala. "your profile looks great however we're too greedy to hire you, please contribute to our software for free").
- dsr_ 9y ago
  It's a good thing for the BBC that Mellanox didn't agree with you, eh? #1 is trivially true for all products. #2 is essentially saying "if your code is crap, you will get crappy contributions". Well, if your code is crap, maybe you want to fix that? #3 is valid. You don't necessarily get great results without putting in a little effort.
  - alacombe 8y ago
    #1 is not true for bazaar-style (ie. rather few inertia) open-source projects, say the Linux kernel. At best, subsystem maintainers can push their own agenda, but even this level of control is rather limited. However, I do concede that it's rather true for non-profit focused cathedral-style (ie. lots of inertia) projects, say, the BSD. #2 does not mean that code is crappy at all, unless you consider that not writing down every single assertion made by a developer who wrote the whole subsystem as "crappy". In which case, I can assure you that all software is crappy. This might sounds as a huge generalization, but unless infinite amount of time and money, everything is a compromise, especially when deliveries are constrained. Devs who are working daily on the codebase tends to know these compromises way better than newcomers. That's my point.
    - 8y ago
imtringued 8y ago
Sales improve because the potential userbase has increased, not because of free contributions. Imagine you are google and you have created your own custom build system. You now want to hire more people that are familiar with it. Unfortunately the potential applicant pool is exactly 0 because nobody outside of google can even start to learn how to use the google specific tools because they simply don't exist outside of google.
- dsr_ 8y ago
  The big barriers to adoption of a new product, service, or whatever are: 1. People knowing about it. 2. People needing it. 3. People trusting it. Like a fire needs heat, oxygen and fuel, new products survive only when they get all three corners of the triangle. If your product is cheap enough, some people will try it when they think that they have a need for something like that; trust comes eventually. Open source builds trust that in the worst case, people using your product will have a glide path out instead of a sudden brick wall. In the instant case, Mellanox gets a highly visible, reputable customer telling everyone in their industry that Mellanox NICs are high-performance, are trusted, and can be adapted to their needs. Anyone who reads this article and thinks about high-performance NICs will have a bit more trust in the Mellanox brand as a flexible system.

greyfox 9y ago

am i reading this right? are they coming up with a way to use IP to broadcast "television" in the sense that, if this was finished and open source that "anyone" could feasibly run their own "television channel/station" (edit: over IP) ?

tiernano 9y ago
I think it's from the cameras and recording gear to the backend where it's peiced together and that is what is broadcast... given they mention uncompressed hd and uhd, that's the impression I get...
sandworm101 9y ago
Also. live TV. Live feeds over IP have always had their issues. Slight delays, even the concept of buffering, is not acceptable. Packets have to come in and out of the system at roughly the speed of light. Any drop of anything will be seen on screens. Parity can help, but that requires processing that is very difficult in the acceptable timeframe. A typical TV studio could have 20 different live feeds coming in and four or five going out. They need to switch between and/or layer any number of these instantly and seamlessly. The bandwidth demands quickly become insane. Remember too that the feed will transit over many different IP-based systems. A seemingly acceptable delay on one system, a few microseconds here and there, adds up or multiplies as data streams between equipment.
cjensen 9y ago
Not exactly. This is about how to pass video between interoperable machines. Traditionally within a broadcast house, video+audio was sent between machines using HD-SDI using BNC coax[1]. For the first generation of HD, this ran at 1.5Gb/s. For 1080p/59.94Hz, a new standard was developed to run at 3Gb/s. For 4K at 59.94Hz, there is a 12Gb/s standard. HD-SDI routers are extremely expensive. Each input and output from a device has to be individually cabled to a router and some devices can have many dozens of I/Os. There has been a push to switch from using ridiculous numbers of cables to using IP solutions and off-the-shelf IP routers. SMPTE 2110 provides a framework for doing this using RTP [2]. 4K needs roughly 12Gb/s, so to carry a single video, you need at least 25Gb/s networking. If you want to carry many signals, the network bandwidth goes up fast -- that's why BBC is interested in 100Gb/s links. Also keep in mind the traffic is not "bursty". These links may be nearly saturated 24/7. Most off-the-shelf routers are not designed with enough buffering to handle fully saturated links on all connectors. [1] https://en.wikipedia.org/wiki/Serial_digital_interface https://en.wikipedia.org/wiki/Serial_digital_interface [2] https://en.wikipedia.org/wiki/SMPTE_2022 https://en.wikipedia.org/wiki/SMPTE_2022 (older IP standard)

kalleboo 9y ago

> unique challenge here involves handling IP packets (around 1500 bytes each) at data rates of between 1 and 8 Gigabits per second This reminded me of something. At this point it seems like Jumbo frames are never going to be widely adopted, are they? Otherwise this seems like the perfect application - massive datarates, controlled hardware/software, high-quality wiring...

Osiris 9y ago
For my home LAN the problem with Jumbo packets is that it seems there isn't a standard size that all NICs support. I have several machines that support 9000 bytes, some 9014 bytes, some only up to 4000 bytes. So I haven't been able to get the whole network on a single MTU. At least this was a case a few years ago.
cremp 9y ago
Working with Mellanox cards for a few weeks; the MTU matters a lot. The difference between an iperf test measuring 14 Gbits/s on MTU 1500 vs. 40 Gbits/s on MTU 9000. We got a couple of Connect-X5 cards, which allow switchless connections, akin to a ring topology. A lot of neat things, at just stupid line speeds, and latency levels I haven't seen in software, ever.
- kevstev 8y ago
  Interesting. I bought a "real" netgear switch a few years back (geared to the small business market) and did some tests with jumbo frames enabled, and saw only a slightly under 10% difference in performance pushing 10GB files from my laptop to my NAS. Everything else was consumer level though. I did ensure that everything was flipped to jumbo frames. I/O shouldn't have been the factor since it was SSD on both sides. Mellanox has some interesting stuff, I used to work with it back in my HFT days.
- shaklee3 8y ago
  Not sure why this would be. Mellanox can do line rate on 40G at 1500 using dpdk easily.
9y ago

chatmasta 9y ago

Can someone ELI5 the major challenges of delivering "broadcast quality" video over IP compared to cable? It seems crazy that pushing HD content over a standard aux cable is faster than downloading HD over IP. We have been pushing HD content to our TV's for 10+ years, but many people still have trouble streaming an HD movie from Netflix. Is this simply due to the overhead of IP transport supporting bidirectional communication? That is, a TV broadcast only needs to support a fixed set of N unidirectional flows (channels), but IP needs to support a dynamic set of N bidirectional flows?

corndoge 9y ago
As you said, broadcast is usually done with IP multicast, so it's not really bandwidth intensive. The netflix example is different, there you have a large group of people who all want different streams. At netflix's scale serving that many simultaneous connections from any reasonable amount of datacenters has the capability to saturate DC uplinks at peak hours. Hence why they've resorted to caching appliances in IXs.
- cnorthwood 9y ago
  It'd be great if it could be done using IP multicast! Sadly over the Internet to consumer devices multicast isn't an option, sometimes it is within a particular ISP for their services, but from an arbitrary broadcaster to a receiver it's all unicast (and often pull-based using HLS or DASH)
  - corndoge 9y ago
    I meant at real scale where it makes sense. You're correct in that your upstream ISP needs to support PIM to the customer edge, which is hard to get, but available for certain customers and especially so in datacenter fabrics.
    - cnorthwood 9y ago
      ah yes, I thought you were talking about distribution rather than inside a production facility. At that case the bandwidth of uncompressed 4K (which you want to do so you're not introducing compression artifacts) is in the order of several Gbps, so bandwidth is pretty high, although multicast is used.

corndoge 9y ago

Wonder why they didn't use DPDK instead of rolling their own MLX specific thing.

xjia 9y ago
DPDK is a very recent thing...
- shaklee3 9y ago
  It's been out for over 5 years, so not too recent.
- corndoge 9y ago
  As another said, been out for quite a while, more or less the standard kernel stack bypass, and certainly less recent than this BBC thing. Wish they'd spent their time contributing better MLX support to DPDK.
- lttlrck 9y ago
  Also netmap is extremely easy to get going. It’s a man page and some great examples and the deva are superbly helpful. I found DPDK daunting in comparison (it really is a different beast though, a real toolkit). One nice thing about Netmap is it can fall-back to emulation mode and work with any NIC. This can be really helpful if you want a single codepath but don’t always need the performance. DPDK might support this too now, I haven’t looked at it for quite a while.
  - shaklee3 8y ago
    DPDK has great documentation and examples. I agree that it's more of a beast compared to netmap and pf_ring, but it's not lacking good documentation. Their documentation on each example are really well-done. DPDK can fall back to emulation mode as well. See the AF_PACKET driver.
    - abainbridge 8y ago
      Hmmm, I found /lots/ of documentation, but lots of gaps. For example, I still have no idea how many rings, how many queues and how many cores I should use for my packet forwarding app. Presumably there's a mental model I need to learn here. It's among the first thing any DPDK user needs to know. It's not the first thing in the docs. In the examples, its magic numbers all the way with no explanation. eg http://dpdk.org/doc/guides/sample_app_ug/skeleton.html http://dpdk.org/doc/guides/sample_app_ug/skeleton.html Also, I can't find a hello world like "send a packet" application, and it's complement of "receive a packet", only forwarding apps. That's not great when I'm trying to test if my config works. I still don't know how many huge pages I need to reserve. The quick start guide says 64: http://dpdk.org/doc/quick-start http://dpdk.org/doc/quick-start That didn't work. The Mellanox rep said 8192, which did work. That's a big difference. I can't find an explanation. I invested about 1% as much time in Netmap to get to the same level of "my app nearly works".

jlebrech 9y ago

All paid by https://en.wikipedia.org/wiki/Television_licensing_in_the_United_Kingdom https://en.wikipedia.org/wiki/Television_licensing_in_the_Un...

lainga 9y ago
I never understood why they stick with this model instead of just levying a tax on citizens (like my beloved CBC). The BBC has been more than just a TV station for a long time; and I hear people in the UK are dead tired of having BBC license "collectors" coming round every 2 months to harass them into paying fees for the flatscreen TV they have hooked up to an XBox.
- jgrahamc 9y ago
  I hear people in the UK are dead tired of having BBC license "collectors" coming round every 2 months I have never heard of this (I am in the UK). Yes, they'll send you letters from time to time (yearly?) but "coming round every two months"?
  - DanBC 9y ago
    The only people who have problems are the people who refuse to tick a box on a webform that says "I do not use iPlayer, I do not watch any tv live as it's broadcast".
    - jgrahamc 9y ago
      I didn't have a TV license for years because I didn't watch live TV etc. and they just sent me regular letters (I think every two years). True that those letters are kind of menacing but...
      - jlebrech 8y ago
        had you been unemployed (or remote working) and been home most weekdays you would have had many interrupting knocks on the door.

decasia 9y ago

One of the sibling articles explains that part of the project involves trying to cut labor costs on covering massively multi-sited events: https://www.bbc.co.uk/rd/projects/nearly-live-production https://www.bbc.co.uk/rd/projects/nearly-live-production The idea is explicitly that you would run the equivalent of a TV control room on a web app that knows how to switch between video feeds (potentially from fixed cameras without human operators) and then can output video to... whoever is your audience. The "moving window" of near-live editorial decisions is pretty interesting as well, and seems geared towards giving non-professional editors a chance to fix mistakes.

decasia 9y ago
relevant TLDR: "Professional, live, multi-camera coverage isn’t practical for all events or venues at a large festival. For our research at Edinburgh Fringe Festival 2015, we experimented with placing three unmanned, static, ultra high definition cameras and two unmanned, static, high definition GoPro cameras around the circumference of the BBC venue. A lightweight video capture rig of this kind, delivering images to a cloud system, could allow a director to crop and cut between these shots in software, over the web and produce good quality coverage ‘nearly live’ at reduced cost."
- deleted 9y ago
  [deleted]
stephen_g 9y ago
How does a web app know how to switch between video feeds, or cameras know where to point? Maybe we're talking about different kinds of events, but there's actually a lot of creativity that goes into camera framing, movement, shot selection, timing of cuts, etc. to end up with something that's actually interesting to watch...
- decasia 9y ago
  Yeah on your second point; I used to do a bit of TV broadcast stuff and it was a marvelous technical world/system of its own. And it seems obvious that the BBC's proposal here is in some sense much more lo-fi than that. I think their proposal was that you could give up on being able to do camera framing and movement, but you would still maintain the ability to cut between shots at aesthetically pleasing moments. I could just about imagine that producing good results, with a relatively predictable performance and a decent technical director. But I think a system like this doesn't compete with professional video production. It competes with someone making a shaky cellphone video in a small venue -- a form of video which a lot of people seem to accept, these days. I could see the BBC's proposal being a step up from that (and a lot cheaper than a mobile TV studio setup).

kierank 9y ago

Lots of people missing the point in this discussion, the goal of this is to send Uncompressed video over closed IP networks in a studio for example. I talked about some of the problems at Demuxed last year (referencing BBC R&D's work): https://www.youtube.com/watch?v=A4L5xEXXlas https://www.youtube.com/watch?v=A4L5xEXXlas

ckdarby 8y ago
I don't think many people missing this point. I got the point from the article itself and even before this when I saw BBC I figured it was going to be related to video.

jwbensley 9y ago

BBC R&D gave a talk on this very project a couple of years ago at a UKNOF meeting, it was very interesting: https://m.youtube.com/watch?v=yLL8wl8YUwA https://m.youtube.com/watch?v=yLL8wl8YUwA

adrianmonk 8y ago

So if this lossless video over ethernet thing ever trickles down to consumers, what happens? Can my computer monitor become just another network device plugged into Ethernet like my printer already is? In my living room, can I ditch the video routing part of my AV receiver and just plug everything (streaming devices, video games, cable box) into a very fast ethernet switch?

tzahola 8y ago
That would be an enormous waste of bandwidth, considering that a fullHD stream (30fps @ 24bpp) would consume 1.4 gigabit/s.
fulafel 8y ago
Quickest Ethernet is up to 400 gbps now, you can probably get by with a pedestrian 10gbit switch :) Hard to see uncompressed consumer video catching on though since compression works so well.
- e12e 8y ago
  Afaik thunderbolt is packet switched?

john37386 8y ago

Many people don't understand how or why the linux kernel is the bottlebeck at speed higher than 10 Gbps. The problem is that the linux kernel can't process many simultaneous small connections. Just to be clear: - linux can easily transfer at 40 Gbps - linux chokes at around 1M packet per second per cpu socket. Thats right. So linux can easily transfer at 40 Gbps one or few simultaneous flows But! It can't transfer several flows at 40 Gbps. The bottlebeck is the number of packets per seconds it can inspect. So if you are sending 100 big transfers, you will reach 40 Gbps. If you send 5M small transfers, linux will die. This is why netmap is handy. It offloads the packets from linux directly to the app Bbc is sending xM small requests per second, hence millions packets per seconds. In conclusion, netmap is good if you need a lot of small simultaneous connections. I reached 40M connections on chelsio 40 Gbps nic using netmap on FreeBSD 11.

sengork 8y ago
Thanks, I've always wondered under which conditions BSD networking performs better compared to Linux. It's the number of simultaneous streams as opposed to raw bandwidth for fewer streams.
- john37386 8y ago
  Well BSD kernel will also choke at more or less the same limits as linux kernel. The key point here is that a netmap driver bypass the kernel and therefore open the door to many millions packets per seconds or millions requests per seconds. Not many Gbps
bogomipz 8y ago
This isn't specific to Linux. PPS is a limiting factor for network hardware vendor's gear as well. Lots of ~100 byte packets will bring pain to Juniper and Cisco products as well.
signa11 8y ago
> linux chokes at around 1M packet per second per cpu socket. which is why kernel bypass mechanisms f.e. dpdk/netmap/... etc. are getting some/lots of traction. fwiw, using dpdk (on x86-64 hw), i have had no problems with pushing minimal 64b sized packets for large (>250k) number of flows at line rate...

ausjke 8y ago

Last kernel hacking I had to do some work with 10Gbps NICs, so 80Gbps is just so attractive to my eyes. There are different pass-through/fastpath patch for different chips to avoid memcpy, or do zero-copys, but they are all kernel patches, kernel-only. An alternative method will be BPF/XDP and DPDK for which you will need modify the kernel drivers somehow for good performance. Wondering if Netmap does that already or is has nothing to do with them. All of them are addressing Dataplane packet move, hope I can have an environment to experience these close-to-100bps network in my next projects. In the meantime, I am wondering, why do you pass 4K uncompressed video using IP packets...

apazgo 8y ago

Wouldn't products like openvpn benefit from using this? Or am I missing something?

jacksmith21006 8y ago

I will be curious to see how well the new Google kernel, Zircon, performs compared to Linux. I have my doubts a micro kernel can come close to the Linux kernel efficiency. Zircon is the new micro kernel that is part of Fuschia. https://github.com/fuchsia-mirror/zircon https://github.com/fuchsia-mirror/zircon

e12e 8y ago
Isn't this a bit of an odd comment, on a story about kernel bypass for high performance networking?
- jacksmith21006 8y ago
  Why? Think it is very relevant.
  - e12e 8y ago
    You doubt a microkernel can approach Linux kernel efficiency, yet here a typical kernel task is delegated to user-space for efficiency under Linux ; an approach that is similar to how networking might be handled by a microkernel? I agree that it'll be interesting to see how fuchsia turns out.
    - jacksmith21006 8y ago
      Yes definitely have my doubts. But hope it works out. Just love Flutter and hope Zircon works out. But we have to see it actually happen.

gnufx 8y ago

If this is for a more-or-less closed system, I wonder why it uses IP, and not an RDMA-type network (Infiniband etc.) which already is kernel-bypass and works routinely at around 100G, 1μs latency. There's nothing special about the Mellanox drivers amongst the Openfabrics ones in having free drivers in Linux, but they generally require blobs or separate firmware.