5 ms·

Does aanyone here work with consuming (not just passing it kn) these amounts of traffic? How do you manage to get anything meaningful out of it?

by bjoli 9y ago

_zoltan_ 9y ago
disclaimer: I work at IBM Research we have just finished setting up our new big data crunching system which sits on 48x100GbE links on the full flash storage side and on 32x100GbE on the client side (split 100GbE -> 4x25GbE). at this point we are saturating most of the links.
walrus01 9y ago
network engineer here: nobody connects a 200 or 400GbE interface linecard off a router to a single anything... The purpose is for ISP-to-ISP interconnections for truly huge amounts of traffic. For example direct peering in the same IX/carrier hotel between a huge source of content (youtube/google, netflix, etc) and a huge ISP (comcast, centurylink). For a big ISP, also things like 100GbE connections from a city's pair of core routers to slightly smaller aggregation routers. edit: I don't know that anything "consumes" data in the way that the questioner is asking. It's more about the aggregate amount of data. One netflix 4K stream to an xbox one s is about 15.75 Mbps. Now multiply that by a hundred thousand netflix subscribers in a typical comcast service area, any of whom at any given time might be sitting around and watching Altered Carbon or Breaking Bad.
- jandrese 9y ago
  The only potential consumers I can imagine are like the Great Firewall of China traffic monitoring systems.
  - icebraining 9y ago
    Or the NSA: https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
- Retric 9y ago
  Raw high frame rate video can into that range. Phantom v2640 for example does 6600 frames per second at a at 2048 x 1952 resolution = 26 Gpx/sec or something like 100Gbit. Currently they simply buffer everything on Camera and use 10GigE, but in theory they could look into 100+GbE connections for newer designs.
  - walrus01 9y ago
    upcoming on-camera flash memory interface https://en.wikipedia.org/wiki/CFexpress https://en.wikipedia.org/wiki/CFexpress basically a way of putting a M.2 interface NVME PCI-Express SSD into a camera. https://petapixel.com/2018/04/09/prograde-digital-unveils-worlds-first-1tb-cfexpress-card/ https://petapixel.com/2018/04/09/prograde-digital-unveils-wo...
    - Retric 9y ago
      According to the link that's currently limited to 2 GB/s which far to slow to use for this. Phantom v2640 needs ~50+x that speed to do it in real time and I assume the bandwidth needs will keep increasing on the next model.
      - walrus01 9y ago
        So what storage device does it use currently? except for RAM, there is not much faster than a NVME based, PCI-Express 3.0 x8 interface SSD. If it's some truly ridiculous bitrate it probably uses proprietary packs of RAM with battery backing.
        Retric 9y ago
        Ok, found more info on that model right now it's got up to 288 GB of internal RAM. At 12 bit per pixel which is lower than I thought that lasts 7.8 seconds for ~40Gbit/second. Anyway, as this is all internal so they have a lot of options.
        9y ago
- kijiki 9y ago
  Cumulus (disclaimer: I co-founded it) has customers that use these: http://www.mellanox.com/page/products_dyn?product_family=260&mtag=connectx_5_en_card http://www.mellanox.com/page/products_dyn?product_family=260... to dual 100G attach their servers to a pair of ToR switches. As a result, they'd really love to have something faster than 100G for their ToR to spine uplinks, but that isn't quite available yet. Insanely (and awesomely), Mellanox already has these: http://www.mellanox.com/page/products_dyn?product_family=266&mtag=connectx_6_en_card http://www.mellanox.com/page/products_dyn?product_family=266... dual 200G NIC.
  - walrus01 9y ago
    100G NICs and dual port of the same version are a really good argument for the need for more pci-express 3.0 lanes in single socket servers. The AMD epyc is a step in the right direction. If you assume a server that might have two m.2 nvme SSD in it, those take up lanes, and then the nic will eat an x16 interface.
deaps 9y ago
Network Engineer here as well - I support one of the world's largest enterprise organizations (although my area of 'expertise' is pretty far down the stack - somewhere between the internet and the WAN circuits) - I can see no personal use for a 400 GbE port on anything. As far as my organization goes, there will always be a bottleneck somewhere else (speaking location to location across the WAN). The only applications I could see this being useful would be maybe huge ISP interconnects or maybe some big players in the video streaming world. That doesn't mean that future-proofing is a bad thing - of course cost will probably be a very large concern in the case of 400GbE...I see a lot of 40 gig ports that go completely unused, personally.
- monocasa 9y ago
  It's not hard to imagine 100GbE being used inside datacenters. AFAIK, 40GbE top of rack switch is pretty common.
  - deaps 9y ago
    I've seen some 160GbE links in datacenters - 4x40's - how much they were actually utilized, I have no idea.
  - walrus01 9y ago
    if you have a really beefy single bare metal hypervisor (typically xen or kvm) it's totally possible to give it 100GbE to the switch now: http://www.silicom-usa.com/pr/server-adapters/networking-adapters/100-gigabit-ethernet-networking-server-adapters/pe3100g2dqir_server_adapter/ http://www.silicom-usa.com/pr/server-adapters/networking-ada... I don't have anything this serious in my own production use, but I'm thinking of machines like dual socket x 32-core per socket xeons, or amd epyc, with 512GB to 1TB of RAM. then pass vlan tagged traffic off one switch port, to each specific guest VM, of which there might be dozens to 100+ VMs, as subinterfaces of that.
- walshemj 9y ago
  HPC Clusters would be one candidate
hinkley 9y ago
Can’t you saturate 100GB easily with a single disk array these days?
- SteveNuts 9y ago
  Yes, I've worked with several that can do that. The interesting thing with disk arrays now is the controller heads are becoming the bottleneck. You used to have many many shelves of SAS drives per controller pair, now the SSDs are so fast that the CPUs are saturated and you have to scale out, not up.
  - hinkley 9y ago
    If I can't (reliably) handle 100GBps coming from the RAID array it sounds like I'm not going to be able to handle 400GBps coming in from the NIC. So these really are just for interconnects for the time being, until server hardware gets faster.
    - buzer 9y ago
      There are some applications which can handle 100Gbps and potentially higher speeds on servers. Usually they require using DPDK or similar frameworks with userland TCP stack (and thus are usually pretty specialized), but at least Netflix has managed to serve 90+Gbps of TLS traffic using "default" FreeBSD TCP stack (https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://medium.com/netflix-techblog/serving-100-gbps-from-an...).
cm2187 9y ago
Not working on anything like that myself, but if the direction going forward it to stream not just 4k TV, but also 4k video games (console in the cloud), you will need some healthy backbones (even if they still run only IPv4 in 5 years!)
simias 9y ago
Streaming uncompressed raw video over ethernet for TV production. It's still commonly done over dark fiber but people are slowly migrating towards IP. A single uncompressed 4K video stream is over 12Gbps, 1080p is 3Gbps and obviously when you're producing content you have a few of these running alongside each other (multiple cameras, feedback, big screens etc... Think something like the world cup or superbowl). It quickly adds up to a tremendous bandwidth. On top of that it needs very low latency and jitter to keep all the streams synchronized.
- tatersolid 9y ago
  Why would anyone stream uncompressed raw video? Even MPEG-2 at q=1 with no B-frames is about 1/10 the size, visually lossless, and adds only a few ms of latency.
  - e12e 9y ago
    Looks like for editing you're looking at up towards 3.5 gbps per camera, not sure how much to add for sound. So a three camera production could easily push over 10 gbps. Pushing past 100 seems to be a bit of a stretch. Maybe for 3d?
fulafel 9y ago
5G base stations cite 50 Gbps bandwidth, connect a bunch of these together in an urban area and you might want 400G for their uplink/backbone/whatever it's called.
- walrus01 9y ago
  More realistically as cellular carriers are upgrading rooftop sites to the latest LTE gear, what I'm seeing as a carrier, they're fed by either a dedicated 1G fiber transport back to an aggregation site, the same at 10G on fiber, or point to point 60GHz (1-5 Gbps) mmwave, or 71-86GHz band ptp mmwave (1 to 10Gbps).
  - fulafel 9y ago
    It sure sounds like 5G deployment is "realistic". Big operators have announced deployments & limited availability this year.
kierank 9y ago
Yes, in my company we generate and receive uncompressed video in software and get to these kinds of bandwidths from a couple of servers running dual 100GbE: https://www.youtube.com/watch?v=A4L5xEXXlas https://www.youtube.com/watch?v=A4L5xEXXlas
drewg123 9y ago
I'm not consuming it, but I can produce over 100Gb/s of traffic from a single CDN node if it is equipped with multiple 100GbE NICS. So 200GbE and 400GbE are exciting to me. (for context: https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://medium.com/netflix-techblog/serving-100-gbps-from-an...)

Does aanyone here work with consuming (not just passing it kn) these amounts of traffic? How do you manage to get anything meaningful out of it?

by bjoli 9y ago

Does aanyone here work with consuming (not just passing it kn) these amounts of traffic? How do you manage to get anything meaningful out of it?

_zoltan_ 9y ago

disclaimer: I work at IBM Research we have just finished setting up our new big data crunching system which sits on 48x100GbE links on the full flash storage side and on 32x100GbE on the client side (split 100GbE -> 4x25GbE). at this point we are saturating most of the links.

walrus01 9y ago

network engineer here: nobody connects a 200 or 400GbE interface linecard off a router to a single anything... The purpose is for ISP-to-ISP interconnections for truly huge amounts of traffic. For example direct peering in the same IX/carrier hotel between a huge source of content (youtube/google, netflix, etc) and a huge ISP (comcast, centurylink). For a big ISP, also things like 100GbE connections from a city's pair of core routers to slightly smaller aggregation routers. edit: I don't know that anything "consumes" data in the way that the questioner is asking. It's more about the aggregate amount of data. One netflix 4K stream to an xbox one s is about 15.75 Mbps. Now multiply that by a hundred thousand netflix subscribers in a typical comcast service area, any of whom at any given time might be sitting around and watching Altered Carbon or Breaking Bad.

jandrese 9y ago
The only potential consumers I can imagine are like the Great Firewall of China traffic monitoring systems.
- icebraining 9y ago
  Or the NSA: https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
Retric 9y ago
Raw high frame rate video can into that range. Phantom v2640 for example does 6600 frames per second at a at 2048 x 1952 resolution = 26 Gpx/sec or something like 100Gbit. Currently they simply buffer everything on Camera and use 10GigE, but in theory they could look into 100+GbE connections for newer designs.
- walrus01 9y ago
  upcoming on-camera flash memory interface https://en.wikipedia.org/wiki/CFexpress https://en.wikipedia.org/wiki/CFexpress basically a way of putting a M.2 interface NVME PCI-Express SSD into a camera. https://petapixel.com/2018/04/09/prograde-digital-unveils-worlds-first-1tb-cfexpress-card/ https://petapixel.com/2018/04/09/prograde-digital-unveils-wo...
  - Retric 9y ago
    According to the link that's currently limited to 2 GB/s which far to slow to use for this. Phantom v2640 needs ~50+x that speed to do it in real time and I assume the bandwidth needs will keep increasing on the next model.
    - walrus01 9y ago
      So what storage device does it use currently? except for RAM, there is not much faster than a NVME based, PCI-Express 3.0 x8 interface SSD. If it's some truly ridiculous bitrate it probably uses proprietary packs of RAM with battery backing.
      - Retric 9y ago
        Ok, found more info on that model right now it's got up to 288 GB of internal RAM. At 12 bit per pixel which is lower than I thought that lasts 7.8 seconds for ~40Gbit/second. Anyway, as this is all internal so they have a lot of options.
        9y ago
kijiki 9y ago
Cumulus (disclaimer: I co-founded it) has customers that use these: http://www.mellanox.com/page/products_dyn?product_family=260&mtag=connectx_5_en_card http://www.mellanox.com/page/products_dyn?product_family=260... to dual 100G attach their servers to a pair of ToR switches. As a result, they'd really love to have something faster than 100G for their ToR to spine uplinks, but that isn't quite available yet. Insanely (and awesomely), Mellanox already has these: http://www.mellanox.com/page/products_dyn?product_family=266&mtag=connectx_6_en_card http://www.mellanox.com/page/products_dyn?product_family=266... dual 200G NIC.
- walrus01 9y ago
  100G NICs and dual port of the same version are a really good argument for the need for more pci-express 3.0 lanes in single socket servers. The AMD epyc is a step in the right direction. If you assume a server that might have two m.2 nvme SSD in it, those take up lanes, and then the nic will eat an x16 interface.

deaps 9y ago

Network Engineer here as well - I support one of the world's largest enterprise organizations (although my area of 'expertise' is pretty far down the stack - somewhere between the internet and the WAN circuits) - I can see no personal use for a 400 GbE port on anything. As far as my organization goes, there will always be a bottleneck somewhere else (speaking location to location across the WAN). The only applications I could see this being useful would be maybe huge ISP interconnects or maybe some big players in the video streaming world. That doesn't mean that future-proofing is a bad thing - of course cost will probably be a very large concern in the case of 400GbE...I see a lot of 40 gig ports that go completely unused, personally.

monocasa 9y ago
It's not hard to imagine 100GbE being used inside datacenters. AFAIK, 40GbE top of rack switch is pretty common.
- deaps 9y ago
  I've seen some 160GbE links in datacenters - 4x40's - how much they were actually utilized, I have no idea.
- walrus01 9y ago
  if you have a really beefy single bare metal hypervisor (typically xen or kvm) it's totally possible to give it 100GbE to the switch now: http://www.silicom-usa.com/pr/server-adapters/networking-adapters/100-gigabit-ethernet-networking-server-adapters/pe3100g2dqir_server_adapter/ http://www.silicom-usa.com/pr/server-adapters/networking-ada... I don't have anything this serious in my own production use, but I'm thinking of machines like dual socket x 32-core per socket xeons, or amd epyc, with 512GB to 1TB of RAM. then pass vlan tagged traffic off one switch port, to each specific guest VM, of which there might be dozens to 100+ VMs, as subinterfaces of that.
walshemj 9y ago
HPC Clusters would be one candidate

hinkley 9y ago

Can’t you saturate 100GB easily with a single disk array these days?

SteveNuts 9y ago
Yes, I've worked with several that can do that. The interesting thing with disk arrays now is the controller heads are becoming the bottleneck. You used to have many many shelves of SAS drives per controller pair, now the SSDs are so fast that the CPUs are saturated and you have to scale out, not up.
- hinkley 9y ago
  If I can't (reliably) handle 100GBps coming from the RAID array it sounds like I'm not going to be able to handle 400GBps coming in from the NIC. So these really are just for interconnects for the time being, until server hardware gets faster.
  - buzer 9y ago
    There are some applications which can handle 100Gbps and potentially higher speeds on servers. Usually they require using DPDK or similar frameworks with userland TCP stack (and thus are usually pretty specialized), but at least Netflix has managed to serve 90+Gbps of TLS traffic using "default" FreeBSD TCP stack (https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://medium.com/netflix-techblog/serving-100-gbps-from-an...).

cm2187 9y ago

Not working on anything like that myself, but if the direction going forward it to stream not just 4k TV, but also 4k video games (console in the cloud), you will need some healthy backbones (even if they still run only IPv4 in 5 years!)

simias 9y ago

Streaming uncompressed raw video over ethernet for TV production. It's still commonly done over dark fiber but people are slowly migrating towards IP. A single uncompressed 4K video stream is over 12Gbps, 1080p is 3Gbps and obviously when you're producing content you have a few of these running alongside each other (multiple cameras, feedback, big screens etc... Think something like the world cup or superbowl). It quickly adds up to a tremendous bandwidth. On top of that it needs very low latency and jitter to keep all the streams synchronized.

tatersolid 9y ago
Why would anyone stream uncompressed raw video? Even MPEG-2 at q=1 with no B-frames is about 1/10 the size, visually lossless, and adds only a few ms of latency.
- e12e 9y ago
  Looks like for editing you're looking at up towards 3.5 gbps per camera, not sure how much to add for sound. So a three camera production could easily push over 10 gbps. Pushing past 100 seems to be a bit of a stretch. Maybe for 3d?

fulafel 9y ago

5G base stations cite 50 Gbps bandwidth, connect a bunch of these together in an urban area and you might want 400G for their uplink/backbone/whatever it's called.

walrus01 9y ago
More realistically as cellular carriers are upgrading rooftop sites to the latest LTE gear, what I'm seeing as a carrier, they're fed by either a dedicated 1G fiber transport back to an aggregation site, the same at 10G on fiber, or point to point 60GHz (1-5 Gbps) mmwave, or 71-86GHz band ptp mmwave (1 to 10Gbps).
- fulafel 9y ago
  It sure sounds like 5G deployment is "realistic". Big operators have announced deployments & limited availability this year.

kierank 9y ago

Yes, in my company we generate and receive uncompressed video in software and get to these kinds of bandwidths from a couple of servers running dual 100GbE: https://www.youtube.com/watch?v=A4L5xEXXlas https://www.youtube.com/watch?v=A4L5xEXXlas

drewg123 9y ago

I'm not consuming it, but I can produce over 100Gb/s of traffic from a single CDN node if it is equipped with multiple 100GbE NICS. So 200GbE and 400GbE are exciting to me. (for context: https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://medium.com/netflix-techblog/serving-100-gbps-from-an...)