6 ms·
New Optical Form Factors for 400 Gigabit Ethernet
- bjoli 9y agoDoes aanyone here work with consuming (not just passing it kn) these amounts of traffic? How do you manage to get anything meaningful out of it?
- _zoltan_ 9y agodisclaimer: I work at IBM Research we have just finished setting up our new big data crunching system which sits on 48x100GbE links on the full flash storage side and on 32x100GbE on the client side (split 100GbE -> 4x25GbE). at this point we are saturating most of the links.
- walrus01 9y agonetwork engineer here: nobody connects a 200 or 400GbE interface linecard off a router to a single anything... The purpose is for ISP-to-ISP interconnections for truly huge amounts of traffic. For example direct peering in the same IX/carrier hotel between a huge source of content (youtube/google, netflix, etc) and a huge ISP (comcast, centurylink). For a big ISP, also things like 100GbE connections from a city's pair of core routers to slightly smaller aggregation routers. edit: I don't know that anything "consumes" data in the way that the questioner is asking. It's more about the aggregate amount of data. One netflix 4K stream to an xbox one s is about 15.75 Mbps. Now multiply that by a hundred thousand netflix subscribers in a typical comcast service area, any of whom at any given time might be sitting around and watching Altered Carbon or Breaking Bad.
- jandrese 9y agoThe only potential consumers I can imagine are like the Great Firewall of China traffic monitoring systems.
- icebraining 9y agoOr the NSA: https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
- Retric 9y agoRaw high frame rate video can into that range. Phantom v2640 for example does 6600 frames per second at a at 2048 x 1952 resolution = 26 Gpx/sec or something like 100Gbit. Currently they simply buffer everything on Camera and use 10GigE, but in theory they could look into 100+GbE connections for newer designs.
- walrus01 9y agoupcoming on-camera flash memory interface https://en.wikipedia.org/wiki/CFexpress https://en.wikipedia.org/wiki/CFexpress basically a way of putting a M.2 interface NVME PCI-Express SSD into a camera. https://petapixel.com/2018/04/09/prograde-digital-unveils-worlds-first-1tb-cfexpress-card/ https://petapixel.com/2018/04/09/prograde-digital-unveils-wo...
- Retric 9y agoAccording to the link that's currently limited to 2 GB/s which far to slow to use for this. Phantom v2640 needs ~50+x that speed to do it in real time and I assume the bandwidth needs will keep increasing on the next model.
- walrus01 9y agoSo what storage device does it use currently? except for RAM, there is not much faster than a NVME based, PCI-Express 3.0 x8 interface SSD. If it's some truly ridiculous bitrate it probably uses proprietary packs of RAM with battery backing.
- Retric 9y agoOk, found more info on that model right now it's got up to 288 GB of internal RAM. At 12 bit per pixel which is lower than I thought that lasts 7.8 seconds for ~40Gbit/second. Anyway, as this is all internal so they have a lot of options.
- 9y ago
- kijiki 9y agoCumulus (disclaimer: I co-founded it) has customers that use these: http://www.mellanox.com/page/products_dyn?product_family=260&mtag=connectx_5_en_card http://www.mellanox.com/page/products_dyn?product_family=260... to dual 100G attach their servers to a pair of ToR switches. As a result, they'd really love to have something faster than 100G for their ToR to spine uplinks, but that isn't quite available yet. Insanely (and awesomely), Mellanox already has these: http://www.mellanox.com/page/products_dyn?product_family=266&mtag=connectx_6_en_card http://www.mellanox.com/page/products_dyn?product_family=266... dual 200G NIC.
- walrus01 9y ago100G NICs and dual port of the same version are a really good argument for the need for more pci-express 3.0 lanes in single socket servers. The AMD epyc is a step in the right direction. If you assume a server that might have two m.2 nvme SSD in it, those take up lanes, and then the nic will eat an x16 interface.
- deaps 9y agoNetwork Engineer here as well - I support one of the world's largest enterprise organizations (although my area of 'expertise' is pretty far down the stack - somewhere between the internet and the WAN circuits) - I can see no personal use for a 400 GbE port on anything. As far as my organization goes, there will always be a bottleneck somewhere else (speaking location to location across the WAN). The only applications I could see this being useful would be maybe huge ISP interconnects or maybe some big players in the video streaming world. That doesn't mean that future-proofing is a bad thing - of course cost will probably be a very large concern in the case of 400GbE...I see a lot of 40 gig ports that go completely unused, personally.
- monocasa 9y agoIt's not hard to imagine 100GbE being used inside datacenters. AFAIK, 40GbE top of rack switch is pretty common.
- deaps 9y agoI've seen some 160GbE links in datacenters - 4x40's - how much they were actually utilized, I have no idea.
- walrus01 9y agoif you have a really beefy single bare metal hypervisor (typically xen or kvm) it's totally possible to give it 100GbE to the switch now: http://www.silicom-usa.com/pr/server-adapters/networking-adapters/100-gigabit-ethernet-networking-server-adapters/pe3100g2dqir_server_adapter/ http://www.silicom-usa.com/pr/server-adapters/networking-ada... I don't have anything this serious in my own production use, but I'm thinking of machines like dual socket x 32-core per socket xeons, or amd epyc, with 512GB to 1TB of RAM. then pass vlan tagged traffic off one switch port, to each specific guest VM, of which there might be dozens to 100+ VMs, as subinterfaces of that.
- walshemj 9y agoHPC Clusters would be one candidate
- hinkley 9y agoCan’t you saturate 100GB easily with a single disk array these days?
- SteveNuts 9y agoYes, I've worked with several that can do that. The interesting thing with disk arrays now is the controller heads are becoming the bottleneck. You used to have many many shelves of SAS drives per controller pair, now the SSDs are so fast that the CPUs are saturated and you have to scale out, not up.
- hinkley 9y agoIf I can't (reliably) handle 100GBps coming from the RAID array it sounds like I'm not going to be able to handle 400GBps coming in from the NIC. So these really are just for interconnects for the time being, until server hardware gets faster.
- buzer 9y agoThere are some applications which can handle 100Gbps and potentially higher speeds on servers. Usually they require using DPDK or similar frameworks with userland TCP stack (and thus are usually pretty specialized), but at least Netflix has managed to serve 90+Gbps of TLS traffic using "default" FreeBSD TCP stack (https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://medium.com/netflix-techblog/serving-100-gbps-from-an...).
- cm2187 9y agoNot working on anything like that myself, but if the direction going forward it to stream not just 4k TV, but also 4k video games (console in the cloud), you will need some healthy backbones (even if they still run only IPv4 in 5 years!)
- simias 9y agoStreaming uncompressed raw video over ethernet for TV production. It's still commonly done over dark fiber but people are slowly migrating towards IP. A single uncompressed 4K video stream is over 12Gbps, 1080p is 3Gbps and obviously when you're producing content you have a few of these running alongside each other (multiple cameras, feedback, big screens etc... Think something like the world cup or superbowl). It quickly adds up to a tremendous bandwidth. On top of that it needs very low latency and jitter to keep all the streams synchronized.
- tatersolid 9y agoWhy would anyone stream uncompressed raw video? Even MPEG-2 at q=1 with no B-frames is about 1/10 the size, visually lossless, and adds only a few ms of latency.
- e12e 9y agoLooks like for editing you're looking at up towards 3.5 gbps per camera, not sure how much to add for sound. So a three camera production could easily push over 10 gbps. Pushing past 100 seems to be a bit of a stretch. Maybe for 3d?
- fulafel 9y ago5G base stations cite 50 Gbps bandwidth, connect a bunch of these together in an urban area and you might want 400G for their uplink/backbone/whatever it's called.
- walrus01 9y agoMore realistically as cellular carriers are upgrading rooftop sites to the latest LTE gear, what I'm seeing as a carrier, they're fed by either a dedicated 1G fiber transport back to an aggregation site, the same at 10G on fiber, or point to point 60GHz (1-5 Gbps) mmwave, or 71-86GHz band ptp mmwave (1 to 10Gbps).
- fulafel 9y agoIt sure sounds like 5G deployment is "realistic". Big operators have announced deployments & limited availability this year.
- kierank 9y agoYes, in my company we generate and receive uncompressed video in software and get to these kinds of bandwidths from a couple of servers running dual 100GbE: https://www.youtube.com/watch?v=A4L5xEXXlas https://www.youtube.com/watch?v=A4L5xEXXlas
- drewg123 9y agoI'm not consuming it, but I can produce over 100Gb/s of traffic from a single CDN node if it is equipped with multiple 100GbE NICS. So 200GbE and 400GbE are exciting to me. (for context: https://medium.com/netflix-techblog/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99 https://medium.com/netflix-techblog/serving-100-gbps-from-an...)
- hesdeadjim 9y agoI remember having to schedule a 780Kb download overnight and hoping no one picked the phone up. Or years later having to download shows in advance (illegally because they weren’t available) because streaming wasn’t possible. Now I casually (and legally) boot up any show I feel like within seconds at HD resolutions. I love thinking about the sheer amount of data that is traversing around the internet now. And it’s more fun to imagine what becomes possible when you bump up another order of magnitude or two. In games for instance, a significant limiting factor for implementing large player counts in fast-paced multiplayer come down to bandwidth. The amount needed, for the most part, scales linearly per-player as the player count increases. Add more bandwidth (and some more CPU power) and games can evolve entirely new experiences. The battle royale games like PUBG are a great example, but due to bandwidth issues and CPU usage the server tick rate is much slower than normal fast-paced games (20hz vs 60hz) and therefore the experience suffers quite a bit.
- Klinky 9y agoMultiplayer gaming has more issues with latency than a lack of raw bandwidth.
- hesdeadjim 9y agoNot necessarily, depending on how the game is implemented a player with bad ping can be the only one who suffers the majority of the effects of their bad connection. I am currently working with this tech: https://www.photonengine.com/en-US/Quantum https://www.photonengine.com/en-US/Quantum And it does deliver on their promises, impressive stuff.
- Lorin 9y agoThis looks neat, think it'd do well with upcoming projects like Star Citizen from CIG? Physics sims have to be limited dramatically atm.
- thathappened 9y agoLatency is usually from bandwidth being queued somewhere along the path
- samstave 9y agoWhat are distance limitations at 400?
- dogecoinbase 9y agoTrans-Pacific :) But more seriously, this article is about the device-side electrical interfaces -- the modules will transceive on any one of a number of physical interfaces depending on the specific needs (ultra-short-range for in-rack/in-DC, though that's a less likely application of this technology). Most multi-lane paths like these are using multi-strand optical cabling for short-range, and typically multiplexing the signal using on-board DWDM in/around the 1550nm band for two-strand long-range (which is primarily constrained by the power available to the module). This band is used because it can also be amplified inline (ref: erbium-doped amplifier) to extend the range. (this is an oversimplification, I am not your network engineer, I am not a network engineer, consult a qualified network engineer in your jurisdiction)
- Osiris 9y agoI wonder when we will start seeing > 1Gpbs Ethernet in consumer products. I've read about initiatives for 5 or 10Gb consumer Ethernet. I'd love to be able to push/pull from my NAS at > 100MB/s.
- dogecoinbase 9y agoIn-wall cabling ends up being the limiting factor for a lot of this -- definitely check out the Nbase-T stuff coming out, which can negotiate from 1Gb up to 10Gb depending on what the cable can support (Ubiquiti just announced an AP using it, Juniper should have switches coming out this quarter IIRC). 10Gb ethernet works fine, but the power requirements for copper (base-t) are high and SFP+ is a bad standard for consumers (finicky connectors, low mating cycle rating, etc).
- dweekly 9y agoIt's a fair question; I suspect most of the reason for the lag is that consumer internet connections rarely offer >1gbps, consumer LAN applications requiring >1gbps of traffic haven't emerged (for NAS/SAN, a directly connected USB 3 is probably the way to go for 10gbps transfer, which is what you're looking for here), and WiFi - how most people are connecting their consumer devices to the network - doesn't offer more than a gigabit of goodput in almost any consumer configuration. (e.g. 802.11ac with two spatial streams and 80MHz channels gets ~700mbps at MCS8) That said, the NBASE-T and MGBASE-T efforts did consolidate to produce a 802.3bz standard that can carry up to 2.5gbps on a Cat 5e line and 5gbps on Cat 6, and we are starting to see consumer and SMB switches, APs, and NICs that support bz. Moral of the story is wire for Cat 6 and look to upgrade to bz in the next year or so.
- e12e 9y agoI see how cat6 works for economics of scale; but when I last year looked to price 10gbs switches for a lan for a school (with a meagre budget) - I was surprised at how expensive second hand 10gps gear was. In the end we stayed with 1gbps even though we had some use-cases where 10gbps would make sense; mostly budget San/nas for video edit/backup. And to a lesser degree audio editing. I think the 1gbps Cisco switches were hand-me-downs from a university - but I saw no indication of reasonably priced 10gbps switches with minimal management (we didn't need "all" of Cisco ios - but minimal remote management and steady performance would be nice). We also threw some lan parties, so budget soho stuff that'd fall over easily wasn't really an option.
- francoisLabonte 9y agoThe real debate is between OSPF and QSFP-DD as they are the only ones with a shot at high density. QSFP-DD is backwards compatible so that you can run your 400G port as a 100G or 40G with 4 lanes at 25G/10G. OSFP will be able to do the same thing but with an adapter from OSFP to QSFP But what this article fails to mention is that OSFP is the connector of the future that will support 8 lanes of 100G to support 800G ethernet with good signal integrity and capability to support high power optics for longer distances. ( This looks to be in 2020 ) So QSFP-DD is a 1 generation connector... OSFP will see us through 400G and 800G. Disclaimer I work at Arista Networks who is bullish on OSFP. For more info see Andy Bechtolsheim's talk on 400G optics at OCP https://www.youtube.com/watch?v=Kotu6B7AQpk https://www.youtube.com/watch?v=Kotu6B7AQpk
- mmmBacon 9y agoThere is no real debate between QSFP-DD and OSFP. OSFP will exist at Google but nearly everyone else is going to use QSFP-DD. The main reason is backwards compatibility for data center upgrades. The only reason to use OSFP might be because you want to use a 400G-ZR plug. But with the advent of 7nm DSP, vendors will be able to fit the ZR into the DD plug. So if the above is true OSFP is not needed now. Honestly, 800G is only incremental at best and frankly it’s not clear that An 800G pluggable is the way to go. The 800G presumes 32 radix and that breaks down power-wise in the datacenter when we get to spine switches where we need 128 radix. The reason being that we need so many power hungry chips to make a high radix switch fully non-blocking (eg Clos topology). 112G SERDES will have limited reach and hence require many retimers which eat power too. So the OSFP is in my view is A solution looking for a problem.
- walrus01 9y agoI am also optimistic about OSFP as compared to the QSFP size for 400G. I'm not a laser engineer but I do have a pretty good understanding of how 400G will be achieved through two strands of singlemode. For medium reach optics it's actually an 8-channel CWDM with prism built into each optic, so the optic body needs to be large enough to dissipate the heat from eight separate 25 Gbps lasers. CFP size will obviously handle that but is a true first generation solution and is huge. The only bad thing about OSFP is that its highly similar name will confuse the hell out of network engineers who never see the OSI layer 1 of a network, and deal with things at OSI layer 3. OSFP != OSPF (the routing protocol).