9 ms·
If you're wondering what this kind of optic costs the rest of us: http://www.fs.com/products/65219.html http://www.fs.com/products/65219.html (£1,081) Although
by thomseddon 9y ago
If you're wondering what this kind of optic costs the rest of us: http://www.fs.com/products/65219.html http://www.fs.com/products/65219.html (£1,081)
Although you'd pay at least 10-20x that if it had a big vendor name on the top.
- ericd 9y agoHa awesome! Is this switch-to-switch only, or are there 100 gig optical connectors for servers as well? I like the 2 km max cable run length.
- erentz 9y agoYou use CWDM4 for switch to switch. Cheaper/easier to use DAC cables for switch to server connections. Though I don't know anyone doing 100G to server yet, so you'd more likely be using DAC breakout cables to deliver 25G to the server.
- ericd 9y agoCool, thanks for the info. Wondering if this is remotely worth it for a home computing cluster yet.
- shouldbworking 9y agoLinux can route traffic over 40G on regular hardware as long as packet sizes are fairly large >~150 bytes. I would guess that serving static assets from ram would be able to saturate that on a single machine
- gonzo 9y ago150 bytes is a fairly small payload. I think you dropped a 0. kernel bypass will get you line rate 40gbps w/64 byte packets. We've been showing IPsec over 40g links during the past week. https://twitter.com/netgateusa/status/853694461456646144 https://twitter.com/netgateusa/status/853694461456646144
- shouldbworking 9y agoNeat! I've heard you can get line rate on slightly bigger packets without bypass using IPVS with direct return since it bypasses part of handling. In a lot of scenarios the big pipe is right at the load balancer so this is useful.
- shaklee3 9y agoYou can do 40gbe even if packet sizes are small using any kernel bypass technique and multiple cores. Mellanox claims up to 130Mpps.
- shouldbworking 9y agoIs this just blind forwarding or does that include some decent routing logic? I've looked into kernel bypass but I keep shying away because it seems to proprietary, especially the virtualized flavor which seems to be Intel only
- shaklee3 9y agoBlind forwarding. It's left to the application to do everything.
- en4bz 9y ago100G, probably not unless you are extremely wealthy and you have a burning desire to be destitute. 40G is definitely doable for a homelab. Used 40Gbe adapters are roughly $200 and DAC cables are < $100. QSPF+ Switches are much harder to find used and/or cheap though. This means that p2p solutions or ring networks are generally preferred unless you are wealthy. If you want fiber, which you don't need for a homelab, you pay the prices listed above.
- jsjohnst 9y agoAgree with other commenters, not sure what the 2.5x increase in traffic would open for you in a home setting. I use 40gb fiber between switches on opposite ends of my place and 10gb connections between switch and computers and I've always found something else to be my bottleneck, not the internal network itself. I can saturate my wifi network easily though (I've got a $20k retail Aruba wifi network, yet still not enough). I buy most of my gear used so it's not that expensive (would've been very expensive retail), but still overkill unless your a network nerd (like me) or are doing something out of your home you likely should be using a DC.
- ericd 9y agoYeah, I think 10 G will be more than good enough for now. Thanks for the info!
- thomseddon 9y agoLooks like you can get 100Gb NICs for servers e.g. https://www.hpe.com/uk/en/product-catalog/servers/server-adapters/pip.hpe-100gb-1-port-op101-qsfp28-x8-pcie-gen3-with-intel-omni-path-architecture-adapter.1008830960.html https://www.hpe.com/uk/en/product-catalog/servers/server-ada... (Intel, Mellanox etc also seem to have their own) On the distance, there are other specs that support 100Gb up to 20km on there!
- ericd 9y agoHaha for when you need another DC that's sort of across town, or you're trying to add artificial latency intra DC with enormous coils of fiber. Not too unreasonably expensive - roughly in line with a GeForce 1080, and that's after the Official HP Hardware markup. I wonder how much the switch/cabling is...
- penagwin 9y ago> trying to add artificial latency intra DC with enormous coils of fiber. Do people really do this?! (I don't have DC experience, serious question)
- ericd 9y agoHeh like the peers said, it was a reference to what IEX does.
- snowwindwaves 9y agoAt least one stock exchange has: https://en.m.wikipedia.org/wiki/IEX https://en.m.wikipedia.org/wiki/IEX
- Dylan16807 9y agoI believe that's a reference to IEX's magic shoebox. 61 kilometers of cable that provide 350 microseconds of delay for stock market voodoo purposes.
- walrus01 9y agoThere are Intel chipset single and dual port 100GbE pci-express 3.0 NICs with good Linux kernel support in v4.8+.
- drewg123 9y agoMy understanding is that this is just a qsfp28 transceiver using a new optical format. There have have been 100GbE server NICs for almost two years (Mellanox ConnectX 4, using mlx5 driver) using QSFP28. We've been using them with great success at Netflix in our flash storage appliances. We are able serve well over 90Gb/s from single machines with these NICs using our tuned/enhanced FreeBSD and nginx.
- shouldbworking 9y ago> We are able serve well over 90Gb/s from single machines with these NICs using our tuned/enhanced FreeBSD and nginx. Incredible
- shaklee3 9y agoCan you describe why you choose freebsd over Linux+DPDK? I realize freebsd has the fast path built in, but I would think it's lacking in other areas.
- toast0 9y agoJust wondering, what kind of things do you think FreeBSD is lacking in? Regardless of those things, the OpenConnect boxes are doing a pretty small subset of possible server tasks, it's basically serving static content from disk, and updating the static content occasionally. This is a task that FreeBSD has been excelling at, since basically forever. FreeBSD is a pretty stable target to tweak on as well, Netflix moved TLS bulk encryption into sendfile(), which helps avoid the transitions from kernel to user space, but by putting more stuff in kernel space, rather than the DPDK method of putting more stuff in user space. They've continued to tweak sendfile, which I imagine helped them get up to nearly 100Gbps out. I haven't had the pleasure of running 100G network, but I had to do very little tuning to saturate 10G with FreeBSD on http download site, and TLS cpu was the primary thing holding me back when moving it to https. Bulk download was moved away from my team before I got new servers with 2x10G and fancier processors, so I was never able to see if I could saturate that too :(
- drewg123 9y agoWe're pretty much the opposite of DPDK. Rather than moving stuff out of the kernel, we move stuff into it. We're actually moving stuff like TLS encryption into the kernel -- see our kernel TLS papers at AsiaBSDCon We do all of our stack "traditionally", in the kernel, whereas DPDK moves things into userspace. By using a traditional stack with async sendfile in the kernel, we benefit from the VM page cache, and reduce IO requirements at peak capacity. There are no memory to memory copies, very little kernel/user boundary crossing, and no AIO. Using single-socket Intel Xeon E5-2697A v4, we serve at 90Gb/s using roughly 35-50% CPU (which will increase as more and more clients adopt HTTPS streaming). There is no question that FreeBSD is lacking in a number of areas. For example, device support is a constant struggle.