5 ms·
We have been moving away from IB for our platform (algorithmic trading) since Ethernet now has almost comparable latency and is a lot easier to understand and m
by abcdef123xyz123 8y ago
We have been moving away from IB for our platform (algorithmic trading) since Ethernet now has almost comparable latency and is a lot easier to understand and manage.
- godelmachine 8y agoMay I ask what’s IB?
- sacheendra 8y agoinfiniband - The high performance interconnect by Mellanox
- SEJeff 8y agoBy Voltaire, the infiniband company acquired by mellanox :) Their ethernet accelerator VMA stands for "Voltaire Messaging Accelerator".
- wmf 8y agoIIRC Mellanox was always doing Infiniband; they made the hardware and Voltaire wrote the drivers (before people understood open source and created OFED).
- SEJeff 8y agoMellanox made the HCAs (aka network cards) and Voltaire made the switches. Mellanox had their own drivers and their own MOFED (their incompatible fork of openfabrics upstream OFED with Mellanox specific enhancements).
- ams6110 8y agoWe just pulled a Voltaire switch from our data center. They definitely made hardware. Or at least put their name on it.
- patrickg_zill 8y agoInfiniband is actually a standard and Mellanox became the go to supplier. However in the early stages there were many more suppliers.
- deleted 8y ago[deleted]
- ckozlowski 8y agoInfiniBand https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand
- shaklee3 8y agoUnderstand, manage, and buy networking equipment for. Infiniband is a thing of the past, especially with the Mellanox VPI adapters that support both Ethernet and Infiniband with a single bit flipped on the adapter.
- jabl 8y agoIDK, IME IB is pretty much plug and play in an HPC setting. Plain ethernet is too, sure, but if you want to do HPC type workloads you'll have to do a lot of configuration and testing to setup DCB, RoCEv2, EVPN+VXLAN+BGP or such. But I think this is the way the market is going in the longer term.
- shaklee3 8y agoIt is, but if you're a large enough company to be buying millions of dollars in adapters and switches, reading a guide from Mellanox to turn on DCB should be fairly seamless. RoCEv2 is API-compatible with IB for the most part, so there is really no configuration on that layer. The other pieces -- not really sure what you're getting at. Most of those are for going across data centers, which IB won't do anyways. At least Ethernet would give you the option to run RDMA from the east coast to the west coast.
- jabl 8y ago> The other pieces -- not really sure what you're getting at. What I'm getting at is setting up clusters larger than what you can fit behind a single switch. So you'll want e.g. a CLOS fabric with multipathing (the typical IB setup, FWIW). As Trill and SPB seem pretty dead, it seems the momentum is to do the multipathing at the L3 level, using the aforementioned EPVN+VXLAN+BGP, or something similar.
- wmf 8y agoYou really don't need EVPN+VXLAN though. (And if you do need it I recommend finding a way to not need it.)
- angry_octet 8y agoLatency really isn't comparable, but now probably isn't an issue for algorithmic trading. Still many other closely coupled codes for which it dominates cluster performance. IB is actually easier to reason about and debug than DCE, but obviously a different community of practice.