4 ms·

The fact that the poster mentioned IP multicast and Aeron implies they are talking about HFT and Stock Exchange environments, where high performance switches wi

by posnet 5y ago

The fact that the poster mentioned IP multicast and Aeron implies they are talking about HFT and Stock Exchange environments, where high performance switches with explicit support for low latency multicast are the norm and not the exception. There is a Signals and Threads podcast episode that goes a bit into the history of it. https://www.youtube.com/watch?v=triyiLwqWUI https://www.youtube.com/watch?v=triyiLwqWUI (Transcript) https://signalsandthreads.com/multicast-and-the-markets/ https://signalsandthreads.com/multicast-and-the-markets/

mgaunard 5y ago
Pretty much any L3 switch supports UDP multicast. What can be more rare is PIM support which is a router feature. If anything in HFT environments, people use L2 switches for latency reasons. Those operate at the Ethernet level, so they don't really care about IP at all. Anyway I don't see why that's specific to electronic trading or even just a low-latency concern. Sending the same traffic to hundreds of people with unicast means using hundreds of times the bandwidth, which is a huge problem.
- posnet 5y ago
  Oh I'm well aware, I've loaded more than enough Metamako alpha firmware releases for one career. But there are a lot of professional software devs with zero real networking experience. Sure, they may understand the text book definition of TCP or they may even have seen pictures of fibre with labels like 'this is how far light travels in a nanosecond'. But would have no idea how to calculate the serialization latency of a 10G link, let alone know the duty cycle required to saturate one. But none of that matters in the cloud (which is where twitter is stacking their jenga tower in the original blog post). And even if both Google and AWS have custom silicon (or at least FPGAs) doing hardware offload for their internal SDN encapsulation protocols at the server level and that their custom switches all support it, it doesn't matter. They hide all of that from you the customer and rarely even acknowledge it's existence.
  - jeffbee 5y ago
    One does wish the cloud network were a little more exposed to the tenant so we can do fancy stuff. Amazon's EFA is as close as they get. On the other hand I suspect Google of having bog-standard merchant silicon switches and maybe custom silicon NICs in their latest and greatest machines but for the most part off-the-shelf stuff at the host as well.

by posnet 5y ago

mgaunard 5y ago

Pretty much any L3 switch supports UDP multicast. What can be more rare is PIM support which is a router feature. If anything in HFT environments, people use L2 switches for latency reasons. Those operate at the Ethernet level, so they don't really care about IP at all. Anyway I don't see why that's specific to electronic trading or even just a low-latency concern. Sending the same traffic to hundreds of people with unicast means using hundreds of times the bandwidth, which is a huge problem.

posnet 5y ago
Oh I'm well aware, I've loaded more than enough Metamako alpha firmware releases for one career. But there are a lot of professional software devs with zero real networking experience. Sure, they may understand the text book definition of TCP or they may even have seen pictures of fibre with labels like 'this is how far light travels in a nanosecond'. But would have no idea how to calculate the serialization latency of a 10G link, let alone know the duty cycle required to saturate one. But none of that matters in the cloud (which is where twitter is stacking their jenga tower in the original blog post). And even if both Google and AWS have custom silicon (or at least FPGAs) doing hardware offload for their internal SDN encapsulation protocols at the server level and that their custom switches all support it, it doesn't matter. They hide all of that from you the customer and rarely even acknowledge it's existence.
- jeffbee 5y ago
  One does wish the cloud network were a little more exposed to the tenant so we can do fancy stuff. Amazon's EFA is as close as they get. On the other hand I suspect Google of having bog-standard merchant silicon switches and maybe custom silicon NICs in their latest and greatest machines but for the most part off-the-shelf stuff at the host as well.