6 ms·

Interesting research paper but multicast is a non-starter for non-trivial production systems. 1. As implemented by most (read: all) network hw, it works fine w

by mritun 10y ago

Interesting research paper but multicast is a non-starter for non-trivial production systems. 1. As implemented by most (read: all) network hw, it works fine when congestion is low. As soon as congestion starts increasing, the packet loss amplifies. 2. If you have large multicast groups, the bloom filters used for membership test become very full, hence false positive rate rapidly increases. This leads to multicast storms. In my experience, anything which is not TCP (or, to lesser degree UDP) is a non-starter for building reliable distributed systems. Despite the claims of "lower overhead" the TCP protocol stack gets most love and hence has been optimized heavily on every significant operating system.

hardwaresofton 10y ago
Would you mind expanding a little more? If I'm understanding your point correctly -- you're saying that the sequencer failure case (whether it's modified switches or programmable switches, or software-based) is more common as congestion increases, and that's what makes this unusable in practice? Wouldn't traditional Paxos already be dead in the water at that point? Also regarding the point about multicast storms -- I was imagining something like <100 nodes in a multicast group -- is that naive? What obvious use case am I missing? I was basically thinking of this scheme in front of a database layer
polskibus 10y ago
Multicast has been used TIBCO Rendezvous to deliver high performance, high volume messaging in financial sector for ages. Can you elaborate on your claims?
- revertts 10y ago
  Messaging in the financial sector has very different requirements from messaging for a general service backend, which I assume is what the OP is referencing. As an example, a certain ecommerce site back in the day was heavily tibco based and migrated off due to inefficiencies at scale, multicast storms, debugability, and the general nature of multicast failures (for the most part you get no graceful degradation - which is fine when you have tight control of the network and do careful capacity planning, but isn't the mode that most tech companies live in). Anecdotally, non-multicast setups are easier to scale in hardware (because you can push commodity farther) and most critically are easier for the normal type of engineer you'd find in the tech sector to reason about (because many aren't comfortable with networking equipment, so it's much easier moving that state out of the network and into layers they can easily debug).
  - sseveran 10y ago
    If its the same ecommerce site I am thinking of, then there was also non-trivial message loss.
  - otterley 10y ago
    I worked on such a site in the mid-2000s. TIBCO Rendezvous was extremely efficient in the steady state, but boy, it had a catastrophic failure mode: during network congestion or receiver saturation, it would start repeating messages, causing the network congestion and saturation to worsen. The only way to stop the bleeding was to shut everything down. At any rate, most cloud SDNs don't support IP multicast (unless you use an overlay like Weave), so it's sort of a nonstarter for them. These days I might use Redis pub/sub along with a multi-level cache (e.g. LMDB + local Redis + remote Redis) if I need to distribute data to a large number of clients efficiently.
    - tptacek 10y ago
      This is my experience with Rendezvous as well. The very largest financial organizations using Tibco tend to have a whole group of engineers whose only job is to keep Rendesvous from throwing a rod. Their networks are also overprovisioned and their traffic rates are pretty straightforward to characterize. You can, of course, use Rendezvous without multicast. And: does anyone still use it? The last exchange I worked with that did was migrating from it to JMS (which sort of gives lie to the idea that Rendezvous's multicast performance was that big a deal).
      - otterley 10y ago
        Back in the 1990s and early 2000s, network bandwidth and server capacity was a lot lower, so IP multicast was the most efficient way to distribute data to hundreds or thousands of clients. Now that 10Gb+ is commonplace and modern servers can service hundreds or thousands of message bus consumers with efficient kernel-managed event loops, yes, it's arguably less needed now. But nobody will claim it's efficient from a network perspective. :)
- mritun 10y ago
  Yes, and with significant outages too. Again, I'm not saying multicast doesn't work. Multicast works well as long as you can guarantee that your network topology, components (switches and routers), servers and usage pattern will be predictable (and you test everything rigorously before going live). I can imagine that being true at some places with extremely tight focussed requirements. In a dynamic environment where you'd be scaling continuously and adding components and servers and building new services, this becomes a non-starter. Multicast implementations doesn't fail gracefully -- there's very little warning before things come to a complete halt and need complete shutdown and reboot of entire network.
jerf 10y ago
But we're not implementing multicast in some general sense, then pushing it to its limits as we try to scale it to thousands of devices. Isn't that the multicast failure case? What we're doing here is replacing something that would right now be using Paxos, which has a lot of message chatter and therefore can become bandwidth limited probably sooner than the multicast channel would be anyhow. You wouldn't use this to try to coordinate thousands of machines either way, you'd be doing this to replace something that ran at Paxos-scale previously. There's a bound here, and might it be below the multicast failure bound? (Honest question; I have no direct experience with multicast.)
- considrdharmful 10y ago
  This is correct.
- mritun 10y ago
  Multicast fails before bandwidth starts becoming an issue. Collisions and congestion are fact of life and do not necessarily mean that links are full (you may just have lots of packets stuck close together in time). Most TCP implementations handles it well and Linux has well written guides on how to tune parameters when the defaults don't match well to your specific situation well. Multicast just fails catastrophically in arcane ways that depend upon your specific hardware, network topology and mix of your firmware version on your switch/routers. It makes it very hard to "design" a product around that kind of fragile system.
- deleted 10y ago
  [deleted]
nickpsecurity 10y ago
Don't forget there's protocols like UDT available, too. HPC field particularly has been doing TCP alternatives either on UDP and custom (eg Active Messages) for a long time to help tackle some of issues you get with TCP. http://udt.sourceforge.net/ http://udt.sourceforge.net/
SEJeff 10y ago
Multicast storms can generally be avoided by IGMP Snooping, which virtually all modern high performance switches will support. Are you saying the above with an understanding of IGMP Snooping, or only in primitive network environments where the hardware / software switches lack it? When IGMP Snooping is enabled, the switches themselves will limit multicast messages only to ports that have actually joined a group, which prevents multicast storms (if I'm understanding you correctly) in the first place.
- mritun 10y ago
  It depends on the network. If the network is 100-1000 nodes and you can guarantee that the switches and routers you use to build it (and upgrade in future) implement it correctly and won't break/introduce bugs in future firmware release then yes. In the kind of environments I worked in, this was a non-starter. edit: also "limit multicast messages only to ports that have actually joined a group" part is usually implemented by bloom-filters that I talked about above. They work well as long as there are not many groups and not many members.
  - SEJeff 10y ago
    Sure in network topologies that size, you often have tiered access, leaf, and spine switches (and you can even build nonblocking topologies if you do things rihgt). Usually doesn't mean always. Many (most?) higher end switches use custom ASICs for IGMP Snooping, which means they scale fine until the ASIC limit at which they don't handle any more. In my career, I've worked in single networks with a max of maybe 4000 server nodes in 1 broadcast domain or so. Not huge, but I've seen no issues with big multicast uses provided you're using the right network cards / HBA and the right network switches. I've seen people build FPGAs to split market data into 10k+ multicast channels and was asked for help making Linux join so many channels (it is a PITA). It was amusing when we had to call around to find switch vendors which even supported it at all, most of them told us we were crazy (this was maybe 9 years ago). Take the Cisco Nexus switch as an example, it supports 32,000 IGMP groups, certain Aristas support 64,000, etc, and that is just for ethernet switches. Lets not get into Infiniband or some of the exotic Cray/IBM interconnects. I would be curious in your experiences if you're able to share it, but do very much believe there are options other than software for implementing multicast filtering on switches. What type of environments that you worked in made IGMP Snooping a non-starter?
liveoneggs 10y ago
Not only these failures but I've had nothing but trouble with multicast implementations on hardware and OS stacks just refusing to talk to each other. It's a highly unstable feature of the networking world and is not to be relied on.

by mritun 10y ago

hardwaresofton 10y ago

Would you mind expanding a little more? If I'm understanding your point correctly -- you're saying that the sequencer failure case (whether it's modified switches or programmable switches, or software-based) is more common as congestion increases, and that's what makes this unusable in practice? Wouldn't traditional Paxos already be dead in the water at that point? Also regarding the point about multicast storms -- I was imagining something like <100 nodes in a multicast group -- is that naive? What obvious use case am I missing? I was basically thinking of this scheme in front of a database layer

polskibus 10y ago

Multicast has been used TIBCO Rendezvous to deliver high performance, high volume messaging in financial sector for ages. Can you elaborate on your claims?

revertts 10y ago
Messaging in the financial sector has very different requirements from messaging for a general service backend, which I assume is what the OP is referencing. As an example, a certain ecommerce site back in the day was heavily tibco based and migrated off due to inefficiencies at scale, multicast storms, debugability, and the general nature of multicast failures (for the most part you get no graceful degradation - which is fine when you have tight control of the network and do careful capacity planning, but isn't the mode that most tech companies live in). Anecdotally, non-multicast setups are easier to scale in hardware (because you can push commodity farther) and most critically are easier for the normal type of engineer you'd find in the tech sector to reason about (because many aren't comfortable with networking equipment, so it's much easier moving that state out of the network and into layers they can easily debug).
- sseveran 10y ago
  If its the same ecommerce site I am thinking of, then there was also non-trivial message loss.
- otterley 10y ago
  I worked on such a site in the mid-2000s. TIBCO Rendezvous was extremely efficient in the steady state, but boy, it had a catastrophic failure mode: during network congestion or receiver saturation, it would start repeating messages, causing the network congestion and saturation to worsen. The only way to stop the bleeding was to shut everything down. At any rate, most cloud SDNs don't support IP multicast (unless you use an overlay like Weave), so it's sort of a nonstarter for them. These days I might use Redis pub/sub along with a multi-level cache (e.g. LMDB + local Redis + remote Redis) if I need to distribute data to a large number of clients efficiently.
  - tptacek 10y ago
    This is my experience with Rendezvous as well. The very largest financial organizations using Tibco tend to have a whole group of engineers whose only job is to keep Rendesvous from throwing a rod. Their networks are also overprovisioned and their traffic rates are pretty straightforward to characterize. You can, of course, use Rendezvous without multicast. And: does anyone still use it? The last exchange I worked with that did was migrating from it to JMS (which sort of gives lie to the idea that Rendezvous's multicast performance was that big a deal).
    - otterley 10y ago
      Back in the 1990s and early 2000s, network bandwidth and server capacity was a lot lower, so IP multicast was the most efficient way to distribute data to hundreds or thousands of clients. Now that 10Gb+ is commonplace and modern servers can service hundreds or thousands of message bus consumers with efficient kernel-managed event loops, yes, it's arguably less needed now. But nobody will claim it's efficient from a network perspective. :)
mritun 10y ago
Yes, and with significant outages too. Again, I'm not saying multicast doesn't work. Multicast works well as long as you can guarantee that your network topology, components (switches and routers), servers and usage pattern will be predictable (and you test everything rigorously before going live). I can imagine that being true at some places with extremely tight focussed requirements. In a dynamic environment where you'd be scaling continuously and adding components and servers and building new services, this becomes a non-starter. Multicast implementations doesn't fail gracefully -- there's very little warning before things come to a complete halt and need complete shutdown and reboot of entire network.

jerf 10y ago

But we're not implementing multicast in some general sense, then pushing it to its limits as we try to scale it to thousands of devices. Isn't that the multicast failure case? What we're doing here is replacing something that would right now be using Paxos, which has a lot of message chatter and therefore can become bandwidth limited probably sooner than the multicast channel would be anyhow. You wouldn't use this to try to coordinate thousands of machines either way, you'd be doing this to replace something that ran at Paxos-scale previously. There's a bound here, and might it be below the multicast failure bound? (Honest question; I have no direct experience with multicast.)

considrdharmful 10y ago
This is correct.
mritun 10y ago
Multicast fails before bandwidth starts becoming an issue. Collisions and congestion are fact of life and do not necessarily mean that links are full (you may just have lots of packets stuck close together in time). Most TCP implementations handles it well and Linux has well written guides on how to tune parameters when the defaults don't match well to your specific situation well. Multicast just fails catastrophically in arcane ways that depend upon your specific hardware, network topology and mix of your firmware version on your switch/routers. It makes it very hard to "design" a product around that kind of fragile system.
deleted 10y ago
[deleted]

nickpsecurity 10y ago

Don't forget there's protocols like UDT available, too. HPC field particularly has been doing TCP alternatives either on UDP and custom (eg Active Messages) for a long time to help tackle some of issues you get with TCP. http://udt.sourceforge.net/ http://udt.sourceforge.net/

SEJeff 10y ago

Multicast storms can generally be avoided by IGMP Snooping, which virtually all modern high performance switches will support. Are you saying the above with an understanding of IGMP Snooping, or only in primitive network environments where the hardware / software switches lack it? When IGMP Snooping is enabled, the switches themselves will limit multicast messages only to ports that have actually joined a group, which prevents multicast storms (if I'm understanding you correctly) in the first place.

mritun 10y ago
It depends on the network. If the network is 100-1000 nodes and you can guarantee that the switches and routers you use to build it (and upgrade in future) implement it correctly and won't break/introduce bugs in future firmware release then yes. In the kind of environments I worked in, this was a non-starter. edit: also "limit multicast messages only to ports that have actually joined a group" part is usually implemented by bloom-filters that I talked about above. They work well as long as there are not many groups and not many members.
- SEJeff 10y ago
  Sure in network topologies that size, you often have tiered access, leaf, and spine switches (and you can even build nonblocking topologies if you do things rihgt). Usually doesn't mean always. Many (most?) higher end switches use custom ASICs for IGMP Snooping, which means they scale fine until the ASIC limit at which they don't handle any more. In my career, I've worked in single networks with a max of maybe 4000 server nodes in 1 broadcast domain or so. Not huge, but I've seen no issues with big multicast uses provided you're using the right network cards / HBA and the right network switches. I've seen people build FPGAs to split market data into 10k+ multicast channels and was asked for help making Linux join so many channels (it is a PITA). It was amusing when we had to call around to find switch vendors which even supported it at all, most of them told us we were crazy (this was maybe 9 years ago). Take the Cisco Nexus switch as an example, it supports 32,000 IGMP groups, certain Aristas support 64,000, etc, and that is just for ethernet switches. Lets not get into Infiniband or some of the exotic Cray/IBM interconnects. I would be curious in your experiences if you're able to share it, but do very much believe there are options other than software for implementing multicast filtering on switches. What type of environments that you worked in made IGMP Snooping a non-starter?

liveoneggs 10y ago

Not only these failures but I've had nothing but trouble with multicast implementations on hardware and OS stacks just refusing to talk to each other. It's a highly unstable feature of the networking world and is not to be relied on.