4 ms·

Architecturally speaking I've always liked building process-group middleware atop multicast, since it has the clearest mechanical sympathy down to the wire and

Architecturally speaking I've always liked building process-group middleware atop multicast, since it has the clearest mechanical sympathy down to the wire and can be built up to handle things like virtual synchrony or tuplespaces with potential for zero-copy I/O if you're careful. My favourite example of off-the-shelf implementations would be the Spread Toolkit¹𝄒², a distant second would be tib/rendezvous². Unfortunately deploying multicast in practice requires having several infrastructure ducks in a row that remain hard to achieve in enterprise environments let alone public clouds, so the devops mafia generally disregard it and deploy sidecar proxies in a unicast mesh instead, with log-structured event stores as the pub/sub core. There has been some continuing research into multicast as middleware³ but to my chagrin (albeit not my surprise) the barriers of infrastructure mean it hasn't developed as an application layer nearly as much as the unicast approaches. [1] http://www.spread.org http://www.spread.org but sadly now either defunct or classified, not sure which, the whole project kinda went suddenly silent ca.2018 [2] not zero-copy; you'd have to roll your own for that [3] e.g. out of the CCRG at UCSC https://ccrg.ucsc.edu/about-our-work/research-areas/routing-and-multicasting-services/ https://ccrg.ucsc.edu/about-our-work/research-areas/routing-...

klabb3 3y ago
Never used multicast outside of local networks, but I was wondering: does you like it for the perf benefits or are there practical advantages that makes multicast systems easier to reason about or program for?
- inopinatus 3y ago
  I like the efficiency, but moreover I like low latency and highly observable solutions, perhaps because my first and most formative job was building hard-real-time interactive devices and this left me with a lifelong distaste for any system that responds more slowly than 100ms. Proxies inevitably introduce the latency of buffering, and this tends to go exponential to the point of timeout beyond 99%ile. Not saying network protocols don’t suffer at their limit too, but it’s more graceful and more directly instrumented. Neither are particularly straightforward to reason about, but ordering guarantees (in the Lamport sense) are (in my experience) harder to ensure at regional scale with proxies and service meshes. There’s a counterexample, of course: at intercontinental and interplanetary scale we lack the ability to stabilise tree topologies with predictable forwarding latency and are forced to use service proxies and bounded timestamps on replicated data to achieve ordering guarantees instead, of which the best known example is possibly TrueTime. This trades interactivity for scale. Multicast is definitely not easier to program for, and although I could attribute that to a shortage of high-level libraries, in practice the shortfall is mediated by the infrastructure dependency. This places it beyond the radar of most system designers in commercial environments, thus completing the elements of a downward popularity spiral. It’s a shame, but that’s tech.

klabb3 3y ago

Never used multicast outside of local networks, but I was wondering: does you like it for the perf benefits or are there practical advantages that makes multicast systems easier to reason about or program for?

inopinatus 3y ago
I like the efficiency, but moreover I like low latency and highly observable solutions, perhaps because my first and most formative job was building hard-real-time interactive devices and this left me with a lifelong distaste for any system that responds more slowly than 100ms. Proxies inevitably introduce the latency of buffering, and this tends to go exponential to the point of timeout beyond 99%ile. Not saying network protocols don’t suffer at their limit too, but it’s more graceful and more directly instrumented. Neither are particularly straightforward to reason about, but ordering guarantees (in the Lamport sense) are (in my experience) harder to ensure at regional scale with proxies and service meshes. There’s a counterexample, of course: at intercontinental and interplanetary scale we lack the ability to stabilise tree topologies with predictable forwarding latency and are forced to use service proxies and bounded timestamps on replicated data to achieve ordering guarantees instead, of which the best known example is possibly TrueTime. This trades interactivity for scale. Multicast is definitely not easier to program for, and although I could attribute that to a shortage of high-level libraries, in practice the shortfall is mediated by the infrastructure dependency. This places it beyond the radar of most system designers in commercial environments, thus completing the elements of a downward popularity spiral. It’s a shame, but that’s tech.