4 ms·

Aeron: Efficient reliable UDP unicast, UDP multicast, and IPC message transport

91 points by todsacerdoti 1y ago

fuzzfactor 1y ago
I like the idea of muticast.
readthenotes1 1y ago
I wonder how many times reliable UDP has been implemented? It's gotta be gazillions...
- pjmlp 1y ago
  Millions of times every year by all university students doing introduction to UNIX networking protocols projects across the planet.
- erikvanoosten 1y ago
  With Aaron I would think the focus should be on 'efficiënt', not reliable. Within a datacenter they are orders of magnitude faster than plan UDP. You get this crazy efficiency at the cost of reduced flexibility in how you send messages.
  - jzwinck 1y ago
    I haven't seen the makers of Aeron (or anyone else) claim it's "orders of magnitude faster than plain UDP." Do you have a link to something about this? It doesn't pass the smell test for me unless you're talking specifically about using Aeron within a single machine (where it uses shared memory instead of the network)...but you said "Within a datacenter" not "Within a computer."
    - erikvanoosten 1y ago
      It has been a while since I saw their presentation. What I remember is that Aaron has an insanely low delay even in the high percentiles, that is orders of magnitude better. Throughput for a large stream of data is probably similar to plain UDP. Please correct me if I remember wrong.
      - buybackoff 1y ago
        It does have great tail latency. But it's not a silver bullet, but careful engineering. And you pay for the latency with spinning threads. It's the architecture that makes it to stand out. In the end, it's just the same old UDP sockets, not even io_uring at least in the free public version. But one can use LD_PRELOAD if hardware has this trick - but again, it's not specific to Aeron.
- mgaunard 1y ago
  In general most people don't do networking right, especially within real-time systems. As people discover the problems with their approach, they rewrite it. I'm my experience it's generally better not to have transparently reliability for lost data as well. The correct handling should be application-specific.
vodou 1y ago
After years of maintaining and using an application suite that relies on multicast for internal communication, I would hesitate to use "reliable" and "multicast" in the same sentence. Multicast is great in theory, but comes with so many pitfalls and grievances in practice. Mostly due to unreliable handling of switches, routers, network adapters and TCP/IP stacks in operating systems. Just to mention a few headaches I've been dealing with over the years: multicast sockets that joins the wrong network adapter interface (due to adapter priorities), losing multicast membership after resume from sleep/hibernate, switches/routers just dropping multicast membership after a while (especially when running in VMs and "enterprise" systems like SUSE Linux and Windows Server, all kinds of socket reuse problems, etc. I don't even dare to think about how many hours I have wasted on issues listed above. I would never rely on multicast again when developing a new system. But that said, the application suite, a mission control system for satellites, works great most of the time (typically on small, controlled subnets, using physical installations instead of VMs) and has served us well.
- mgaunard 1y ago
  "reliable" means that if one of the recipients observes a gap, it can ask for a replay of the missing packets.
  - vodou 1y ago
    Sure. But if you don't have reliability at the network layer you don't have any chance to have a reliable transport layer.
    - fanf2 1y ago
      TCP begs to differ. https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf https://web.mit.edu/Saltzer/www/publications/endtoend/endtoe...
    - deleted 1y ago
      [deleted]
- DanielHB 1y ago
buybackoff 1y ago
One of the greatest things about Aeron is just the fact it exists. If one goes e.g. to StackOverflow or a place with the same patronizing attitude of "experts", they will tell you no one needs UDP, even in the same DC on reliable network, especially no one needs multicast. Any sane person should use TCP with a loop for multiple destinations, they would say, and one should measure before optimizing, they would say. But they themselves probably never had a chance to measure. Yet Aeron guys, who are real expert in low-latency systems, just delivered an ultra-fast thing that is quite simple in design. Aeron latency histograms vs TCP are quite nice in the same DC on enterprise-grade networking hardware. But it really makes sense to use if a single-digit or low-double digit microsecond latency improvement on P50 is worth the effort. Or if the long tail with TCP is a dealbreaker, as Aeron has much nicer P99+ regardless of how well optimized a TCP setup is. Also, if one can leverage multicast that's nice, but not only clouds have it disabled, and Aeron works fine with unicast to N. However, there are gotchas with threading and configuration overall. Cross-DC setup may surprise in a bad way if buffers are not configured to account for bandwidth-delay product. Any packet loss on high-latency network leads to a nasty NACK storm that is slow to recover under load. It's better to set the highest QoS and ensure the network is never dropping packets, e.g. calculate the real peak instant load vs hardware capacity. Relative latency savings cross-DC become less interesting the longer the distance, so there's nothing wrong with TCP there. Another note is that, e.g. ZMQ is slow not because of TCP but because of its internals, almost 2x slower for small packets than raw well-tuned TCP sockets, which are not that bad vs Aeron. Also, Aeron is not for sending big blobs around, the best is to use it with small payloads. Aeron is designed with mechanical sympathy in mind by the guys who coined this term and have been evangelizing it for years, and it's visible. Lots to learn from the design & implementation (tons of resources on the web) even without using it in prod.
- lowwave 1y ago
  >Also, if one can leverage multicast that's nice, but not only clouds have it disabled, and Aeron works fine with unicast to N. How did that happened? Seems multicast is already built in, just use that for massive broadcast. Is TCP used just so we can get an ACK that it is received. Seems multicast and UDP shouldn't be a problem if we just want massive people to listen in on it, but if we want to also track these people then that is another story. From a user perspective, use UDP/multicast all the way. Let the client to request something if it is dropped or missing or otherwise just multicast for everything.
arnold_palmur 1y ago
At least in terms of open-source, I often say that the Aeron code base is one of the best projects to study in terms of software quality (especially Java). The real-logic (now Adaptive) guys are a skilled and knowledgeable bunch.
- pgwhalen 1y ago
  It’s a superb codebase for sure, I’ve spent dozens or hundreds of hours reading it. But I would caution against using it as an example of Java the language because of how non-idiomatic it can be to achieve the absurd performance it does. Most Java shouldn’t use a lot of the techniques they use (and the authors would be the first to admit that).
neomantra 1y ago
It’s easy to focus on the reliable UDP protocol and the multicast support, but what’s important about Aeron is its system architecture. As noted elsewhere, it all combines together in “mechanical sympathy” and once you have that you can interconnect with high performance transports [1]. So you set up an Aeron Server on your machine. That handles all external network communication at the message layer (NAKs, etc). Every “Aeron Client” process communicates with that Server to stand up shared memory pipes. The messaging client solely deals with those pipes (cache efficient, etc). They subscribe to channels at host:port/topic but it is not direct network delivery to them (the client is transport agnostic, besides the subscription). The network service directs data to the clients shared memory queue. Once they have that base networked-IPC setup, you can maximize performance with it using reliable UDP and fan out common data (eg marlet data) using multicast. Then Aeron further adds Archiving (to persist) and Clustering (to scale late join / state replication) components. This stuff works well with kernel bypass :) [1] It could work with RDMA as well, but the ticket regarding that was closed 8 years ago. Maybe if there was AI/GPU workload interest.
- jauntywundrkind 1y ago
  A whole different tamale, but Apache Iggy feels similar-ish, as a persistent message streaming system. Also using UDP, this time with QUIC, which is a pretty future-forward protocol that I love to see targetted (also has a rest API and its own binary protocol). https://iggy.apache.org/ https://iggy.apache.org/
  - neomantra 1y ago
    I had not heard of Iggy, love the mascot! Looks to be pretty comprehensive messaging system focusing on persistent streams; lots of batteries included like auth and monitoring. While that has some sick throughput benchmarks, it is a pretty different architecture and Aeron will have way (orders) lower and more stable latencies. But it won’t work on as many networks as Iggy. Iggy client SDKs are easy to make because they just need to speak socket. Aeron client SDKs are “easy” to make because they just need to figure out framing shared memory. (And the orchestrator handles the network getting it there).
Veserv 1y ago
Is it actually high performance? Looking at their transport protocol benchmarks on AWS [1][2], they average ~3 million 288-byte messages per second on c5.9xlarge (36 vCPU) instances. When increasing to their MTU limit of 1,344 bytes per message that drops to 700 thousand messages per second [2] or ~1 GB/s (~7.5 Gbps) over 36 cores. That is just ~200 Mbps per core assuming it is significantly parallel. Looking at their transport protocol benchmarks on GCP [3], they average ~4.7 million 288 byte messages per second on C3 (unspecified type) instances. Assuming it scales proportionally to the AWS test, as they do not provide a max message size throughput number for GCP, that would be ~1 million messages per second or ~1.5 GB/s (~12 Gbps). TCP stacks can routinely average 10 Gbps per individual core even without aggressive tuning, but Aeron appears to struggle to achieve parity with 36x as many cores. That is not to say that there might not be other advantages to Aeron such as latency, multicast support, or whatever their higher levels are doing, but 36x worse performance than basic off-the-shelf protocols does not sound like "high performance". [1] https://hub.aeron.io/hubfs/Aeron-Assets/Aeron_AWS_Performance_Testing_Guide_Nov23.pdf https://hub.aeron.io/hubfs/Aeron-Assets/Aeron_AWS_Performanc... Page 13 [2] https://aws.amazon.com/blogs/industries/aeron-performance-enables-capital-markets-to-move-to-the-cloud-on-aws/ https://aws.amazon.com/blogs/industries/aeron-performance-en... Search "Test Results" [3] https://aeron.io/other/aeron-google-cloud-performance-testing/ https://aeron.io/other/aeron-google-cloud-performance-testin...
- sethev 1y ago
  A c5.9xlarge instance has 12 gigabits/s of bandwidth available [1]. A fair comparison to TCP would need to look at end to end message delivery, including framing / parsing of the messages. On the face of it, the ability to use the majority of the bandwidth of the instance with small messages is impressive. [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...
- nly 1y ago
  > That is just ~200 Mbps per core assuming it is significantly parallel. It's not. The Aeron media driver has 1 RX thread and 1 TX thread in the most heavily threaded configuration (+1 admin thread)

91 points by todsacerdoti 1y ago

fuzzfactor 1y ago

I like the idea of muticast.

readthenotes1 1y ago

I wonder how many times reliable UDP has been implemented? It's gotta be gazillions...

pjmlp 1y ago
Millions of times every year by all university students doing introduction to UNIX networking protocols projects across the planet.
erikvanoosten 1y ago
With Aaron I would think the focus should be on 'efficiënt', not reliable. Within a datacenter they are orders of magnitude faster than plan UDP. You get this crazy efficiency at the cost of reduced flexibility in how you send messages.
- jzwinck 1y ago
  I haven't seen the makers of Aeron (or anyone else) claim it's "orders of magnitude faster than plain UDP." Do you have a link to something about this? It doesn't pass the smell test for me unless you're talking specifically about using Aeron within a single machine (where it uses shared memory instead of the network)...but you said "Within a datacenter" not "Within a computer."
  - erikvanoosten 1y ago
    It has been a while since I saw their presentation. What I remember is that Aaron has an insanely low delay even in the high percentiles, that is orders of magnitude better. Throughput for a large stream of data is probably similar to plain UDP. Please correct me if I remember wrong.
    - buybackoff 1y ago
      It does have great tail latency. But it's not a silver bullet, but careful engineering. And you pay for the latency with spinning threads. It's the architecture that makes it to stand out. In the end, it's just the same old UDP sockets, not even io_uring at least in the free public version. But one can use LD_PRELOAD if hardware has this trick - but again, it's not specific to Aeron.
mgaunard 1y ago
In general most people don't do networking right, especially within real-time systems. As people discover the problems with their approach, they rewrite it. I'm my experience it's generally better not to have transparently reliability for lost data as well. The correct handling should be application-specific.

vodou 1y ago

After years of maintaining and using an application suite that relies on multicast for internal communication, I would hesitate to use "reliable" and "multicast" in the same sentence. Multicast is great in theory, but comes with so many pitfalls and grievances in practice. Mostly due to unreliable handling of switches, routers, network adapters and TCP/IP stacks in operating systems. Just to mention a few headaches I've been dealing with over the years: multicast sockets that joins the wrong network adapter interface (due to adapter priorities), losing multicast membership after resume from sleep/hibernate, switches/routers just dropping multicast membership after a while (especially when running in VMs and "enterprise" systems like SUSE Linux and Windows Server, all kinds of socket reuse problems, etc. I don't even dare to think about how many hours I have wasted on issues listed above. I would never rely on multicast again when developing a new system. But that said, the application suite, a mission control system for satellites, works great most of the time (typically on small, controlled subnets, using physical installations instead of VMs) and has served us well.

mgaunard 1y ago
"reliable" means that if one of the recipients observes a gap, it can ask for a replay of the missing packets.
- vodou 1y ago
  Sure. But if you don't have reliability at the network layer you don't have any chance to have a reliable transport layer.
  - fanf2 1y ago
    TCP begs to differ. https://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf https://web.mit.edu/Saltzer/www/publications/endtoend/endtoe...
  - deleted 1y ago
    [deleted]
DanielHB 1y ago

buybackoff 1y ago

One of the greatest things about Aeron is just the fact it exists. If one goes e.g. to StackOverflow or a place with the same patronizing attitude of "experts", they will tell you no one needs UDP, even in the same DC on reliable network, especially no one needs multicast. Any sane person should use TCP with a loop for multiple destinations, they would say, and one should measure before optimizing, they would say. But they themselves probably never had a chance to measure. Yet Aeron guys, who are real expert in low-latency systems, just delivered an ultra-fast thing that is quite simple in design. Aeron latency histograms vs TCP are quite nice in the same DC on enterprise-grade networking hardware. But it really makes sense to use if a single-digit or low-double digit microsecond latency improvement on P50 is worth the effort. Or if the long tail with TCP is a dealbreaker, as Aeron has much nicer P99+ regardless of how well optimized a TCP setup is. Also, if one can leverage multicast that's nice, but not only clouds have it disabled, and Aeron works fine with unicast to N. However, there are gotchas with threading and configuration overall. Cross-DC setup may surprise in a bad way if buffers are not configured to account for bandwidth-delay product. Any packet loss on high-latency network leads to a nasty NACK storm that is slow to recover under load. It's better to set the highest QoS and ensure the network is never dropping packets, e.g. calculate the real peak instant load vs hardware capacity. Relative latency savings cross-DC become less interesting the longer the distance, so there's nothing wrong with TCP there. Another note is that, e.g. ZMQ is slow not because of TCP but because of its internals, almost 2x slower for small packets than raw well-tuned TCP sockets, which are not that bad vs Aeron. Also, Aeron is not for sending big blobs around, the best is to use it with small payloads. Aeron is designed with mechanical sympathy in mind by the guys who coined this term and have been evangelizing it for years, and it's visible. Lots to learn from the design & implementation (tons of resources on the web) even without using it in prod.

lowwave 1y ago
>Also, if one can leverage multicast that's nice, but not only clouds have it disabled, and Aeron works fine with unicast to N. How did that happened? Seems multicast is already built in, just use that for massive broadcast. Is TCP used just so we can get an ACK that it is received. Seems multicast and UDP shouldn't be a problem if we just want massive people to listen in on it, but if we want to also track these people then that is another story. From a user perspective, use UDP/multicast all the way. Let the client to request something if it is dropped or missing or otherwise just multicast for everything.

arnold_palmur 1y ago

At least in terms of open-source, I often say that the Aeron code base is one of the best projects to study in terms of software quality (especially Java). The real-logic (now Adaptive) guys are a skilled and knowledgeable bunch.

pgwhalen 1y ago
It’s a superb codebase for sure, I’ve spent dozens or hundreds of hours reading it. But I would caution against using it as an example of Java the language because of how non-idiomatic it can be to achieve the absurd performance it does. Most Java shouldn’t use a lot of the techniques they use (and the authors would be the first to admit that).

neomantra 1y ago

It’s easy to focus on the reliable UDP protocol and the multicast support, but what’s important about Aeron is its system architecture. As noted elsewhere, it all combines together in “mechanical sympathy” and once you have that you can interconnect with high performance transports [1]. So you set up an Aeron Server on your machine. That handles all external network communication at the message layer (NAKs, etc). Every “Aeron Client” process communicates with that Server to stand up shared memory pipes. The messaging client solely deals with those pipes (cache efficient, etc). They subscribe to channels at host:port/topic but it is not direct network delivery to them (the client is transport agnostic, besides the subscription). The network service directs data to the clients shared memory queue. Once they have that base networked-IPC setup, you can maximize performance with it using reliable UDP and fan out common data (eg marlet data) using multicast. Then Aeron further adds Archiving (to persist) and Clustering (to scale late join / state replication) components. This stuff works well with kernel bypass :) [1] It could work with RDMA as well, but the ticket regarding that was closed 8 years ago. Maybe if there was AI/GPU workload interest.

jauntywundrkind 1y ago
A whole different tamale, but Apache Iggy feels similar-ish, as a persistent message streaming system. Also using UDP, this time with QUIC, which is a pretty future-forward protocol that I love to see targetted (also has a rest API and its own binary protocol). https://iggy.apache.org/ https://iggy.apache.org/
- neomantra 1y ago
  I had not heard of Iggy, love the mascot! Looks to be pretty comprehensive messaging system focusing on persistent streams; lots of batteries included like auth and monitoring. While that has some sick throughput benchmarks, it is a pretty different architecture and Aeron will have way (orders) lower and more stable latencies. But it won’t work on as many networks as Iggy. Iggy client SDKs are easy to make because they just need to speak socket. Aeron client SDKs are “easy” to make because they just need to figure out framing shared memory. (And the orchestrator handles the network getting it there).

Veserv 1y ago

Is it actually high performance? Looking at their transport protocol benchmarks on AWS [1][2], they average ~3 million 288-byte messages per second on c5.9xlarge (36 vCPU) instances. When increasing to their MTU limit of 1,344 bytes per message that drops to 700 thousand messages per second [2] or ~1 GB/s (~7.5 Gbps) over 36 cores. That is just ~200 Mbps per core assuming it is significantly parallel. Looking at their transport protocol benchmarks on GCP [3], they average ~4.7 million 288 byte messages per second on C3 (unspecified type) instances. Assuming it scales proportionally to the AWS test, as they do not provide a max message size throughput number for GCP, that would be ~1 million messages per second or ~1.5 GB/s (~12 Gbps). TCP stacks can routinely average 10 Gbps per individual core even without aggressive tuning, but Aeron appears to struggle to achieve parity with 36x as many cores. That is not to say that there might not be other advantages to Aeron such as latency, multicast support, or whatever their higher levels are doing, but 36x worse performance than basic off-the-shelf protocols does not sound like "high performance". [1] https://hub.aeron.io/hubfs/Aeron-Assets/Aeron_AWS_Performance_Testing_Guide_Nov23.pdf https://hub.aeron.io/hubfs/Aeron-Assets/Aeron_AWS_Performanc... Page 13 [2] https://aws.amazon.com/blogs/industries/aeron-performance-enables-capital-markets-to-move-to-the-cloud-on-aws/ https://aws.amazon.com/blogs/industries/aeron-performance-en... Search "Test Results" [3] https://aeron.io/other/aeron-google-cloud-performance-testing/ https://aeron.io/other/aeron-google-cloud-performance-testin...

sethev 1y ago
A c5.9xlarge instance has 12 gigabits/s of bandwidth available [1]. A fair comparison to TCP would need to look at end to end message delivery, including framing / parsing of the messages. On the face of it, the ability to use the majority of the bandwidth of the instance with small messages is impressive. [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...
nly 1y ago
> That is just ~200 Mbps per core assuming it is significantly parallel. It's not. The Aeron media driver has 1 RX thread and 1 TX thread in the most heavily threaded configuration (+1 admin thread)