4 ms·

> BEAM is by default tuned for throughput over latency I thought it's exactly the other way around. I Always assumed BEAM is built for concurrent, consistent

by hopia 7y ago

> BEAM is by default tuned for throughput over latency I thought it's exactly the other way around. I Always assumed BEAM is built for concurrent, consistent low latency IO. The benchmarks seem to vaguely reflect that.

derefr 7y ago
Right; we're saying the same thing in different ways, I think. BEAM's concerns, in descending order, are: 1. that an actor that wants to run is never de-scheduled for too long (= 99th percentile latency); 2. that every actor that wants to issue an IOP to the kernel gets to do so efficiently (= whole-system throughput)†; 3. that any individual actor makes actual appreciable progress (= 50th percentile TTFB latency.) This is basically the definition of a soft-realtime system. If latency always came first, it'd be a hard-realtime system; if throughput always came first, it wouldn't be a realtime system at all. † BEAM is actually quite bad about this for disk IO in the normal case, because all disk IO by default goes through the bottleneck of sending messages to a single file-server actor. This has the advantage of virtualizing the filesystem, allowing for e.g. diskless BEAM nodes that boot by using a peer's file server as their own. But it's not very good for filling up your disk's IO queue. Luckily, you can bypass the file server and use file-IO ports/BIFs directly. Or, if you're working against network storage anyway, just skip your own OS kernel and have BEAM speak iSCSI to the disk server directly (like Erlang-on-Xen does.) That approach will likely give you more disk IOPS than you could achieve even in C, presuming the C program is relying on OS syscalls to talk to the remote disk through the OS VFS layer.
- hopia 7y ago
  That's a great explanation for a soft real-time system! However, I meant that I always assumed that BEAM prioritized low consistent latency at the expense of high throughout (but without providing hard real-time guarantees). And the mechanism being that preemptive scheduling and counting those reductions so no single process is able to hog the entire system's resources for expensive computations. While these benchmarks may not be reflective of the true performance it seems that Phoenix has a smaller throughput but lower latencies than most of its peer group in them.
  - derefr 7y ago
    I would say that BEAM prioritizes throughput over 50th-percentile latency, because BEAM prioritizes serving “port” pseudo-actors (distinct scheduled interactions with the kernel through syscalls for queuing/resolving IO ops) over serving actual userland execution-thread actors. One actor queuing up a heavy amount of IOPS through many different ports, can actually cause the BEAM node to generate a visible latency spike, as it always tries to schedule all the ready ports assigned to the given OS scheduler-thread per scheduler-timeslice, before trying to schedule any user-defined actors; and so will find itself with very little time left in that timeslice to schedule the execution threads of actual actors. A single actor can accomplish this “uneven” loading because BEAM’s reduction scheduling means that just queuing IO ops through ports in a linear block of code can never cause an actor to be descheduled, since the port BIFs are not function calls per se, and therefore have no ability to trigger scheduler yield-checking. Mind you, this is a highly-unusual usage pattern. Actors don’t usually batch up huge binary payloads to send and then blast them all through at once into many different ports in an unrolled loop. And, usually, one actor doesn’t even usually hold multiple ports; the idiomatic approach is for each port to have its own separate “owner” actor (because this means that these “owner” actors will be scheduled semi-synchronously in response to activity on their respective ports, sort of like Go with goroutines getting scheduled in response to activity on channels.) In an idiomatic usage pattern, you could say that the node doesn’t become IOPS-bound but rather actor-scheduling-bound, where a scheduler-saturating number of ready ports can’t be brought about, because as ready ports per timeslice increase, the amount of the scheduler-timeslice available to actors decreases, and so fewer actors that would generate IOPS are getting run, meaning that more of the next timeslice will be available. In other words, with idiomatic use of ports, this is a self-limiting problem. But, well, we’re talking about how the BEAM is built, rather than about what Erlang/OTP systems do in practice. The BEAM itself has these priorities, even if, in practice, they’re papered over with other ones :)

by hopia 7y ago

derefr 7y ago

Right; we're saying the same thing in different ways, I think. BEAM's concerns, in descending order, are: 1. that an actor that wants to run is never de-scheduled for too long (= 99th percentile latency); 2. that every actor that wants to issue an IOP to the kernel gets to do so efficiently (= whole-system throughput)†; 3. that any individual actor makes actual appreciable progress (= 50th percentile TTFB latency.) This is basically the definition of a soft-realtime system. If latency always came first, it'd be a hard-realtime system; if throughput always came first, it wouldn't be a realtime system at all. † BEAM is actually quite bad about this for disk IO in the normal case, because all disk IO by default goes through the bottleneck of sending messages to a single file-server actor. This has the advantage of virtualizing the filesystem, allowing for e.g. diskless BEAM nodes that boot by using a peer's file server as their own. But it's not very good for filling up your disk's IO queue. Luckily, you can bypass the file server and use file-IO ports/BIFs directly. Or, if you're working against network storage anyway, just skip your own OS kernel and have BEAM speak iSCSI to the disk server directly (like Erlang-on-Xen does.) That approach will likely give you more disk IOPS than you could achieve even in C, presuming the C program is relying on OS syscalls to talk to the remote disk through the OS VFS layer.

hopia 7y ago
That's a great explanation for a soft real-time system! However, I meant that I always assumed that BEAM prioritized low consistent latency at the expense of high throughout (but without providing hard real-time guarantees). And the mechanism being that preemptive scheduling and counting those reductions so no single process is able to hog the entire system's resources for expensive computations. While these benchmarks may not be reflective of the true performance it seems that Phoenix has a smaller throughput but lower latencies than most of its peer group in them.
- derefr 7y ago
  I would say that BEAM prioritizes throughput over 50th-percentile latency, because BEAM prioritizes serving “port” pseudo-actors (distinct scheduled interactions with the kernel through syscalls for queuing/resolving IO ops) over serving actual userland execution-thread actors. One actor queuing up a heavy amount of IOPS through many different ports, can actually cause the BEAM node to generate a visible latency spike, as it always tries to schedule all the ready ports assigned to the given OS scheduler-thread per scheduler-timeslice, before trying to schedule any user-defined actors; and so will find itself with very little time left in that timeslice to schedule the execution threads of actual actors. A single actor can accomplish this “uneven” loading because BEAM’s reduction scheduling means that just queuing IO ops through ports in a linear block of code can never cause an actor to be descheduled, since the port BIFs are not function calls per se, and therefore have no ability to trigger scheduler yield-checking. Mind you, this is a highly-unusual usage pattern. Actors don’t usually batch up huge binary payloads to send and then blast them all through at once into many different ports in an unrolled loop. And, usually, one actor doesn’t even usually hold multiple ports; the idiomatic approach is for each port to have its own separate “owner” actor (because this means that these “owner” actors will be scheduled semi-synchronously in response to activity on their respective ports, sort of like Go with goroutines getting scheduled in response to activity on channels.) In an idiomatic usage pattern, you could say that the node doesn’t become IOPS-bound but rather actor-scheduling-bound, where a scheduler-saturating number of ready ports can’t be brought about, because as ready ports per timeslice increase, the amount of the scheduler-timeslice available to actors decreases, and so fewer actors that would generate IOPS are getting run, meaning that more of the next timeslice will be available. In other words, with idiomatic use of ports, this is a self-limiting problem. But, well, we’re talking about how the BEAM is built, rather than about what Erlang/OTP systems do in practice. The BEAM itself has these priorities, even if, in practice, they’re papered over with other ones :)