3 ms·
BEAM is by default tuned for throughput over latency, but also for soft-realtime responsiveness over throughput. The right benchmark for BEAM performance isn’t
by derefr 7y ago
BEAM is by default tuned for throughput over latency, but also for soft-realtime responsiveness over throughput. The right benchmark for BEAM performance isn’t how fast it can do a single-threaded or even multi-threaded CPU-bound task; but rather the 99th percentile packet-to-pcket latency + makespan + memory-requirement volatility of a C10M workload. In other words: if the VM were hosting a million VoIP calls, how many frames would it deliver “on time” such that the client wouldn’t discard them?
(This is not to say you can’t tune BEAM to the needs of other workloads; those are just the defaults. Though, because that is the default assumed workload, it’s also the one most optimization effort goes into.)
BEAM is also pure-functional in a way that a lot of VMs hosting pure-functional languages aren’t; things like mutable atomic memory handles were only introduced recently, and there’s no plan to rewrite core operations in terms of them, because these primitives are less predictable in their time costs than equivalent functional primitives.
You can go fast under BEAM, but the people who want to do so, often decide that it’s too hard to go fast while also retaining soft-real-time guarantees they began using a BEAM language to attain; so they instead use one of BEAM’s many IPC bridges to hook up a fast native “port program” written in another language to the BEAM node, ensuring that any hiccups or crashes in the native code are isolated to its own address space and execution threads.
This last consideration means that very few people are really driving BEAM’s performance forward, because so many instead take the “escape hatch” of low-overhead IPC.
Taken as whole systems, though, you’ll find BEAM in a lot of services that go very fast indeed—you just won’t usually find BEAM itself handling the performance-critical part!
- taspeotis 7y agoThank you for the explanation of BEAM + its objectives and internals. > It's incredibly good at multiprocessing, which is really useful in the websphere So on that benchmark I linked, if you switched to the latency tab Phoenix clocks in at 14.6ms average latency with a standard deviation of 23ms and a max of 460.7ms. The best looking ASP.NET Core result from a standard deviation and max point of view is 1.5ms average latency, 1.4ms standard deviation and 49ms max. So again it's hard for me to tell whether it's BEAM or Phoenix that's responsible for the relative performance hit.
- derefr 7y agoVery likely Phoenix (Phoenix is “more” software than ASP.Net is; a better comparison might be to Erlang’s cowboy library); but also very likely that the author of that benchmark didn’t really tune BEAM. By default, BEAM has runtime tracing hooks enabled; native code compilation (HiPE) disabled; no core pinning; low enough process-heap-size upon spawn that an immediate grow is usually necessary; and a slew of other things. There’s also the fact that Elixir and even Erlang don’t take full advantage of the possibilities of BEAM-bytecode whole-program optimization. For example, any list you walk using Erlang’s lists module or Elixir’s Enum module is going to generate a back-and-forth of remote (symbolic) calls, rather than resulting in the body of that function of lists/Enum being inlined and fused into the caller. Which in turn means, even if you use HiPE, you’ll be thunking in and out of native code so much that the overhead will eat any potential benefit; and each side will be opaque to the static analysis of the other, so even a wholesale native recompilation of the runtime won’t save you. (Same with gen_servers: remote callbacks through library code, rather than fused module-local receive statements; ergo, 10x optimization opportunity lost.) You’ll see real optimization in code generated by e.g. Erlang’s lexer-generator library leex; or in specific parts of each language’s stdlib, like Elixir’s Unicode-handling functions. But such optimized code emission is thin-on-the-ground compared to “naive” coding in Erlang/Elixir. (And I don’t blame the language devs: the languages are fast enough for almost everything people try to do; and for the rest, you can ignore the language-as-framework and write entirely self-contained modules that only rely on BEAM primitives.)
- gregors 7y agolooking at latencies for framework vs framework tells another story https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=plaintext&l=zg24n1-f&c=6 https://www.techempower.com/benchmarks/#section=data-r18&hw=... Where ASP.NET MVC clocks in with 39.3ms average latency, 154.1ms standard deviation
- hopia 7y ago> BEAM is by default tuned for throughput over latency I thought it's exactly the other way around. I Always assumed BEAM is built for concurrent, consistent low latency IO. The benchmarks seem to vaguely reflect that.
- derefr 7y agoRight; we're saying the same thing in different ways, I think. BEAM's concerns, in descending order, are: 1. that an actor that wants to run is never de-scheduled for too long (= 99th percentile latency); 2. that every actor that wants to issue an IOP to the kernel gets to do so efficiently (= whole-system throughput)†; 3. that any individual actor makes actual appreciable progress (= 50th percentile TTFB latency.) This is basically the definition of a soft-realtime system. If latency always came first, it'd be a hard-realtime system; if throughput always came first, it wouldn't be a realtime system at all. † BEAM is actually quite bad about this for disk IO in the normal case, because all disk IO by default goes through the bottleneck of sending messages to a single file-server actor. This has the advantage of virtualizing the filesystem, allowing for e.g. diskless BEAM nodes that boot by using a peer's file server as their own. But it's not very good for filling up your disk's IO queue. Luckily, you can bypass the file server and use file-IO ports/BIFs directly. Or, if you're working against network storage anyway, just skip your own OS kernel and have BEAM speak iSCSI to the disk server directly (like Erlang-on-Xen does.) That approach will likely give you more disk IOPS than you could achieve even in C, presuming the C program is relying on OS syscalls to talk to the remote disk through the OS VFS layer.
- hopia 7y agoThat's a great explanation for a soft real-time system! However, I meant that I always assumed that BEAM prioritized low consistent latency at the expense of high throughout (but without providing hard real-time guarantees). And the mechanism being that preemptive scheduling and counting those reductions so no single process is able to hog the entire system's resources for expensive computations. While these benchmarks may not be reflective of the true performance it seems that Phoenix has a smaller throughput but lower latencies than most of its peer group in them.