3 ms·

Rust however does not offer nearly the same level of power when it comes to runtime introspection that BEAM does. Coming from a company that uses elixir heavil

by jhgg 8y ago

Rust however does not offer nearly the same level of power when it comes to runtime introspection that BEAM does. Coming from a company that uses elixir heavily and has made a significant investment in rust, I don’t think we would ever use solely rust on our distributed systems. However, we have rewritten some code that elixir was too slow at in rust and exposed it as a NIF on BEAM - and that has worked well. (Blog post on that soon hopefully). I do admit, we are also going to be ditching mnesia for one of our clusters for our own in-house simpler system (ETS replication with different consistency/netsplit guarantees for our use-case), we've had to write our own cross-node process monitoring solution (at peak we see 200M+ cross-node monitors on our cluster), and we've also had to overcome the limits of message-fanout on distribution as well (https://github.com/discordapp/manifold https://github.com/discordapp/manifold). However, for operating at our scale (peak 9m ccu @ 5m events/sec fanout to clients), we run a surprisingly small number of servers for our real time system (~120). EDIT: I can't reply to your post below, but I think the runtime introspection we run into is not dealing with OS level metrics, but application level introspection. Introspecting the state of processes, writing code in the repl to debug issues within the cluster, benchmarking to find hot functions or where specific processes are spending a lot of their time. Capturing traffic to replay on a test cluster to simulate production load, all becomes very trivial with BEAM.

ilovecaching 8y ago
Yes, we can do that through flamegraphs. For any container running on our system, we can get the call stack of the running process, time taken, resources used etc. We can attach to containers, proxy traffic etc, all through Kubernetes. I said this earlier, but we've essentially separated operation concerns from our applications, and that opens us up to relying more on knowledge of Linux which is easier to hire for, and we can reuse that knowledge and all our tooling with any other languages we want to use.
- jhgg 8y ago
  We run between 200k and 2m processes per beam VM. I don't know how it'd be possible to get as precise metrics as we need just from relying on linux utilities. And in the same cluster, some processes although identical in code have dramatically different workloads.
ilovecaching 8y ago
We couldn't get enough introspection. Now we use Rust and we can use a much larger toolkit. We have BCC, kprobes, flamegraphs... When you use Rust the OS is your machine. With Elixir you need to understand the BEAM, a slightly esoteric platform with a rather small community working on it.

by jhgg 8y ago

ilovecaching 8y ago

Yes, we can do that through flamegraphs. For any container running on our system, we can get the call stack of the running process, time taken, resources used etc. We can attach to containers, proxy traffic etc, all through Kubernetes. I said this earlier, but we've essentially separated operation concerns from our applications, and that opens us up to relying more on knowledge of Linux which is easier to hire for, and we can reuse that knowledge and all our tooling with any other languages we want to use.

jhgg 8y ago
We run between 200k and 2m processes per beam VM. I don't know how it'd be possible to get as precise metrics as we need just from relying on linux utilities. And in the same cluster, some processes although identical in code have dramatically different workloads.

ilovecaching 8y ago

We couldn't get enough introspection. Now we use Rust and we can use a much larger toolkit. We have BCC, kprobes, flamegraphs... When you use Rust the OS is your machine. With Elixir you need to understand the BEAM, a slightly esoteric platform with a rather small community working on it.