4 ms·

Preemptive scheduling is great, but there are some times when it is bad. For example, I was having problems in my Elixir code for live streaming that it was ta

Preemptive scheduling is great, but there are some times when it is bad. For example, I was having problems in my Elixir code for live streaming that it was taking too many reductions to fully deserialize and process the incoming message (nothing major was in this code path, just a parsing the TCP packet and unwrapping the different layers). This ended up causing the process to get (what seems to be) overbalanced and I ended up at having delays transferring 5+mbps video through it. I broke the deserialization up into 3 separate processes (1 for handling the I/O, 1 for packet deserialization, 1 for message deserialization) and while this fixed all the issues (presumably because it can balance the processes more effectively since no single process went over the reduction limit) it turned the effectively synchronous parsing process into a split async process and caused a lot of extra complexity around that. I'm sure there's more to what caused the issue than just reduction limits but it also made me wary about the magic happening under the hood.

bitwalker 9y ago
You may have been able to bump the process priority to give that process more scheduler time if it's a critical process in a hot path. Perhaps you tried that though. Definitely have to be careful with it, but always an option to keep in mind!
- KallDrexx 9y ago
  I haven't tried that, but I'd have to make sure that every inbound and outbound RTMP connection had high enough priority to correctly execute, and that seemed like a lot of tweaking that had to be done in order to maintain the constant flow of video throughout the app with low latency. At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly (not just with that but other potential issues I had in the back of my mind as well).
  - randomstudent 9y ago
    > At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly From someone with zero experience in building concurrent systems: isn't this true for every sufficiently complex system? If your system is complex enough, then no general framework will help much, right?
    - KallDrexx 9y ago
      At some point sure, but I can get concurrent systems up and running pretty easily in Go, C# and Rust without going too deep into the underlying details of the asynchronous framework. Erlang's VM does a lot more than any of those frameworks or runtimes do though because of how it handles per process garbage collection yet globally shared binaries, the preemptive scheduler and how it balances process load, the hot-reload capabilities etc... It does a lot of things really well but there's a lot of magic behind it, and I seemed to brush against that much faster than I have in other languages.
dguaraglia 9y ago
Do you mind me asking what kind of throughput you are seeing for a process like that where you unpack video across some processes? I'm working on something similar using Go and it'd be great to know if Elixir is a good alternative or not.
- KallDrexx 9y ago
  I know of some Erlang based platforms that achieve 10gbps throughput. I hadn't done exhaustive testing on my prototype as I've changed gears away from my home-grown solution to modifying some open source C++ media servers that are a bit more proven than my own stuff.
lobster_johnson 9y ago
Not just scheduling — the ability to forcibly terminate processes itself is a key part of why Erlang can be resilient to failures. It's a key area where I think Go went wrong. Without the ability to kill a goroutine, you can't implement caller-enforced timeouts and canceling — anything that should be able to be canceled has to explicitly check for some signal sent by the controlling party. This is why Go ended up with Context, which infects everything it touched. And you also can't create Erlang-style supervisor trees; you can't create self-repairing hiearchies of control. And you can't do things like use a debugger to attach to a process and terminate runaway goroutines.

bitwalker 9y ago

You may have been able to bump the process priority to give that process more scheduler time if it's a critical process in a hot path. Perhaps you tried that though. Definitely have to be careful with it, but always an option to keep in mind!

KallDrexx 9y ago
I haven't tried that, but I'd have to make sure that every inbound and outbound RTMP connection had high enough priority to correctly execute, and that seemed like a lot of tweaking that had to be done in order to maintain the constant flow of video throughout the app with low latency. At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly (not just with that but other potential issues I had in the back of my mind as well).
- randomstudent 9y ago
  > At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly From someone with zero experience in building concurrent systems: isn't this true for every sufficiently complex system? If your system is complex enough, then no general framework will help much, right?
  - KallDrexx 9y ago
    At some point sure, but I can get concurrent systems up and running pretty easily in Go, C# and Rust without going too deep into the underlying details of the asynchronous framework. Erlang's VM does a lot more than any of those frameworks or runtimes do though because of how it handles per process garbage collection yet globally shared binaries, the preemptive scheduler and how it balances process load, the hot-reload capabilities etc... It does a lot of things really well but there's a lot of magic behind it, and I seemed to brush against that much faster than I have in other languages.

dguaraglia 9y ago

Do you mind me asking what kind of throughput you are seeing for a process like that where you unpack video across some processes? I'm working on something similar using Go and it'd be great to know if Elixir is a good alternative or not.

KallDrexx 9y ago
I know of some Erlang based platforms that achieve 10gbps throughput. I hadn't done exhaustive testing on my prototype as I've changed gears away from my home-grown solution to modifying some open source C++ media servers that are a bit more proven than my own stuff.

lobster_johnson 9y ago

Not just scheduling — the ability to forcibly terminate processes itself is a key part of why Erlang can be resilient to failures. It's a key area where I think Go went wrong. Without the ability to kill a goroutine, you can't implement caller-enforced timeouts and canceling — anything that should be able to be canceled has to explicitly check for some signal sent by the controlling party. This is why Go ended up with Context, which infects everything it touched. And you also can't create Erlang-style supervisor trees; you can't create self-repairing hiearchies of control. And you can't do things like use a debugger to attach to a process and terminate runaway goroutines.