5 ms·

It's not the distributed system bits. It's the heavy abstractions over the distributed system bits that make me question if operating it requires specialized kn

by ctvo 4y ago

It's not the distributed system bits. It's the heavy abstractions over the distributed system bits that make me question if operating it requires specialized knowledge. I have a performance issue or an edge case with actors from the OTP. How do I debug and resolve it? Say network partitions aren't behaving like I expect and there seems to be data inconsistency or lost. I think you're doing a disservice when you pretend this is the same class of issue as a three tier app you can Google and StackOverflow for. This dependency should raise questions for anyone looking to adopt this, and I want to know how people have attempted to mitigate this vs. handwaving.

toast0 4y ago
The central abstraction isn't IMHO that heavy. It's just this: A process has an isolated execution state, a message queue, and an id. Messages can be added to the queue using the process id (from local or remote processes), and messages can be pulled from the queue by the process (with pattern matching, first matching message is removed from the queue) If you have a performance issue, introspect with erlang:process_info to see what your processes are doing, and language indepedent tools to see what the VM is doing. process_info gives tons of information, IMHO, much more operable than other systems I'm familiar with. Ex: if your process has a large message queue, either it's not processing it's messages fast enough, it's selectively processing messages and leaving junk in the queue, or some combination; you can get inspect the queue with process_info and maybe figure that out. Lots of possibilities for why too slow, maybe it's waiting, maybe it's slow numerically, etc; process_info can tell you what function is running, which is usually helpful, etc. Network partitions and their effects are highly application dependant, but if you have multiple nodes in your existing system, you have to think about it already, or choose to ignore it, which you can also do here, although if you use mnesia, you shouldn't ignore it: the default is for partitioned and then rejoined nodes to stay apart, waiting for an intervention, which may be the only reasonable default (because there's no right answer), but it's not what people usually want.
- Fire-Dragon-DoL 4y ago
  One note, the message queue has a limit in size,going over the limit will drop messages.
  - toast0 4y ago
    If there's a limit, it's new and I can't find it in a quick search; I did find a 2014 PR that was rejected because dropping messages violates core semantics, IMHO killing the recipient would be OK though. Traditionally it's unbounded, but when you exhaust the memory of the host (or the ulimit, if set), you'll likely loose the whole VM.
    - Fire-Dragon-DoL 4y ago
      Turns out it was specific to the application I worked on, sorry for the misinformation. It is configurable: https://www.erlang.org/eeps/eep-0042 https://www.erlang.org/eeps/eep-0042
      - toast0 4y ago
        EEP 42 is not implemented in mainline OTP. If it were, the proposal would likely link to the implementation, and the EEP index in EEP 0 would show what version it was first included in: https://www.erlang.org/eeps/eep-0000 https://www.erlang.org/eeps/eep-0000 If your application is using something like this, you're running a modified beam (which is fine; my professional experience is on a modified beam where we added a process_info option to drop all messages in the queue of a given process, and to allow for messages to be added to the front of the queue, neither of which would be acceptable upstream, but both of which were super handy for us)
        Fire-Dragon-DoL 4y ago
        It's been a long time, so i could be wrong, but I remember we did face the problem at some point. Either way, not having a limit makes it even easier to use, which I really appreciate
Fire-Dragon-DoL 4y ago
I see your point, I still disagree, I encountered some bugs that were incredibly surprising with postgres + sidekiq combo (made perfect sense though) Elixir doesn't distribute anything unless you tell it to, keep that in mind. You can run it like a normal rails app. The OTP by default runs on one machine and as such, it's not subject to distributed systems law. And you can still scale horizontally in the traditional way (deploy the app to multiple machines), no need to learn or use the multi-nodes. The debugging experience is the same. Of course you have to learn the OTP library at some point, but it's equivalent to study the ruby standard library: it's natural, must be done, like for any programming language. That being said, the tools provided are really nice, given erlang is very old, the tools do exist. And you can connect to a production machine with an "irb session", except that the one in elixir is in the same memory space as the main process, so you can actually inspect in memory stuff, very powerful and dangerous.

by ctvo 4y ago

toast0 4y ago

The central abstraction isn't IMHO that heavy. It's just this: A process has an isolated execution state, a message queue, and an id. Messages can be added to the queue using the process id (from local or remote processes), and messages can be pulled from the queue by the process (with pattern matching, first matching message is removed from the queue) If you have a performance issue, introspect with erlang:process_info to see what your processes are doing, and language indepedent tools to see what the VM is doing. process_info gives tons of information, IMHO, much more operable than other systems I'm familiar with. Ex: if your process has a large message queue, either it's not processing it's messages fast enough, it's selectively processing messages and leaving junk in the queue, or some combination; you can get inspect the queue with process_info and maybe figure that out. Lots of possibilities for why too slow, maybe it's waiting, maybe it's slow numerically, etc; process_info can tell you what function is running, which is usually helpful, etc. Network partitions and their effects are highly application dependant, but if you have multiple nodes in your existing system, you have to think about it already, or choose to ignore it, which you can also do here, although if you use mnesia, you shouldn't ignore it: the default is for partitioned and then rejoined nodes to stay apart, waiting for an intervention, which may be the only reasonable default (because there's no right answer), but it's not what people usually want.

Fire-Dragon-DoL 4y ago
One note, the message queue has a limit in size,going over the limit will drop messages.
- toast0 4y ago
  If there's a limit, it's new and I can't find it in a quick search; I did find a 2014 PR that was rejected because dropping messages violates core semantics, IMHO killing the recipient would be OK though. Traditionally it's unbounded, but when you exhaust the memory of the host (or the ulimit, if set), you'll likely loose the whole VM.
  - Fire-Dragon-DoL 4y ago
    Turns out it was specific to the application I worked on, sorry for the misinformation. It is configurable: https://www.erlang.org/eeps/eep-0042 https://www.erlang.org/eeps/eep-0042
    - toast0 4y ago
      EEP 42 is not implemented in mainline OTP. If it were, the proposal would likely link to the implementation, and the EEP index in EEP 0 would show what version it was first included in: https://www.erlang.org/eeps/eep-0000 https://www.erlang.org/eeps/eep-0000 If your application is using something like this, you're running a modified beam (which is fine; my professional experience is on a modified beam where we added a process_info option to drop all messages in the queue of a given process, and to allow for messages to be added to the front of the queue, neither of which would be acceptable upstream, but both of which were super handy for us)
      - Fire-Dragon-DoL 4y ago
        It's been a long time, so i could be wrong, but I remember we did face the problem at some point. Either way, not having a limit makes it even easier to use, which I really appreciate

Fire-Dragon-DoL 4y ago

I see your point, I still disagree, I encountered some bugs that were incredibly surprising with postgres + sidekiq combo (made perfect sense though) Elixir doesn't distribute anything unless you tell it to, keep that in mind. You can run it like a normal rails app. The OTP by default runs on one machine and as such, it's not subject to distributed systems law. And you can still scale horizontally in the traditional way (deploy the app to multiple machines), no need to learn or use the multi-nodes. The debugging experience is the same. Of course you have to learn the OTP library at some point, but it's equivalent to study the ruby standard library: it's natural, must be done, like for any programming language. That being said, the tools provided are really nice, given erlang is very old, the tools do exist. And you can connect to a production machine with an "irb session", except that the one in elixir is in the same memory space as the main process, so you can actually inspect in memory stuff, very powerful and dangerous.