3 ms·

(In Elixir) I've most recently found myself lamenting the indirection of processes as a ton of third party dependencies spin off processes to manage their state

(In Elixir) I've most recently found myself lamenting the indirection of processes as a ton of third party dependencies spin off processes to manage their state, and when something goes wrong (usually some argument I'm incorrectly passing in to a function call or something) the stack trace I get doesn't lead back to my code so I have to try and figure out the closest point to where I'm invoking it. Probably still a good tradeoff for immutability and the actor model, but an unfortunate downside that has pained me.

throwawaymaths 3y ago
I haven't found most 3rd party dependencies doing that unless they're making, for example, a request pool. Should we be shaming those libraries?
- freedomben 3y ago
  Maybe so, although there's usually at least some justification for it, whether it's performance related or just as simple as "the API can get a lot better when it can track some state so you don't have to pass every initialization value in every time." I particularly don't agree with the latter most of the time, but I can see how they arrive at the gen server conclusion.
dqv 3y ago
What libraries? You might be able to do something with the process dict depending on how the libraries are set up. Basically you create a process id that you use to track what's going on: @type trace_id :: nonempty_binary() @spec put_trace_id(trace_id()) :: term() | nil def put_trace_id(id), do: Process.put({:my_app_name, :trace_id}, id) @spec get_trace_id :: trace_id() | nil def get_trace_id, do: Process.get({:my_app_name, :trace_id}) @spec make_trace_id :: trace_id() def make_trace_id, do: Base.hex_encode32(:crypto.strong_rand_bytes(16), padding: false) It does require that, along the call path, you set that trace id whenever a new process is started. It gives you some visibility into what's going on, especially if you also do something like Logger.metadata(trace_id: get_trace_id()) after you call put_trace_id/1 That said, I think any library that does start processes like that should have some mechanism that lets us identify the provenance of the crash. "Crashed when trying to fribulate. Trace id xxxxxx"
- yuliyp 3y ago
  Libraries in a single process are tricky because so many things are dependent on messages. So a piece of code running inside another process has to rely on the "hosting" process to deliver all of the messages it needs. So if it monitors another process it needs to have its DOWNs delivered to it. If it wants to use a timer it needs the timer messages delivered to it. If it wants to make synchronous GenServer.calls, the parent process needs to be aware of that and plan around it (i.e. realizing that it may be stuck while waiting on that call to complete). A separate process can own its responsibilities.
  - throwawaymaths 3y ago
    A Library can just be a bundle of code. Suppose you're parsing xml. There is no fucking reason for that code to be in its own process, and it does not need to do anything with messages.
    - freedomben 3y ago
      In this example, I could see a library parsing xml spin off a process so it can track state. I could also see a performance argument for not wanting to do potentially computationally expensives (or even I/O expensive) work on the main process blocking other things. I'm not enough of a parser expert to know whether stateless processing is possible, but that's how I can see a simple xml parser ending up in a separate process.
      - throwawaymaths 3y ago
        "parsing xml spin off a process so it can track state" Don't do that. Use state tracking data type. "the main process" That's not a thing in the BEAM. For something pure compute (i.e. no need for connection pools), if your user really needs that, then they can run the xml parsing in their own spawned process. Even if it's I/O bound, Erlang VM already makes great decisions about yielding, you will almost certainly not do better. You should not make concurrency decision on the users behalf.

throwawaymaths 3y ago

I haven't found most 3rd party dependencies doing that unless they're making, for example, a request pool. Should we be shaming those libraries?

freedomben 3y ago
Maybe so, although there's usually at least some justification for it, whether it's performance related or just as simple as "the API can get a lot better when it can track some state so you don't have to pass every initialization value in every time." I particularly don't agree with the latter most of the time, but I can see how they arrive at the gen server conclusion.

dqv 3y ago

What libraries? You might be able to do something with the process dict depending on how the libraries are set up. Basically you create a process id that you use to track what's going on: @type trace_id :: nonempty_binary() @spec put_trace_id(trace_id()) :: term() | nil def put_trace_id(id), do: Process.put({:my_app_name, :trace_id}, id) @spec get_trace_id :: trace_id() | nil def get_trace_id, do: Process.get({:my_app_name, :trace_id}) @spec make_trace_id :: trace_id() def make_trace_id, do: Base.hex_encode32(:crypto.strong_rand_bytes(16), padding: false) It does require that, along the call path, you set that trace id whenever a new process is started. It gives you some visibility into what's going on, especially if you also do something like Logger.metadata(trace_id: get_trace_id()) after you call put_trace_id/1 That said, I think any library that does start processes like that should have some mechanism that lets us identify the provenance of the crash. "Crashed when trying to fribulate. Trace id xxxxxx"

yuliyp 3y ago
Libraries in a single process are tricky because so many things are dependent on messages. So a piece of code running inside another process has to rely on the "hosting" process to deliver all of the messages it needs. So if it monitors another process it needs to have its DOWNs delivered to it. If it wants to use a timer it needs the timer messages delivered to it. If it wants to make synchronous GenServer.calls, the parent process needs to be aware of that and plan around it (i.e. realizing that it may be stuck while waiting on that call to complete). A separate process can own its responsibilities.
- throwawaymaths 3y ago
  A Library can just be a bundle of code. Suppose you're parsing xml. There is no fucking reason for that code to be in its own process, and it does not need to do anything with messages.
  - freedomben 3y ago
    In this example, I could see a library parsing xml spin off a process so it can track state. I could also see a performance argument for not wanting to do potentially computationally expensives (or even I/O expensive) work on the main process blocking other things. I'm not enough of a parser expert to know whether stateless processing is possible, but that's how I can see a simple xml parser ending up in a separate process.
    - throwawaymaths 3y ago
      "parsing xml spin off a process so it can track state" Don't do that. Use state tracking data type. "the main process" That's not a thing in the BEAM. For something pure compute (i.e. no need for connection pools), if your user really needs that, then they can run the xml parsing in their own spawned process. Even if it's I/O bound, Erlang VM already makes great decisions about yielding, you will almost certainly not do better. You should not make concurrency decision on the users behalf.