5 ms·

Libraries can start processes too

70 points by sandbags 5y ago

dceddia 5y ago
In earlier versions of Elixir where you had to explicitly add the libraries as additional processes to start, I remember being very confused why things like e.g. an HTTP client needed its own process. To be honest I still felt a little uneasy about it every time. Thinking of those processes as just “the library needs some internal state” makes so much more sense!
- dnautics 5y ago
  Problem is, it's not entirely clear that an http client library needs external state. (And for example, mint does not).
  - masklinn 5y ago
    Persistent connections? Cookies? Response cache? Conditional requests?
    - chii 5y ago
      depends on how low level you want the library to be - for example, cURL doesn't have state.
      - masklinn 5y ago
        That is incorrect. Libcurl has a connection cache for persistent connections enabled by default. It also supports automatically recording and resending cookies (including optionally storing those cookies to disk for cross-session usage), though that is not enabled by default. In 2020, the CLI cURL utility also added support for conditional requests, although that is not native to the underlying libcurl.
AlchemistCamp 5y ago
Many, many libraries do this! It's a key feature. (to clarify, these are light-weight Elixir processes, not OS processes)
OskarS 5y ago
I think Erlang and Elixir are incredible, and this particular feature of libraries starting BEAM processes seems cool in the abstract. However, this PARTICULAR task, Erlang/Elixir might be the worst possible language you could choose: > The Elixir approach to shared mutable state is wrapping it in a process. In this case, I needed a counter and the easiest way to implement it is to use an Agent which is a kind of process designed to handle simple state. In this case, the get_and_update function allows me to return the counter and increment it as an atomic operation. This is literally just an atomic counter. It’s a single CPU instruction that is guaranteed to be safe. I don’t care how lightweight a BEAM process is, it’s not faster than updating an atomic counter. Doing it this way is also absurdly more complicated than using, say, a std::atomic in C++ (or the equivalent type in other languages). Again, I think Elixir is cool, but if you want to show off how cool it is, maybe don’t use an example that is incredibly much slower and more complicated than it should be. It’s not a great look for Elixir.
- enjo 5y ago
  Of course you're right, but I think anyone who is capable of getting their head around Elixir is capable of understanding that this is a trivial example meant to showcase a powerful language feature. What you choose to use it for is up to you, not the article's author.
  - OskarS 5y ago
    I see that point, but it was pretty jarring reading this Elixir article as a curious outsider that they had to make these kinds of very advanced contortions to do something so simple in such a non-performant way. It’s something that is only impressive if you were already totally sold in on the BEAM way of doing things, but looks insane to any C/C++/Rust/Java/whatever developer.
    - bcrosby95 5y ago
      Processes are a core fundamental concept in Elixir - I wouldn't call them advanced. But they certainly have been shoved to the background for a lot of developers due to things like the Phoenix framework where you might not have to make your own.
jolux 5y ago
This is a huge part of the power of Elixir/Erlang. Libraries can be written to use the same powers of isolation, resiliency, and concurrency that application code has, at no extra cost to the consumer and with total transparency. You quickly realize that every function you call could be working asynchronously underneath, and it’s not scary because the whole system is designed to function like this.
dnautics 5y ago
I really wish that in elixir we drew a distinction between dependencies that do and don't spawn their own supervision trees. It would be nice if this were reflected in, say, hex.pm, possibly using different tags for them.
- lostcolony 5y ago
  It's not obvious from use? It's been a long time since I was in this world (well, Erlang), but it was application:ensure_started(App) that let me know "this dependency has its own supervision tree".
  - dnautics 5y ago
    Httpoison, for example, starts its own supervision tree to manage client process pools (not obvious that an http client should do that, definitely usecases where it shouldn't) and there is no indication either in mix.exs, or `MyApplication.Application.start/2` that Httpoison needs the tree. For most deployments, it's probably not a big deal. Some process will try to read http content, fail if httpoison isn't quite ready, and be restarted by its supervisor. However, if you try to use it early in compilation or test, say, in your test_helper.exs file, speaking from experience, you could wind up with a very difficult to understand race condition where the httpoison process tree hasn't fully booted and you're trying to fetch something off the internet, and you don't have the same level of supervision protection -- if test_helper fails the whole test suite gives up and doesn't restart -- for obvious reason. For the http case thankfully the elixir ecosystem is getting Mint as a base http library, which doesn't require a process tree out the gate, and several interesting explicit process-pool-libaries (finch, mojito) which are tuned for their own use cases that derive from mint.
- QuinnWilton 5y ago
  Agreed. I also wish fewer libraries started their own supervision tree, and instead gave you a child spec to drop into your supervision tree. There's definitely use-cases where shipping libraries as an application makes sense, but oftentimes that sort of design causes problems for me, because it means not being able to start multiple copies of the dependency with different configurations. I think Phoenix PubSub is a perfect example of how libraries should be structured, in that you just need to drop the module + options into your supervision tree, and you have the freedom of starting multiple independent copies of the tree, in different contexts, and with their own configurations: https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html#module-getting-started https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html#module...

70 points by sandbags 5y ago

In earlier versions of Elixir where you had to explicitly add the libraries as additional processes to start, I remember being very confused why things like e.g. an HTTP client needed its own process. To be honest I still felt a little uneasy about it every time. Thinking of those processes as just “the library needs some internal state” makes so much more sense!

dnautics 5y ago
Problem is, it's not entirely clear that an http client library needs external state. (And for example, mint does not).
- masklinn 5y ago
  Persistent connections? Cookies? Response cache? Conditional requests?
  - chii 5y ago
    depends on how low level you want the library to be - for example, cURL doesn't have state.
    - masklinn 5y ago
      That is incorrect. Libcurl has a connection cache for persistent connections enabled by default. It also supports automatically recording and resending cookies (including optionally storing those cookies to disk for cross-session usage), though that is not enabled by default. In 2020, the CLI cURL utility also added support for conditional requests, although that is not native to the underlying libcurl.

AlchemistCamp 5y ago

Many, many libraries do this! It's a key feature. (to clarify, these are light-weight Elixir processes, not OS processes)

OskarS 5y ago

I think Erlang and Elixir are incredible, and this particular feature of libraries starting BEAM processes seems cool in the abstract. However, this PARTICULAR task, Erlang/Elixir might be the worst possible language you could choose: > The Elixir approach to shared mutable state is wrapping it in a process. In this case, I needed a counter and the easiest way to implement it is to use an Agent which is a kind of process designed to handle simple state. In this case, the get_and_update function allows me to return the counter and increment it as an atomic operation. This is literally just an atomic counter. It’s a single CPU instruction that is guaranteed to be safe. I don’t care how lightweight a BEAM process is, it’s not faster than updating an atomic counter. Doing it this way is also absurdly more complicated than using, say, a std::atomic in C++ (or the equivalent type in other languages). Again, I think Elixir is cool, but if you want to show off how cool it is, maybe don’t use an example that is incredibly much slower and more complicated than it should be. It’s not a great look for Elixir.

enjo 5y ago
Of course you're right, but I think anyone who is capable of getting their head around Elixir is capable of understanding that this is a trivial example meant to showcase a powerful language feature. What you choose to use it for is up to you, not the article's author.
- OskarS 5y ago
  I see that point, but it was pretty jarring reading this Elixir article as a curious outsider that they had to make these kinds of very advanced contortions to do something so simple in such a non-performant way. It’s something that is only impressive if you were already totally sold in on the BEAM way of doing things, but looks insane to any C/C++/Rust/Java/whatever developer.
  - bcrosby95 5y ago
    Processes are a core fundamental concept in Elixir - I wouldn't call them advanced. But they certainly have been shoved to the background for a lot of developers due to things like the Phoenix framework where you might not have to make your own.

jolux 5y ago

This is a huge part of the power of Elixir/Erlang. Libraries can be written to use the same powers of isolation, resiliency, and concurrency that application code has, at no extra cost to the consumer and with total transparency. You quickly realize that every function you call could be working asynchronously underneath, and it’s not scary because the whole system is designed to function like this.

dnautics 5y ago

I really wish that in elixir we drew a distinction between dependencies that do and don't spawn their own supervision trees. It would be nice if this were reflected in, say, hex.pm, possibly using different tags for them.

lostcolony 5y ago
It's not obvious from use? It's been a long time since I was in this world (well, Erlang), but it was application:ensure_started(App) that let me know "this dependency has its own supervision tree".
- dnautics 5y ago
  Httpoison, for example, starts its own supervision tree to manage client process pools (not obvious that an http client should do that, definitely usecases where it shouldn't) and there is no indication either in mix.exs, or `MyApplication.Application.start/2` that Httpoison needs the tree. For most deployments, it's probably not a big deal. Some process will try to read http content, fail if httpoison isn't quite ready, and be restarted by its supervisor. However, if you try to use it early in compilation or test, say, in your test_helper.exs file, speaking from experience, you could wind up with a very difficult to understand race condition where the httpoison process tree hasn't fully booted and you're trying to fetch something off the internet, and you don't have the same level of supervision protection -- if test_helper fails the whole test suite gives up and doesn't restart -- for obvious reason. For the http case thankfully the elixir ecosystem is getting Mint as a base http library, which doesn't require a process tree out the gate, and several interesting explicit process-pool-libaries (finch, mojito) which are tuned for their own use cases that derive from mint.
QuinnWilton 5y ago
Agreed. I also wish fewer libraries started their own supervision tree, and instead gave you a child spec to drop into your supervision tree. There's definitely use-cases where shipping libraries as an application makes sense, but oftentimes that sort of design causes problems for me, because it means not being able to start multiple copies of the dependency with different configurations. I think Phoenix PubSub is a perfect example of how libraries should be structured, in that you just need to drop the module + options into your supervision tree, and you have the freedom of starting multiple independent copies of the tree, in different contexts, and with their own configurations: https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html#module-getting-started https://hexdocs.pm/phoenix_pubsub/Phoenix.PubSub.html#module...