20 ms·

A Brief Guide to OTP in Elixir

242 points by NaeosPsy 6y ago

jeremyjh 6y ago
I'd just like to point out that while knowledge of OTP is a good thing for an Elixir developer, it isn't really needed for most application development. There are excellent libraries and frameworks that leverage OTP to provide a lot of value to their users, and most of us who use Elixir in industry just use those libraries.
- matt2000 6y ago
  Would you mind linking to a couple of them? I don't know anything about Elixir (but am very comfortable in a lot of other languages), and am interested in the kind of thing more experienced people are using with Elixir. Thanks!
  - jeremyjh 6y ago
    For web applications, Phoenix is very popular: https://github.com/phoenixframework/phoenix; https://github.com/phoenixframework/phoenix; it uses some OTP directly and its dependencies like the web-server Cowboy use it as well. DB development usually uses Ecto: https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto, it has OTP servers for connection pooling and other tasks. For managing background tasks I use Honeydew: https://github.com/koudelka/honeydew https://github.com/koudelka/honeydew
    - alickz 6y ago
      Your Phoenix link is broken (includes the semi-colon) for the lazy: https://github.com/phoenixframework/phoenix https://github.com/phoenixframework/phoenix
  - pythonaut_16 6y ago
    Phoenix is a good example. It makes great use of OTP, but you don't really need to touch OTP when using it to build web applications. You can, and your life as an Elixir dev will certainly be easier if you have some understanding of OTP, but it works great as a web framework even if you don't.
- ing33k 6y ago
  depends on the type of project for sure. possessing knowledge of OTP is a superpower. when working on any serious project, it will always help if there is at least one developer in the team who has knowledge about it.
- dnautics 6y ago
  I would go so far as to say a beginner should not write genservers, and using genservers that aren't part of a library is usually an antipattern. When you're ready to write a serious library, then you're ready to write genserver. Of course it's always good to understand genservers.
- bcrosby95 6y ago
  I'm working on a game backend (which I'm finding Elixir/Erlang pretty good for) and I don't see how I could reasonably do this without using my own GenServers. I can see not writing my own GenServer for webapps, but when it comes to concurrency webapps tend to be dead simple. Elixir is a fine choice for these but I feel like its killer features don't shine that much over the alternatives.
- darksavant 6y ago
  unless you're only building web apps i don't know how you can build systems in elixir without touching otp
abetlen 6y ago
OTP = Open Telecom Platform [1], though I think the full name is avoided nowadays. On an unrelated note, I've always been fascinated with the Erlang VM. The idea that you could hot-load modules and have multiple versions of the same module running at once seems really useful. I wonder why other runtimes haven't adopted these features? [1] https://en.wikipedia.org/wiki/Open_Telecom_Platform https://en.wikipedia.org/wiki/Open_Telecom_Platform
- yetihehe 6y ago
  > I wonder why other runtimes haven't adopted these features? Probably mainly due to shared memory. When you know your data can be modified only in one place, changing data structure is easier. Try changing a struct when some other code is using it. It would require a lock for every object in memory.
  - macintux 6y ago
    One of the great things about Erlang and the BEAM is that by designing a very opinionated language around concepts like immutability and ubiquitous messaging between lightweight actors, you can define the VM to support those. So there’s this collection of features between the language and VM that support each other, and are generally hard to reproduce outside that ecosystem because they are so interdependent.
  - dnautics 6y ago
    the fact that looping through a process is a tail call which by its very nature creates well-defined boundaries (both in "code segment" and in "data segment") also helps a gazillion. Try doing hot loading in the middle of a while loop... Not saying it's impossible, but you're not gonna have a good time.
- rectang 6y ago
  Or "One Time Password" if you're an idiot like me. Gotta love an introductory article on "OTP" which even includes a section entitled "What is OTP?" but doesn't expand the acronym.
yetihehe 6y ago
> Every time the loop runs, it will check the top of the mailbox (mailbox is last-in-first-out) for messages that match what we need and process them. Nope, erlang mailbox is fifo with filtering. When a process sends messages, they are delivered in the same order they were sent.
- yetihehe 6y ago
  That sentence was fixed in the article and now reads: > Every time the loop runs, it will check from the bottom of the mailbox (in order they were received) for messages that match what we need and process them.
pmontra 6y ago
> In real life, we don’t need to write code with receive do loops. Instead, we use one of the behaviours created by people much smarter than us. I make more than half of my income from Elixir. That said, the naive receive loop is much easier to understand than any of the GenServer examples. They pollute the module logic with all that handle_* boilerplate. I believe that neither Erlang nor Elixir got the right abstraction there. Method definition in any OO language is easier to understand. At least Elixir should have taken Agent and made it an invisible part of the language. All those calls to Agent in this example https://elixir-lang.org/getting-started/mix-otp/agent.html https://elixir-lang.org/getting-started/mix-otp/agent.html are still boilerplate.
- jeremyjh 6y ago
  There are libraries that do this, like https://github.com/sasa1977/exactor https://github.com/sasa1977/exactor but the Elixir community nowadays seems more conservative about macro "magic" than it was a few years ago. Sasa has written an entire book about OTP and if he doesn't recommend using his own library I wouldn't argue with him, but I haven't written a Genserver myself in years despite writing Elixir for a living the past 3.5 years. If you are going to learn how they work though, it makes sense to stick to how Erlang actually implements them.
  - buzzerbetrayed 6y ago
    > There are libraries that do this, like https://github.com/sasa1977/exactor https://github.com/sasa1977/exactor At the very top of the Extractor readme: > I don't maintain this project anymore. In hindsight, I don't think it was a good idea in the first place. I haven't been using ExActor myself for years, and I recommend sticking with regular GenServer instead :-) This both confirms what you said about the Elixir community shying away from macro magic, and makes this library a very bad idea to use in new projects, imo.
    - jeremyjh 6y ago
      Right, thats why I said he doesn't recommend using his own library. My point is people have been down this road, and moved away from it, not that anyone should use it.
nickjj 6y ago
> Forget using a million different technologies for things like background jobs, OTP can supply you with everything. It's true but in practice this usually doesn't pan out. For example with just background jobs alone there's the idea of queues, tracking failures / successes, exponential back-off retries, guaranteeing uniqueness, draining, periodic tasks and everything else you'd likely want in a production ready app. Typically you'd use Redis, Postgres or something else to help with this. Fortunately https://github.com/sorentwo/oban https://github.com/sorentwo/oban exists and uses Postgres as a back-end with close to 10,000 lines of Elixir.
- sergiotapia 6y ago
  This is my experience as well, working with Elixir fulltime for about 5 years.
- atonse 6y ago
  Yep exactly. We process millions of jobs using Exq, mainly because of the retry, persistence of jobs, etc. I don't want to risk leaving that in elixir only. We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).
  - pselbert 6y ago
    A lot of people have been moving from Exq to Oban. Come by the #oban channel on Slack if you have any questions.
- rozap 6y ago
  I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points: Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time Tracking failures/sucesses: postgres Exponential backoff: a function Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2 https://erlang.org/doc/man/global.html#trans-2) draining: remote console periodic tasks: a simple library that knows how to cron I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff. Erlang has plenty of drawbacks, but this isn't it.
mrdoops 6y ago
OTP is definitely more of an expert's toolset than a framework for throwing together features quickly. This is a good thing because it's not trying to be more of an abstraction than necessary. With OTP you can make fault-tolerant, concurrent, self-healing applications that handle stateful operations with ease, but you'll still need a fair bit of need-to-know about the callbacks, how to manage process lifecycles, etc. What a lot of people seem to want is a batteries included framework that does the OTP for you, but doing at a high level of abstraction is difficult. So more commonly you'll see the OTP managed by libraries for more concrete use cases (web servers, database connection pooling, back pressured job processing, etc). Trying to wrap or rebuild OTP from the primitives available in the BEAM is possible, but definitely not an easy task. Could we potentially have an OTP-like library that cuts down on the boilerplate significantly? Yes. But how much effort would that take and is it much of a win over OTP as-is? Not sure the trade-off is entirely there yet - OTP really isn't that bad. You're still writing less boilerplate than Java in general. I think we're much better off writing good library-level abstractions over OTP that can be embedded in a supervision tree or used in more specific use cases. If we were to actually attempt some kind of higher level abstraction in the scope of OTP - maybe we can do so with a very different approach like decorated dataflow graphs and runtime property based testing.
- dnautics 6y ago
  Elixir already started the process of extending OTP: Task is a hugely important addition to the family of genserver, genstatem, etc.
  - filmor 6y ago
    Isn't Task an entirely different beast to the gen_* behaviours? To me it looks like a convenience wrapper around Erlang's primitives. I usually use them directly if I need the flexibility to "await" later. If I don't, I have a small function "run_in_subprocess" (mostly to not run into binary gc problems) that spawns and receives within one function. The only real advantage I see is that it is separately supervised. When avoiding trap_exit, "spawn_link"ing within a process that is itself supervised has a similar effect, though.
davidw 6y ago
Knowing how to use OTP is definitely something worthwhile if you're going to stick with the BEAM platform: https://ferd.ca/ten-years-of-erlang.html https://ferd.ca/ten-years-of-erlang.html
thelazydogsback 6y ago
I assume because every process has its own GC that messages are always copied, even on the same node? AFAICT, there is no shared/copy-on-write messages in the BEAM, except for a few cases where you explicitly pass a shared byte buffer for special circumstances? I would think this is worth the cost in most cases, but for some workloads would be prohibitive.
- toast0 6y ago
  Some binaries (byte buffers as you called them) are stored outside the process heap, and those are messaged by reference, not by copying; but all other terms are messaged by copying. This is a lot of copying, but it makes the GC very simple, so it's usually a positive tradeoff. Although, there's probably some use cases where it's not great. For those, it might makes sense to force things into the off process heap binaries (RefC), or possibly by doing terrible things with native code (Nifs). You might also put things into the included key-value memory storage (ets), and more selectively pull things out to avoid copying a large chunk data to every process that needs a small piece.
  - dnautics 6y ago
    It is possible to create nif objects which are tied into the garbage collector and reference counted, and bound to a destructor function that gets called when the VM is done with it!
freedomben 6y ago
> In a sense, supervisors are very similar to Kubernetes, but they work on the application level instead of the cluster level. Good article, and very worth sharing, but it could have done without that part. That's a terribly unhelpful analogy. For those who Kubernetes it's wildly misleading, and for those who don't it's meaningless.
- jakeva 6y ago
  I use both kubernetes and OTP supervisors and don't find it misleading, rather I quite agree but maybe I'm missing something. I'm curious what you mean by that?
  - freedomben 6y ago
    Interesting, thanks for the feedback. I find it misleading because Kubernetes is a massive platform that does all manner of things besides just supervising your Pods, and in fact I had to think about it for a while before concluding that the supervisory function is likely what the author was alluding to. It strikes me as saying that an oxygen mask in a hospital room is like an airplane: sure they both provide a facility for providing oxygen to the user, but the airplane is significantly more complex than simply an oxygen delivery system. Probably a terrible analogy, just thinking off the top of my head.
    - dfinninger 6y ago
      I agree with your sentiment. I think a more apt metaphor than Kubernetes in its entirety would be comparing it to a Kubernetes ReplicaSet[1]. I can see some similarities there. For fun, to expound on your analogy, maybe it's like comparing an airport to a hospital. They both feature a person you check in with (triage/ticket counter) to direct you to an appropriate room/gate depending on your needs, but the core mission of each are rather different. [1] https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ https://kubernetes.io/docs/concepts/workloads/controllers/re...
  - jeremyjh 6y ago
    You don't find it misleading because you already know what an OTP Supervisor's role is limited to, and you know that Kubernetes (among a thousand other things) also provides that one feature. But its not a good analogy since it can only be understood by someone who doesn't need it to be explained.
fredliu 6y ago
IMO, Erlang/Elixir/BEAM/OTP really is the toolset that can be used to build what the mystical 10x/100x programmer can achieve. But that's also the reason why it's hard to push for its adoption into bigger teams. When the value proposition is "you can do all that by only using Erlang/Elixir/OTP...", while the alternatives are based on multiple, but more "traditional" and well-understood systems (which you are easier to hire for) ,"using one thing for all" sounds more like a risk (and potential tech/dev lock-in) than an advantage to the management. Love to hear success stories of how Erlang/Elixir are pushed into "mainstream" within a relative large org.
- Thaxll 6y ago
  The problem is that the runtime does too much magic and it's not portable / standard compared to other languages / runtimes. Lot of what BEAM/OTP is doing is done in other tools and it's language neutral. Not everything is good tbh, when you see the pain it is to deploy an Erlang app in 2020 ...
  - cutety 6y ago
    Regarding portability, I recently came across this project: https://github.com/spawnfest/bakeware https://github.com/spawnfest/bakeware I haven’t gotten around to actually trying it myself, but it advertises that it can compile Elixir projects into a single binary you can copy, paste & run (on Linux & MacOS). It’s not the greatest solution, and the mentioned ~0.5s startup time doesn’t make it great for cli tools when compared to Go/Rust/C. I’m mostly just happy to see there are people out there trying to make Elixir/Erlang easier to use, as much as I love the language, some of the tooling and deployment methods (releases) make me groan.
  - dnautics 6y ago
    is it a pain? mix release Compiles everything, tar/gz's it, sends it to a private s3 bucket. On your server side, you periodically watch the s3 bucket, and when one of your nodes detects a relup, it downloads from the s3 bucket, kills itself. Systemd then restarts it, kicking into the newest version. That's it. This is maybe about 50-100 lines of code, one external library (pick your AWS library of choice), and one systemd script. I think there's even a library for managing systemd from within the BEAM now. If you want to be more sophisticated (a rolling blue-green deploy across an erlang cluster) you could probably do it with transactional locks with the :global module in about an additional 100-ish lines of code, including fully verifying the soundness of newly upgraded nodes using telemetry.

242 points by NaeosPsy 6y ago

I'd just like to point out that while knowledge of OTP is a good thing for an Elixir developer, it isn't really needed for most application development. There are excellent libraries and frameworks that leverage OTP to provide a lot of value to their users, and most of us who use Elixir in industry just use those libraries.

matt2000 6y ago
Would you mind linking to a couple of them? I don't know anything about Elixir (but am very comfortable in a lot of other languages), and am interested in the kind of thing more experienced people are using with Elixir. Thanks!
- jeremyjh 6y ago
  For web applications, Phoenix is very popular: https://github.com/phoenixframework/phoenix; https://github.com/phoenixframework/phoenix; it uses some OTP directly and its dependencies like the web-server Cowboy use it as well. DB development usually uses Ecto: https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto, it has OTP servers for connection pooling and other tasks. For managing background tasks I use Honeydew: https://github.com/koudelka/honeydew https://github.com/koudelka/honeydew
  - alickz 6y ago
    Your Phoenix link is broken (includes the semi-colon) for the lazy: https://github.com/phoenixframework/phoenix https://github.com/phoenixframework/phoenix
- pythonaut_16 6y ago
  Phoenix is a good example. It makes great use of OTP, but you don't really need to touch OTP when using it to build web applications. You can, and your life as an Elixir dev will certainly be easier if you have some understanding of OTP, but it works great as a web framework even if you don't.
ing33k 6y ago
depends on the type of project for sure. possessing knowledge of OTP is a superpower. when working on any serious project, it will always help if there is at least one developer in the team who has knowledge about it.
dnautics 6y ago
I would go so far as to say a beginner should not write genservers, and using genservers that aren't part of a library is usually an antipattern. When you're ready to write a serious library, then you're ready to write genserver. Of course it's always good to understand genservers.
bcrosby95 6y ago
I'm working on a game backend (which I'm finding Elixir/Erlang pretty good for) and I don't see how I could reasonably do this without using my own GenServers. I can see not writing my own GenServer for webapps, but when it comes to concurrency webapps tend to be dead simple. Elixir is a fine choice for these but I feel like its killer features don't shine that much over the alternatives.
darksavant 6y ago
unless you're only building web apps i don't know how you can build systems in elixir without touching otp

abetlen 6y ago

OTP = Open Telecom Platform [1], though I think the full name is avoided nowadays. On an unrelated note, I've always been fascinated with the Erlang VM. The idea that you could hot-load modules and have multiple versions of the same module running at once seems really useful. I wonder why other runtimes haven't adopted these features? [1] https://en.wikipedia.org/wiki/Open_Telecom_Platform https://en.wikipedia.org/wiki/Open_Telecom_Platform

yetihehe 6y ago
> I wonder why other runtimes haven't adopted these features? Probably mainly due to shared memory. When you know your data can be modified only in one place, changing data structure is easier. Try changing a struct when some other code is using it. It would require a lock for every object in memory.
- macintux 6y ago
  One of the great things about Erlang and the BEAM is that by designing a very opinionated language around concepts like immutability and ubiquitous messaging between lightweight actors, you can define the VM to support those. So there’s this collection of features between the language and VM that support each other, and are generally hard to reproduce outside that ecosystem because they are so interdependent.
- dnautics 6y ago
  the fact that looping through a process is a tail call which by its very nature creates well-defined boundaries (both in "code segment" and in "data segment") also helps a gazillion. Try doing hot loading in the middle of a while loop... Not saying it's impossible, but you're not gonna have a good time.
rectang 6y ago
Or "One Time Password" if you're an idiot like me. Gotta love an introductory article on "OTP" which even includes a section entitled "What is OTP?" but doesn't expand the acronym.

yetihehe 6y ago

> Every time the loop runs, it will check the top of the mailbox (mailbox is last-in-first-out) for messages that match what we need and process them. Nope, erlang mailbox is fifo with filtering. When a process sends messages, they are delivered in the same order they were sent.

yetihehe 6y ago
That sentence was fixed in the article and now reads: > Every time the loop runs, it will check from the bottom of the mailbox (in order they were received) for messages that match what we need and process them.

pmontra 6y ago

> In real life, we don’t need to write code with receive do loops. Instead, we use one of the behaviours created by people much smarter than us. I make more than half of my income from Elixir. That said, the naive receive loop is much easier to understand than any of the GenServer examples. They pollute the module logic with all that handle_* boilerplate. I believe that neither Erlang nor Elixir got the right abstraction there. Method definition in any OO language is easier to understand. At least Elixir should have taken Agent and made it an invisible part of the language. All those calls to Agent in this example https://elixir-lang.org/getting-started/mix-otp/agent.html https://elixir-lang.org/getting-started/mix-otp/agent.html are still boilerplate.

jeremyjh 6y ago
There are libraries that do this, like https://github.com/sasa1977/exactor https://github.com/sasa1977/exactor but the Elixir community nowadays seems more conservative about macro "magic" than it was a few years ago. Sasa has written an entire book about OTP and if he doesn't recommend using his own library I wouldn't argue with him, but I haven't written a Genserver myself in years despite writing Elixir for a living the past 3.5 years. If you are going to learn how they work though, it makes sense to stick to how Erlang actually implements them.
- buzzerbetrayed 6y ago
  > There are libraries that do this, like https://github.com/sasa1977/exactor https://github.com/sasa1977/exactor At the very top of the Extractor readme: > I don't maintain this project anymore. In hindsight, I don't think it was a good idea in the first place. I haven't been using ExActor myself for years, and I recommend sticking with regular GenServer instead :-) This both confirms what you said about the Elixir community shying away from macro magic, and makes this library a very bad idea to use in new projects, imo.
  - jeremyjh 6y ago
    Right, thats why I said he doesn't recommend using his own library. My point is people have been down this road, and moved away from it, not that anyone should use it.

nickjj 6y ago

> Forget using a million different technologies for things like background jobs, OTP can supply you with everything. It's true but in practice this usually doesn't pan out. For example with just background jobs alone there's the idea of queues, tracking failures / successes, exponential back-off retries, guaranteeing uniqueness, draining, periodic tasks and everything else you'd likely want in a production ready app. Typically you'd use Redis, Postgres or something else to help with this. Fortunately https://github.com/sorentwo/oban https://github.com/sorentwo/oban exists and uses Postgres as a back-end with close to 10,000 lines of Elixir.

sergiotapia 6y ago
This is my experience as well, working with Elixir fulltime for about 5 years.
atonse 6y ago
Yep exactly. We process millions of jobs using Exq, mainly because of the retry, persistence of jobs, etc. I don't want to risk leaving that in elixir only. We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).
- pselbert 6y ago
  A lot of people have been moving from Exq to Oban. Come by the #oban channel on Slack if you have any questions.
rozap 6y ago
I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points: Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time Tracking failures/sucesses: postgres Exponential backoff: a function Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2 https://erlang.org/doc/man/global.html#trans-2) draining: remote console periodic tasks: a simple library that knows how to cron I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff. Erlang has plenty of drawbacks, but this isn't it.

mrdoops 6y ago

OTP is definitely more of an expert's toolset than a framework for throwing together features quickly. This is a good thing because it's not trying to be more of an abstraction than necessary. With OTP you can make fault-tolerant, concurrent, self-healing applications that handle stateful operations with ease, but you'll still need a fair bit of need-to-know about the callbacks, how to manage process lifecycles, etc. What a lot of people seem to want is a batteries included framework that does the OTP for you, but doing at a high level of abstraction is difficult. So more commonly you'll see the OTP managed by libraries for more concrete use cases (web servers, database connection pooling, back pressured job processing, etc). Trying to wrap or rebuild OTP from the primitives available in the BEAM is possible, but definitely not an easy task. Could we potentially have an OTP-like library that cuts down on the boilerplate significantly? Yes. But how much effort would that take and is it much of a win over OTP as-is? Not sure the trade-off is entirely there yet - OTP really isn't that bad. You're still writing less boilerplate than Java in general. I think we're much better off writing good library-level abstractions over OTP that can be embedded in a supervision tree or used in more specific use cases. If we were to actually attempt some kind of higher level abstraction in the scope of OTP - maybe we can do so with a very different approach like decorated dataflow graphs and runtime property based testing.

dnautics 6y ago
Elixir already started the process of extending OTP: Task is a hugely important addition to the family of genserver, genstatem, etc.
- filmor 6y ago
  Isn't Task an entirely different beast to the gen_* behaviours? To me it looks like a convenience wrapper around Erlang's primitives. I usually use them directly if I need the flexibility to "await" later. If I don't, I have a small function "run_in_subprocess" (mostly to not run into binary gc problems) that spawns and receives within one function. The only real advantage I see is that it is separately supervised. When avoiding trap_exit, "spawn_link"ing within a process that is itself supervised has a similar effect, though.

davidw 6y ago

Knowing how to use OTP is definitely something worthwhile if you're going to stick with the BEAM platform: https://ferd.ca/ten-years-of-erlang.html https://ferd.ca/ten-years-of-erlang.html

thelazydogsback 6y ago

I assume because every process has its own GC that messages are always copied, even on the same node? AFAICT, there is no shared/copy-on-write messages in the BEAM, except for a few cases where you explicitly pass a shared byte buffer for special circumstances? I would think this is worth the cost in most cases, but for some workloads would be prohibitive.

toast0 6y ago
Some binaries (byte buffers as you called them) are stored outside the process heap, and those are messaged by reference, not by copying; but all other terms are messaged by copying. This is a lot of copying, but it makes the GC very simple, so it's usually a positive tradeoff. Although, there's probably some use cases where it's not great. For those, it might makes sense to force things into the off process heap binaries (RefC), or possibly by doing terrible things with native code (Nifs). You might also put things into the included key-value memory storage (ets), and more selectively pull things out to avoid copying a large chunk data to every process that needs a small piece.
- dnautics 6y ago
  It is possible to create nif objects which are tied into the garbage collector and reference counted, and bound to a destructor function that gets called when the VM is done with it!

freedomben 6y ago

> In a sense, supervisors are very similar to Kubernetes, but they work on the application level instead of the cluster level. Good article, and very worth sharing, but it could have done without that part. That's a terribly unhelpful analogy. For those who Kubernetes it's wildly misleading, and for those who don't it's meaningless.

jakeva 6y ago
I use both kubernetes and OTP supervisors and don't find it misleading, rather I quite agree but maybe I'm missing something. I'm curious what you mean by that?
- freedomben 6y ago
  Interesting, thanks for the feedback. I find it misleading because Kubernetes is a massive platform that does all manner of things besides just supervising your Pods, and in fact I had to think about it for a while before concluding that the supervisory function is likely what the author was alluding to. It strikes me as saying that an oxygen mask in a hospital room is like an airplane: sure they both provide a facility for providing oxygen to the user, but the airplane is significantly more complex than simply an oxygen delivery system. Probably a terrible analogy, just thinking off the top of my head.
  - dfinninger 6y ago
    I agree with your sentiment. I think a more apt metaphor than Kubernetes in its entirety would be comparing it to a Kubernetes ReplicaSet[1]. I can see some similarities there. For fun, to expound on your analogy, maybe it's like comparing an airport to a hospital. They both feature a person you check in with (triage/ticket counter) to direct you to an appropriate room/gate depending on your needs, but the core mission of each are rather different. [1] https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ https://kubernetes.io/docs/concepts/workloads/controllers/re...
- jeremyjh 6y ago
  You don't find it misleading because you already know what an OTP Supervisor's role is limited to, and you know that Kubernetes (among a thousand other things) also provides that one feature. But its not a good analogy since it can only be understood by someone who doesn't need it to be explained.

fredliu 6y ago

IMO, Erlang/Elixir/BEAM/OTP really is the toolset that can be used to build what the mystical 10x/100x programmer can achieve. But that's also the reason why it's hard to push for its adoption into bigger teams. When the value proposition is "you can do all that by only using Erlang/Elixir/OTP...", while the alternatives are based on multiple, but more "traditional" and well-understood systems (which you are easier to hire for) ,"using one thing for all" sounds more like a risk (and potential tech/dev lock-in) than an advantage to the management. Love to hear success stories of how Erlang/Elixir are pushed into "mainstream" within a relative large org.

Thaxll 6y ago
The problem is that the runtime does too much magic and it's not portable / standard compared to other languages / runtimes. Lot of what BEAM/OTP is doing is done in other tools and it's language neutral. Not everything is good tbh, when you see the pain it is to deploy an Erlang app in 2020 ...
- cutety 6y ago
  Regarding portability, I recently came across this project: https://github.com/spawnfest/bakeware https://github.com/spawnfest/bakeware I haven’t gotten around to actually trying it myself, but it advertises that it can compile Elixir projects into a single binary you can copy, paste & run (on Linux & MacOS). It’s not the greatest solution, and the mentioned ~0.5s startup time doesn’t make it great for cli tools when compared to Go/Rust/C. I’m mostly just happy to see there are people out there trying to make Elixir/Erlang easier to use, as much as I love the language, some of the tooling and deployment methods (releases) make me groan.
- dnautics 6y ago
  is it a pain? mix release Compiles everything, tar/gz's it, sends it to a private s3 bucket. On your server side, you periodically watch the s3 bucket, and when one of your nodes detects a relup, it downloads from the s3 bucket, kills itself. Systemd then restarts it, kicking into the newest version. That's it. This is maybe about 50-100 lines of code, one external library (pick your AWS library of choice), and one systemd script. I think there's even a library for managing systemd from within the BEAM now. If you want to be more sophisticated (a rolling blue-green deploy across an erlang cluster) you could probably do it with transactional locks with the :global module in about an additional 100-ish lines of code, including fully verifying the soundness of newly upgraded nodes using telemetry.