4 ms·

> Forget using a million different technologies for things like background jobs, OTP can supply you with everything. It's true but in practice this usually doe

by nickjj 6y ago

> Forget using a million different technologies for things like background jobs, OTP can supply you with everything. It's true but in practice this usually doesn't pan out. For example with just background jobs alone there's the idea of queues, tracking failures / successes, exponential back-off retries, guaranteeing uniqueness, draining, periodic tasks and everything else you'd likely want in a production ready app. Typically you'd use Redis, Postgres or something else to help with this. Fortunately https://github.com/sorentwo/oban https://github.com/sorentwo/oban exists and uses Postgres as a back-end with close to 10,000 lines of Elixir.

sergiotapia 6y ago
This is my experience as well, working with Elixir fulltime for about 5 years.
atonse 6y ago
Yep exactly. We process millions of jobs using Exq, mainly because of the retry, persistence of jobs, etc. I don't want to risk leaving that in elixir only. We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).
- pselbert 6y ago
  A lot of people have been moving from Exq to Oban. Come by the #oban channel on Slack if you have any questions.
rozap 6y ago
I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points: Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time Tracking failures/sucesses: postgres Exponential backoff: a function Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2 https://erlang.org/doc/man/global.html#trans-2) draining: remote console periodic tasks: a simple library that knows how to cron I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff. Erlang has plenty of drawbacks, but this isn't it.
- toast0 6y ago
  > There's usually (almost always) durable storage required somewhere. Despite the name, mnesia can provide durable storage, without using anything outside of OTP and your nodes' (hopefully durable) filesystems.
  - riquito 6y ago
    Despite? It pretty much means "to remember", it doesn't have the alpha privative of amnesia, "to forget"
    - patrec 6y ago
      IIRC original name was Amnesia, but Joe was made to change it for marketing reasons (it's in one of his books or blog posts). Anyway, I do not think Mnesia is a particularly good way to persist stuff, just use postgres.
  - simscitizen 6y ago
    At WhatsApp we had a bunch of mnesia-backed services. I can't say that it was the best experience. Pretty much anything non-trivial, such as healing a cluster after a partition, or expanding a cluster, or cross-DC replication was a complete PITA to do properly. There's also very few people that have operational experience with it and very little documentation about it compared to pretty much any other storage engine. The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database. (Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)
    - toast0 6y ago
      I think we would have found any database to be a PITA at write volumes and data volumes we were using, with our expectations of availability. I don't think we would have been able to get that performance out of MySQL, or Postgres; and anyway, we didn't have a dedicated team for database management to make sure nobody wrote bad queries if we did. Sync, status and reg ran on MySQL and php when I started, and it was a lot easier to run them and scale them on Erlang, IMHO. Repartitioning MySQL like we repartitioned mnesia would be a lot harder, I think. I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year. Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :) (Hmmm, now I'm trying to figure out you are, send me an email if you will :)
laser 6y ago
Phoenix ships with Ecto ORM-style-binding to Postgres underneath (usually) for a good reason! :) And if you need that in memory magic where the serial nature of a genserver won't suffice for your load, you basically have built-in Redis with Erlang's ets. And if you really need insane performance on longer-cached items than ets is optimal for, Discord released https://github.com/discord/fastglobal https://github.com/discord/fastglobal And if you don't want to put all those pieces together, like you said there's a rich set of modules people have built, like oban. Long rise and live the elixir monoliths ;) (although admittedly even Discord had to swap out some of their most intense functions with Rust, but how many people are writing systems that need to handle millions of concurrents?)
- nickjj 6y ago
  > you basically have built-in Redis with Erlang's ets The big difference there is if you have a Dockerized app, Redis won't be restart on every deploy but the BEAM will be so you will lose your ETS cache or whatever you're storing. Now, I know cache is meant to be disposable but this model of using ETS instead of Redis reminds me of a Python or Ruby app storing its cache in memory. I mean, sure you can do it but for the longest time (7-8 years?) the community as a whole kind of landed on keeping that stuff out of your app's process.

by nickjj 6y ago

sergiotapia 6y ago

This is my experience as well, working with Elixir fulltime for about 5 years.

atonse 6y ago

Yep exactly. We process millions of jobs using Exq, mainly because of the retry, persistence of jobs, etc. I don't want to risk leaving that in elixir only. We use Exq because I've used Sidekiq for years. I should check out Oban, since it could reduce the complexity in our stack (we wouldn't need redis anymore).

pselbert 6y ago
A lot of people have been moving from Exq to Oban. Come by the #oban channel on Slack if you have any questions.

rozap 6y ago

I think these blog posts usually pretend it's simpler than it is. There's usually (almost always) durable storage required somewhere. But to respond to your example specifically, OTP offers quite a bit. To your points: Queues: a coordinator genserver process per node, which uses poolboy to only run X jobs at a time Tracking failures/sucesses: postgres Exponential backoff: a function Uniqueness across the cluster: stdlib global locking (https://erlang.org/doc/man/global.html#trans-2 https://erlang.org/doc/man/global.html#trans-2) draining: remote console periodic tasks: a simple library that knows how to cron I built a sharded cron-like scheduler in a few hundred lines of code. It's not doing a billion transactions per second, but it's been running in production for a few years without much drama. It's not complicated because it's just postgres and some standard erlang/elixir stuff. Erlang has plenty of drawbacks, but this isn't it.

toast0 6y ago
> There's usually (almost always) durable storage required somewhere. Despite the name, mnesia can provide durable storage, without using anything outside of OTP and your nodes' (hopefully durable) filesystems.
- riquito 6y ago
  Despite? It pretty much means "to remember", it doesn't have the alpha privative of amnesia, "to forget"
  - patrec 6y ago
    IIRC original name was Amnesia, but Joe was made to change it for marketing reasons (it's in one of his books or blog posts). Anyway, I do not think Mnesia is a particularly good way to persist stuff, just use postgres.
- simscitizen 6y ago
  At WhatsApp we had a bunch of mnesia-backed services. I can't say that it was the best experience. Pretty much anything non-trivial, such as healing a cluster after a partition, or expanding a cluster, or cross-DC replication was a complete PITA to do properly. There's also very few people that have operational experience with it and very little documentation about it compared to pretty much any other storage engine. The main benefit was the ability to load and store Erlang terms natively, along with the fact that it's just a library. The fact that your compute is colocated with your storage lends itself to a different architecture than when you use, say, a stateless web service in front of a relational database. (Edit: I realize now that I'm replying to a WA old-timer so I'd be curious to know what your assessment of mnesia is with the benefit of hindsight.)
  - toast0 6y ago
    I think we would have found any database to be a PITA at write volumes and data volumes we were using, with our expectations of availability. I don't think we would have been able to get that performance out of MySQL, or Postgres; and anyway, we didn't have a dedicated team for database management to make sure nobody wrote bad queries if we did. Sync, status and reg ran on MySQL and php when I started, and it was a lot easier to run them and scale them on Erlang, IMHO. Repartitioning MySQL like we repartitioned mnesia would be a lot harder, I think. I'm not too worried about a lack of experts and documentation. If you operate at the limits, you'll just need to become an expert, and the source code will be your documentation; mnesia is all written in Erlang, and not too hard to follow; most people won't hit the limits of ets, I'd think. If you don't operate at the limits, you might not need an expert, and if you keep up with OTP releases and hardware releases, the limits get bigger every year. Looking back, I think we should have spent more time earlier on using the hooks mnesia provides to make healing clusters take less effort; and maybe some automation on cluster expansion. Of course, WA never shied away from manual work by server engineers. :) (Hmmm, now I'm trying to figure out you are, send me an email if you will :)

laser 6y ago

Phoenix ships with Ecto ORM-style-binding to Postgres underneath (usually) for a good reason! :) And if you need that in memory magic where the serial nature of a genserver won't suffice for your load, you basically have built-in Redis with Erlang's ets. And if you really need insane performance on longer-cached items than ets is optimal for, Discord released https://github.com/discord/fastglobal https://github.com/discord/fastglobal And if you don't want to put all those pieces together, like you said there's a rich set of modules people have built, like oban. Long rise and live the elixir monoliths ;) (although admittedly even Discord had to swap out some of their most intense functions with Rust, but how many people are writing systems that need to handle millions of concurrents?)

nickjj 6y ago
> you basically have built-in Redis with Erlang's ets The big difference there is if you have a Dockerized app, Redis won't be restart on every deploy but the BEAM will be so you will lose your ETS cache or whatever you're storing. Now, I know cache is meant to be disposable but this model of using ETS instead of Redis reminds me of a Python or Ruby app storing its cache in memory. I mean, sure you can do it but for the longest time (7-8 years?) the community as a whole kind of landed on keeping that stuff out of your app's process.