5 ms·

Since this thread is going to be bait to more experienced Elixir devs, can someone tell me books, blogs, etc that you like that discuss performance in Elixir?

Since this thread is going to be bait to more experienced Elixir devs, can someone tell me books, blogs, etc that you like that discuss performance in Elixir? I am working on an Elixir project that in no circumstance should be taxing but it's falling over miserably at like 100 - 200 events per minute. The detail is it is distributed-ish IoT. I didn't write it, the person(s) who did write it are gone, and no one else writes Elixir. I've gotten some good gains already but I'd like to squeeze the 80 - 90% of juice I think I can get before resorting to beefier hardware. I've gotten into instrumenting and measuring it and I have some ideas but I'd love to hear others point me to other ideas. The real problem is that the hardware is miserably underpowered and it is real-time, by that I mean I can't defer, schedule for later, or de-prioritize anything. --- To actually contribute, I really like Elixir. I am not yet sure why I would advocate for it over something more 'simple' like nodejs (My background is, accidentally, Javascript World) but it's certainly a very nice language to write in. It feels magical but not too magical where you get scared it's trapping you into its web. Before anyone jumps too much on me for it, I gauge "simplicity" by how many people can I hire to write it. You can barely swing a cat without hitting 3 competent Javascript developers. I tried for many years to hire another golang dev so I could write it professionally, I only encountered a few despite having been in most interviews my employers would do. With that said, it may just be that the Venn diagram between "writes Javascript" and "writes golang" is small.

throwaway81523 2y ago
Disclaimer: I've written Erlang but not Elixir. From what I can tell the issues are similar though. I'd say look at Fred Hebert's Erlang blog (ferd.ca) and "Erlang in Anger" for some tuning hints. Erlang has been around a lot longer and its devs have gotten more into nitty gritty issues, it seems to me.
- madamelic 2y ago
  Thanks! That is a helpful tip in general I should remember: search for Erlang, not Elixir ;)
elcritch 2y ago
Interesting, Elixir should scale far more than that. Are you doing a lot of non-io processing or computations? I run Elixir on raspberry pi 4s doing IoT and they easily handle say generatings graphs with hundred of thousands data points. One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. Though if the hardware is very underpowered and say single core, I could see it becoming problematic.
- madamelic 2y ago
  > Are you doing a lot of non-io processing or computations? Unfortunately. From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache. The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware. One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion. > One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. This is done by pushing it to a task ie: `Task.Supervisor.async_nolink`? That's largely where I found my gains actually. It took a dive into how things schedule, because a big issue that was happening was the queue would get massively backed up, and I realized that I needed to apparently toggle on a flag telling it to pack the scheduler more (`+scl true`). I also looked into the wake-up lengths of threads. I am starting to get my head around "dirty schedulers" but I am not entirely sure how to affect those or how I can besides it doing it forever me. The other one just for posterity is that I believe events get unnecessarily queued because they don't / didn't have locks. So if event A gets queued then creates a timer to re-queue it in 5 minutes, event A (c|w)ould continue to get queued despite the fact the first event A hadn't been processed yet. So the queue would just continue to compound and starve itself.
DoesntMatter22 2y ago
This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix. I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework. Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose. And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work? I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong". I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV Proof below https://www.techempower.com/benchmarks/#section=data-r22&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r22&hw=...
- josevalim 2y ago
  > And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2]. My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/application.rb#L33-L53 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... But almost nobody runs a Rails application like that in production. And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests. I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 [2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.
imjonse 2y ago
Probably unrelated to Elixir itself but caused by doing too much/redundant work on network/disk so it needs an algo change.

throwaway81523 2y ago

Disclaimer: I've written Erlang but not Elixir. From what I can tell the issues are similar though. I'd say look at Fred Hebert's Erlang blog (ferd.ca) and "Erlang in Anger" for some tuning hints. Erlang has been around a lot longer and its devs have gotten more into nitty gritty issues, it seems to me.

madamelic 2y ago
Thanks! That is a helpful tip in general I should remember: search for Erlang, not Elixir ;)

elcritch 2y ago

Interesting, Elixir should scale far more than that. Are you doing a lot of non-io processing or computations? I run Elixir on raspberry pi 4s doing IoT and they easily handle say generatings graphs with hundred of thousands data points. One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. Though if the hardware is very underpowered and say single core, I could see it becoming problematic.

madamelic 2y ago
> Are you doing a lot of non-io processing or computations? Unfortunately. From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache. The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware. One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion. > One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. This is done by pushing it to a task ie: `Task.Supervisor.async_nolink`? That's largely where I found my gains actually. It took a dive into how things schedule, because a big issue that was happening was the queue would get massively backed up, and I realized that I needed to apparently toggle on a flag telling it to pack the scheduler more (`+scl true`). I also looked into the wake-up lengths of threads. I am starting to get my head around "dirty schedulers" but I am not entirely sure how to affect those or how I can besides it doing it forever me. The other one just for posterity is that I believe events get unnecessarily queued because they don't / didn't have locks. So if event A gets queued then creates a timer to re-queue it in 5 minutes, event A (c|w)ould continue to get queued despite the fact the first event A hadn't been processed yet. So the queue would just continue to compound and starve itself.

DoesntMatter22 2y ago

This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix. I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework. Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose. And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work? I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong". I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV Proof below https://www.techempower.com/benchmarks/#section=data-r22&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r22&hw=...

josevalim 2y ago
> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2]. My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/application.rb#L33-L53 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... But almost nobody runs a Rails application like that in production. And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests. I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 [2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.

imjonse 2y ago

Probably unrelated to Elixir itself but caused by doing too much/redundant work on network/disk so it needs an algo change.