3 ms·
I should have been clearer in my final note. My Elixir vs Python example was for standalone batch (script) processing. It was not in the context of a webapp.
by blunte 6y ago
I should have been clearer in my final note. My Elixir vs Python example was for standalone batch (script) processing. It was not in the context of a webapp.
The case was a recurring calculation system which would take a few hundred thousand records and do about a dozen different calculations against those (some calculations requiring additional lookups).
The original script was in Python, and then after the new webapp was built in Elixir, I rewrote it also in Elixir. I discovered that even tuning the batch sizes, the Elixir/Ecto version was much slower than the Python version. Then I looking into per-thread Elixir performance and found out that that is not its strong suit :). So the answer there of course is to leverage the strength of the tools and parallelize it; but I had no need to do that since I already had a working Python version. Could Elixir have been faster if multi-threaded? Probably. Single? No, I doubt it.
- dnautics 6y agoHm, if you have a chance to revisit it, you may want to consider Flow, which will likely take a lot of the pain of parallelization away.
- ch4s3 6y agoAhh, ok. You've sort of hit a weird spot for Elixir. Python will just start up faster and that will be noticeable. Math is much better optimized in Python, though this is changing with NX. The real gain here would probably be streaming chunks of rows out of ecto and maybe firing those off to tasks per CPU core. Which is easier than it sounds. That said, you had working code so no real reason to rewrite it except as a learning exercise.
- toast0 6y agoTypical ways to start beam end up doing a lot of work you might not need. Taking some time to fiddle with the startup settings for a short running task can make a big difference. You might actually get better results by setting it to single cpu, because there's less setup that way. Really depends on how much total cpu time your work takes. If your VM is going to run for a year (or more), startup time hardly matters and so the defaults aren't highly tuned for a 30 second script.