12 ms·

Ask HN: Why Isn't Elixir More Popular?

87 points by wkyleg 2y ago

It seems that when implemented correctly, running full-stack Elixir can greatly simplify complicated distributed systems. In-memory storage can replace a cache, and the architecture itself can replace a message queue. Horizontal or even vertical scaling can replace complicated scaling mechanisms. There are plenty of advantages to running data pipelines in parallel too. There's also a huge trend now away from managed cloud services (ZIRP), and it seems like Elixir with a database on the same machine or network could deliver great performance and scalability. So why don't more people use Elixir? I've read some complaints about the lack of static types or not having as many developers. But if those were the only constraints, one would think that they could be overcome, or some service could just greatly simplify Elixir deployments.

johnklos 2y ago
Got me. Running Akkoma on Elixir is a piece of cake, relatively speaking. Trying to run Mastodon on Ruby by simply using the latest version of the required Ruby components hasn't worked for me, ever. The idea that we'd need to keep a snapshot of modules from a specific date in order to run certain software is ridiculous, and the idea of running it in a container like Docker is just giving in to bad practices. Too often it means things are too fragile to update when there are security issues. I haven't seen this happen with Elixir. What do they do better / differently from Ruby that updating doesn't cause the house of cards to come crashing down?
- wkyleg 2y ago
  Yes! That's a good point Docker is another technology that is sort of ridiculous from this perspective. Deploying an entire OS just to make dealing with dependencies easier
  - abhisgup 2y ago
    You are not exactly deploying an entire OS as the overhead of that would be huge. You are deploying an isolated process group having the bells and whistles of the OS you like.
sergiotapia 2y ago
It's a wonderful language. Unfortunately the only places I've been able to use it and get paid were places where I ultimately had the decision making ability of the tech stack. Why isn't it used? It's niche and betting on such a small community is risky for the majority of companies. Why use Elixir when you could hire 10 engineers to pump out javascript. That's the mentality of most. Hiring for elixir was great, it self selected people who wrote code as their craft. You kind of have to be curious about code to even be aware of Elixir, know how to write elixir. These types of devs would pick it up really quickly because the language is just so damn ergonomic. I'm using Elixir now, and I wake up so happy that I get paid to do this. I am really blessed.
- wkyleg 2y ago
  It's funny I think I remember explicitly reading this argument somewhere, that startups need to meet headcount as a metric for VC funding so they will end up picking "worse" tech for that reason.
  - Terr_ 2y ago
    I don't follow, wouldn't it be faster/cheaper to meet an arbitrary headcount goal using a more-common language or stack?
    - julienmarie 2y ago
      I think @wkyleg meant that an Elixir dev can be as productive as multiple js devs because of the specificity of the language and the beam virtual machine. Complicated things are trivial in Elixir, dependency management is a breather, and all in all, a really simple and productive environment as the mental model is pretty simple once it clicks.
- sgarland 2y ago
  > it self selected people who wrote code as their craft I’ve often thought that would be true for niche languages. I interviewed at a fintech whose backend is written in Haskell, and definitely got that vibe from the interviewers.
yen223 2y ago
The simple reason is that Elixir is competing in the backend web-services space, which is a very crowded space with many equally-compelling alternatives. As someone who likes working with off-meta languages, it's clear that the default state for any new language is to die an early death. The fact that Elixir hasn't died is testament to its strengths.
madamelic 2y ago
Since this thread is going to be bait to more experienced Elixir devs, can someone tell me books, blogs, etc that you like that discuss performance in Elixir? I am working on an Elixir project that in no circumstance should be taxing but it's falling over miserably at like 100 - 200 events per minute. The detail is it is distributed-ish IoT. I didn't write it, the person(s) who did write it are gone, and no one else writes Elixir. I've gotten some good gains already but I'd like to squeeze the 80 - 90% of juice I think I can get before resorting to beefier hardware. I've gotten into instrumenting and measuring it and I have some ideas but I'd love to hear others point me to other ideas. The real problem is that the hardware is miserably underpowered and it is real-time, by that I mean I can't defer, schedule for later, or de-prioritize anything. --- To actually contribute, I really like Elixir. I am not yet sure why I would advocate for it over something more 'simple' like nodejs (My background is, accidentally, Javascript World) but it's certainly a very nice language to write in. It feels magical but not too magical where you get scared it's trapping you into its web. Before anyone jumps too much on me for it, I gauge "simplicity" by how many people can I hire to write it. You can barely swing a cat without hitting 3 competent Javascript developers. I tried for many years to hire another golang dev so I could write it professionally, I only encountered a few despite having been in most interviews my employers would do. With that said, it may just be that the Venn diagram between "writes Javascript" and "writes golang" is small.
- throwaway81523 2y ago
  Disclaimer: I've written Erlang but not Elixir. From what I can tell the issues are similar though. I'd say look at Fred Hebert's Erlang blog (ferd.ca) and "Erlang in Anger" for some tuning hints. Erlang has been around a lot longer and its devs have gotten more into nitty gritty issues, it seems to me.
  - madamelic 2y ago
    Thanks! That is a helpful tip in general I should remember: search for Erlang, not Elixir ;)
- elcritch 2y ago
goralph 2y ago
In a commercial setting (i.e. not a side-project) the choice of programming language is also a business decision. The main factors being: - How large is the pool of available candidates for this language? A recruiting risk. - How mature is this language? A business continuity risk.
- viccis 2y ago
  >How large is the pool of available candidates for this language? A recruiting risk. I worked at a company with a massive Erlang codebase. Really nasty, not really following good OTP practices, etc. HUGE hiring problem and it took 6-12 months to spin up new devs to the point that they could really have ownership over things. And this is not a system that even remotely needed this. Elixir might seem better until you write enough of it to realize that you do basically have to learn Erlang or else you'll always be at a disadvantage when it comes to really understanding it.
  - josevalim 2y ago
    You don't have to learn the Erlang language (and this is commonly echoed by the community). At a certain scale, you have to learn the runtime, but this is the same for any technology. You have to learn JVM if you are doing Kotlin, there are plenty of discussions on V8 internals if you are running Node, etc.
  - victorbjorklund 2y ago
    You def dont need to learn Erlang to code in Elixir (other than maybe the very basics to interact with some Erlang libraries like ETS).
    - mtndew4brkfst 2y ago
      As José answered more gracefully, you don't need to learn the syntax but you do need to learn the semantics, the performance gotchas, and for most projects I'd say you'll probably need to consume Erlang documentation which is certainly a learned skill.
rsrsrs86 2y ago
I regret using it. Recruiting risk. Smaller ecosystem. I should have gone for Python.
- throwawaymaths 2y ago
  Hired a python dev at an elixir gig because I was forced to by the CTO. Picked by the CTO "because we need someone that can do python". Dev was absolute trash. Never hired a shit elixir dev. Python is a huge recruiting risk.
  - rsrsrs86 2y ago
    Don’t get me wrong. Phoenix is wonderful, Elixir kicks serious ass as a language. Jose valim is one of the kindest and most prolific guys I met on the net. I’d use elixir for a solo project any time.
- elixir_newbie 2y ago
  Will you consider hiring me (remote, non-US) if I pickup the basics in 15 days? I'm open to technical rounds that don't assume elixir familiarity.
hakunin 2y ago
Here’s my anecdote. I built 3 production projects in elixir around 2015-2018, and it was a blast to learn and work with. Those were interesting projects that were a great fit for the stack. One was March Madness bracket game which required a huge throughput on day one, and another was a football game audience play calling each play from their phone as the team is playing live at the stadium. This one needed a lot of timing coordination and poor connection handling. I even put out an open source fast/customizable leaderboard on top of ETS. However, I didn’t feel the need for this stack on most projects, and if I’m totally honest, I never got good at the novel way of building applications in it. While it was very enjoyable, everything felt a little awkward. Even 3 years in I constantly felt that I’m not doing it right, and I’m fighting the language to do data transformations in a purely functional style. I never got used to writing Ecto queries, and always had to look up their syntax. Plus, there didn’t seem to be a good architecture story, just isolated praise of OTP. And Phoenix further fueled the confusion, making it hard to understand whether I’m supposed to reason about my app like a Rails app (just build controllers, models, and views) with no regard for processes, or I should orchestrate some creative supervision trees, that I can’t even tell how they would be arranged in a typical web app. Going back to Ruby on Rails was kind of back to super productive comfort zone after a bit, and I just continued staying there. I’m still looking back at Elixir with nostalgia, wondering if I’m going to have a chance to go back to it and really make it an extension of my arms/brain like Ruby had become. And whether I can do all that amazing supervision-based architecting I keep theorizing in my mind. I loved Dave Thomas’s vision (and agreed with his controversial takes) and really wished I could get as good as Sasa Juric at really deeply reasoning in this framework. Maybe one day.
- steve_adams_86 2y ago
  I have to say this is a really good summary from my perspective too. Different exposure and projects, but the experience in terms of ergonomics and other developer experience was much the same. I never felt truly proficient. I recall years ago interviewing for a role and straight up telling the interviewer I wasn’t very good with Elixir. That’s just how it made me feel. In retrospect I was fine, I built some interesting stuff. I probably could have promoted myself for that. But the language kind of left me feeling like I wasn’t getting it. Even so, great fun to learn and apart from those aspects, I enjoyed building with it a lot.
ddyevf635372 2y ago
Because it is not strictly typed language.
- kylecazar 2y ago
  a lot of extremely popular languages aren't.
- KingMob 2y ago
  You probably meant "static" or "strong", not "strictly". Regardless, both are still not right, since Elixir is getting a gradually-typed set-theoretic type system. It's been in the works for a few years now.
maz1b 2y ago
From what i've read, it is the "upgrade" to rails for people who want higher scale. I don't know if I agree, as Rails definitely does scale if you know what you're doing. Would love to hear more about others experience using Elixir for fast, low-latency high throughput scale.
- DoesntMatter22 2y ago
  Rails is pretty fast today and as you said scales well. I don't think think the addressable elixir market was really ever that big. Even companies like Bleecher report migrated off
ipnon 2y ago
No manager is going to lose their job by hiring 10 Python and JavaScript engineers to accomplish what could be done with 1 Elixir engineer. And that’s how most time and effort is allocated in our tech labor market economy. It’s by working backwards from “I want to still have a job in tech next year.” The number of people who are calculating which language to write in by thinking “how can I write a fault-tolerant distributed system with less time and energy so that I can quickly release performant products” is minuscule. The lack of popularity of Elixir is evidence of this I think. In regards to your point on ZIRP: billions of dollars have been poured in to LLMs that are biased toward legacy languages like Python and JavaScript. Even the file structure of these languages is conducive to LLMs. A HTTP server can be and is frequently defined in a single file or function, where everything from socket creation to connection handling to request parsing to database queries to response can be composed in literally a few lines of code. This immense expressiveness is a testament to the power of HTTP. But as I’m sure we’re all aware there are limits to what can be accomplished with a single machine serving these stateless requests, and the limit can be reached very quickly. But LLMs gravitate towards producing these haiku-like incantations, it’s trivial for them. Elixir’s power comes from having a well-defined API and explicit failure modes for each of these layers, each in their own expressive modules. This makes it difficult for LLMs to write Phoenix code when they’ve been optimized to output a 3-line FastAPI decorator-definition-query endpoint. Each of these layers in Phoenix is by itself quite simple, each layer has about 2-4 required functions to be implemented, and Phoenix can generate all of these for you. But ChatGPT doesn’t seem to be able to grok it all at once the way a good engineer can after readings the docs for an afternoon. Will Elixir survive an era of programming where RTFM is a lost art? I suppose we will find out soon enough!
- abhisgup 2y ago
  So everything is a conspiracy?
  - moooo99 2y ago
    Whenever it suits our own personal beliefs. If not, then no.
- 2y ago
7bit 2y ago
I'm Not a Professional dev, but a sysadmin WHO Likes to code. I learned a bunch of languages over the year. C#, Java, Python, JS/TS, PowerShell, Rust. Not professionally, but enough to finish some Personal projects. Elixir was so vastly different, had such a small community that I lost interest very, very fast. It just felt like a waste of time. Any of the other languages I learned I could and did apply at my job. Even just knowing the languages helps a great deal when evaluating the next tech stack to implement. And Elixir just doesn't. It's a niche language for niche applications and doesn't personally help me in the least. I also just did not like the DX. I did fail to understand how to use the VS Code debugger to step through functions. Printing to console to inspect variables felt like being in the stone age again, no thank you. I still believe Phoenix LiveView is a really, really good tech and currently wish that Svelte could implement something like that. And that's my personal reason for not looking into it.
- h0l0cube 2y ago
  > I still believe Phoenix LiveView is a really, really good tech and currently wish that Svelte could implement something like that. Not what you meant, but there's this: https://github.com/woutdp/live_svelte https://github.com/woutdp/live_svelte > I did fail to understand how to use the VS Code debugger to step through functions. Did you have any luck with the ElixirLS extension? https://github.com/elixir-lsp/elixir-ls https://github.com/elixir-lsp/elixir-ls
- mau013 2y ago
  I started coding in Elixir about a year ago, and my experience with the community has been great. I was coming from Python and JS, where evidently the community size is orders of magnitude larger, but in my experience that is more than compensated by how engaging the Elixir community is. I’ve had very helpful interactions on the Discord channel and through the ElixirForum… also, you tend to get very good/thoughtful answers. So to me community size has not been an issue. Also, just out of curiosity, what do you mean by “niche”? As far as I’m aware the use case of Elixir is pretty general purpose (obviously with its strengths and weaknesses - like any other language). Anyways, to each his own ;)
ram_rar 2y ago
A limited job market, considerably smaller ecosystem (compared to java/python), and lack of support from major tech corporations (unlike Go or Rust) makes it challenging to adopt Elixir. At my company , we inherited a legacy Elixir service, but we're actively moving away from it. We found it more practical to rewrite the service in Go (entire team is proficient in + LLMs make it easier to migrate) rather than invest time and resources into learning Elixir and its associated ecosystem. The return on investment for mastering Elixir didn't justify the effort required, given our specific circumstances and needs. I do like to tinker with Elixir for hobby projects. But supporting a whole new language in a company is an uphill battle, that's incredibly hard to justify.
- innocentoldguy 2y ago
  I've had the opposite experience. I've worked at several companies that migrated from Go to Elixir because the return on investment DID justify the effort. For example, Elixir allows us to do hot code deployments, uses a tweezer approach to self-healing as opposed to Go's sledgehammer approach (when paired with K8s, otherwise Go can't self-heal at all), and Elixir's immutability makes concurrency less error prone than Go's imperative nature.
jatins 2y ago
I can't say about mass popularity but for me personally the lack of static types has held me back from using it. Now that it's being gradually introduced I am looking forward to trying it
pansa2 2y ago
Ultimately, Elixir is a functional programming language. Has any functional language ever really become popular?
- reducesuffering 2y ago
  The way people write Typescript these days is very functional. A huge amount of modern code are React apps written functionally
andrewstuart 2y ago
Straight up - you should not use it without much more compelling reasons than those you have outlined. It's not OK as CTO to push your company to use for favorite pet slightly obscure technology. The company needs ordinary technologies to get its job done - Java, C#, TypeScript, Golang, Python, something like that. Maybe even Ruby but frankly that causes hiring headaches too. Unless the company is doing something so extraordinary that it NEEDS Elixir over the technologies above, and it probably doesn't need it. I know companies that went all in on Elixir - they spend all their time trying to recruit Elixir developers from a tiny recruiting pool, or trying to find people willing to learn it and bring them up to speed. This is poor business practice - there was nothing they were doing that could not be done just as effectively with any of the other major languages. Just use normal technologies, have all the people and documentation and training and other resources available and get on with the job of building the software that the business needs. I guarantee you that the company that went with Elixir due to the enthusiasm of some current or past CTO will have to go through the painful process of getting rid of it. Elixir is for personal projects. Corporate software development is mature enough now that ordinary technologies are extraordinarily powerful and there's almost no reason sufficiently compelling to not use them. Toys are for home, tools are for work.
toldyouso2022 2y ago
Used elixir a few years ago for tutorials and such, worked fine. Tried to use it again in 2023, vscode extension would break. Tried on 2 windows machine and one linux machine, exetension always broke. I then figured out why (I don't remember the exact reason, it was one of those "yeah I can see why it didn't work and it's my fault but it kinda isn't really" situations) but at that point I was done because I had better stuff to do. Point being, if you want your stuff to be adopted make sure the tooling is noob friendly. The noobs of today can be the professionals of tomorrow. Especially today when there is so much stuff to learn and elixir is so far from the usual and requires a certain time investment
- gregors 2y ago
  You might have missed this recent announcement https://elixir-lang.org/blog/2024/08/15/welcome-elixir-language-server-team/ https://elixir-lang.org/blog/2024/08/15/welcome-elixir-langu...
  - toldyouso2022 2y ago
    Good news, I really want elixir to become popular and improve the dev experience
binary132 2y ago
Most people just want to get something done and reach for the tools that will most straightforwardly and simply get them to their destination. Niche stuff doesn’t have the market fit because you’ll always have a hard time finding people to work on your product that don’t cost twice as much as everyone else, unless it’s 10x faster gtm / 10x cheaper. Seems obvious enough. Personally? I like good typesystems and very sharp tooling. It has seemed to me that Elixir has neither, and doesn’t really offer me a particularly big advantage that would outweigh those disadvantages.
- innocentoldguy 2y ago
  In my experience, Elixir is one of the easiest languages to pick up and get stuff done quickly. It has excellent tooling and deployments ARE 10X cheaper (if not more). I've never converted a project to Elixir and not had the associated costs decrease dramatically. Also, I don't find typing to be an issue in Elixir due to the way the language works (pattern-matching, guard clauses, immutable data, etc.). Maybe I just have a talent for remembering types, though, because strong typing has always been a huge meh for me.
  - mtndew4brkfst 2y ago
    I've never converted a project to Elixir and not had the associated costs decrease dramatically. It's useful to remind that any full rewrite without notable changes in feature scope will probably wind up performing better than the original organically developed solution, even if you never change tech stacks. That's just the nature of going in with a total picture of the problem space and knowledge of the warts already encountered.
pjmlp 2y ago
From my point of view, it builds on BEAM, and while Erlang might have been a success story in telecommunications, there is no reason for any of my employers to use it instead of JVM or CLR, and the languages available to them.
hauxir 2y ago
We're using it with great success at kosmi.io
gregors 2y ago
As an in-the-code CTO that runs small teams, I've used Elixir/Phoenix quite successfully. Hiring has never been an issue, in fact - Elixir was a tool to recruit a specific senior developer we wanted but couldn't pay them what they wanted due to budget. Elixir interest got them to come on board. I've trained a number JR developers with no prior experience with great success. I was all in on Rails for over a decade, Elixir/Phoenix has definitely replaced that stack for me. I think it hits a sweet spot in terms of runtime performance and developer productivity. The latter is not something you can afford to ignore on small teams.
- innocentoldguy 2y ago
  Same! I've had a lot of success training Python, Ruby, and JavaScript developers in Elixir and Phoenix. I've never had the "You can't find devs who know Elixir" problem that I occasionally see.
deleted 2y ago
[deleted]
kugelblitz 2y ago
I use Symfony (PHP based framework) and it works fine. I've been able to get into Django (Python), Laravel (PHP), Java (Spring) and even Grails (Groovy) because they either had similar concepts or even similar syntax (I mostly do web development, so this is a very biased take). Being a freelancer, I need to focus on what's marketable. Sure, Elixir will get me into a niche, but I will have way less projects to choose from. And when I start a project for a company, if I start with Elixir, I will also have a smaller pool of devs to recruit from. It's a chicken-and-egg problem. Nowadays, if I start a project, I would try to build on monolith and full framework with a PaaS. Unfortunately, most projects want to start out "the right way", which means separate backend (e.g. Java), separate frontend (React), rented server (e.g. Hetzner server) and custom deployment (some pipeline an outside agency built when they first started the project). I'd rather spend 400 USD on tools each month, but then only need 1-2 full stack devs instead of 6-8 people (1 sys admin, 1-2 deployment, 2 backend, 2 frontend) and with all the overhead that comes with it.
- wkyleg 2y ago
  Do you really find Java backends to be that great? I don't think they're intrinsically bad, and Kotlin or Groovy can be nice, but I've always found the community and lack of open source tools to be wanting. I've found myself to be much more productive in JavaScript/TypeScript or Python, in part because of the languages, but also because the open source libraries have been way better
kdemetter 2y ago
I love Elixir, and I've been learning it because I see the potential. But, it is a steep learning curve, and it's quite a different paradigm than most developers are used to. Which means it would be very difficult to get it adopted in say a company, because it would require lots of money to retrain people and it's harder to find people for it. I think it might also have a marketing problem, in that it uses the Erlang VM which is pretty old. That's actually evidence of it's strength, but it's less sexy than doing something completely new
kdemetter 2y ago
I love Elixir, and I've been learning it because I see the potential. But, it is a steep learning curve, and it's quite a different paradigm than most developers are used to. Which means it would be very difficult to get it adopted in say a company, because it would require lots of money to retrain people and it's harder to find people for it. I think it might also have a marketing problem, in that it uses the Erlang VM which is pretty old. That's actually evidence of it's strength, but it's less sexy than doing something completely new. However, I think it's a great language to just try out yourself. Even if you won't be able to use it professionally, you'll still learning something you can use in your job as well.
aristofun 2y ago
One of the reasons is that FP languages just don’t feel natural to most beginners. It gives the language a very niche vibes. And rightfully so, I suppose. But the key reason is that no giant has sponsored and promoted it. Products are virtually never popular because of their internal qualities.
shortrounddev2 2y ago
I think functional programming languages are at an inherent disadvantage

madamelic 2y ago
> Are you doing a lot of non-io processing or computations? Unfortunately. From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache. The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware. One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion. > One possibility is you're using a single process instead of parallelizing things. For example, you may want to use one process per event, etc. This is done by pushing it to a task ie: `Task.Supervisor.async_nolink`? That's largely where I found my gains actually. It took a dive into how things schedule, because a big issue that was happening was the queue would get massively backed up, and I realized that I needed to apparently toggle on a flag telling it to pack the scheduler more (`+scl true`). I also looked into the wake-up lengths of threads. I am starting to get my head around "dirty schedulers" but I am not entirely sure how to affect those or how I can besides it doing it forever me. The other one just for posterity is that I believe events get unnecessarily queued because they don't / didn't have locks. So if event A gets queued then creates a timer to re-queue it in 5 minutes, event A (c|w)ould continue to get queued despite the fact the first event A hadn't been processed yet. So the queue would just continue to compound and starve itself.
- elcritch 2y ago
  > From metrics, computing AWS signatures takes up an absurdly large amount of CPU time. The actual processing of events is quite minimal and honestly well-architected, a lot of stuff is loaded into memory rather than read from disk. There's syncing that happens fairly frequently from the internet which refreshes the cache. Oh, sounds nice! Caching in Elixir really is nice. Okay, that makes sense. Elixir isn't fast at pure compute. It can actually be slower than Python or Ruby. However, the signatures likely are NIFs (native code). If the AWS signs are computed using NIFs then the CPUs are likely just can't keep up with them. Tokens would make sense in that scenario. But you should check the lib or code you're using for them. > The big problem is each event computes a new signature to send back to the API. I do have to wonder if the AWS signature is 99% of the problem and once I take that burden off, the entire system will roar to life. That's what makes me so confused because I had heard Erlang / Elixir could do on the scale of significantly more per minute even with pretty puny hardware. Yeah, crypto compute can be expensive especially on older / smaller cpus without builtin primitives. Usually I find Elixir performs better than equivalent NodeJS, Python, etc due to it's built in parallelism. Also one thing to lookout for would be NIF C functions blocking the BEAM VM. The VM can now do "dirty nifs", but if they're not used and the code assumes the AWS signs will run fast, it could create knock on effects by blocking the Beam VM's schedulers. That's also not always easy to find with Beams built in tools. On that note, make sure you've tried the `:observe` tooling. It's fantastic. > One thing I am working on is batching then I am considering dropping the AWS signatures in favor of short-lived tokens since either way, it's game over if someone gets onto the system anyway since they could exploit the privilege. The systems are air-gapped anyway so the risk is minimal in my opinion. Definitely, seems logical to me.
  - madamelic 2y ago
    Thank you! You gave me a great term that I can jump off from (NIF). I'll have to rig up Observer. I've been using recon because I was being lazy overall.
    - gregors 2y ago
      Once upon a time I needed to do hashes en masse for a specific blockchain projects. Just a tad of Rust (via nif) really helped the performance. Might be of help to you, check this out (not my lib) https://github.com/ExWeb3/ex_keccak https://github.com/ExWeb3/ex_keccak for my usage, benchmarks went from 2.79K ips to 346.97K ips
    - elcritch 2y ago
      You might also look into Elixir's broadway as provides back-pressure [1]. 1: https://hexdocs.pm/broadway/Broadway.html https://hexdocs.pm/broadway/Broadway.html > I also looked into the wake-up lengths of threads. I am starting to get my head around "dirty schedulers" but I am not entirely sure how to affect those or how I can besides it doing it forever me. Note that dirty schedulers really only affect NIFs which run longer than what the BEAM schedulers expect. I mentioned that in regards the possibility that the AWS sigs are taking longer than they should, then they'd cause havoc on the scheduler.
- innocentoldguy 2y ago
  It looks like others have address the first 90% of your post, so I'll refrain from commenting on that. I am curious about your timer code, though, because the timer shouldn't be firing at all unless the task associated with it has completed successfully. You shouldn't run into an issue where a timer is re-queueing the same task in Elixir.
- asa400 2y ago
  I don't know the specifics of your app so I don't feel commenting in more than generalities, but generally speaking, if you are doing work in native code, and if that native code work is CPU-bound (roughly, more than a millisecond of CPU time) you should try to do it in a dirty scheduler. If you don't, what will happen is that that CPU-bound code will interfere with the "regular" BEAM schedulers, meaning it will start to interfere with how the BEAM schedules all of the other work in your app, from regular function calls to IO to job queuing to serving requests, and whatever else. I'm also suspicious of the `+scl true` setting as maybe being a bit of a red herring. I've been using BEAM off and on for 10 years both professionally and as a hobbyist and I've never used this myself nor seen anyone else ever need to use it. I'm sure there are circumstances where someone, somewhere has used this, but in a given Elixir app it is extremely likely that there is lower-hanging fruit than messing with scheduler flags. In terms of queuing, are you using Oban or Broadway or something hand-built? It's common for folks to DIY this kind DAG/queuing stuff when 99.9% of the time using something like Oban or Broadway would be better than hand-rolling it.

DoesntMatter22 2y ago

This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix. I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework. Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose. And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work? I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong". I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV Proof below https://www.techempower.com/benchmarks/#section=data-r22&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r22&hw=...

josevalim 2y ago
> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2]. My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/application.rb#L33-L53 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... But almost nobody runs a Rails application like that in production. And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests. I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 [2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.
- DoesntMatter22 2y ago
  Sounds like you are agreeing with me. You tried to work on it years ago and didn't fix it. Even when it was completing last year it's performance was half of what Rails was, which isn't known to be fast [1]. The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2] Even if, somehow, Elixir is actually faster, which seemingly lacks evidence at this point other than some anecdotes by people who have a vested interest in people adopting the language (primarily Elixir dev shops and framework/language authors), the fact that the community of Elixir devs cannot get decent results on these tests says a lot. If these people can't get good results in tuning Elixir, who can? Not a surprise to see people ITT also noting that they ran into the same types of problems that the tests reveal. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/database.yml https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2]: https://www.techempower.com/benchmarks/#section=data-r21&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r21&hw=...
  - josevalim 2y ago
    > The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2] Rails concurrency model is per process. So you will have `n_workers` pools and the amount of connections will be `n_workers * 3` based on the code you shared. The production machine has 28 total threads [1], which means Rails is starting 35 workers, according to their auto tuning code [2]. Overall, Rails uses 35x more pools than the Phoenix application and has twice the number of connections (105). It is absurd to imply that Rails is actually running with far less. Second of all, the database pool needs to be 512 or 1024 if your app server can push traffic quickly enough to the database. If you are increasing the Rails pool size and it is not getting faster, then Rails is your bottleneck. Based on the runs I did last week, it seems the pool is the bottleneck in Elixir's case. We will know for sure in a couple weeks once the PRs land. > If these people can't get good results in tuning Elixir, who can? You can look at the results from the latest community efforts in main: https://www.techempower.com/benchmarks/#section=test&runid=176ba510-3607-4faa-996e-74f0778b88d4&hw=ph&test=query https://www.techempower.com/benchmarks/#section=test&runid=1... If you want to compare with Rails in particular, we beat Rails in every benchmark except two: 1. Single query: which will be tackled by better sizing our pools 2. Plain text: where we are currently sending the wrong payload (but when we sent the proper payload, as in the last published round, we beat Rails too) And it does not include any of my proposed changes yet. My point is exactly that the feedback cycle is long, so it takes time, and I was not able to invest time before. But given how intent some folks are to misinterpret benchmarks (and assign malice to the whole community), it is probably better to continue looking into it. [1]: https://ark.intel.com/content/www/us/en/ark/products/120474/intel-xeon-gold-5120-processor-19-25m-cache-2-20-ghz.html https://ark.intel.com/content/www/us/en/ark/products/120474/... [2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/auto_tune.rb https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

ipnon 2y ago
I have this same feeling using Elixir, and I think it’s because it’s excellent at showing you exactly where you have undefined behavior in your application, much in the same way you spend most of your time writing Rust by wrangling with the compiler that keeps showing you all the holes in your logic. You can write a seemingly equivalent program in Python, and it runs just fine because it’s interpreted, so it seems like you hit the bullseye first try. But as we’re all aware, there are still countless bugs in your system, they just don’t surface until later when it’s Friday night and your website finally went viral. It’s the downside of the “fail fast, fail often” OTP paradigm: you develop the program by working through your faulty logic step by step as determined by the (many) errors raised by the BEAM. The upside being that unlike Python none of these errors are usually strong enough to be fatal to your application.
- steve_adams_86 2y ago
  This is a huge part of it, for sure. I think this is why people often dislike state machines too, for example. They force you to really think through what you're trying to do, and there are no shortcuts. The irony is that the state machine pattern itself kind of is a shortcut to stable, reliable, safe code. People find them overly complex and full of ceremony, but it's like... Do you really want to try getting the same assurances from your code without this kind of abstraction? I'd rather know my I can't shoot myself in the foot, because I guarantee you that I will. Elixir also has the added layer of unfamiliarity for a lot of people (myself included) where you might intuitively know how to do something imperatively, but you need to do it within the FP paradigm. I like FP and have adopted a lot of its patterns all over the place, but I still find I'm not a great FP thinker in general. After something like 10 years. My FP-fu is limited to relatively basic operations, and when I go into advanced territory I'm constantly leaning on reference material. That's more so a skill issue than an Elixir issue.
  - throwawaymaths 2y ago
    For the most part you really only need to know Enum.map, Enum.reduce and if you really hate recursion, Enum.reduce_while And the nice thing is that those are highly imperative/declarative functionals. Edit: ok, also "returning stuff out of if and case statements". That one kept triggering a python developer I had, had to remind him something like 20 times. Elixir would be way better if it had 'no shadowing variables' rules.
- michalbcz 2y ago
  > The upside being that unlike Python none of these errors are usually strong enough to be fatal to your application. That's a great point. Each language has its domain where it excels. The reality is that 90% (or more) of all apps don't deal with the traffic scale of YouTube, the complexity of an operating system, or the mission-critical constraints of software for the F-35. For most businesses, Python is more than sufficient in 2024. At the same time, for learners, it's much easier to pick up. It's a sweet spot.