5 ms·

>It’s unfortunate it isn’t more popular what might explain this? I mean hypes and infatuations are common but tangible advantages usually win the day because i

>It’s unfortunate it isn’t more popular what might explain this? I mean hypes and infatuations are common but tangible advantages usually win the day because in the end people will pick the right tool for the right job. is it that distributed systems are not actually as big a domain so it does not get much press?

Jtsummers 3y ago
> tangible advantages usually win the day because in the end people will pick the right tool for the right job No, they don't. They pick the familiar and the easy-to-hire-for. I've worked on systems (written in C++, Java, others) that would have been perfect for Erlang, but at most I was able to present a prototype. No one is interested in a better solution that they can't hire for or have to retrain their staff for. (The familiar and the easy-to-hire-for may, by some metric, be the right tool for the job, but it isn't necessarily the right tool by technical metrics.)
- devoutsalsa 3y ago
  I am pretty strong with Elixir, and I've done a little bit of Erlang. Anyone who understand the basics of Erlang features that get used in Elixir should be able to program in Erlang with a bit of a ramp up period. Not too long ago I interviewed with a company that only uses Erlang, not Elixir, and they rejected me because I am not an Erlang expert. There wasn't anything I could say that would change their minds. I even volunteered to do a coding challenge of some type & they still wouldn't consider me further. Sometimes a talent shortage is a state of mind.
  - ollien 3y ago
    That's what I did! I learned Elixir in my spare time and then started a job that used Erlang. There's some quirks that I had to get over, but it wasn't a huge change over.
    - devoutsalsa 3y ago
      Shoot me an email if you'd like to connect!
  - davidw 3y ago
    > Erlang, not Elixir, and they rejected me because I am not an Erlang expert Wow... was this an actual hiring manager or someone, or like an HR person with a checklist? I've interviewed for one or two Elixir positions where the opposite thing happened: I talked with someone who had a checklist. I haven't done Elixir professionally even though I have extensive Erlang experience and that was that. There is a pretty stark difference between "has architected an application with OTP components" and "can write linear code in Erlang or Elixir", but even then you'd probably be able to fit someone in unless you're hiring for that kind of senior/architect position.
    - devoutsalsa 3y ago
      It was a team decision. I interviewed with the hiring manager and three developers. It sounded like I was the first Elixir person they'd spoken to, and just decided it was outside of their comfort zone.
  - hosh 3y ago
    When I worked at an Erlang shop, the director’s hiring strategy was to hire generalists willing to learn Erlang. He also hired remote, years before the lockdowns. He built distributed teams that builds distributed systems.
    - devoutsalsa 3y ago
      Sounds like a nice gig.
- Verdex 3y ago
  > > [...] people will pick the right tool for the right job > No, they don't. I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical). I assert the following: Scenario A: I have a literal mathematical proof that erlang is the best choice. I implement the solution, but due to the market changing (or an earthquake or ninjas literally destroy the server room) the overall project fails. * Me: But, I had math proving everything out. Ultimate Boss: I don't believe in math, you're fired. * Boss: Well, it failed, but look at this incomprehensible math that I don't understand. Ultimate Boss: You've obviously been hoodwinked, you're fired. Scenario B: I have a really good story that includes phrases like: "best practices" and "industry standard", and is doing exactly the same thing as everyone else. The project fails because the tool isn't great. * Me: It failed, but here's a good story. Ultimate Boss: Well, it looks exactly like everything else, so I'll look for a different nail that's standing out from the crowd to blame for this so that the muggles don't get nervous about taking non-chances in the future. * Boss: Here's this great story that I totally understand and can embellish upon because it operates on storytelling and not category theory. Ultimate Boss: That's a really good story. I'll have to steal it myself to give to the board to handle some unrelated problems of my own. Better luck next time. Even the scenario of master craftsman using erlang because it's the right tool for the job is more about a story about an uber expert who is off doing wizard stuff than it is about an actual merited set of reasoning.
  - Jtsummers 3y ago
    > I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical). The "nobody ever got fired for buying IBM" defense. Which has since been updated to Oracle, Microsoft, Java, C# (for a large portion of the enterprise world), Python, and Javascript. Best-in-class that isn't mainstream is a hard sell to employers if they are strongly risk averse (which many people, if not most, are).
    - 3y ago
- deleted 3y ago
  [deleted]
- ralphc 3y ago
  Sounds like they're the prototype for a new company. WhatsApp ran rings around competitors with a small team using Erlang; can you do a better job than the company that rejected it?
  - Jtsummers 3y ago
    Too low a margin and too few customers to make it worth it for me to try and jump into those niches. Honestly, many of the benefits of using Erlang for those systems would have been similar to any other more service-oriented or decomposition of monoliths. They had complex systems doing too much, but that would have benefited from the really good binary processing portion of Erlang and it's relatively easy horizontal scaling story. But when the number of customer (organizations) can be counted on one hand, and the number of users is less than 1000, and there's already competition you know it's going to be hard to break in. They'd already hit rock bottom pricing on their contracts.
bcrosby95 3y ago
Have you seen Erlang syntax? That's why. Even some of the best programmers I know are very superficial.
- zelphirkalt 3y ago
  I don't actually find the syntax to be so bad. It can even be pretty neat with the pattern matching.
  - 0x457 3y ago
    I don't think it's bad, but it looks very alien. It's also very simple compared to say JavaScript or really anything else. I understand why Elixir was born.
- aabbcc1241 3y ago
  Maybe a transpiler and type checker that convert typescript into erlang can help to drive the adoption
  - masklinn 3y ago
    That’s Elixir, pretty much.
  - cjallen88 3y ago
    Purescript has an erlang backend FYI
    - aabbcc1241 3y ago
      That's interesting! I think Typescript would be "easier" for a larger group of developers to hook on
- datadeft 3y ago
  Have you seen the semantical aspects of that syntax?
deleted 3y ago
[deleted]
toast0 3y ago
Distributed systems are everywhere in big tech. So that's not really the issue. A lot of it is unfamiliarity with the tool and unfamiliarity with the paradigm. And of course, you can do distributed systems with other tools that people tend to be more familiar with, so go with what you know. Erlang is adaptable, but was built for a Telcom setting with a lot more permanence than today's popular cloud model with nodes coming and going. On the other hand, an awful lot of large scale chat systems start as ejabberd and adjust from there; sometimes rewritten in another language, sometimes rebuilt in company specific Erlang while the service is running, sometimes without muvh adjustment.
- hosh 3y ago
  Elixir/Erlang/OTP actually holds up well even with modern cloud environments, including Kubernetes. For example, you can effectively architect with microservices without actually creating more deployable artifacts. This is because each GenServer is already a microservice. You build it from the ground up reasoning through failure modes and coordination. Those are easier to separate out into its own deployable artifact when the time comes, if it ever does. People hear about code hot reloainding (where you deploy new code without shutting down the production environment) and think that’s the killer feature of the BEAM vm. Code hot reloading is a narrow use—case of a more general capability — you can directly write or patch new code in a running system in a live environment through the REPL. That has been very useful for improving the system, to stop the bleeding or identifying bottlenecks and trying out fixes, even when deployed as ephemeral pods in Kubernetes.
hosh 3y ago
Last year, according to Stackoverflow polls, Elixir was the second most loved language, just under Rust. There is a reason why people who knows it, loves it. Elixir’s runtime, unlike many others, uses preemptive lightweight processes. Unlike async-only systems, a runaway process won’t block out other processes. That’s just the start of the kind of advantages in writing with Elixir/Erlang/OTP It’s been battle tested for years. I think many of the hype cycles for distributed systems reexplore or reinvent what Erlang and Elixir has to offer.
nixlim 3y ago
I work for a smallish supermarket chain (I assume, by the US standards) - we have about a 1000 stores in 5 countries, about 40-45K tills. At peak trading my teams' services handle a combined 250K tps. We run a bunch of distributed systems in the cloud, across two DCs and some other weird legacy locations (there are even a few mainframes) - the business is really old and went digital before some of my colleagues were born :) (Anyone remember teletext? Yep, we were on that too). So, Elixir/Erlang combo seems like a perfect fit. Instead, we are having all remaining Elixir and Erlang systems ripped out and re-written in Java. The reason is simple - readability and hire-ability. The cherries on the cake - Java's "good enough" for what we do, the tooling, the quality and availability of reference resources and subject matter experts - are just that, cherries. But the ability to hire and onboard new engineers is THE thing - it took us 3 years to fully hire out the headcount we were allocated for just our little team and that is in Java. Hiring for Elixir and Erlang has been such a pain that we are having to pay contractors while we re-write the current Elixir and Erlang systems into Java. Our domain is somewhat complicated - most people who have not worked in food retail generally would not imagine it to be so - so onboarding people to the domain knowledge takes 3-6 months and that is without "This is what? Erlang?" barrier. That is where readability of Java and non-FP vs Elixir/Erlang and FP comes in as well. I guess what I am trying to say is that, yes, the right tool for the job and all that but good enough tool with a significantly higher hiring pool is proving so much better.
- travisgriggs 3y ago
  It would be interesting to see where things are at in a few years. I get the problem. But not sure the right decision was made. I've seen the "leave esoteric/boutique/niche language for more commodity platforms" play out a number of times over the years. The results are very mixed, and hard to infer because of the many variables. But I don't usually see the "we're so much better now" after the original zeal for the exodus pans out. Usually it ends up being more of a "you had a problem so you did X. Now you have two problems" in my experience. I wish you and yours the best of luck.

Jtsummers 3y ago

> tangible advantages usually win the day because in the end people will pick the right tool for the right job No, they don't. They pick the familiar and the easy-to-hire-for. I've worked on systems (written in C++, Java, others) that would have been perfect for Erlang, but at most I was able to present a prototype. No one is interested in a better solution that they can't hire for or have to retrain their staff for. (The familiar and the easy-to-hire-for may, by some metric, be the right tool for the job, but it isn't necessarily the right tool by technical metrics.)

devoutsalsa 3y ago
I am pretty strong with Elixir, and I've done a little bit of Erlang. Anyone who understand the basics of Erlang features that get used in Elixir should be able to program in Erlang with a bit of a ramp up period. Not too long ago I interviewed with a company that only uses Erlang, not Elixir, and they rejected me because I am not an Erlang expert. There wasn't anything I could say that would change their minds. I even volunteered to do a coding challenge of some type & they still wouldn't consider me further. Sometimes a talent shortage is a state of mind.
- ollien 3y ago
  That's what I did! I learned Elixir in my spare time and then started a job that used Erlang. There's some quirks that I had to get over, but it wasn't a huge change over.
  - devoutsalsa 3y ago
    Shoot me an email if you'd like to connect!
- davidw 3y ago
  > Erlang, not Elixir, and they rejected me because I am not an Erlang expert Wow... was this an actual hiring manager or someone, or like an HR person with a checklist? I've interviewed for one or two Elixir positions where the opposite thing happened: I talked with someone who had a checklist. I haven't done Elixir professionally even though I have extensive Erlang experience and that was that. There is a pretty stark difference between "has architected an application with OTP components" and "can write linear code in Erlang or Elixir", but even then you'd probably be able to fit someone in unless you're hiring for that kind of senior/architect position.
  - devoutsalsa 3y ago
    It was a team decision. I interviewed with the hiring manager and three developers. It sounded like I was the first Elixir person they'd spoken to, and just decided it was outside of their comfort zone.
- hosh 3y ago
  When I worked at an Erlang shop, the director’s hiring strategy was to hire generalists willing to learn Erlang. He also hired remote, years before the lockdowns. He built distributed teams that builds distributed systems.
  - devoutsalsa 3y ago
    Sounds like a nice gig.
Verdex 3y ago
> > [...] people will pick the right tool for the right job > No, they don't. I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical). I assert the following: Scenario A: I have a literal mathematical proof that erlang is the best choice. I implement the solution, but due to the market changing (or an earthquake or ninjas literally destroy the server room) the overall project fails. * Me: But, I had math proving everything out. Ultimate Boss: I don't believe in math, you're fired. * Boss: Well, it failed, but look at this incomprehensible math that I don't understand. Ultimate Boss: You've obviously been hoodwinked, you're fired. Scenario B: I have a really good story that includes phrases like: "best practices" and "industry standard", and is doing exactly the same thing as everyone else. The project fails because the tool isn't great. * Me: It failed, but here's a good story. Ultimate Boss: Well, it looks exactly like everything else, so I'll look for a different nail that's standing out from the crowd to blame for this so that the muggles don't get nervous about taking non-chances in the future. * Boss: Here's this great story that I totally understand and can embellish upon because it operates on storytelling and not category theory. Ultimate Boss: That's a really good story. I'll have to steal it myself to give to the board to handle some unrelated problems of my own. Better luck next time. Even the scenario of master craftsman using erlang because it's the right tool for the job is more about a story about an uber expert who is off doing wizard stuff than it is about an actual merited set of reasoning.
- Jtsummers 3y ago
  > I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical). The "nobody ever got fired for buying IBM" defense. Which has since been updated to Oracle, Microsoft, Java, C# (for a large portion of the enterprise world), Python, and Javascript. Best-in-class that isn't mainstream is a hard sell to employers if they are strongly risk averse (which many people, if not most, are).
  - 3y ago
deleted 3y ago
[deleted]
ralphc 3y ago
Sounds like they're the prototype for a new company. WhatsApp ran rings around competitors with a small team using Erlang; can you do a better job than the company that rejected it?
- Jtsummers 3y ago
  Too low a margin and too few customers to make it worth it for me to try and jump into those niches. Honestly, many of the benefits of using Erlang for those systems would have been similar to any other more service-oriented or decomposition of monoliths. They had complex systems doing too much, but that would have benefited from the really good binary processing portion of Erlang and it's relatively easy horizontal scaling story. But when the number of customer (organizations) can be counted on one hand, and the number of users is less than 1000, and there's already competition you know it's going to be hard to break in. They'd already hit rock bottom pricing on their contracts.

bcrosby95 3y ago

Have you seen Erlang syntax? That's why. Even some of the best programmers I know are very superficial.

zelphirkalt 3y ago
I don't actually find the syntax to be so bad. It can even be pretty neat with the pattern matching.
- 0x457 3y ago
  I don't think it's bad, but it looks very alien. It's also very simple compared to say JavaScript or really anything else. I understand why Elixir was born.
aabbcc1241 3y ago
Maybe a transpiler and type checker that convert typescript into erlang can help to drive the adoption
- masklinn 3y ago
  That’s Elixir, pretty much.
- cjallen88 3y ago
  Purescript has an erlang backend FYI
  - aabbcc1241 3y ago
    That's interesting! I think Typescript would be "easier" for a larger group of developers to hook on
datadeft 3y ago
Have you seen the semantical aspects of that syntax?

deleted 3y ago

[deleted]

toast0 3y ago

Distributed systems are everywhere in big tech. So that's not really the issue. A lot of it is unfamiliarity with the tool and unfamiliarity with the paradigm. And of course, you can do distributed systems with other tools that people tend to be more familiar with, so go with what you know. Erlang is adaptable, but was built for a Telcom setting with a lot more permanence than today's popular cloud model with nodes coming and going. On the other hand, an awful lot of large scale chat systems start as ejabberd and adjust from there; sometimes rewritten in another language, sometimes rebuilt in company specific Erlang while the service is running, sometimes without muvh adjustment.

hosh 3y ago
Elixir/Erlang/OTP actually holds up well even with modern cloud environments, including Kubernetes. For example, you can effectively architect with microservices without actually creating more deployable artifacts. This is because each GenServer is already a microservice. You build it from the ground up reasoning through failure modes and coordination. Those are easier to separate out into its own deployable artifact when the time comes, if it ever does. People hear about code hot reloainding (where you deploy new code without shutting down the production environment) and think that’s the killer feature of the BEAM vm. Code hot reloading is a narrow use—case of a more general capability — you can directly write or patch new code in a running system in a live environment through the REPL. That has been very useful for improving the system, to stop the bleeding or identifying bottlenecks and trying out fixes, even when deployed as ephemeral pods in Kubernetes.

hosh 3y ago

Last year, according to Stackoverflow polls, Elixir was the second most loved language, just under Rust. There is a reason why people who knows it, loves it. Elixir’s runtime, unlike many others, uses preemptive lightweight processes. Unlike async-only systems, a runaway process won’t block out other processes. That’s just the start of the kind of advantages in writing with Elixir/Erlang/OTP It’s been battle tested for years. I think many of the hype cycles for distributed systems reexplore or reinvent what Erlang and Elixir has to offer.

nixlim 3y ago

I work for a smallish supermarket chain (I assume, by the US standards) - we have about a 1000 stores in 5 countries, about 40-45K tills. At peak trading my teams' services handle a combined 250K tps. We run a bunch of distributed systems in the cloud, across two DCs and some other weird legacy locations (there are even a few mainframes) - the business is really old and went digital before some of my colleagues were born :) (Anyone remember teletext? Yep, we were on that too). So, Elixir/Erlang combo seems like a perfect fit. Instead, we are having all remaining Elixir and Erlang systems ripped out and re-written in Java. The reason is simple - readability and hire-ability. The cherries on the cake - Java's "good enough" for what we do, the tooling, the quality and availability of reference resources and subject matter experts - are just that, cherries. But the ability to hire and onboard new engineers is THE thing - it took us 3 years to fully hire out the headcount we were allocated for just our little team and that is in Java. Hiring for Elixir and Erlang has been such a pain that we are having to pay contractors while we re-write the current Elixir and Erlang systems into Java. Our domain is somewhat complicated - most people who have not worked in food retail generally would not imagine it to be so - so onboarding people to the domain knowledge takes 3-6 months and that is without "This is what? Erlang?" barrier. That is where readability of Java and non-FP vs Elixir/Erlang and FP comes in as well. I guess what I am trying to say is that, yes, the right tool for the job and all that but good enough tool with a significantly higher hiring pool is proving so much better.

travisgriggs 3y ago
It would be interesting to see where things are at in a few years. I get the problem. But not sure the right decision was made. I've seen the "leave esoteric/boutique/niche language for more commodity platforms" play out a number of times over the years. The results are very mixed, and hard to infer because of the many variables. But I don't usually see the "we're so much better now" after the original zeal for the exodus pans out. Usually it ends up being more of a "you had a problem so you did X. Now you have two problems" in my experience. I wish you and yours the best of luck.