9 ms·

Erlang's not about lightweight processes and message passing (2023)

331 points by todsacerdoti 2y ago

whalesalad 2y ago
I disagree. Interfaces are a trivial concept that can get bolted-on to any language. Even in languages without an official interface construct, you can replicate them in the program space. The BEAM succeeds because you can run 1M processes on a single node, represent complex distributed state machines with ease, and restart portions of the system with zero downtime. Among many other things. I really don't think behaviors/interfaces is the most critical piece.
- hinkley 2y ago
  I haven’t used it enough to be able to say yet, but I believe the BEAM avoids part of the problem Ian Cooper (Where Did It All Go Wrong?) rediscovered, which is that microservices don’t min-max the inter- versus intra-modular friction in systems. I would not say that Beam eliminates this problem in any way, but I do think it lowers the slope of the line. The self-consistent idioms and functionality, especially with deployment, auto recovery and load balancing, reduce the inter-module friction. It makes a system where 12 engineers can easily manage 30 endpoints, and your surface area can still follow a power law.
- rdtsc 2y ago
  I see your point to a degree. That's kind of how Erlang is. At first, anything Erlang has, some other system has too: Isolated process heaps? - Just use OS processes Supervision trees? - Use kubernetes. Message passing? - Not a big deal, I can write two threads and a shared queue in Java. Hot code loading? - Java can do that too Low latency processing? - I can tune my LMAX disruptor to kick Erlang's butt any day. Now getting all that into one platform or library that's the main idea. OS processes are heavyweight. Running 2M of them on a server is not easy. You could use some green threads or promises but now you lost the isolated heap bit. You can use kubernetes to some degree but it does not do nested supervision trees well. I guess it would work, but now you have your code, and you have pods and controllers, and volumes and all the shit. You can do message passing with an "actors" libraries in many language. But you cannot do pattern matching on receive, and it doesn't transparently integrate with sending it across nodes to another thread. You can do hot code loading, but how do you deal with runtime data structure and state. Erlang is built around that: gen_servers since the state is immutable and explicit has callbacks to upgrade not just the code but the state itself.
  - whalesalad 2y ago
    > Isolated process heaps? - Just use OS processes OS procs are heavyweight. Erlang procs are ~2KB each. You can spin up millions on one box while traditional OS procs would melt your machine. Not even in the same league. > Supervision trees? - Use kubernetes. Comparing Kubernetes to Erlang supervision trees misses the mark. K8s is infrastructure that requires significant configuration and maintenance outside your codebase. Erlang's supervision is just code - it's part of your application logic. And those nested supervision trees? Good luck implementing that cleanly in K8s without a ton of custom work. > Message passing? - Not a big deal, I can write two threads and a shared queue in Java. Basic threads and queues don't compare to Erlang's sophisticated message passing. Pattern matching on receive makes for cleaner, more maintainable code, and the transparent distribution across nodes comes standard. Building equivalent functionality from scratch would require significantly more code and infrastructure. "Any sufficiently complicated concurrent program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Erlang." > Hot code loading? - Java can do that too Java can reload classes, but Erlang was designed for seamless evolution of running systems. The gen_server callbacks for upgrading state during hot code loading show how deeply this concept is integrated into Erlang's core.
    - hackernudes 2y ago
      It's funny reading your response because I think you skipped the second half of the comment! Your rebuttals to the rebuttals are pretty similar.
      - whalesalad 2y ago
        its friday my brain is cooked - you're right.
        rdtsc 2y ago
        No worries, at all I had figured you'd notice eventually. But I essentially agreed with you was just saying that common perceptions and how Erlang managed to do well there.
myth_drannon 2y ago
"In February 1998 Erlang was banned for new product development within Ericsson—the main reason for the ban was that Ericsson wanted to be a consumer of software technologies rather than a producer." - The creator of the language banned any use of it internally.
- vvpan 2y ago
  But from the quote it seems that for reasons unrelated to the language itself?
  - bluGill 2y ago
    Right. Companies need to figure out what they are going to be good at and what you buy from someone else. There is no one correct answer, and often the best answer changes over time (both reasons that different companies will have different answers). My company architects have been debating what IPC framework we should adopt for longer than it would take me to just write one from scratch - but they are correct to debate this instead of writing one because adopting means we don't have to maintain it and eventually that should cost more than all the debate. (note I didn't say will, though I think it is likely)
- zdragnar 2y ago
  Being a consumer rather than a producer of tech is strictly a business decision. There's significant cost to producing and maintaining a language, and Ericsson no longer wanted to pay the upkeep. That's not necessarily an indictment on the language itself. The alternative would have been to keep using it while also open sourcing it, but I'm guessing they just wanted to be able to hire cheaper C developers or whatever the flavor of the time was.
- adamkittelson 2y ago
  It is wildly disingenuous to just copy paste that line from wikipedia and not the rest of the paragraph. > In February 1998, Ericsson Radio Systems banned the in-house use of Erlang for new products, citing a preference for non-proprietary languages.[15] The ban caused Armstrong and others to make plans to leave Ericsson.[16] In March 1998 Ericsson announced the AXD301 switch,[8] containing over a million lines of Erlang and reported to achieve a high availability of nine "9"s.[17] In December 1998, the implementation of Erlang was open-sourced and most of the Erlang team resigned to form a new company, Bluetail AB.[8] Ericsson eventually relaxed the ban and re-hired Armstrong in 2004. - edit, poster was quoting a quote in the article, not wikipedia, the article is the one omitting the context
aanet 2y ago
Thanks for posting this. What a fantastic post on the history or Erlang. Very illuminating (all of this is new to me).
bcardarella 2y ago
The amazing thing about Erlang and the BEAM is it's depth of features. To the OP the Behaviour/Interface of Erlang is their biggest take away. For me I believe it is how you require far far less development resources to build complex systems than you would require in any other language (provided comparable experience in both stacks). And for many the lightweight processes and programming model. OTP itself has so much in it. We've been working on compiling Elixir to run on iOS devices. Not only can we do that through the release process but through using the ei library provided in Erlang we can compile a Node in C that will interface with any other Erlang node over a typical distributed network as you would for Erlang, Elixir, Gleam, etc... furthermore there is a rpc library in Erlang where from C we can make function calls and interface with our Elixir application. Yes, the encoding/decoding has an overhead and FFI would be faster but we're still way within our latency budget and we got this stood up in a few days without even have heard of it before. The larger point here is that Erlang has been solving many of the problems that modern tech stacks are struggling with and it has solved for scale and implementation cost and it solved these problems decades ago. I know HN has a bit of a click-bait love relationship with Erlang/Elixir but it hasn't translated over to adoption and there are companies that are just burning money trying to do what you get out of the box for free with the Erlang stack.
- relistan 2y ago
  C nodes are under appreciated. We have one (Cgo) for communicating between Go and Elixir services running in the same Kubernetes pod. The docs are also pretty good for Erlang and its C libs.
- agent281 2y ago
  > I know HN has a bit of a click-bait love relationship with Erlang/Elixir but it hasn't translated over to adoption and there are companies that are just burning money trying to do what you get out of the box for free with the Erlang stack. Do you or the community have a sense why that is?
  - bsder 2y ago
    Erlang looks weird--Prolog-base, tail recursive loops, extensive pattern matching. Also, a lot of the power of Erlang is the OTP (Open Telecom Platform) even more than Erlang, itself. You have to internalize those architectural decisions (expect crashes--do fast restart) to get the full power of Erlang. Elixir seems like it has been finding more traction by looking more like mainstream languages. In addition, languages on the BEAM (like Elixir) made the BEAM much better documented, understood and portable.
jerf 2y ago
"This begs the question: why aren’t language and library designers stealing the structure behind Erlang’s behaviours, rather than copying the ideas of lightweight processes and message passing?" Because the function signatures of Erlang's behaviors are critically tied to Erlang's other functionality, specifically its unusual use of immutability. You need a separate init call for its servers because of that, and a very distinct use of the state management to work exactly the same way. But to achieve the same goals in other languages, you almost always shouldn't directly copy what Erlang is doing. In fact when I see "Look! I ported gen_server into $SOME_OTHER_LANGUAGE" and I see exactly and precisely the exact interface Erlang has, I know that the port doesn't deeply understand what Erlang is doing. When I ported the idea of supervisor trees into Go [1], I did so idiomatically. It turns out in modern Go the correct interface for "a thing that can be supervised" is not precisely the same signature that Erlang has, but type Service interface { Serve(context.Context) } That's all you need and all you should use... in Go. Your other language may vary. Go doesn't need a "handle_event/2" because it has channels, and you should use those, not because they are "better" or "worse" but because that's what this language does. In another language you may use something else. In another infrastructure you may end up sending things over Kafka or some cloud event bus rather than "calling a handle_event/2". The key is in building an event-based system, not copying the exact implementation Erlang has. A peculiar issue the Erlang community has is getting excessively convinced that there's something super-mega-special about the exact way Erlang does it, and that if you do it any other way it is ipso facto wrong and therefore not reliable. This may have been true in 2005; it is not true in 2025. Where once Erlang had almost the only sensible answer, in 2025 the problem is poking through the ocean of answers deluging us! While I recommend learning from Erlang about reliable software, I strongly recommend against just blind-porting out the exact way Erlang achieves it into any other language. It is in almost any other language context the wrong answer. Even other immutable languages generally vary enough that they can't just copy the same structure. [1]: https://jerf.org/iri/post/2930/ https://jerf.org/iri/post/2930/
- asa400 2y ago
  To follow on from your excellent post, I think a reasonable next question is, "why have these kinds of approaches and ideas in other languages and systems succeeded in gaining market adoption, but Erlang/Elixir has not?" This to me is the most interesting question about Erlang, and I say this as someone who works professionally in Elixir. It's _clear_ that there is incredible appetite for tools that help us design reliable concurrent systems given the wild success of things like k8s, Kafka, AWS's distributed systems products, etc., but why hasn't Erlang/Elixir been able to capture that share? My friends and I debate this all the time, but I don't know the answer.
senderista 2y ago
For me the most interesting concept in Erlang/BEAM is that partial recovery is built in from the ground up. When an unexpected state is encountered, instead of either killing the entire process or trying to proceed and risking corruption, you just roll back to a known good state, at the most granular level possible. This idea was researched many years ago under the name of "microreboots"(associated with "crash-only software"), but only Erlang/BEAM made it a first-class concept in a production system.
- benmmurphy 2y ago
  You still have to be careful with supervision trees and parts of the tree restarting. For example your system might work if the whole erlang operating system process is suddenly killed and restarted but your system might start corrupting data if parts of the erlang process tree is restarted. Erlang gives you a good model to work with these problems but it doesn't allow you to completely turn off your brain. If you walk in thinking that you can just let things restart and everything will be fine then you might end up getting burnt.
  - juped 2y ago
    Yes, you can design your system pathologically to make it wrong.
  - alienthrowaway 2y ago
    > You still have to be careful with supervision trees and parts of the tree restarting [...] Erlang gives you a good model to work with these problems but it doesn't allow you to completely turn off your brain. Erlang gives architects the tools to restart as little, or as much of the tree as they like, so I hope they have their brains fully engaged when working on the infrastructure that underlies their projects. For complex projects, it's vital think long and hard about state-interactions and sub-system dependencies, but the upside for Erlang is that this infrastructure is separated from sequential code via behaviors, and if the organization is big enough, the behaviors will be owned by a dedicated infrastructure team (or person) and consumed by product teams, with clear demarcations of responsibilities.
- 2y ago
Towaway69 2y ago
I've just gotten back into Erlang becuase of the lightweight processes and message passing, so far behaviour has been secondary (i.e. just learning about them)! The project is about bring visual Flow Based Programming(FBP)[1] to Erlang. FBP seems to be made for Erlang and I was surprised there was something already but there does not seem to be. My goto tool for FBP is Node-RED and hence the basic idea is to bolt a Node-RED frontend on to an Erlang backend and to have every node being a process. Node-REDs frontend is great for modelling message passing between nodes, hence there is a very simply one-to-one mapping to Erlangs processes and messages. I've implemented some basics and started to create some unit tests as flows to slowly build up functionality. I would really like this to be 100% compatiable to Node-RED the NodeJS backend. For more details, the github repo --> https://github.com/gorenje/erlang-red https://github.com/gorenje/erlang-red Overall Erlang is amazingly well suited to this and astonished that no one else has done anything like this - or have they? [1] = https://jpaulm.github.io/fbp/index.html https://jpaulm.github.io/fbp/index.html
- __jonas 2y ago
  Oh that's really cool to see! I always thought a visual programming language on the BEAM would be fun
- mcintyre1994 2y ago
  This is a really cool idea!
  - Towaway69 2y ago
    Thank you, it's also a lot of fun to do :) Hopeful I can get some useful functionality together without hitting my Erlang coding limits! Any help is greatly appreciated :+1:
- runlaszlorun 2y ago
  Love the idea as well! Would I be wrong in thinking that, at a high-level, fbp is like erlang processes where message flow is one way?
hinkley 2y ago
I’ve worked with a few individuals, mostly managers, who intended to write books informed by our experiences. It was always frustrating for me to see that we disagreed about what aspects of our work made us successful. There was always something they minimized as being nice that I felt was essential. And here we see someone claiming that lightweight processes and message passing aren’t the secret sauce, missing that Erlang as Communicating Sequential Processes is indivisible from those qualities, and then repeatedly mentioning CSP as part of the secret sauce. Examples: > The application programmer writes sequential code, all concurrency is hidden away in the behaviour; > Easier for new team members to get started: business logic is sequential, similar structure that they might have seen before elsewhere; > Supervisors and the “let it crash” philosophy, appear to produce reliable systems. Joe uses the Ericsson AXD301 telephone switch example again (p. 191): Behaviors are interesting and solve a commonly encountered problem in the 80’s that was still being solved in some cases in the 00’s, but it’s a means as much as an end in Erlang. It’s how they implemented those other qualities. But I don’t know if they had to, to make Erlang still mostly be Erlang.
- sitkack 2y ago
  Managers make up their own narrative based on vibes.
- silisili 2y ago
  Is Erlang considered CSP? I've always thought it wasn't really, and had its own thing called 'actors' which are id'd and can communicate directly, vs CSP which are anonymous and use channel messaging. I've always thought the actor model made more sense, but highly YMMV.
  - hinkley 2y ago
    The erlang docs only go as far as saying it’s functionally similar to CSP. I think the term Actor Model has been so semantically diluted at this point that the phrase also understates what Erlang has as well. Neither CSP nor AM require process isolation to work, which means they can work when they work but fail much much worse. They are necessary but insufficient.
Supersaiyan_IV 2y ago
"In February 1998 Erlang was banned for new product development within Ericsson" False statement. Ericsson still uses Erlang, for example in their MME. Source: I used to work at Ericsson.
- bee_rider 2y ago
  Is there any additional context here? (Is this a common misperception that you’ve come across?)
  - macintux 2y ago
    Not the original commenter, but from hearing Joe Armstrong speak on multiple occasions my recollection is that Erlang was blacklisted for some length of time (which allowed it to become an open source language), but eventually the company realized it still offered significant value.
  - sbuttgereit 2y ago
    I posted an excerpt in a different comment, but a decent source for this is section 5.2 of the document: https://lfe.io/papers/%5B2007%5D%20Armstrong%20-%20HOPL%20III%20A%20History%20of%20Erlang.pdf https://lfe.io/papers/%5B2007%5D%20Armstrong%20-%20HOPL%20II...
- 4ad 2y ago
  It is simultaneously possible that Ericsson banned Erlang in 1998 (a statement claimed multiple times by the creators of Erlang) and that Ericsson rescinded the ban later in 2004, when they hired back Joe Armstrong.
- jesperwe 2y ago
  And there is a small team of Ericsson full time devs working on developing the language itself and the BEAM.
- lysace 2y ago
  My impression from Ericssonland: Around year 2008 being an Erlang coder was often more or less seen as being a COBOL coder in Sweden. Bluetail had sort of failed, having burned lots of VC, iirc. So Erlang was something weird and custom that Ericsson used to build software for legacy phone exchanges. I remember that a colleague's wife working at Ericsson had received on-the-job training from essentially zero programming knowledge to become an Erlang developer in order to maintain some phone exchange software. It's been fascinating to see it morph into something cool. Whatsapp, etc.
behnamoh 2y ago
Is it just me or does Erlang's syntax look a little bit nicer than Elixir's?
- Capricorn2481 2y ago
  I'm an outsider to this ecosystem, but I've seen a few people share that same opinion. They prefer the explicitness of Erlang.
  - sodapopcan 2y ago
    Elixir is still very explicit from a syntactic point of view. Macros allow hiding a significant amount of boilerplate in certain behaviours, though. So it's a matter of preference, of course.
- SoftTalker 2y ago
  It's inspired/descended from Prolog, and my impression is that many people find it a bit odd. It is at first, but I quickly adjusted to it and quite like it now.
- whalesalad 2y ago
  gleam is probably my favorite middle ground between elixir and erlang.
- bmitc 2y ago
  Elixir came from Ruby developers and thus has similarly verbose syntax and macros. Erlang's syntax came from Prolog, which was used to implement the first compiler and is why Erlang's syntax is more concise.
- ValtteriL 2y ago
  I learned Erlang at school and used to prefer its syntax for years. However, after giving Elixir a chance and writing 1000 loc I was converted. Now I look at snippets of Erlang in docs with mild disgust.
- pton_xd 2y ago
nailer 2y ago
"In 1998 Ericsson decided to ban all use of Erlang. The people responsible for developing it argued that if they were going to ban it, then they might as well open source it. Which Ericsson did and shortly after most of the team that created Erlang quit and started their own company." Bwahahaha. Reminds me of the JRuby team, who left Sun as a single unit and resumed work as a team at another company (I can't remember where) when Oracle acquired Sun.
deleted 2y ago
[deleted]
hk__2 2y ago
(2023)
layer8 2y ago
From this article and others, it’s still unclear to me what the state-handling and state-sharing model of Erlang is. Presumably, the granularity of the crashing/restarting sequential processes is also the granularity of in-memory state sharing. But what about external state, like databases, queues, file systems? For example, if a process has taken an item off a queue and then crashes before having fully processed it, how is that accounted for? Or you might not even know from the outside if it has been fully, partially, or not at all processed yet. This is an example where correct error handling or not crashing is crucial, in my experience. Or what about processing pipelines where a component in the middle crashes. Is there something like that in Erlang? Is there an article explaining Erlang from that perspective?
- fidotron 2y ago
  > For example, if a process has taken an item off a queue and then crashes before having fully processed it, how is that accounted for? I have worked with people that had deployed huge amounts on the BEAM that had a real problem with the answer to that, and resort to magical thinking. When erlang processes "crash", assuming the whole system didn't crash, they almost certainly alerted a monitoring process of the fact, so that a process can be quickly restarted. This is the core of how supervision trees in erlang are built. There are a lot of subtleties to that. The whole system may or may not be a single BEAM instance, and if more than one then they can be distributed, i.e. processes on one machine receive failure messages from processes on others, and can restart the processes elsewhere. These mechanisms on a practical basis are sufficient to automatically pick up the majority of transient failures. (I should add there are two classic ways to blow up a BEAM instance which make this less good than it should be: a bad C function call "NIF" for native something function, or posting messages to a process faster than it can consume them, which will eventually cause an OOM). But this differs from the underlying philosophy of the runtime, which is that things are only done when they're done, and you should expect failures at any time. This maps on to their messaging paradigm. What you actually sound like you want is a universe more like FoundationDB and QuiCK https://www.foundationdb.org/files/QuiCK.pdf https://www.foundationdb.org/files/QuiCK.pdf where the DB and worker queue all live in one single transactional space, which certainly makes reasoning about a lot of these things easier, but have nothing to do with erlang.
deleted 2y ago
[deleted]
whorleater 2y ago
One thing I haven't seen being discussed is the BEAM internals becoming a little long in the tooth. We still have static reductions before the scheduler switches to another task, the priority system in scheduling is a bit dodgy, flipping vmargs is kinda complex, lock counting and crash dump tooling kinda suck, etc. BEAM is great, although it's definitely missing something like pprof for go or java flight recorder.
spintin 2y ago
[dead]
geophile 2y ago
In 2003 I joined a startup building a horizontally scalable archive. You could add nodes to add capacity for storing data and metadata, and the system could tolerate up to a configured number of failures and carry on without loss of data or service. (This was not a general-purpose file system, it was for write-once/read-many objects.) We built the system in Java and C. The distribution layer was done completely in Java. It was only after the system was done that I discovered Erlang. I REALLY wish I had known about it earlier. Erlang solved so many of the problems we had to solve by ourselves.
- DarkNova6 2y ago
  Even these says, now that Java got Virtual Threads?
  - geophile 2y ago
    I haven't been following recent Java releases, but unless Java has some equivalent to Erlang's OTP library (at a minimum), I stand by my statement
pkilgore 2y ago
Interesting. It strikes me that some of this rhymes with the platform abstraction of roc[1] [1] https://www.roc-lang.org/platforms https://www.roc-lang.org/platforms
jFriedensreich 2y ago
TLDR title for erlang people: erlang is not just erlang but erlang + OTP.
jiggawatts 2y ago
Someone explain to me why I should prefer Erlang/BEAM/Elixir over something like Akka.NET? With the latter I get a huge ecosystem of packages and wide compatibility with platforms and tooling and also a robust and scalable actor model. Learning Erlang or any related language meanwhile feels like learning Tolkien’s Elvish for the purposes of international trade.
- neonsunset 2y ago
  _Supposedly_ they are more convenient if you are willing to tolerate abysmally subpar efficiency, exotic semantics and lacking ecosystem.
- dqv 2y ago
  No, we can't explain to you why our blub language should be preferred to your blub language. It's your job to make that determination on your own. I can come back in 5 years to explain to you what is annoying about Akka.NET compared to the BEAM and vice versa. An expert in the BEAM who lacks experience in C# is not going to be able explain to an expert in C# who lacks experience in the BEAM why BEAM is better. You're asking for something incredibly rare - a person who is an expert in both runtimes and can concisely explain to you the tradeoffs of each.
  - deleted 2y ago
    [deleted]
- neonsunset 2y ago
  If you want to do exclusively distributed computing at the application level - Erlang/Elixir will be better. They can offer nice Northstar of where the UX of Akka.net/Orleans should sit at (and, arguably, Orleans is not exactly nice to use in comparison). Otherwise, aside from educational purposes, they are not worth spending your time on. Just skip to F# over Elixir because Elixir is not a serious language, lacking base language primitives and operations one would expect standard library to offer. It's not productive nor fast.
travisgriggs 2y ago
To me, Erlang/Elixir’s power is not necessarily the Actor model implementation, the matching from prolog, immutability, behaviors, etc, but Joes desire to demonstrate you could do more with less. It is a well thought out and trued system of computation that has a consistency rarely witnessed in other languages, much less the “web”. It is not perfect. But it is pretty impressive. Unfortunately, I find the appreciation and uptake for what simplicity empowers in the software world pretty under appreciated. Complexity allows people to become specialists, managers to have big teams and lots of meetings, experts to stay experts. Erlang was being developed in a period where companies were trying to implement software solutions with smaller headcounts, limited horsepower, etc. A multi decade outpouring of cash into the domain has made the value of “less will mean more for all of us in good ways” less of an attractor.
- zelphirkalt 2y ago
  Reminds me of Rich Hickey's talk about Simple VS Easy.
- runlaszlorun 2y ago
  You've just convinced me to spend some more time with Erlang! I've dabbled a bit and, at least on the surface, prefer erlang syntax over elixir.
  - LtdJorge 2y ago
    Me too, as weird as it might sound
    - travisgriggs 2y ago
      I mostly prefer the Elixir syntax. But I don’t care for all of the “ends”. Wish they had taken more inspiration from Python in that department.
      - mg74 2y ago
        And I am grateful for the ends, wish Python had them as well. Python already marks the start of every block with a :, one more keyword and the semantic whitespace madness could end.
HeavyRain266 2y ago
Erlang, OTP, and the BEAM offer much more than just behaviours. The VM is similar to a virtual kernel with supervisor, isolated processes, and distributed mode that treats multiple (physical or virtual) machines as a single pool of resources. OTP provides numerous useful modes, such as Mnesia (database) and atomic counters/ETS tables (for caching), among others. The runtime also supports bytecode hot-reloading, a feature used to apply patches without any system downtime. While the syntax is not very screen reader-friendly, it is digestable. Apache Mesos[1] is the only thing that comes to my mind as a similar platform to BEAM in its ability to treat multi-machine resources as a single pool. Over a year ago, my private consulting company decided to adopt Erlang as our backend language. After some time, we started exploring BEAM's internals to, for example, replace the TCP-based stack with QUIC and integrate some Rust patches. A truly fantastic choice for lightweight and high-throughput systems that are only failing in case of kernel panic or power loss. We are currently working on very "busy", concurrent software like a film/game production tracker and pipeline manager, and are now also preparing R&D for a private hospital management services. [1]: https://mesos.apache.org/ https://mesos.apache.org/
- HeavyRain266 2y ago
  Before you ask, we're not going to ever fully adopt Elixir (or Gleam) as its ecosystem is built around Phoenix framework and external services/databases. We would have to maintain internal bindings/implementations of things that are unmaintained on Elixir's side. Also worth to mention that it has a large amount of syntax sugar and its users have that weird fetish for abstracting stuff into DSL interfaces.
  - deleted 2y ago
    [deleted]
  - pdimitar 2y ago
    Couldn't understand your comment well but I am making a SQLite library for Elixir (via Rust bindings) so that would be one less dependency on external systems. I happen to believe that most projects don't need a full-blown database server.
throwawaymaths 2y ago
I don't think behaviours are all that interesting; after all other programming languages have them. Rather, what is interesting about the BEAM is that throwing an error is so graceful that it's not such a sin to just throw an error. In otherwords, a component that CAN error or get into a weird state can be shoved into a behaviour that CANNOT. And by default you are safe from certain operational errors becoming logic or business errors. For example. You might have a defined "get" interface that doesn't return an error -- let's say it starts as an in-memory K/V store and it returns an optional(value), which is NULL in the case that the key didn't exist. But suppose you want to have two datastores that the same interface targets, so you might abstract that to a filesystem, and you could have a permission error. And returning "NULL" is not actually "correct". You should throw, because that bubbles up the error to ops teams instead of swallowing it whole. A panic in this case is probably fine. What if now you're going over a filesystem that's over the network, and the line to the datacenter was backhoe'd and there was a 10 millisecond failover by your SDN -- returning "NULL" is really not correct, because consumers of your getter are liable to have a bad time managing real consistency business cases that could cost $$$. And in this case a panic is not necessarily great, because you bring down everything over a minor hiccup. The other power with throwing errors + behaviors is that it makes trapping errors with contextual information reporting (e.g. a user-bound 500 error with stack trace information sent somewhere where ops can take a gander) really easy and generically composable, that's not so for error monads or panics. Anyways it was always strange to me that erlang-inspired actor system programming languages came out that obsessed over "never having errors" as a principle (like ponylang) because that's throwing out a big part of erlang.
- deleted 2y ago
  [deleted]
dang 2y ago
Discussed at the time: Erlang's not about lightweight processes and message passing - https://news.ycombinator.com/item?id=34545061 https://news.ycombinator.com/item?id=34545061 - Jan 2023 (274 comments)
cmdrk 2y ago
Erlang is my favorite language but getting a job writing Erlang feels impossible. I make it a habit to ctrl-F every Who’s Hiring? thread and find Elixir occasionally and Erlang never.
- gavmor 2y ago
  Can you articulate the kinds of business problems Erlang is particularly well-suited to solve? When you choose Erlang for a project, what kind of return on investment do you think it typically offers? Does it lead to significant cost savings or help generate more revenue in ways that other languages might not? In situations where Erlang is chosen, what are some concrete examples of how it has demonstrably increased efficiency, reduced errors, or enabled new business opportunities that wouldn't have been as feasible with other technologies? Edit: I guess if I'd done any research myself before asking, I might've found this: https://www.erlang-solutions.com/blog/which-companies-are-using-erlang-and-why-mytopdogstatus/ https://www.erlang-solutions.com/blog/which-companies-are-us...
unoti 2y ago
I came here looking for information about why Ericsson stopped using Erlang, and for more information about Joe's firing. The short answer seems to be that they pivoted to Java for new projects, which marginalized Erlang. Then Joe and colleagues formed Bluetail in 1998. They were bought by Nortel. Nortel was a telecom giant forming about a third of the value of the Toronto Stock Exchange. In 2000 Nortel's stock reached $125 per share, but by 2002 the stock had gone down to less than $1. This was all part of the dot com crash, and Nortel was hit particularly hard because of the dot com bubble burst corresponding with a big downturn in telecom spending. It seems safe to look at Joe's layoff as more of a "his unit was the first to slip beneath the waves on a sinking ship" situation, as they laid off 60,000 employees or more than two thirds of their workforce. The layoff was not a sign that he may not have been pulling his weight. It was part of a big move of desperation not to be taken as a sign of the ineffectiveness of that business unit.
- cmrdporcupine 2y ago
  It's very weird to me to see the word "fired" in this context. "Laid off" is more appropriate. "Fired" is very value-laden and implies fault and termination with cause. Which I'm sure if that was somehow actually true the original article author would know nothing about, nor would it be any of their business.
spott 2y ago
A question about erlang: Haskell taught me a lot about programming, things that I still use now, even though I only write Python. Does learning erlang teach you a new way of thinking? Or does it just make you wish you had erlang language features and libraries when not writing erlang?
- lgas 2y ago
  IMHO it will teach you a new way of thinking but that way is not as generally applicable as what most people take away from Haskell.
kerkeslager 2y ago
I think if you ever find yourself saying Erlang is about one thing, you've lost the plot. Erlang has a bunch of features that each would be powerful in their own right, but they aren't special--there are other ways to solve the same problems, and some of those ways even have merit. But Erlang chose the set of features they did with a very high level understanding of how those varous features would interact. I honestly think it's one of the greater achievements humanity has accomplished. It lacks the obvious elegance that, say, Lisp, has, but I think the sum of parts in Erlang was probably a lot harder to make.
ggm 2y ago
Only ever played with ejabberd but felt it was a marvel of high speed engineering. I also felt it had strong "halt and catch fire on error" properties. Am I maligning it, and Erlang?
kamma4434 2y ago
I do not agree with the contents of this article. Behaviors are possible because of the underlying architecture of the system. Behaviors are not interfaces – they are more like abstract object in a language like Java so they implement basic and self-contained functionalities that are hidden behind a collaboration interface, but they couldn’t do much without the underlying infrastructure that makes sure that every process is totally separate from other processes, the that all processes can be safely closed without leaking memory, or resources, and that you cannot just share a rogue pointer between two different processes. What Joe did in his thesis is to show you how you can build reliable systems (and up point, reliable distribuited systems) by using a given set of Lego blocks. The reason why you need the erlang vm to implement something like that appropriately – and that you cannot do that fully on a different VM – is that without the underlying plumbing, supervision trees would be leaky - in Java, you cannot kill a thread that is holding up to resources and hope that everything will always go well, And do not have ways to monitor different processes.
coldtea 2y ago
Haven't digged deep into Erlang, but from what I read, I never thought Erlang's big idea is "lightweight processes and message passing". Always thought the key differentiator is having a kind of "orchestrator/supervisor" for those processes. (Which "lightweight processes and message passing" facilitates, sure, but it's more than those)
LtdJorge 2y ago
To me the most important aspect of Erlang is the runtime's scheduler, which is preemptive instead of cooperative. This allows the message passing, sequential code and lightweight processes to be much more effective than in any other general language or framework using cooperative scheduling (like async runtimes or coroutines in Rust, .Net, Kotlin, Lua). You can write actually synchronous code in Erlang and the runtime makes it so that no process blocks any other process by preempting them on a schedule.
- assbuttbuttass 2y ago
  Sounds a lot like Go
  - LtdJorge 2y ago
    Yes, goroutines are preemptive, too. And they took a lot of inspiration from Erlang and the BEAM when they designed goroutines and channels. In Go you manage your goroutines and channels explicitly, while the BEAM runs all processes for you, and I've seen Robert Virding run an infinite loop in one Erlang process while the rest were serving requests, the core with the loop stayed at 100% but 0 requests were dropped and the latency and throughput was more or less the same, pretty crazy capabilities. You can do the same in Go but it's a lot more manual.
ksec 2y ago
Does anyone knows if Whatsapp is still running on Erlang? I know it used to be on FreeBSD but when Meta bought them they switched to Linux. Not sure if they switched away from Erlang as well.

331 points by todsacerdoti 2y ago

I disagree. Interfaces are a trivial concept that can get bolted-on to any language. Even in languages without an official interface construct, you can replicate them in the program space. The BEAM succeeds because you can run 1M processes on a single node, represent complex distributed state machines with ease, and restart portions of the system with zero downtime. Among many other things. I really don't think behaviors/interfaces is the most critical piece.

hinkley 2y ago
I haven’t used it enough to be able to say yet, but I believe the BEAM avoids part of the problem Ian Cooper (Where Did It All Go Wrong?) rediscovered, which is that microservices don’t min-max the inter- versus intra-modular friction in systems. I would not say that Beam eliminates this problem in any way, but I do think it lowers the slope of the line. The self-consistent idioms and functionality, especially with deployment, auto recovery and load balancing, reduce the inter-module friction. It makes a system where 12 engineers can easily manage 30 endpoints, and your surface area can still follow a power law.
rdtsc 2y ago
I see your point to a degree. That's kind of how Erlang is. At first, anything Erlang has, some other system has too: Isolated process heaps? - Just use OS processes Supervision trees? - Use kubernetes. Message passing? - Not a big deal, I can write two threads and a shared queue in Java. Hot code loading? - Java can do that too Low latency processing? - I can tune my LMAX disruptor to kick Erlang's butt any day. Now getting all that into one platform or library that's the main idea. OS processes are heavyweight. Running 2M of them on a server is not easy. You could use some green threads or promises but now you lost the isolated heap bit. You can use kubernetes to some degree but it does not do nested supervision trees well. I guess it would work, but now you have your code, and you have pods and controllers, and volumes and all the shit. You can do message passing with an "actors" libraries in many language. But you cannot do pattern matching on receive, and it doesn't transparently integrate with sending it across nodes to another thread. You can do hot code loading, but how do you deal with runtime data structure and state. Erlang is built around that: gen_servers since the state is immutable and explicit has callbacks to upgrade not just the code but the state itself.
- whalesalad 2y ago
  > Isolated process heaps? - Just use OS processes OS procs are heavyweight. Erlang procs are ~2KB each. You can spin up millions on one box while traditional OS procs would melt your machine. Not even in the same league. > Supervision trees? - Use kubernetes. Comparing Kubernetes to Erlang supervision trees misses the mark. K8s is infrastructure that requires significant configuration and maintenance outside your codebase. Erlang's supervision is just code - it's part of your application logic. And those nested supervision trees? Good luck implementing that cleanly in K8s without a ton of custom work. > Message passing? - Not a big deal, I can write two threads and a shared queue in Java. Basic threads and queues don't compare to Erlang's sophisticated message passing. Pattern matching on receive makes for cleaner, more maintainable code, and the transparent distribution across nodes comes standard. Building equivalent functionality from scratch would require significantly more code and infrastructure. "Any sufficiently complicated concurrent program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Erlang." > Hot code loading? - Java can do that too Java can reload classes, but Erlang was designed for seamless evolution of running systems. The gen_server callbacks for upgrading state during hot code loading show how deeply this concept is integrated into Erlang's core.
  - hackernudes 2y ago
    It's funny reading your response because I think you skipped the second half of the comment! Your rebuttals to the rebuttals are pretty similar.
    - whalesalad 2y ago
      its friday my brain is cooked - you're right.
      - rdtsc 2y ago
        No worries, at all I had figured you'd notice eventually. But I essentially agreed with you was just saying that common perceptions and how Erlang managed to do well there.

myth_drannon 2y ago

"In February 1998 Erlang was banned for new product development within Ericsson—the main reason for the ban was that Ericsson wanted to be a consumer of software technologies rather than a producer." - The creator of the language banned any use of it internally.

vvpan 2y ago
But from the quote it seems that for reasons unrelated to the language itself?
- bluGill 2y ago
  Right. Companies need to figure out what they are going to be good at and what you buy from someone else. There is no one correct answer, and often the best answer changes over time (both reasons that different companies will have different answers). My company architects have been debating what IPC framework we should adopt for longer than it would take me to just write one from scratch - but they are correct to debate this instead of writing one because adopting means we don't have to maintain it and eventually that should cost more than all the debate. (note I didn't say will, though I think it is likely)
zdragnar 2y ago
Being a consumer rather than a producer of tech is strictly a business decision. There's significant cost to producing and maintaining a language, and Ericsson no longer wanted to pay the upkeep. That's not necessarily an indictment on the language itself. The alternative would have been to keep using it while also open sourcing it, but I'm guessing they just wanted to be able to hire cheaper C developers or whatever the flavor of the time was.
adamkittelson 2y ago
It is wildly disingenuous to just copy paste that line from wikipedia and not the rest of the paragraph. > In February 1998, Ericsson Radio Systems banned the in-house use of Erlang for new products, citing a preference for non-proprietary languages.[15] The ban caused Armstrong and others to make plans to leave Ericsson.[16] In March 1998 Ericsson announced the AXD301 switch,[8] containing over a million lines of Erlang and reported to achieve a high availability of nine "9"s.[17] In December 1998, the implementation of Erlang was open-sourced and most of the Erlang team resigned to form a new company, Bluetail AB.[8] Ericsson eventually relaxed the ban and re-hired Armstrong in 2004. - edit, poster was quoting a quote in the article, not wikipedia, the article is the one omitting the context

aanet 2y ago

Thanks for posting this. What a fantastic post on the history or Erlang. Very illuminating (all of this is new to me).

bcardarella 2y ago

The amazing thing about Erlang and the BEAM is it's depth of features. To the OP the Behaviour/Interface of Erlang is their biggest take away. For me I believe it is how you require far far less development resources to build complex systems than you would require in any other language (provided comparable experience in both stacks). And for many the lightweight processes and programming model. OTP itself has so much in it. We've been working on compiling Elixir to run on iOS devices. Not only can we do that through the release process but through using the ei library provided in Erlang we can compile a Node in C that will interface with any other Erlang node over a typical distributed network as you would for Erlang, Elixir, Gleam, etc... furthermore there is a rpc library in Erlang where from C we can make function calls and interface with our Elixir application. Yes, the encoding/decoding has an overhead and FFI would be faster but we're still way within our latency budget and we got this stood up in a few days without even have heard of it before. The larger point here is that Erlang has been solving many of the problems that modern tech stacks are struggling with and it has solved for scale and implementation cost and it solved these problems decades ago. I know HN has a bit of a click-bait love relationship with Erlang/Elixir but it hasn't translated over to adoption and there are companies that are just burning money trying to do what you get out of the box for free with the Erlang stack.

relistan 2y ago
C nodes are under appreciated. We have one (Cgo) for communicating between Go and Elixir services running in the same Kubernetes pod. The docs are also pretty good for Erlang and its C libs.
agent281 2y ago
> I know HN has a bit of a click-bait love relationship with Erlang/Elixir but it hasn't translated over to adoption and there are companies that are just burning money trying to do what you get out of the box for free with the Erlang stack. Do you or the community have a sense why that is?
- bsder 2y ago
  Erlang looks weird--Prolog-base, tail recursive loops, extensive pattern matching. Also, a lot of the power of Erlang is the OTP (Open Telecom Platform) even more than Erlang, itself. You have to internalize those architectural decisions (expect crashes--do fast restart) to get the full power of Erlang. Elixir seems like it has been finding more traction by looking more like mainstream languages. In addition, languages on the BEAM (like Elixir) made the BEAM much better documented, understood and portable.

jerf 2y ago

"This begs the question: why aren’t language and library designers stealing the structure behind Erlang’s behaviours, rather than copying the ideas of lightweight processes and message passing?" Because the function signatures of Erlang's behaviors are critically tied to Erlang's other functionality, specifically its unusual use of immutability. You need a separate init call for its servers because of that, and a very distinct use of the state management to work exactly the same way. But to achieve the same goals in other languages, you almost always shouldn't directly copy what Erlang is doing. In fact when I see "Look! I ported gen_server into $SOME_OTHER_LANGUAGE" and I see exactly and precisely the exact interface Erlang has, I know that the port doesn't deeply understand what Erlang is doing. When I ported the idea of supervisor trees into Go [1], I did so idiomatically. It turns out in modern Go the correct interface for "a thing that can be supervised" is not precisely the same signature that Erlang has, but type Service interface { Serve(context.Context) } That's all you need and all you should use... in Go. Your other language may vary. Go doesn't need a "handle_event/2" because it has channels, and you should use those, not because they are "better" or "worse" but because that's what this language does. In another language you may use something else. In another infrastructure you may end up sending things over Kafka or some cloud event bus rather than "calling a handle_event/2". The key is in building an event-based system, not copying the exact implementation Erlang has. A peculiar issue the Erlang community has is getting excessively convinced that there's something super-mega-special about the exact way Erlang does it, and that if you do it any other way it is ipso facto wrong and therefore not reliable. This may have been true in 2005; it is not true in 2025. Where once Erlang had almost the only sensible answer, in 2025 the problem is poking through the ocean of answers deluging us! While I recommend learning from Erlang about reliable software, I strongly recommend against just blind-porting out the exact way Erlang achieves it into any other language. It is in almost any other language context the wrong answer. Even other immutable languages generally vary enough that they can't just copy the same structure. [1]: https://jerf.org/iri/post/2930/ https://jerf.org/iri/post/2930/

asa400 2y ago
To follow on from your excellent post, I think a reasonable next question is, "why have these kinds of approaches and ideas in other languages and systems succeeded in gaining market adoption, but Erlang/Elixir has not?" This to me is the most interesting question about Erlang, and I say this as someone who works professionally in Elixir. It's _clear_ that there is incredible appetite for tools that help us design reliable concurrent systems given the wild success of things like k8s, Kafka, AWS's distributed systems products, etc., but why hasn't Erlang/Elixir been able to capture that share? My friends and I debate this all the time, but I don't know the answer.

senderista 2y ago

For me the most interesting concept in Erlang/BEAM is that partial recovery is built in from the ground up. When an unexpected state is encountered, instead of either killing the entire process or trying to proceed and risking corruption, you just roll back to a known good state, at the most granular level possible. This idea was researched many years ago under the name of "microreboots"(associated with "crash-only software"), but only Erlang/BEAM made it a first-class concept in a production system.

benmmurphy 2y ago
You still have to be careful with supervision trees and parts of the tree restarting. For example your system might work if the whole erlang operating system process is suddenly killed and restarted but your system might start corrupting data if parts of the erlang process tree is restarted. Erlang gives you a good model to work with these problems but it doesn't allow you to completely turn off your brain. If you walk in thinking that you can just let things restart and everything will be fine then you might end up getting burnt.
- juped 2y ago
  Yes, you can design your system pathologically to make it wrong.
- alienthrowaway 2y ago
  > You still have to be careful with supervision trees and parts of the tree restarting [...] Erlang gives you a good model to work with these problems but it doesn't allow you to completely turn off your brain. Erlang gives architects the tools to restart as little, or as much of the tree as they like, so I hope they have their brains fully engaged when working on the infrastructure that underlies their projects. For complex projects, it's vital think long and hard about state-interactions and sub-system dependencies, but the upside for Erlang is that this infrastructure is separated from sequential code via behaviors, and if the organization is big enough, the behaviors will be owned by a dedicated infrastructure team (or person) and consumed by product teams, with clear demarcations of responsibilities.
2y ago

Towaway69 2y ago

I've just gotten back into Erlang becuase of the lightweight processes and message passing, so far behaviour has been secondary (i.e. just learning about them)! The project is about bring visual Flow Based Programming(FBP)[1] to Erlang. FBP seems to be made for Erlang and I was surprised there was something already but there does not seem to be. My goto tool for FBP is Node-RED and hence the basic idea is to bolt a Node-RED frontend on to an Erlang backend and to have every node being a process. Node-REDs frontend is great for modelling message passing between nodes, hence there is a very simply one-to-one mapping to Erlangs processes and messages. I've implemented some basics and started to create some unit tests as flows to slowly build up functionality. I would really like this to be 100% compatiable to Node-RED the NodeJS backend. For more details, the github repo --> https://github.com/gorenje/erlang-red https://github.com/gorenje/erlang-red Overall Erlang is amazingly well suited to this and astonished that no one else has done anything like this - or have they? [1] = https://jpaulm.github.io/fbp/index.html https://jpaulm.github.io/fbp/index.html

__jonas 2y ago
Oh that's really cool to see! I always thought a visual programming language on the BEAM would be fun
mcintyre1994 2y ago
This is a really cool idea!
- Towaway69 2y ago
  Thank you, it's also a lot of fun to do :) Hopeful I can get some useful functionality together without hitting my Erlang coding limits! Any help is greatly appreciated :+1:
runlaszlorun 2y ago
Love the idea as well! Would I be wrong in thinking that, at a high-level, fbp is like erlang processes where message flow is one way?

hinkley 2y ago

I’ve worked with a few individuals, mostly managers, who intended to write books informed by our experiences. It was always frustrating for me to see that we disagreed about what aspects of our work made us successful. There was always something they minimized as being nice that I felt was essential. And here we see someone claiming that lightweight processes and message passing aren’t the secret sauce, missing that Erlang as Communicating Sequential Processes is indivisible from those qualities, and then repeatedly mentioning CSP as part of the secret sauce. Examples: > The application programmer writes sequential code, all concurrency is hidden away in the behaviour; > Easier for new team members to get started: business logic is sequential, similar structure that they might have seen before elsewhere; > Supervisors and the “let it crash” philosophy, appear to produce reliable systems. Joe uses the Ericsson AXD301 telephone switch example again (p. 191): Behaviors are interesting and solve a commonly encountered problem in the 80’s that was still being solved in some cases in the 00’s, but it’s a means as much as an end in Erlang. It’s how they implemented those other qualities. But I don’t know if they had to, to make Erlang still mostly be Erlang.

sitkack 2y ago
Managers make up their own narrative based on vibes.
silisili 2y ago
Is Erlang considered CSP? I've always thought it wasn't really, and had its own thing called 'actors' which are id'd and can communicate directly, vs CSP which are anonymous and use channel messaging. I've always thought the actor model made more sense, but highly YMMV.
- hinkley 2y ago
  The erlang docs only go as far as saying it’s functionally similar to CSP. I think the term Actor Model has been so semantically diluted at this point that the phrase also understates what Erlang has as well. Neither CSP nor AM require process isolation to work, which means they can work when they work but fail much much worse. They are necessary but insufficient.

Supersaiyan_IV 2y ago

"In February 1998 Erlang was banned for new product development within Ericsson" False statement. Ericsson still uses Erlang, for example in their MME. Source: I used to work at Ericsson.

bee_rider 2y ago
Is there any additional context here? (Is this a common misperception that you’ve come across?)
- macintux 2y ago
  Not the original commenter, but from hearing Joe Armstrong speak on multiple occasions my recollection is that Erlang was blacklisted for some length of time (which allowed it to become an open source language), but eventually the company realized it still offered significant value.
- sbuttgereit 2y ago
  I posted an excerpt in a different comment, but a decent source for this is section 5.2 of the document: https://lfe.io/papers/%5B2007%5D%20Armstrong%20-%20HOPL%20III%20A%20History%20of%20Erlang.pdf https://lfe.io/papers/%5B2007%5D%20Armstrong%20-%20HOPL%20II...
4ad 2y ago
It is simultaneously possible that Ericsson banned Erlang in 1998 (a statement claimed multiple times by the creators of Erlang) and that Ericsson rescinded the ban later in 2004, when they hired back Joe Armstrong.
jesperwe 2y ago
And there is a small team of Ericsson full time devs working on developing the language itself and the BEAM.
lysace 2y ago
My impression from Ericssonland: Around year 2008 being an Erlang coder was often more or less seen as being a COBOL coder in Sweden. Bluetail had sort of failed, having burned lots of VC, iirc. So Erlang was something weird and custom that Ericsson used to build software for legacy phone exchanges. I remember that a colleague's wife working at Ericsson had received on-the-job training from essentially zero programming knowledge to become an Erlang developer in order to maintain some phone exchange software. It's been fascinating to see it morph into something cool. Whatsapp, etc.

behnamoh 2y ago

Is it just me or does Erlang's syntax look a little bit nicer than Elixir's?

Capricorn2481 2y ago
I'm an outsider to this ecosystem, but I've seen a few people share that same opinion. They prefer the explicitness of Erlang.
- sodapopcan 2y ago
  Elixir is still very explicit from a syntactic point of view. Macros allow hiding a significant amount of boilerplate in certain behaviours, though. So it's a matter of preference, of course.
SoftTalker 2y ago
It's inspired/descended from Prolog, and my impression is that many people find it a bit odd. It is at first, but I quickly adjusted to it and quite like it now.
whalesalad 2y ago
gleam is probably my favorite middle ground between elixir and erlang.
bmitc 2y ago
Elixir came from Ruby developers and thus has similarly verbose syntax and macros. Erlang's syntax came from Prolog, which was used to implement the first compiler and is why Erlang's syntax is more concise.
ValtteriL 2y ago
I learned Erlang at school and used to prefer its syntax for years. However, after giving Elixir a chance and writing 1000 loc I was converted. Now I look at snippets of Erlang in docs with mild disgust.
pton_xd 2y ago

nailer 2y ago

"In 1998 Ericsson decided to ban all use of Erlang. The people responsible for developing it argued that if they were going to ban it, then they might as well open source it. Which Ericsson did and shortly after most of the team that created Erlang quit and started their own company." Bwahahaha. Reminds me of the JRuby team, who left Sun as a single unit and resumed work as a team at another company (I can't remember where) when Oracle acquired Sun.

deleted 2y ago

[deleted]

hk__2 2y ago

(2023)

layer8 2y ago

From this article and others, it’s still unclear to me what the state-handling and state-sharing model of Erlang is. Presumably, the granularity of the crashing/restarting sequential processes is also the granularity of in-memory state sharing. But what about external state, like databases, queues, file systems? For example, if a process has taken an item off a queue and then crashes before having fully processed it, how is that accounted for? Or you might not even know from the outside if it has been fully, partially, or not at all processed yet. This is an example where correct error handling or not crashing is crucial, in my experience. Or what about processing pipelines where a component in the middle crashes. Is there something like that in Erlang? Is there an article explaining Erlang from that perspective?

fidotron 2y ago
> For example, if a process has taken an item off a queue and then crashes before having fully processed it, how is that accounted for? I have worked with people that had deployed huge amounts on the BEAM that had a real problem with the answer to that, and resort to magical thinking. When erlang processes "crash", assuming the whole system didn't crash, they almost certainly alerted a monitoring process of the fact, so that a process can be quickly restarted. This is the core of how supervision trees in erlang are built. There are a lot of subtleties to that. The whole system may or may not be a single BEAM instance, and if more than one then they can be distributed, i.e. processes on one machine receive failure messages from processes on others, and can restart the processes elsewhere. These mechanisms on a practical basis are sufficient to automatically pick up the majority of transient failures. (I should add there are two classic ways to blow up a BEAM instance which make this less good than it should be: a bad C function call "NIF" for native something function, or posting messages to a process faster than it can consume them, which will eventually cause an OOM). But this differs from the underlying philosophy of the runtime, which is that things are only done when they're done, and you should expect failures at any time. This maps on to their messaging paradigm. What you actually sound like you want is a universe more like FoundationDB and QuiCK https://www.foundationdb.org/files/QuiCK.pdf https://www.foundationdb.org/files/QuiCK.pdf where the DB and worker queue all live in one single transactional space, which certainly makes reasoning about a lot of these things easier, but have nothing to do with erlang.

deleted 2y ago

[deleted]

whorleater 2y ago

One thing I haven't seen being discussed is the BEAM internals becoming a little long in the tooth. We still have static reductions before the scheduler switches to another task, the priority system in scheduling is a bit dodgy, flipping vmargs is kinda complex, lock counting and crash dump tooling kinda suck, etc. BEAM is great, although it's definitely missing something like pprof for go or java flight recorder.

spintin 2y ago

[dead]

geophile 2y ago

In 2003 I joined a startup building a horizontally scalable archive. You could add nodes to add capacity for storing data and metadata, and the system could tolerate up to a configured number of failures and carry on without loss of data or service. (This was not a general-purpose file system, it was for write-once/read-many objects.) We built the system in Java and C. The distribution layer was done completely in Java. It was only after the system was done that I discovered Erlang. I REALLY wish I had known about it earlier. Erlang solved so many of the problems we had to solve by ourselves.

DarkNova6 2y ago
Even these says, now that Java got Virtual Threads?
- geophile 2y ago
  I haven't been following recent Java releases, but unless Java has some equivalent to Erlang's OTP library (at a minimum), I stand by my statement

pkilgore 2y ago

Interesting. It strikes me that some of this rhymes with the platform abstraction of roc[1] [1] https://www.roc-lang.org/platforms https://www.roc-lang.org/platforms

jFriedensreich 2y ago

TLDR title for erlang people: erlang is not just erlang but erlang + OTP.

jiggawatts 2y ago

Someone explain to me why I should prefer Erlang/BEAM/Elixir over something like Akka.NET? With the latter I get a huge ecosystem of packages and wide compatibility with platforms and tooling and also a robust and scalable actor model. Learning Erlang or any related language meanwhile feels like learning Tolkien’s Elvish for the purposes of international trade.

neonsunset 2y ago
_Supposedly_ they are more convenient if you are willing to tolerate abysmally subpar efficiency, exotic semantics and lacking ecosystem.
dqv 2y ago
No, we can't explain to you why our blub language should be preferred to your blub language. It's your job to make that determination on your own. I can come back in 5 years to explain to you what is annoying about Akka.NET compared to the BEAM and vice versa. An expert in the BEAM who lacks experience in C# is not going to be able explain to an expert in C# who lacks experience in the BEAM why BEAM is better. You're asking for something incredibly rare - a person who is an expert in both runtimes and can concisely explain to you the tradeoffs of each.
- deleted 2y ago
  [deleted]
neonsunset 2y ago
If you want to do exclusively distributed computing at the application level - Erlang/Elixir will be better. They can offer nice Northstar of where the UX of Akka.net/Orleans should sit at (and, arguably, Orleans is not exactly nice to use in comparison). Otherwise, aside from educational purposes, they are not worth spending your time on. Just skip to F# over Elixir because Elixir is not a serious language, lacking base language primitives and operations one would expect standard library to offer. It's not productive nor fast.

travisgriggs 2y ago

To me, Erlang/Elixir’s power is not necessarily the Actor model implementation, the matching from prolog, immutability, behaviors, etc, but Joes desire to demonstrate you could do more with less. It is a well thought out and trued system of computation that has a consistency rarely witnessed in other languages, much less the “web”. It is not perfect. But it is pretty impressive. Unfortunately, I find the appreciation and uptake for what simplicity empowers in the software world pretty under appreciated. Complexity allows people to become specialists, managers to have big teams and lots of meetings, experts to stay experts. Erlang was being developed in a period where companies were trying to implement software solutions with smaller headcounts, limited horsepower, etc. A multi decade outpouring of cash into the domain has made the value of “less will mean more for all of us in good ways” less of an attractor.

zelphirkalt 2y ago
Reminds me of Rich Hickey's talk about Simple VS Easy.
runlaszlorun 2y ago
You've just convinced me to spend some more time with Erlang! I've dabbled a bit and, at least on the surface, prefer erlang syntax over elixir.
- LtdJorge 2y ago
  Me too, as weird as it might sound
  - travisgriggs 2y ago
    I mostly prefer the Elixir syntax. But I don’t care for all of the “ends”. Wish they had taken more inspiration from Python in that department.
    - mg74 2y ago
      And I am grateful for the ends, wish Python had them as well. Python already marks the start of every block with a :, one more keyword and the semantic whitespace madness could end.

HeavyRain266 2y ago

Erlang, OTP, and the BEAM offer much more than just behaviours. The VM is similar to a virtual kernel with supervisor, isolated processes, and distributed mode that treats multiple (physical or virtual) machines as a single pool of resources. OTP provides numerous useful modes, such as Mnesia (database) and atomic counters/ETS tables (for caching), among others. The runtime also supports bytecode hot-reloading, a feature used to apply patches without any system downtime. While the syntax is not very screen reader-friendly, it is digestable. Apache Mesos[1] is the only thing that comes to my mind as a similar platform to BEAM in its ability to treat multi-machine resources as a single pool. Over a year ago, my private consulting company decided to adopt Erlang as our backend language. After some time, we started exploring BEAM's internals to, for example, replace the TCP-based stack with QUIC and integrate some Rust patches. A truly fantastic choice for lightweight and high-throughput systems that are only failing in case of kernel panic or power loss. We are currently working on very "busy", concurrent software like a film/game production tracker and pipeline manager, and are now also preparing R&D for a private hospital management services. [1]: https://mesos.apache.org/ https://mesos.apache.org/

HeavyRain266 2y ago
Before you ask, we're not going to ever fully adopt Elixir (or Gleam) as its ecosystem is built around Phoenix framework and external services/databases. We would have to maintain internal bindings/implementations of things that are unmaintained on Elixir's side. Also worth to mention that it has a large amount of syntax sugar and its users have that weird fetish for abstracting stuff into DSL interfaces.
- deleted 2y ago
  [deleted]
- pdimitar 2y ago
  Couldn't understand your comment well but I am making a SQLite library for Elixir (via Rust bindings) so that would be one less dependency on external systems. I happen to believe that most projects don't need a full-blown database server.

throwawaymaths 2y ago

I don't think behaviours are all that interesting; after all other programming languages have them. Rather, what is interesting about the BEAM is that throwing an error is so graceful that it's not such a sin to just throw an error. In otherwords, a component that CAN error or get into a weird state can be shoved into a behaviour that CANNOT. And by default you are safe from certain operational errors becoming logic or business errors. For example. You might have a defined "get" interface that doesn't return an error -- let's say it starts as an in-memory K/V store and it returns an optional(value), which is NULL in the case that the key didn't exist. But suppose you want to have two datastores that the same interface targets, so you might abstract that to a filesystem, and you could have a permission error. And returning "NULL" is not actually "correct". You should throw, because that bubbles up the error to ops teams instead of swallowing it whole. A panic in this case is probably fine. What if now you're going over a filesystem that's over the network, and the line to the datacenter was backhoe'd and there was a 10 millisecond failover by your SDN -- returning "NULL" is really not correct, because consumers of your getter are liable to have a bad time managing real consistency business cases that could cost $$$. And in this case a panic is not necessarily great, because you bring down everything over a minor hiccup. The other power with throwing errors + behaviors is that it makes trapping errors with contextual information reporting (e.g. a user-bound 500 error with stack trace information sent somewhere where ops can take a gander) really easy and generically composable, that's not so for error monads or panics. Anyways it was always strange to me that erlang-inspired actor system programming languages came out that obsessed over "never having errors" as a principle (like ponylang) because that's throwing out a big part of erlang.

deleted 2y ago
[deleted]

dang 2y ago

Discussed at the time: Erlang's not about lightweight processes and message passing - https://news.ycombinator.com/item?id=34545061 https://news.ycombinator.com/item?id=34545061 - Jan 2023 (274 comments)

cmdrk 2y ago

Erlang is my favorite language but getting a job writing Erlang feels impossible. I make it a habit to ctrl-F every Who’s Hiring? thread and find Elixir occasionally and Erlang never.

gavmor 2y ago
Can you articulate the kinds of business problems Erlang is particularly well-suited to solve? When you choose Erlang for a project, what kind of return on investment do you think it typically offers? Does it lead to significant cost savings or help generate more revenue in ways that other languages might not? In situations where Erlang is chosen, what are some concrete examples of how it has demonstrably increased efficiency, reduced errors, or enabled new business opportunities that wouldn't have been as feasible with other technologies? Edit: I guess if I'd done any research myself before asking, I might've found this: https://www.erlang-solutions.com/blog/which-companies-are-using-erlang-and-why-mytopdogstatus/ https://www.erlang-solutions.com/blog/which-companies-are-us...

unoti 2y ago

I came here looking for information about why Ericsson stopped using Erlang, and for more information about Joe's firing. The short answer seems to be that they pivoted to Java for new projects, which marginalized Erlang. Then Joe and colleagues formed Bluetail in 1998. They were bought by Nortel. Nortel was a telecom giant forming about a third of the value of the Toronto Stock Exchange. In 2000 Nortel's stock reached $125 per share, but by 2002 the stock had gone down to less than $1. This was all part of the dot com crash, and Nortel was hit particularly hard because of the dot com bubble burst corresponding with a big downturn in telecom spending. It seems safe to look at Joe's layoff as more of a "his unit was the first to slip beneath the waves on a sinking ship" situation, as they laid off 60,000 employees or more than two thirds of their workforce. The layoff was not a sign that he may not have been pulling his weight. It was part of a big move of desperation not to be taken as a sign of the ineffectiveness of that business unit.

cmrdporcupine 2y ago
It's very weird to me to see the word "fired" in this context. "Laid off" is more appropriate. "Fired" is very value-laden and implies fault and termination with cause. Which I'm sure if that was somehow actually true the original article author would know nothing about, nor would it be any of their business.

spott 2y ago

A question about erlang: Haskell taught me a lot about programming, things that I still use now, even though I only write Python. Does learning erlang teach you a new way of thinking? Or does it just make you wish you had erlang language features and libraries when not writing erlang?

lgas 2y ago
IMHO it will teach you a new way of thinking but that way is not as generally applicable as what most people take away from Haskell.

kerkeslager 2y ago

I think if you ever find yourself saying Erlang is about one thing, you've lost the plot. Erlang has a bunch of features that each would be powerful in their own right, but they aren't special--there are other ways to solve the same problems, and some of those ways even have merit. But Erlang chose the set of features they did with a very high level understanding of how those varous features would interact. I honestly think it's one of the greater achievements humanity has accomplished. It lacks the obvious elegance that, say, Lisp, has, but I think the sum of parts in Erlang was probably a lot harder to make.

ggm 2y ago

Only ever played with ejabberd but felt it was a marvel of high speed engineering. I also felt it had strong "halt and catch fire on error" properties. Am I maligning it, and Erlang?

kamma4434 2y ago

I do not agree with the contents of this article. Behaviors are possible because of the underlying architecture of the system. Behaviors are not interfaces – they are more like abstract object in a language like Java so they implement basic and self-contained functionalities that are hidden behind a collaboration interface, but they couldn’t do much without the underlying infrastructure that makes sure that every process is totally separate from other processes, the that all processes can be safely closed without leaking memory, or resources, and that you cannot just share a rogue pointer between two different processes. What Joe did in his thesis is to show you how you can build reliable systems (and up point, reliable distribuited systems) by using a given set of Lego blocks. The reason why you need the erlang vm to implement something like that appropriately – and that you cannot do that fully on a different VM – is that without the underlying plumbing, supervision trees would be leaky - in Java, you cannot kill a thread that is holding up to resources and hope that everything will always go well, And do not have ways to monitor different processes.

coldtea 2y ago

Haven't digged deep into Erlang, but from what I read, I never thought Erlang's big idea is "lightweight processes and message passing". Always thought the key differentiator is having a kind of "orchestrator/supervisor" for those processes. (Which "lightweight processes and message passing" facilitates, sure, but it's more than those)

LtdJorge 2y ago

To me the most important aspect of Erlang is the runtime's scheduler, which is preemptive instead of cooperative. This allows the message passing, sequential code and lightweight processes to be much more effective than in any other general language or framework using cooperative scheduling (like async runtimes or coroutines in Rust, .Net, Kotlin, Lua). You can write actually synchronous code in Erlang and the runtime makes it so that no process blocks any other process by preempting them on a schedule.

assbuttbuttass 2y ago
Sounds a lot like Go
- LtdJorge 2y ago
  Yes, goroutines are preemptive, too. And they took a lot of inspiration from Erlang and the BEAM when they designed goroutines and channels. In Go you manage your goroutines and channels explicitly, while the BEAM runs all processes for you, and I've seen Robert Virding run an infinite loop in one Erlang process while the rest were serving requests, the core with the loop stayed at 100% but 0 requests were dropped and the latency and throughput was more or less the same, pretty crazy capabilities. You can do the same in Go but it's a lot more manual.

ksec 2y ago

Does anyone knows if Whatsapp is still running on Erlang? I know it used to be on FreeBSD but when Meta bought them they switched to Linux. Not sure if they switched away from Erlang as well.