25 ms·

Erlang's not about lightweight processes and message passing

543 points by stevan 4y ago

notamy 4y ago
> Please don't share this repo or any of the linked to documents or repos just yet! Oh no :P Very interesting read though, thanks for sharing!
- pietroppeter 4y ago
  hopefully stevan is same as stevana from github
- boomskats 4y ago
  When I started my first job at a grad scheme some 15 years ago, on the first day, we were asked to introduce ourselves to the group and say what name we'd prefer to be called by. This one guy says: "My name is XXX XXXXXXX. I don't care what you call me, just don't call me Ziggy... I HATE being called Ziggy". I heard Ziggy still works there.
  - sriram_sun 4y ago
    This is not the first time I'm reading about Ziggy!
  - giancarlostoro 4y ago
    Wonder if he did it intentionally or did not realize when he opened his mouth and said that.
  - Pet_Ant 4y ago
    I'm guessing that guys real name was "Zbigniew"
  - layer8 4y ago
    > I heard Ziggy still works there. He can’t have hated it that much, then. ;)
  - pnt12 4y ago
    This reminds of lord of the flies: I'm not a native English speaker, and only read it recently. When Piggy says he doesn't want to be called Piggy, I figured that's what he would be called the whole book. Alas, he was, and I cannot remember his real name.
dmitriid 4y ago
The main insight is in Joe Armstrong's thesis. "Making reliable distributed systems in the presence of software errors" https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf (I think the original title was "In the presence of hardware errors", emphasis mine) All the other things flow from that thesis and understanding. You can recreate behaviours described in the repo doc using Erlang primitives very easily, and they are very hard to recreate in pretty much any other language. Because Erlang is very much literally about lightweight processes and message passing. Only: - every part of the system knows about lightweight processes and messages - every part of the system is engineered around them - the underlying VM is not only re-entrant in almost every single function, it's also extremely resilient, and can almost guarantee that when something dies, only that process dies, and the rest of the system receives a guaranteed notification that this process dies There are more, but without at least that you can't really recreate Erlang's standard behaviours. For example, you can't recreate this in Go or Rust because `panic` will kill your entire program unconditionally.
- ThemalSpan 4y ago
  I agree up to the last point. You can catch panics at the thread level right? I mean, more generally, what is Erlang implemented in?
  - Arch-TK 4y ago
    Erlang is implemented on top of BEAM which has its own scheduler and schedules its own idea of lightweight processes.
    - dolmen 4y ago
      It looks like features that the Go runtime provides.
  - ModernMech 4y ago
    Of course you could implement a language like Erlang in Rust, but I think the point is that you would have to do exactly that in order to do in Rust what Erlang does at the language level.
worthless-trash 4y ago
One thing I like about "behaviours" is that I can immediately recognise, discard boiler plate, skim quickly, and see the difference from expected pattern just by knowing the behaviour that the process implements.
practal 4y ago
Very interesting read. Also very timely for me (I am always amazed how HN often has posts that resonate strongly with what I am currently doing), as I am just now designing a programming language based on a generalisation of Algebra, which I call Abstraction Algebra. I think there is a strong connection to this post: Behaviours are just an abstraction algebra you program against. Switch out the algebra implementation against another one, and you can adapt the same program to different scenarios without changing the program itself.
- what-no-tests 4y ago
  > I am just now designing a programming language based on a generalisation of Algebra, which I call Abstraction Algebra. I admire your ambition!
  - practal 4y ago
    To be honest, I am quite amazed by how nicely all the pieces are coming together now. It all just feels right, it feels like collecting fruit from under a huge apple tree.
- survirtual 4y ago
  I always thought it would be neat to have a system that takes math symbols / formulas / functions etc in the notation of a mathematician, and automatically generates highly performant implementations — sort of like Taichi but instead of using python, you just use mathematics. Seems to me all the tools are available to accomplish this effectively now days. Good luck on your project, in any case.
  - practal 4y ago
    Thank you! Yes, a large part of programming will be just a special case of doing mathematics.
- samsquire 4y ago
  I am also designing a language based on algebra. https://GitHub.com/samsquire/algebralang https://GitHub.com/samsquire/algebralang It's based on the idea there are relations between variables and every function is a concurrent process.
adamddev1 4y ago
> In 1998 Ericsson decided to ban all use of Erlang. Does anyone know why?
- doikor 4y ago
  Easier to find C/C++ devs according to one friend who used to work there in early 2000s.
  - bluGill 4y ago
    That is a cop out. You always need to train your people. I've been doing C++ for 20 years, and consider myself somewhat and expert, but if I join your team I will still need to learn a lot of things that are just how your team does it. If I also need to learn Erlang/java/go/ (pick any language I haven't used much) that only adds a short time. Even as complex as C++ is, I can train a great programmer C++ a lot faster than I can train a future great junior C++ programmer to be a great programmer. Yes you will encounter the rough edges of whatever language often in the first 5 years, but a great programmer will be great in any language quickly. You need one expert in the language on the team for the weird complex stuff, but most code isn't that complex.
    - sillysaurusx 4y ago
      I mean… sort of. It’d be like trying to write an ML stack in lisp circa 2023. Sure, you can find devs willing to learn it, but it’ll be at least two orders or magnitude easier to find python devs. I love lisp, but the thought of being forced to use it exclusively for ML makes me uneasy. And I’ve tried building lisp ML stacks. :)
      - sitkack 4y ago
        I’d probably use Hy and JAX, :) Should be beautiful.
        sillysaurusx 4y ago
peoplefromibiza 4y ago
Erlang isn't, but the BEAM is.
- rhodin 4y ago
  Yes came to say this. The Erlang VM is about lightweight processes and message passing.
mapcars 4y ago
As someone who worked with Erlang for a few of years I still wonder if using parallel processing in program top level is worth it or not. Because there are many ways to process data efficiently using queues, pipelines, etc and being clear on when it happens rather than a "wild west" of Erlang where you need to manage processes, links, restarts, and it's harder to focus on the business logic.
- stevan 4y ago
  Do I understand you correctly in that you'd like more structure? E.g. that you can only deploy an `application` (= supervisor tree)?
  - mapcars 4y ago
    I'm talking about development, deployment should be automated anyways. As a developer I think it's easier to think in terms of how to send data to hadoop, sqs queue etc for processing and read results later than keep in mind the supervision tree, messages, mailbox size, linking and so on. And the "processing" side can be as well implemented in Erlang just I don't feel Erlang's features are needed in the "top level development" and create more problems and barriers.
- Kototama 4y ago
  What do you mean with "process data efficiently"? You speak about performance? But the main goal of the design of Erlang is fault tolerance, not performance. You can still use queues or whatever in Erlang if you want, but having a fault-tolerant system with process tree and process supervision in other languages is a different story.
  - mapcars 4y ago
    Right, I was mainly describing arguments for parallel processing which Erlang is also famous for. In terms of being fault-tolerant I think the modern approach with (micro)services is quite similar, one can have multiple services running and communicating using something like protobuf, having restart strategies, fallbacks and so on. From my experience Erlang doesn't offer any killer features in this case, does it?
natdempk 4y ago
Relevant, I’ve written a bit about the various actor models here including Erlang under process-based actors: http://dist-prog-book.com/chapter/3/message-passing.html http://dist-prog-book.com/chapter/3/message-passing.html
- styluss 4y ago
  opening the root page http://dist-prog-book.com/ http://dist-prog-book.com/ seems to load Tufte CSS documentation
valenterry 4y ago
> The application programmer writes sequential code, all concurrency is hidden away in the behaviour; Yeah, until the business problem itself involves inherent concurrency, which usually happens much faster than people think. Or until I, as the non-expert, want to dig in to make changes or debug a problem. This distinction into "expert" and "lowlife (SCNR) using the expert abstractions" is really one that doesn't hold in practice most of the time. I think it's much better to embrace that concurrency is a cross-cutting-concern and reality is that it can happen on any level, so the language should better support reality.
- jeremyjh 4y ago
  I think you're talking about two different things. When they say concurrency is hidden away, I think they mean the programmer isn't dealing directly with locks/mutexes. They are also lying, since async functionality like `handle_cast` is built into the `gen_server` behaviour and every caller is calling either `gen_server:cast` or `gen_server:call` depending on whether or not it is going to block on a response. Whenever you invoke async behaviour concurrency will invade all the domain logic and can't be ignored.
JustSomeNobody 4y ago
I get that this is probably written for people familiar with Erlang, but a quick suggestion if I may: In the Behaviors section you talk about behaviors being similar to interfaces in Go and give a Go example. But then you switch examples for Erlang. Maybe show the Joe/Mike example written in Erlang and then say, here's a more complicated example (the key-value example) that really describes behaviors better.
jerf 4y ago
There is a very common pattern in the world where people conflate goals with results. Of course, when I say that, it's obvious that the two things aren't the same, but the unexamined assumption that they are the same sneaks in anyhow when you aren't looking, a cognitive shortcut hard to catch yourself making and harder yet to get in front of and deal with. In the light of this statement, the answer to what I think is the thesis question of that entire piece: "This begs the question: why aren't language and library designers stealing the structure behind Erlang's behaviours, rather than copying the ideas of lightweight processes and message passing?" Is that while Erlang has a lot of good goals, the results of how they got there are simply not the state of the art. Or, to put it another way, language designers are not copying Erlang, and they are correct to not copy Erlang. I respect Erlang a lot. They were a good 10-15 years ahead of their time. However, you will note that if you add 10-15 years to the creation date of Erlang, you still end up in the past. If Erlang were to come out today, fresh, nobody had seen it before, into an otherwise identical programming language environment, I would say it makes several mistakes. One I've written about before is that Erlang is a non-Algol language for no reason: https://news.ycombinator.com/item?id=7277957 https://news.ycombinator.com/item?id=7277957 (Note the library mentioned in that post, suture, is now mature, and I use it all the time. It works for what I need.) But in the context of this post, that's less critical. The other major mistake I'd say it made if it came out in 2023 is that it is a totalizing environment. By that I mean that it has this built in implicit assumption that it is responsible for all the reliability in the system, and you don't get Erlang's features very conveniently if you don't use it as the complete control backplane for your entire system. You run an Erlang cluster, and it bundles all the message passing, restarting, reliability, cluster management, software deploy, and everything into one system. But for the world we live in today, that's a price not worth paying. We don't need the Erlang message bus to be the only message bus. The Erlang message bus is, frankly, not very good, and it's actively terrible if you want to use it for one non-Erlang process to communicate to another. We don't need the Erlang message bus. We have a dozen message busses, some in the cloud, some commercial, some open source, some that double as retention (Kafka), all of which scale better, none of which tie you to Erlang's funky data types. And then, within the matrix of those message busses, we don't need Erlang's restart capability. We have an abundance of ways to restart processes, from systemd, to kubernetes, to any number of other ways. We don't need Erlang clusters for redundancy any more. You just run multiple copies of a service against the message bus, on multiple systems for redundancy. We don't need Erlang's behaviors. We have interfaces, traits, object orientation, and even just pushing that entire problem up to the OS process level, or writing a cloud function, and any number of ways of achieving the same goal. Erlang's software deploy is interesting, but we have a lot of options for it. The whole attempt to do live updates is interesting, but it also imposed a lot of constraints that systems that don't have that need, which is the vast majority of them, don't need or want. This is perhaps the space where the state of the art isn't that far ahead of Erlang. It's still a mess, despite all the churn in this space. But even so, with all the options available, you can probably find something better for your system than the Erlang way of upgrading software, even if it isn't necessarily much easier. The cognitive hazard that Erlang presents the community in 2023 is that it has some very good writing on the topic of reliability and its other goals, and then, naturally, one segues into the discussion of how Erlang solved the problem. And it was a very interesting solution for the time. I used Erlang for many, many years back when it was effectively the only solution to these problems. But it isn't the only solution anymore. The space has exploded with options. Unsurprisingly, the ones that a highly innovative pioneer tried out first are not the best, or the only. They chose well. Let me again emphasize my respect for the project. But it's not on the cutting edge anymore. Granted, the diversity of options does mean the real world has gotten quite chaotic, where you may have three message busses attaching systems implemented in a dozen different languages, but that's something you can take up with Conway's Law. Erlang couldn't work with Conway's Law without totally converting your entire company to it, which just isn't going to happen. The reason why language designers aren't rushing to copy Erlang is that what was excellent and amazing in 2000 (and, again let me underline, I mean that very seriously, it was a cutting edge platform built with a lot of vision and moxie) is, in 2023, mediocre. Erlang is a mediocre language (Elixir is at least "good", Erlang is mediocre), attached to a mediocre message bus, with a type system that doesn't even reach mediocre, with a mediocre totalizing approach to system design where there's a very significant impedence mismatch between it and the rest of the world, with an at-par-at-best VM (I won't call that mediocre, but where it used to be head-and-shoulders above everything else in certain ways, it is now merely competitive), with mediocre standard libraries, and a mediocre product fit to its own stated goals. It just isn't the best any more. The state of the art right now is super chaotic. I can hardly get two systems deployed on the same infrastructure any more, because there's always some reason something in that list has changed. But when the chaos settles and best practices emerge, something that I'd say is at least a good 5 years away, the result will clearly have Erlang inspiration in it for sure... but it won't look a lot like Erlang on the surface. What is worth copying has largely been copied. It doesn't look exactly like Erlang, but this turns out to be a good thing.
revskill 4y ago
What i learn from Erlang, is, naming is hard. What does `gen` mean in `gen_server` ? Thanks for replies for "Generic". So this article is not well organized in that case. With some simple naming explanation, it should be obvious to guess what the system does. Abbreviation doesn't save the writings. Some more examples, in Ruby, instead of calling `implement Module`, it uses `include Module` . I do think include is not as clear as implement.
- mtremsal 4y ago
  generic, I believe. i.e. it’s an interface that has to be implemented.
  - sodapopcan 4y ago
    Yep, “generic” it is.
- deleted 4y ago
  [deleted]
- dolmen 4y ago
  generic?
- lopatin 4y ago
  I always thought it was “generate” and that something about the implementation/immutableness required the server to be “generated” at runtime. I never did any actual Erlang programming, it’s just how my brain plugged in the gaps when I heard about all the gen stuff.
- mabbo 4y ago
  If they called it 'generic_server' then people would make fun of it for being too verbose. If they call it 'gen_server' then everyone pretends they know what it means and stays quiet.
  - edgyquant 4y ago
    gen_server to me is “generate a server”
xwowsersx 4y ago
> In 1998 Ericsson decided to ban all use of Erlang. What's the story there? Why did they decide to ban it? EDIT: doh, this has been answered elsewhere in this thread.
crad 4y ago
Erlang vs OTP
sb8244 4y ago
> just like the concurrent code of gen_server I found this a really interesting read, but this stuck out because it doesn't jive with my mental model of gen_server. gen_server is fully serialized. Even the code underpinning it is not concurrent. Now I guess gen_server does expose some top level functions to simplify sending/receiving a message into the process, but the process itself is serial. And this is part of the genius of gen_server to me. You don't need to think about your state concurrently because it processes a single message at a time. Your system executes concurrently, but the individual components do not. Maybe that is what the post means and I misinterpreted it.
- wefarrell 4y ago
  Yeah they mention that in 2 of the 6 points arguing in favor of behaviors: 2. The application programmer writes sequential code, all concurrency is hidden away in the behaviour; 4. Easier for new team members to get started: business logic is sequential, similar structure that they might have seen before elsewhere;
  - sb8244 4y ago
    Thanks for that. My thought is probably just semantics then. I think it's even more nuanced, but doesn't really matter for this type of post. (I am actively writing a chapter exploring this topic, so is has been top of mind.)
  - toast0 4y ago
    Adding on to sb8244's comments; writing sequential code is very nice, but it's not the gen_server behavior that gets you that. It's the lightweight processes. gen_server is useful, but it's not much more than receive in a tail recursive loop and conventions around messages, most specifically about if the sender expects to receive a reply or not. It's not magic, and it's written in clear Erlang that anyone with a week of Erlang fiddling can understand (which kind of is magic; almost everything in OTP is clearly readable) Concurrency comes from running many processes each with their own message queue.
- btbuildem 4y ago
amelius 4y ago
Isn't Erlang's model a giant ball of microservices?
- lupire 4y ago
  Nanoservices.
throwawaymaths 4y ago
Wait till this person learns about links, monitors, call's default automatic backpressure mechanism, etc...
hosh 4y ago
I think it is misleading to frame this as "behavior". I think what we really have are battle-tested concurrency patterns, which then uses the language feature called "behavior" to implement. This collection of battle-tested concurrency patterns that has been in use for a long time, in a variety of settings, is called OTP. It's why you'll hear long-time Erlang folks talk about how, it's really about OTP. You can implement those same patterns, if you also have lightweight processes, messages, and message queues. There's a Rust library that aims to do exactly that. The new WASM runtime (Firefly, formerly known as Lumen) aims to bring those patterns into the WASM ecosystem, potentially using this kind of concurrency pattern client-side. What's potentially novel is adding in newer concurrency patterns that are in use in the Kubernetes community -- use of selectors, services, scaling with replicasets -- so that, instead of a static supervision tree, it can be a dynamic supervision tree, spread out across the entire pool, even as that pool expands or shrinks.
- imachine1980_ 4y ago
  Please, can you share the rust library??
  - linkdd 4y ago
    Not sure this is what GP is talking about but to implement the actor model in https://letlang.dev https://letlang.dev I use tokio.
  - hosh 4y ago
    I don’t remember the name and I don’t remember if it is tokio or something else. However, I did find this: https://docs.rs/genserver/latest/genserver/ https://docs.rs/genserver/latest/genserver/ I will keep looking. Keep in mind, BEAM uses a preemptive actor rather than async, so it avoids the “storms” that can happen when an async reactor runs out of resources. EDIT: but I guess Tokio is capable of preemption through work-stealing too?
    - hosh 4y ago
      This looks interesting too, though no OTP out-of-the-box: https://docs.rs/axiom/latest/axiom/ https://docs.rs/axiom/latest/axiom/
brudgers 4y ago
I agree with the headline because uptime was the goal of Erlang. Lightweight processes using message passing is how Erlang stays up (and perhaps ironically, let-it-crash is how Erlang avoids going down). Lightweight processes and message passing are the architectural decisions that address the business problem Erlang was designed to solve.
peter_d_sherman 4y ago
>"The next interesting behavior is supervisor. Supervisors are processes which sole job is to make sure that other processes are healthy and doing their job. If a supervised process fails then the supervisor can restart it according to some predefined strategy." This is an interesting software pattern -- especially applied to having it entirely supported entirely inside of one programming language... We see this software pattern in such things as Windows Services (they restart automatically if they fail), Unix/Linux daemons, as one of the original purposes of the original Unix 'init' process, and in high-availability, mission-critical 24/7 systems (databases, etc.). Having all of that fault-tolerant infrastructure entirely inside of a programming language is indeed, somewhat novel. But let's work up what a given language must have as prerequisites to support this... First, we need the ability to run multiple processes inside of a language. Many languages can accomplish this with threads, but of the languages that do this, many need additional cumbersome programming to guarantee thread safety... So the language needs to support the notion of a threaded process inside of that code -- without having to code extra to support this threaded process. Next, the language needs some form of communication between supervisor (process A) and supervised (process B) code. Message passing is the solution -- but that requires each process to have its own message queue. And message passing requires interfaces... Here we're starting to sound like we're duplicating a mini Operating System inside a language(!) (should the language have pipes, too?) -- and/or that the complexity of most OS's would be deeply mitigated by designing them inside of a language that supported these constructs... Whatever the case, I think it's a fascinating software pattern. Yes, Go exists and supports features like this, Rust exists, and people are using it to write Operating Systems. And there are probably all other kinds of languages (both existing and existing in the future) that do/will support some form of these constructs... But there's another interesting, purely academic, purely theoretical question here... That is: What is the simplest full-featured OS (processes, IPC, synchronization primitives, memory management, hardware resource management, syscalls, scheduling, etc.) that could be written in the smallest amount of lines of code if the language it was written in had intrinsic knowledge of all of those underlying constructs? Anyway, a fascinating article about Erlang, and it definitely gave the impression that Erlang was a whole lot more than I thought it was... I will be checking out Erlang for future projects!
- 4y ago
jb3689 4y ago
To me the real amazing thing about Erlang is more the fact that these patterns based on queues and messages and supervisors scale so well. They are easy to implement on a small, single node application. You can compose them. You don't need to complete reauthor your calls to start distributing them and fanning out responsibility across your nodes because you've been doing RPCs from the start. A multi-node application is not that much more difficult to reasonable than a single-node application
finder83 4y ago
There's a component that seems to be missing here which is preemptive task scheduling. I've not seen another non-OS system do it like the BEAM VM (the VM behind Erlang), though there may be something I'm not aware of. It really prevents a whole class of concurrency issues where a hung process can freeze or slow down the entire system. For example, if you recreated gen_server in a cooperative concurrency environment, one gen_server could use up all of the CPU and have an impact on the performance of the rest of the system. Maybe the other threads (microthreads, not OS threads) would still respond in <500ms, but if every request takes 500ms when they normally take 15ms you could essentially have outage-like conditions, particularly with upstream timeouts. Instead, because BEAM is preemptive that one (or 10) hung gen_server doesn't hang up everything else on a node. Sure at some point performance will degrade, but that point is much further down the line than in cooperative concurrency models. There was a fantastic talk by Sasa Juric that demonstrates this in Erlang. [1] Otherwise you run a higher risk of even the supervisors being starved for CPU time, particularly if you are launching hundreds of processes that are all locking the CPU. It's really the combination of the behaviors (OTP), the scheduler, lightweight threads, message passing, and immutability to me that makes the Erlang (Elixir for me) concurrency model so appealing. Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time. [1] https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE
- Fire-Dragon-DoL 4y ago
  If I'm not wrong, Go added uncooperative task scheduling that behaves similarly?
  - finder83 4y ago
    Interesting, at least when I was using it the goroutines themselves were cooperative, and of course the OS threads were preemptive when GO_MAXPROCS > 1. Not finding much with a search, there was a proposal to make them preemptive. Curious if others will chime in that it's now preemptive. Even so I like Elixir better as a language, but the processing speed of Go with a preemptive scheduler would be tempting for some use cases.
didip 4y ago
These days, if you have Kubernetes, queues, and worker queues, does OTP still have an edge?
- bsaul 4y ago
  I believe the only argument left is that erlang /OTP provide all of those in a single cohesive environment.
- deleted 4y ago
  [deleted]
- sb8244 4y ago
  Yes. Being able to do this at the programming language level is extremely powerful and creates an entirely different way of building applications. My go-to analogy is building a city (lots of individual, isolated, separate stack processes) instead of a big skyscraper (deep stack requests, concurrency difficult, error recovery manual). You can build that city in a limited way with k8s, but there's way more overhead along the way, to the point of it not being enjoyable for me.
- weatherlight 4y ago
  Yes because concurrency and fault tolerance are literally at the core of the the BEAM. you get all the above with zero configuration, all through convention and you only need to learn one set of tools. When some new container technology comes along, or new worker queue platform etc. Erlang/the beam will be exactly the same. One of the advantages of working with Erlang I found is that it's just as easy to scale down as it is to scale up. How does one scale down from Kafka once you have vendor lock in with them?
ComputerGuru 4y ago
I don't mean to disparage the article (quite the contrary, it's quite the eye opener if you're not accustomed to architecting your code in this fashion) but choosing go of all languages to mockup the ideas was a terrible choice. It's syntax and type system are really quite the red-headed stepchild and for anyone unfamiliar with its syntax, its incredibly hard to make heads or tails of. (And I say this as someone that's at least passably familiar with go and have worked on/contributed to golang projects in the past.)
- rahilb 4y ago
  As far as I can tell the article contains only erlang snippets directly quoted from the paper reviewed.
- preseinger 4y ago
  I think you may be alone in this assessment. Go is uniquely easy to understand, due to its small set of keywords and lack of esoteric e.g. sigils.
  - weatherlight 4y ago
    Yeah, I had no idea how to read the Go code. I'm also coming from a ruby/js/ocaml/elixir/erlang background.
    - preseinger 4y ago
      The Go code, in its entirety: type HasName interface { Name() string } func Greet(n HasName) { fmt.Printf("Hello %s!\n", n.Name()) } type Joe struct {} func (_ *Joe) Name() string { return "Joe" } type Mike struct {} func (_ *Mike) Name() string { return "Mike" } func main() { joe := &Joe{} mike := &Mike{} Greet(mike) Greet(joe) } You have no idea how to read this?
      - weatherlight 4y ago
lippingoff 4y ago
This is fucking cool, thanks for sharing
eternalban 4y ago
J2EE could have had all that but it was missing an OTP. [Java got the competing 'application server' vendors competing themselves out of a massive market.] Components with well defined life cycles, request/reply and message processing, dynamic remote interface binding, isolated execution environment, pure single threaded application level code ('business logic'), declarative composition ... that's all in the specs. Erlang is mainly about OTP. OTP delivered an opionated take on distributed components -- think of Erlang on OTP as Ruby on Rails -- and did it exceptionally well. But one could still do this with Java btw. The specs are still there and they are solid.
jFriedensreich 4y ago
It makes me sentimental to see some erlang code, truly miss writing erlang, as its probably the closest to how i think about systems. But it is so hard to justify using it when most of my problems can be solved with javascript edge workers. These days it's rare for me to even come across a backend heavy project, yet something that would fit perfect to erlang. Does anyone else fell this way or is it because of my drift in the industry? When was the last project you started and thought this is perfect to solve with erlang?
- noloblo 4y ago
  We need backend heavy erlang prototype
  - yurishimo 4y ago
    Arguably that’s what Elixir is. They’re really trying to push Phoenix as a batteries included full stack solution. I know it’s not exactly Erlang, but the interfaces for things like gen_server are nearly identical. It’s effectively Erlang with a more modern syntax.
    - throwawaymaths 4y ago
      Its also got much better tooling, and a few features that don't exist in Erlang yet (Tasks - and no, tasks are not just proc_lib, they're actually special.
lippingoff 4y ago
This fucking rocks, great work and thanks for sharing.
samsquire 4y ago
I wrote a multithreaded message passing system in Java and I was trying to get async await to work The thing I want to define with behaviours is observable effects on objects. Such as interactions between tasks and objects between them. Not necessarily method calls but state interactions across multiple objects. Some objects should be colocated on threads and send behaviours to arbitrary groups of objects. Think of it as a collection that responds to events. I wrote a state machine serialization that looks like this. This defined a state progression between threads and async/await state machine. I think organised state machines would mean actor programming is so much easier to program. next_free_thread = 2 task(A) thread(1) assignment(A, 1) = running_on(A, 1) | paused(A, 1) running_on(A, 1) thread(1) assignment(A, 1) thread_free(next_free_thread) = fork(A, B) | send_task_to_thread(B, next_free_thread) | running_on(B, 2) paused(B, 1) running_on(A, 1) | { yield(B, returnvalue) | paused(B, 2) } { await(A, B, returnvalue) | paused(A, 1) } | send_returnvalue(B, A, returnvalue) I think this serialization is truly powerful and easy to understand.
dominicl 4y ago
IMHO the differentiator is deeper yet everywhere engrained in OTP/Behaviours and the famous "Let it crash" slogan. It is the OS-Style process and resource isolation. That is something you can't port with a library into a language that doesn't have it built-in. Lightweight processes, are not the same if they can crash each other, or messup some shared objects/resources/... You can implement actor behaviours in go or node or even c, but without that lower level support it will never give you the stability guarantees that Erlang process isolation is giving. To draw a weird comparison Elixir (with Erlang process isolation) brings two world together. First it's a PHP/Ruby level of fire-and-forget productivity because each http request is handled in an independent isolated process, which if it crashes won't affect the system, but instead provide automatically a nicely debuggable crash log. And second it provides natively all distributed tools for long-lived systems. E.g. PubSub , Sessions and database connections don't have to be rebuilt like in ruby/PHP on a per request basis but can be their own long-lived processes. If there would be a library that could bring this easy to use process isolation+communication e.g. to C programming it would be a game changer. But the best you get in all other languages I'm aware of is to use actual process isolation (fork multiple node/ruby/go processes) and then use some kind of IPC manually or redis or k8s...
matthiasl 4y ago
Not important to the conclusion or reasoning... but Stevena's post says: "the whole team working on Erlang quit and started their own company." The same event as described in "A history of Erlang": "In December, most of the group that created Erlang resigned from Ericsson and started a new company called Bluetail AB." https://www.labouseur.com/courses/erlang/history-of-erlang-armstrong.pdf 'most of the group that _created_ Erlang' is not the same thing as 'the whole team _working_ on Erlang'. Or, quantifying it, at the end of the 'history' paper, there's a list of 45 people under 'implementation' and 'tools'. Around 35 were at Ericsson in 1998. Of those, nine or ten quit to form Bluetail, and another two or three left for Bluetail later on. (In 1998, there were two connected groups working on Erlang in the same building in Älvsjö, Stockholm. One was the computer science laboratory (CSLAB), where Erlang was created, the other was "Open Systems", which had more of a development role. A significant part of CSLAB left. Almost all of 'Open Systems' stayed. Many that stayed were already doing a stellar job on Erlang and many still are.)
- stevan 4y ago
  Fixed, thanks!
lll-o-lll 4y ago
Fascinating! Long ago I came to the conclusion that multithreaded applications in the sense of shared memory protected by locks was simply too difficult for humans to reliably work with. As a result, it turns out I reinvented erlang behaviours in C++ (along with light processes and message passing in the form of Queues and Threads). I may have been influenced by erlang at the time. All of the behaviours listed were replicated. Convergent design I suppose. So, you can probably create erlang behaviours in any language, but it requires building a framework and then training every developer in how to use it. There is probably value in having a standard version of erlang behaviours for a range of different languages.
- hoosieree 4y ago
  I predict the next several generations of programmers will spend a lot of time and energy to reimplement most of Erlang/OTP in curly brace languages in a variation of Greenspun's 10th rule.
  - tormeh 4y ago
    That's already the case. Why stop now?
  - lll-o-lll 4y ago
    Is there a rust equivalent?
  - hoosieree 4y ago
    Haha, great timing: https://news.ycombinator.com/item?id=34557116 https://news.ycombinator.com/item?id=34557116
divs1210 4y ago
Without those lightweight processes that process messages sequentially from their inbox these behaviors won't give the same concurrency guarantees. I don't know why the author is turning this into a competition of whether processes are more important or behaviors - they both are parts of a well designed system that work well together. GenServers etc can't be written equivalently in Go/Java since goroutines and Java threads (even the new virtual threads) are not pre-emptive, whereas Erlang processes are truly independent.
rvirding 4y ago
OTP came much later than the design of the language. The language was designed around our I ideas of what the problem really was and the best way of solving it. The massive and extremely lightweight concurrency were a critical part of attacking the problem which was/is extremely concurrent. There are an enormous number of things going on in a switch which have to be handled concurrently, sometime over 10k calls plus running the switch. So the concurrency was fundamental. The error handling primitives were of course also a a critical part of the solution as fault tolerance was a critical part of the problem. A lot of effort went in to working out what the concurrency and error handling should look like to be right for how to attack the issues and solve the problems. Fortunately we worked with a user group looking at designing a new architecture who tested using our ideas and came with extremely important feedback on the ideas, what was good or bad or just not necessary. They then became the first group to build a product using Erlang. It didn't use OTP as it didn't exist then. OTP came later and is “just” a formalised, generic wrapper around these ideas. We had client-servers, state machines and the equivalent to supervisors in products before OTP was developed. Behaviours came with OTP. And in the beginning there were many different versions of OTP alive at the same time. Behaviours could not have been developed as they are without lightweight processes and communication. OTP is of course fundamnetal to the distribution and use of the language as it does encapsulate most of of our original ideas in how Erlang should/could be used to build systems.
- cmdrk 4y ago
  It would be really interesting to know what sort of things the Erlang team considered but then dropped as bad ideas!
- krmboya 4y ago
  I found this page on one of the LFE books that explains in brief the original problem and constraints https://lfe.io/books/sicp/fm/preface-3/erlang.html https://lfe.io/books/sicp/fm/preface-3/erlang.html

543 points by stevan 4y ago

> Please don't share this repo or any of the linked to documents or repos just yet! Oh no :P Very interesting read though, thanks for sharing!

pietroppeter 4y ago
hopefully stevan is same as stevana from github
boomskats 4y ago
When I started my first job at a grad scheme some 15 years ago, on the first day, we were asked to introduce ourselves to the group and say what name we'd prefer to be called by. This one guy says: "My name is XXX XXXXXXX. I don't care what you call me, just don't call me Ziggy... I HATE being called Ziggy". I heard Ziggy still works there.
- sriram_sun 4y ago
  This is not the first time I'm reading about Ziggy!
- giancarlostoro 4y ago
  Wonder if he did it intentionally or did not realize when he opened his mouth and said that.
- Pet_Ant 4y ago
  I'm guessing that guys real name was "Zbigniew"
- layer8 4y ago
  > I heard Ziggy still works there. He can’t have hated it that much, then. ;)
- pnt12 4y ago
  This reminds of lord of the flies: I'm not a native English speaker, and only read it recently. When Piggy says he doesn't want to be called Piggy, I figured that's what he would be called the whole book. Alas, he was, and I cannot remember his real name.

dmitriid 4y ago

The main insight is in Joe Armstrong's thesis. "Making reliable distributed systems in the presence of software errors" https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf (I think the original title was "In the presence of hardware errors", emphasis mine) All the other things flow from that thesis and understanding. You can recreate behaviours described in the repo doc using Erlang primitives very easily, and they are very hard to recreate in pretty much any other language. Because Erlang is very much literally about lightweight processes and message passing. Only: - every part of the system knows about lightweight processes and messages - every part of the system is engineered around them - the underlying VM is not only re-entrant in almost every single function, it's also extremely resilient, and can almost guarantee that when something dies, only that process dies, and the rest of the system receives a guaranteed notification that this process dies There are more, but without at least that you can't really recreate Erlang's standard behaviours. For example, you can't recreate this in Go or Rust because `panic` will kill your entire program unconditionally.

ThemalSpan 4y ago
I agree up to the last point. You can catch panics at the thread level right? I mean, more generally, what is Erlang implemented in?
- Arch-TK 4y ago
  Erlang is implemented on top of BEAM which has its own scheduler and schedules its own idea of lightweight processes.
  - dolmen 4y ago
    It looks like features that the Go runtime provides.
- ModernMech 4y ago
  Of course you could implement a language like Erlang in Rust, but I think the point is that you would have to do exactly that in order to do in Rust what Erlang does at the language level.

worthless-trash 4y ago

One thing I like about "behaviours" is that I can immediately recognise, discard boiler plate, skim quickly, and see the difference from expected pattern just by knowing the behaviour that the process implements.

practal 4y ago

Very interesting read. Also very timely for me (I am always amazed how HN often has posts that resonate strongly with what I am currently doing), as I am just now designing a programming language based on a generalisation of Algebra, which I call Abstraction Algebra. I think there is a strong connection to this post: Behaviours are just an abstraction algebra you program against. Switch out the algebra implementation against another one, and you can adapt the same program to different scenarios without changing the program itself.

what-no-tests 4y ago
> I am just now designing a programming language based on a generalisation of Algebra, which I call Abstraction Algebra. I admire your ambition!
- practal 4y ago
  To be honest, I am quite amazed by how nicely all the pieces are coming together now. It all just feels right, it feels like collecting fruit from under a huge apple tree.
survirtual 4y ago
I always thought it would be neat to have a system that takes math symbols / formulas / functions etc in the notation of a mathematician, and automatically generates highly performant implementations — sort of like Taichi but instead of using python, you just use mathematics. Seems to me all the tools are available to accomplish this effectively now days. Good luck on your project, in any case.
- practal 4y ago
  Thank you! Yes, a large part of programming will be just a special case of doing mathematics.
samsquire 4y ago
I am also designing a language based on algebra. https://GitHub.com/samsquire/algebralang https://GitHub.com/samsquire/algebralang It's based on the idea there are relations between variables and every function is a concurrent process.

adamddev1 4y ago

> In 1998 Ericsson decided to ban all use of Erlang. Does anyone know why?

doikor 4y ago
Easier to find C/C++ devs according to one friend who used to work there in early 2000s.
- bluGill 4y ago
  That is a cop out. You always need to train your people. I've been doing C++ for 20 years, and consider myself somewhat and expert, but if I join your team I will still need to learn a lot of things that are just how your team does it. If I also need to learn Erlang/java/go/ (pick any language I haven't used much) that only adds a short time. Even as complex as C++ is, I can train a great programmer C++ a lot faster than I can train a future great junior C++ programmer to be a great programmer. Yes you will encounter the rough edges of whatever language often in the first 5 years, but a great programmer will be great in any language quickly. You need one expert in the language on the team for the weird complex stuff, but most code isn't that complex.
  - sillysaurusx 4y ago
    I mean… sort of. It’d be like trying to write an ML stack in lisp circa 2023. Sure, you can find devs willing to learn it, but it’ll be at least two orders or magnitude easier to find python devs. I love lisp, but the thought of being forced to use it exclusively for ML makes me uneasy. And I’ve tried building lisp ML stacks. :)
    - sitkack 4y ago
      I’d probably use Hy and JAX, :) Should be beautiful.
      - sillysaurusx 4y ago

peoplefromibiza 4y ago

Erlang isn't, but the BEAM is.

rhodin 4y ago
Yes came to say this. The Erlang VM is about lightweight processes and message passing.

mapcars 4y ago

As someone who worked with Erlang for a few of years I still wonder if using parallel processing in program top level is worth it or not. Because there are many ways to process data efficiently using queues, pipelines, etc and being clear on when it happens rather than a "wild west" of Erlang where you need to manage processes, links, restarts, and it's harder to focus on the business logic.

stevan 4y ago
Do I understand you correctly in that you'd like more structure? E.g. that you can only deploy an `application` (= supervisor tree)?
- mapcars 4y ago
  I'm talking about development, deployment should be automated anyways. As a developer I think it's easier to think in terms of how to send data to hadoop, sqs queue etc for processing and read results later than keep in mind the supervision tree, messages, mailbox size, linking and so on. And the "processing" side can be as well implemented in Erlang just I don't feel Erlang's features are needed in the "top level development" and create more problems and barriers.
Kototama 4y ago
What do you mean with "process data efficiently"? You speak about performance? But the main goal of the design of Erlang is fault tolerance, not performance. You can still use queues or whatever in Erlang if you want, but having a fault-tolerant system with process tree and process supervision in other languages is a different story.
- mapcars 4y ago
  Right, I was mainly describing arguments for parallel processing which Erlang is also famous for. In terms of being fault-tolerant I think the modern approach with (micro)services is quite similar, one can have multiple services running and communicating using something like protobuf, having restart strategies, fallbacks and so on. From my experience Erlang doesn't offer any killer features in this case, does it?

natdempk 4y ago

Relevant, I’ve written a bit about the various actor models here including Erlang under process-based actors: http://dist-prog-book.com/chapter/3/message-passing.html http://dist-prog-book.com/chapter/3/message-passing.html

styluss 4y ago
opening the root page http://dist-prog-book.com/ http://dist-prog-book.com/ seems to load Tufte CSS documentation

valenterry 4y ago

> The application programmer writes sequential code, all concurrency is hidden away in the behaviour; Yeah, until the business problem itself involves inherent concurrency, which usually happens much faster than people think. Or until I, as the non-expert, want to dig in to make changes or debug a problem. This distinction into "expert" and "lowlife (SCNR) using the expert abstractions" is really one that doesn't hold in practice most of the time. I think it's much better to embrace that concurrency is a cross-cutting-concern and reality is that it can happen on any level, so the language should better support reality.

jeremyjh 4y ago
I think you're talking about two different things. When they say concurrency is hidden away, I think they mean the programmer isn't dealing directly with locks/mutexes. They are also lying, since async functionality like `handle_cast` is built into the `gen_server` behaviour and every caller is calling either `gen_server:cast` or `gen_server:call` depending on whether or not it is going to block on a response. Whenever you invoke async behaviour concurrency will invade all the domain logic and can't be ignored.

JustSomeNobody 4y ago

I get that this is probably written for people familiar with Erlang, but a quick suggestion if I may: In the Behaviors section you talk about behaviors being similar to interfaces in Go and give a Go example. But then you switch examples for Erlang. Maybe show the Joe/Mike example written in Erlang and then say, here's a more complicated example (the key-value example) that really describes behaviors better.

jerf 4y ago

There is a very common pattern in the world where people conflate goals with results. Of course, when I say that, it's obvious that the two things aren't the same, but the unexamined assumption that they are the same sneaks in anyhow when you aren't looking, a cognitive shortcut hard to catch yourself making and harder yet to get in front of and deal with. In the light of this statement, the answer to what I think is the thesis question of that entire piece: "This begs the question: why aren't language and library designers stealing the structure behind Erlang's behaviours, rather than copying the ideas of lightweight processes and message passing?" Is that while Erlang has a lot of good goals, the results of how they got there are simply not the state of the art. Or, to put it another way, language designers are not copying Erlang, and they are correct to not copy Erlang. I respect Erlang a lot. They were a good 10-15 years ahead of their time. However, you will note that if you add 10-15 years to the creation date of Erlang, you still end up in the past. If Erlang were to come out today, fresh, nobody had seen it before, into an otherwise identical programming language environment, I would say it makes several mistakes. One I've written about before is that Erlang is a non-Algol language for no reason: https://news.ycombinator.com/item?id=7277957 https://news.ycombinator.com/item?id=7277957 (Note the library mentioned in that post, suture, is now mature, and I use it all the time. It works for what I need.) But in the context of this post, that's less critical. The other major mistake I'd say it made if it came out in 2023 is that it is a totalizing environment. By that I mean that it has this built in implicit assumption that it is responsible for all the reliability in the system, and you don't get Erlang's features very conveniently if you don't use it as the complete control backplane for your entire system. You run an Erlang cluster, and it bundles all the message passing, restarting, reliability, cluster management, software deploy, and everything into one system. But for the world we live in today, that's a price not worth paying. We don't need the Erlang message bus to be the only message bus. The Erlang message bus is, frankly, not very good, and it's actively terrible if you want to use it for one non-Erlang process to communicate to another. We don't need the Erlang message bus. We have a dozen message busses, some in the cloud, some commercial, some open source, some that double as retention (Kafka), all of which scale better, none of which tie you to Erlang's funky data types. And then, within the matrix of those message busses, we don't need Erlang's restart capability. We have an abundance of ways to restart processes, from systemd, to kubernetes, to any number of other ways. We don't need Erlang clusters for redundancy any more. You just run multiple copies of a service against the message bus, on multiple systems for redundancy. We don't need Erlang's behaviors. We have interfaces, traits, object orientation, and even just pushing that entire problem up to the OS process level, or writing a cloud function, and any number of ways of achieving the same goal. Erlang's software deploy is interesting, but we have a lot of options for it. The whole attempt to do live updates is interesting, but it also imposed a lot of constraints that systems that don't have that need, which is the vast majority of them, don't need or want. This is perhaps the space where the state of the art isn't that far ahead of Erlang. It's still a mess, despite all the churn in this space. But even so, with all the options available, you can probably find something better for your system than the Erlang way of upgrading software, even if it isn't necessarily much easier. The cognitive hazard that Erlang presents the community in 2023 is that it has some very good writing on the topic of reliability and its other goals, and then, naturally, one segues into the discussion of how Erlang solved the problem. And it was a very interesting solution for the time. I used Erlang for many, many years back when it was effectively the only solution to these problems. But it isn't the only solution anymore. The space has exploded with options. Unsurprisingly, the ones that a highly innovative pioneer tried out first are not the best, or the only. They chose well. Let me again emphasize my respect for the project. But it's not on the cutting edge anymore. Granted, the diversity of options does mean the real world has gotten quite chaotic, where you may have three message busses attaching systems implemented in a dozen different languages, but that's something you can take up with Conway's Law. Erlang couldn't work with Conway's Law without totally converting your entire company to it, which just isn't going to happen. The reason why language designers aren't rushing to copy Erlang is that what was excellent and amazing in 2000 (and, again let me underline, I mean that very seriously, it was a cutting edge platform built with a lot of vision and moxie) is, in 2023, mediocre. Erlang is a mediocre language (Elixir is at least "good", Erlang is mediocre), attached to a mediocre message bus, with a type system that doesn't even reach mediocre, with a mediocre totalizing approach to system design where there's a very significant impedence mismatch between it and the rest of the world, with an at-par-at-best VM (I won't call that mediocre, but where it used to be head-and-shoulders above everything else in certain ways, it is now merely competitive), with mediocre standard libraries, and a mediocre product fit to its own stated goals. It just isn't the best any more. The state of the art right now is super chaotic. I can hardly get two systems deployed on the same infrastructure any more, because there's always some reason something in that list has changed. But when the chaos settles and best practices emerge, something that I'd say is at least a good 5 years away, the result will clearly have Erlang inspiration in it for sure... but it won't look a lot like Erlang on the surface. What is worth copying has largely been copied. It doesn't look exactly like Erlang, but this turns out to be a good thing.

revskill 4y ago

What i learn from Erlang, is, naming is hard. What does `gen` mean in `gen_server` ? Thanks for replies for "Generic". So this article is not well organized in that case. With some simple naming explanation, it should be obvious to guess what the system does. Abbreviation doesn't save the writings. Some more examples, in Ruby, instead of calling `implement Module`, it uses `include Module` . I do think include is not as clear as implement.

mtremsal 4y ago
generic, I believe. i.e. it’s an interface that has to be implemented.
- sodapopcan 4y ago
  Yep, “generic” it is.
deleted 4y ago
[deleted]
dolmen 4y ago
generic?
lopatin 4y ago
I always thought it was “generate” and that something about the implementation/immutableness required the server to be “generated” at runtime. I never did any actual Erlang programming, it’s just how my brain plugged in the gaps when I heard about all the gen stuff.
mabbo 4y ago
If they called it 'generic_server' then people would make fun of it for being too verbose. If they call it 'gen_server' then everyone pretends they know what it means and stays quiet.
- edgyquant 4y ago
  gen_server to me is “generate a server”

xwowsersx 4y ago

> In 1998 Ericsson decided to ban all use of Erlang. What's the story there? Why did they decide to ban it? EDIT: doh, this has been answered elsewhere in this thread.

crad 4y ago

Erlang vs OTP

sb8244 4y ago

> just like the concurrent code of gen_server I found this a really interesting read, but this stuck out because it doesn't jive with my mental model of gen_server. gen_server is fully serialized. Even the code underpinning it is not concurrent. Now I guess gen_server does expose some top level functions to simplify sending/receiving a message into the process, but the process itself is serial. And this is part of the genius of gen_server to me. You don't need to think about your state concurrently because it processes a single message at a time. Your system executes concurrently, but the individual components do not. Maybe that is what the post means and I misinterpreted it.

wefarrell 4y ago
Yeah they mention that in 2 of the 6 points arguing in favor of behaviors: 2. The application programmer writes sequential code, all concurrency is hidden away in the behaviour; 4. Easier for new team members to get started: business logic is sequential, similar structure that they might have seen before elsewhere;
- sb8244 4y ago
  Thanks for that. My thought is probably just semantics then. I think it's even more nuanced, but doesn't really matter for this type of post. (I am actively writing a chapter exploring this topic, so is has been top of mind.)
- toast0 4y ago
  Adding on to sb8244's comments; writing sequential code is very nice, but it's not the gen_server behavior that gets you that. It's the lightweight processes. gen_server is useful, but it's not much more than receive in a tail recursive loop and conventions around messages, most specifically about if the sender expects to receive a reply or not. It's not magic, and it's written in clear Erlang that anyone with a week of Erlang fiddling can understand (which kind of is magic; almost everything in OTP is clearly readable) Concurrency comes from running many processes each with their own message queue.
btbuildem 4y ago

amelius 4y ago

Isn't Erlang's model a giant ball of microservices?

lupire 4y ago
Nanoservices.

throwawaymaths 4y ago

Wait till this person learns about links, monitors, call's default automatic backpressure mechanism, etc...

hosh 4y ago

I think it is misleading to frame this as "behavior". I think what we really have are battle-tested concurrency patterns, which then uses the language feature called "behavior" to implement. This collection of battle-tested concurrency patterns that has been in use for a long time, in a variety of settings, is called OTP. It's why you'll hear long-time Erlang folks talk about how, it's really about OTP. You can implement those same patterns, if you also have lightweight processes, messages, and message queues. There's a Rust library that aims to do exactly that. The new WASM runtime (Firefly, formerly known as Lumen) aims to bring those patterns into the WASM ecosystem, potentially using this kind of concurrency pattern client-side. What's potentially novel is adding in newer concurrency patterns that are in use in the Kubernetes community -- use of selectors, services, scaling with replicasets -- so that, instead of a static supervision tree, it can be a dynamic supervision tree, spread out across the entire pool, even as that pool expands or shrinks.

imachine1980_ 4y ago
Please, can you share the rust library??
- linkdd 4y ago
  Not sure this is what GP is talking about but to implement the actor model in https://letlang.dev https://letlang.dev I use tokio.
- hosh 4y ago
  I don’t remember the name and I don’t remember if it is tokio or something else. However, I did find this: https://docs.rs/genserver/latest/genserver/ https://docs.rs/genserver/latest/genserver/ I will keep looking. Keep in mind, BEAM uses a preemptive actor rather than async, so it avoids the “storms” that can happen when an async reactor runs out of resources. EDIT: but I guess Tokio is capable of preemption through work-stealing too?
  - hosh 4y ago
    This looks interesting too, though no OTP out-of-the-box: https://docs.rs/axiom/latest/axiom/ https://docs.rs/axiom/latest/axiom/

brudgers 4y ago

I agree with the headline because uptime was the goal of Erlang. Lightweight processes using message passing is how Erlang stays up (and perhaps ironically, let-it-crash is how Erlang avoids going down). Lightweight processes and message passing are the architectural decisions that address the business problem Erlang was designed to solve.

peter_d_sherman 4y ago

>"The next interesting behavior is supervisor. Supervisors are processes which sole job is to make sure that other processes are healthy and doing their job. If a supervised process fails then the supervisor can restart it according to some predefined strategy." This is an interesting software pattern -- especially applied to having it entirely supported entirely inside of one programming language... We see this software pattern in such things as Windows Services (they restart automatically if they fail), Unix/Linux daemons, as one of the original purposes of the original Unix 'init' process, and in high-availability, mission-critical 24/7 systems (databases, etc.). Having all of that fault-tolerant infrastructure entirely inside of a programming language is indeed, somewhat novel. But let's work up what a given language must have as prerequisites to support this... First, we need the ability to run multiple processes inside of a language. Many languages can accomplish this with threads, but of the languages that do this, many need additional cumbersome programming to guarantee thread safety... So the language needs to support the notion of a threaded process inside of that code -- without having to code extra to support this threaded process. Next, the language needs some form of communication between supervisor (process A) and supervised (process B) code. Message passing is the solution -- but that requires each process to have its own message queue. And message passing requires interfaces... Here we're starting to sound like we're duplicating a mini Operating System inside a language(!) (should the language have pipes, too?) -- and/or that the complexity of most OS's would be deeply mitigated by designing them inside of a language that supported these constructs... Whatever the case, I think it's a fascinating software pattern. Yes, Go exists and supports features like this, Rust exists, and people are using it to write Operating Systems. And there are probably all other kinds of languages (both existing and existing in the future) that do/will support some form of these constructs... But there's another interesting, purely academic, purely theoretical question here... That is: What is the simplest full-featured OS (processes, IPC, synchronization primitives, memory management, hardware resource management, syscalls, scheduling, etc.) that could be written in the smallest amount of lines of code if the language it was written in had intrinsic knowledge of all of those underlying constructs? Anyway, a fascinating article about Erlang, and it definitely gave the impression that Erlang was a whole lot more than I thought it was... I will be checking out Erlang for future projects!

4y ago

jb3689 4y ago

To me the real amazing thing about Erlang is more the fact that these patterns based on queues and messages and supervisors scale so well. They are easy to implement on a small, single node application. You can compose them. You don't need to complete reauthor your calls to start distributing them and fanning out responsibility across your nodes because you've been doing RPCs from the start. A multi-node application is not that much more difficult to reasonable than a single-node application

finder83 4y ago

There's a component that seems to be missing here which is preemptive task scheduling. I've not seen another non-OS system do it like the BEAM VM (the VM behind Erlang), though there may be something I'm not aware of. It really prevents a whole class of concurrency issues where a hung process can freeze or slow down the entire system. For example, if you recreated gen_server in a cooperative concurrency environment, one gen_server could use up all of the CPU and have an impact on the performance of the rest of the system. Maybe the other threads (microthreads, not OS threads) would still respond in <500ms, but if every request takes 500ms when they normally take 15ms you could essentially have outage-like conditions, particularly with upstream timeouts. Instead, because BEAM is preemptive that one (or 10) hung gen_server doesn't hang up everything else on a node. Sure at some point performance will degrade, but that point is much further down the line than in cooperative concurrency models. There was a fantastic talk by Sasa Juric that demonstrates this in Erlang. [1] Otherwise you run a higher risk of even the supervisors being starved for CPU time, particularly if you are launching hundreds of processes that are all locking the CPU. It's really the combination of the behaviors (OTP), the scheduler, lightweight threads, message passing, and immutability to me that makes the Erlang (Elixir for me) concurrency model so appealing. Creating a language with the feel of a lisp, the environment of Smalltalk, and the concurrency of Erlang has been my dream for a long time. [1] https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE

Fire-Dragon-DoL 4y ago
If I'm not wrong, Go added uncooperative task scheduling that behaves similarly?
- finder83 4y ago
  Interesting, at least when I was using it the goroutines themselves were cooperative, and of course the OS threads were preemptive when GO_MAXPROCS > 1. Not finding much with a search, there was a proposal to make them preemptive. Curious if others will chime in that it's now preemptive. Even so I like Elixir better as a language, but the processing speed of Go with a preemptive scheduler would be tempting for some use cases.

didip 4y ago

These days, if you have Kubernetes, queues, and worker queues, does OTP still have an edge?

bsaul 4y ago
I believe the only argument left is that erlang /OTP provide all of those in a single cohesive environment.
deleted 4y ago
[deleted]
sb8244 4y ago
Yes. Being able to do this at the programming language level is extremely powerful and creates an entirely different way of building applications. My go-to analogy is building a city (lots of individual, isolated, separate stack processes) instead of a big skyscraper (deep stack requests, concurrency difficult, error recovery manual). You can build that city in a limited way with k8s, but there's way more overhead along the way, to the point of it not being enjoyable for me.
weatherlight 4y ago
Yes because concurrency and fault tolerance are literally at the core of the the BEAM. you get all the above with zero configuration, all through convention and you only need to learn one set of tools. When some new container technology comes along, or new worker queue platform etc. Erlang/the beam will be exactly the same. One of the advantages of working with Erlang I found is that it's just as easy to scale down as it is to scale up. How does one scale down from Kafka once you have vendor lock in with them?

ComputerGuru 4y ago

I don't mean to disparage the article (quite the contrary, it's quite the eye opener if you're not accustomed to architecting your code in this fashion) but choosing go of all languages to mockup the ideas was a terrible choice. It's syntax and type system are really quite the red-headed stepchild and for anyone unfamiliar with its syntax, its incredibly hard to make heads or tails of. (And I say this as someone that's at least passably familiar with go and have worked on/contributed to golang projects in the past.)

rahilb 4y ago
As far as I can tell the article contains only erlang snippets directly quoted from the paper reviewed.
preseinger 4y ago
I think you may be alone in this assessment. Go is uniquely easy to understand, due to its small set of keywords and lack of esoteric e.g. sigils.
- weatherlight 4y ago
  Yeah, I had no idea how to read the Go code. I'm also coming from a ruby/js/ocaml/elixir/erlang background.
  - preseinger 4y ago
    The Go code, in its entirety: type HasName interface { Name() string } func Greet(n HasName) { fmt.Printf("Hello %s!\n", n.Name()) } type Joe struct {} func (_ *Joe) Name() string { return "Joe" } type Mike struct {} func (_ *Mike) Name() string { return "Mike" } func main() { joe := &Joe{} mike := &Mike{} Greet(mike) Greet(joe) } You have no idea how to read this?
    - weatherlight 4y ago

lippingoff 4y ago

This is fucking cool, thanks for sharing

eternalban 4y ago

J2EE could have had all that but it was missing an OTP. [Java got the competing 'application server' vendors competing themselves out of a massive market.] Components with well defined life cycles, request/reply and message processing, dynamic remote interface binding, isolated execution environment, pure single threaded application level code ('business logic'), declarative composition ... that's all in the specs. Erlang is mainly about OTP. OTP delivered an opionated take on distributed components -- think of Erlang on OTP as Ruby on Rails -- and did it exceptionally well. But one could still do this with Java btw. The specs are still there and they are solid.

jFriedensreich 4y ago

It makes me sentimental to see some erlang code, truly miss writing erlang, as its probably the closest to how i think about systems. But it is so hard to justify using it when most of my problems can be solved with javascript edge workers. These days it's rare for me to even come across a backend heavy project, yet something that would fit perfect to erlang. Does anyone else fell this way or is it because of my drift in the industry? When was the last project you started and thought this is perfect to solve with erlang?

noloblo 4y ago
We need backend heavy erlang prototype
- yurishimo 4y ago
  Arguably that’s what Elixir is. They’re really trying to push Phoenix as a batteries included full stack solution. I know it’s not exactly Erlang, but the interfaces for things like gen_server are nearly identical. It’s effectively Erlang with a more modern syntax.
  - throwawaymaths 4y ago
    Its also got much better tooling, and a few features that don't exist in Erlang yet (Tasks - and no, tasks are not just proc_lib, they're actually special.

lippingoff 4y ago

This fucking rocks, great work and thanks for sharing.

samsquire 4y ago

I wrote a multithreaded message passing system in Java and I was trying to get async await to work The thing I want to define with behaviours is observable effects on objects. Such as interactions between tasks and objects between them. Not necessarily method calls but state interactions across multiple objects. Some objects should be colocated on threads and send behaviours to arbitrary groups of objects. Think of it as a collection that responds to events. I wrote a state machine serialization that looks like this. This defined a state progression between threads and async/await state machine. I think organised state machines would mean actor programming is so much easier to program. next_free_thread = 2 task(A) thread(1) assignment(A, 1) = running_on(A, 1) | paused(A, 1) running_on(A, 1) thread(1) assignment(A, 1) thread_free(next_free_thread) = fork(A, B) | send_task_to_thread(B, next_free_thread) | running_on(B, 2) paused(B, 1) running_on(A, 1) | { yield(B, returnvalue) | paused(B, 2) } { await(A, B, returnvalue) | paused(A, 1) } | send_returnvalue(B, A, returnvalue) I think this serialization is truly powerful and easy to understand.

dominicl 4y ago

IMHO the differentiator is deeper yet everywhere engrained in OTP/Behaviours and the famous "Let it crash" slogan. It is the OS-Style process and resource isolation. That is something you can't port with a library into a language that doesn't have it built-in. Lightweight processes, are not the same if they can crash each other, or messup some shared objects/resources/... You can implement actor behaviours in go or node or even c, but without that lower level support it will never give you the stability guarantees that Erlang process isolation is giving. To draw a weird comparison Elixir (with Erlang process isolation) brings two world together. First it's a PHP/Ruby level of fire-and-forget productivity because each http request is handled in an independent isolated process, which if it crashes won't affect the system, but instead provide automatically a nicely debuggable crash log. And second it provides natively all distributed tools for long-lived systems. E.g. PubSub , Sessions and database connections don't have to be rebuilt like in ruby/PHP on a per request basis but can be their own long-lived processes. If there would be a library that could bring this easy to use process isolation+communication e.g. to C programming it would be a game changer. But the best you get in all other languages I'm aware of is to use actual process isolation (fork multiple node/ruby/go processes) and then use some kind of IPC manually or redis or k8s...

matthiasl 4y ago

Not important to the conclusion or reasoning... but Stevena's post says: "the whole team working on Erlang quit and started their own company." The same event as described in "A history of Erlang": "In December, most of the group that created Erlang resigned from Ericsson and started a new company called Bluetail AB." https://www.labouseur.com/courses/erlang/history-of-erlang-armstrong.pdf 'most of the group that _created_ Erlang' is not the same thing as 'the whole team _working_ on Erlang'. Or, quantifying it, at the end of the 'history' paper, there's a list of 45 people under 'implementation' and 'tools'. Around 35 were at Ericsson in 1998. Of those, nine or ten quit to form Bluetail, and another two or three left for Bluetail later on. (In 1998, there were two connected groups working on Erlang in the same building in Älvsjö, Stockholm. One was the computer science laboratory (CSLAB), where Erlang was created, the other was "Open Systems", which had more of a development role. A significant part of CSLAB left. Almost all of 'Open Systems' stayed. Many that stayed were already doing a stellar job on Erlang and many still are.)

stevan 4y ago
Fixed, thanks!

lll-o-lll 4y ago

Fascinating! Long ago I came to the conclusion that multithreaded applications in the sense of shared memory protected by locks was simply too difficult for humans to reliably work with. As a result, it turns out I reinvented erlang behaviours in C++ (along with light processes and message passing in the form of Queues and Threads). I may have been influenced by erlang at the time. All of the behaviours listed were replicated. Convergent design I suppose. So, you can probably create erlang behaviours in any language, but it requires building a framework and then training every developer in how to use it. There is probably value in having a standard version of erlang behaviours for a range of different languages.

hoosieree 4y ago
I predict the next several generations of programmers will spend a lot of time and energy to reimplement most of Erlang/OTP in curly brace languages in a variation of Greenspun's 10th rule.
- tormeh 4y ago
  That's already the case. Why stop now?
- lll-o-lll 4y ago
  Is there a rust equivalent?
- hoosieree 4y ago
  Haha, great timing: https://news.ycombinator.com/item?id=34557116 https://news.ycombinator.com/item?id=34557116

divs1210 4y ago

Without those lightweight processes that process messages sequentially from their inbox these behaviors won't give the same concurrency guarantees. I don't know why the author is turning this into a competition of whether processes are more important or behaviors - they both are parts of a well designed system that work well together. GenServers etc can't be written equivalently in Go/Java since goroutines and Java threads (even the new virtual threads) are not pre-emptive, whereas Erlang processes are truly independent.

rvirding 4y ago

OTP came much later than the design of the language. The language was designed around our I ideas of what the problem really was and the best way of solving it. The massive and extremely lightweight concurrency were a critical part of attacking the problem which was/is extremely concurrent. There are an enormous number of things going on in a switch which have to be handled concurrently, sometime over 10k calls plus running the switch. So the concurrency was fundamental. The error handling primitives were of course also a a critical part of the solution as fault tolerance was a critical part of the problem. A lot of effort went in to working out what the concurrency and error handling should look like to be right for how to attack the issues and solve the problems. Fortunately we worked with a user group looking at designing a new architecture who tested using our ideas and came with extremely important feedback on the ideas, what was good or bad or just not necessary. They then became the first group to build a product using Erlang. It didn't use OTP as it didn't exist then. OTP came later and is “just” a formalised, generic wrapper around these ideas. We had client-servers, state machines and the equivalent to supervisors in products before OTP was developed. Behaviours came with OTP. And in the beginning there were many different versions of OTP alive at the same time. Behaviours could not have been developed as they are without lightweight processes and communication. OTP is of course fundamnetal to the distribution and use of the language as it does encapsulate most of of our original ideas in how Erlang should/could be used to build systems.

cmdrk 4y ago
It would be really interesting to know what sort of things the Erlang team considered but then dropped as bad ideas!
krmboya 4y ago
I found this page on one of the LFE books that explains in brief the original problem and constraints https://lfe.io/books/sicp/fm/preface-3/erlang.html https://lfe.io/books/sicp/fm/preface-3/erlang.html