6 ms·

Ergo: Erlang/OTP patterns implemented in Golang

125 points by rapnie 4y ago

fishgoesblub 4y ago
I originally thought this had something to due with the IRCd written in Go also named Ergo https://github.com/ergochat/ergo https://github.com/ergochat/ergo
kaycebasques 4y ago
I will forever think of Will Ferrell's parody of the Matrix whenever I hear "ergo": https://youtu.be/iRkiyy3EDl4?t=360 https://youtu.be/iRkiyy3EDl4?t=360
prophesi 4y ago
This is just a tongue-in-cheek response that I have no stock in, but I find it funny that the page says "You don't have to reinvent the wheel" after they rewrite Erlang processes in another language.
- tete 4y ago
  Being compatible with what's there is exactly the opposite of reinventing the wheel (mechanisms, protocols, etc.).
  - prophesi 4y ago
    That's true, but it'd be even simpler to implement a node in x language that can communicate with Erlang/OTP, and let Erlang work its fault-tolerant and scalable magic.
    - gfodor 4y ago
      This is why analogies like the wheel analogy are silly - to stretch it closer to reality, sometimes you are eyeing a wheel in Go and a wheel in Erlang, so you’ve gotta pick which one to port.
lll-o-lll 4y ago
There was another recent Erlang post which posited the suggestion that the power of erlang was not in lightweight processes and message parsing, but in generic components erlang calls behaviours. These generic patterns are what allows for building reliable distributed systems, and Joe Armstrong wrote a PhD thesis about it. I gave an anecdote of having built a c++ framework that unwittingly followed the same principles for “big corp” long ago (convergent design), and posited that the same concepts could probably be built in any language. Here it is for go. So is it a good thing? Previously I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure. The problem I have is that as you start looking at these higher level patterns involving distributed computing, the landscape of frameworks seems to explode. Is this because the erlang/otp concepts are insufficient or outdated?
- digilypse 4y ago
  Maybe it’s because the frameworks themselves are less important than the *human language used to describe them. To someone who knows what a Genserver is, or has used one before, what does it matter which of the many frameworks they’re using?
  - lll-o-lll 4y ago
    It just seems like we reinvent the same things again and again. Maybe that’s because at this level you can’t have a standard and things need to be bespoke. It could, on the other hand, be an indication that there is a huge amount of wasted effort in revisiting long ago solved problems.
    - digilypse 4y ago
      I agree, I just think it’s part of the evolutionary process. We reinvent them because the core concepts are right, but there isn’t enough connective tissue to re-use the implementations directly. I don’t see it as a bad thing. The concepts persist and gain momentum and with each iteration we get closer to something that can actually be relied on consistently without replication. There’s wasted effort, but something about the original was missing that would have made it the obvious choice. I think it feels wasted mostly because it isn’t yet obvious what the missing pieces were.
jerf 4y ago
On the one hand, this framework's front page almost undersells the biggest feature it has, which is that I believe it can function as an actual Erlang node. You can use this to join a Go system to your Erlang cluster directly. This is pretty impressive because that's nontrivial functionality. On the other hand... this is a pure an example of a transliteration as an idiomatic port as I can think of. There is simply no need, in pure Go at least, to blindly copy every last detail of the Erlang implementation for gen_server for general usage, and more than you should copy every last such thing for Javascript or Python or any other language. If you are joining to an Erlang node and working in the Erlang ecosystem, you have no real choice, but if you aren't doing that, this library is going to add a lot of friction to a lot of things in Go, all over the place. It's a very niche thing. If you need it it's probably worth all the money, but I'd pay to not get it put into a greenfield Go project.
- Idk__Throwaway 4y ago
  As someone without a lot of knowledge of GO and with only a trivial understanding of OTP/the gen_server impl, how would this add friction? What exactly is wrong with it and what would be a better alternative?
  - jerf 4y ago
    I go over this fairly deeply in https://www.jerf.org/iri/post/2930/ https://www.jerf.org/iri/post/2930/ . The Big Idea is that it's rarely a good idea to transliterate a library. You should translate a library. When I was done translating into Go, I found that what was idiomatic in Go was much simpler than Erlang. To some extent this is because Go simply can't do some things Erlang can do (in particular Erlang is one of the very short list of languages that can safely use "asynchronous exceptions", or to put it another way, can let one thread simply reach out and nuke another thread, so safely it can be incorporated into the core design of systems rather than treated as a dangerous special case)... but if that is the case, why port over the aspects of the Erlang design that are now dangling free in space, unconnected to anything?
    - lll-o-lll 4y ago
moomoo11 4y ago
dumbo here, what are some use cases for this?
- waynesonfire 4y ago
  probably none for you, but a bullet item on the authors resume.
- pejrich 4y ago
  I imagine similar to the use cases for Erlang. Erlang can be a polarizing language, but what it's designed to do, it does exceptionally well, in fact, maybe the best. And that's not concurrency, multicore programming, or distribution. That was all a consequence of the main goal, which was fault tolerance. If you're a Swedish telecom company, you don't want a little bug in the code, that happens during someone's phone call, to knock out a switch, which takes down 50k phone calls, only for some guy to have to trek out into the middle of the forest to restart the computer running everything. So the model they came up with was high isolation, via processes(no OS processes, but super tiny, like a few hundred bytes processes, all managed by the VM), and distribution(can't have reliability if one node can knock out the system). The other unique part is the supervision trees that manage those processes. Let's say you have a chat server. You may have a process per device connected to the server, a supervisor for each chat room(group of processes), and a supervisor for all the chat rooms(supervisor of supervisors). Supervisors are themselves processes, but with a specific job. If your phone's connection/process on the server has some issue, it'll just crash, that's the erlang philosphy "let it crash". You as the user likely can't fix it. The code can't magically fix itself. But presumably through testing we know that initial connections with a clean state are stable, so let it crash, your phone reconnects. Then it crashes again. Reconnects. Crashes again. (This isn't the ideal state obviously, but a worst case for demonstration). Eventually that'll trigger some limit, where the chat group supervisor will notice, somethings wrong here, there's some state that's in a mess, it's time to crash so we can get a fresh try at this. So it'll crash. Usually the first phone reconnection would have been enough to fix it and the group supervisor wouldn't need to restart, but for demonstration purposes we'll say it does, and let's pretend when it comes back up it crashes again. This will propagate up, until enough restarts have fixed the state of the system to the point where it's stable again. I believe Erlang was the first system to achieve nine 9s of reliability, which is 31.56 milliseconds of downtime a year. Not saying this lib will get you that, but that's the spiel for Erlang.
gfodor 4y ago
Anything like this for Python?
- rcarmo 4y ago
  Not for the patterns, but Thespian is a pretty decent actor library.
  - metadat 4y ago
    Thanks, I'd never heard of Thespian for Python: https://thespianpy.com/doc/ https://thespianpy.com/doc/
- beagle3 4y ago
  This included the concurrency primitives (and a little bit more) almost 20 years ago. Probably requires porting to Python 3. https://candygram.sourceforge.net/ https://candygram.sourceforge.net/
yumraj 4y ago
Does anyone know what tool was used to make these diagrams.. I really like the style.
- Existenceblinks 4y ago
  I wasn't impressed but I look at it again .. it could be more pretty with right colors. Anyone has answer?
cutler 4y ago
So that's it for Elixir then.
- Alifatisk 4y ago
  No?
rad_gruchalski 4y ago
Looks cool. However, since this is a paid product… if one wants an actor framework for go without the need to connect to Erlang nodes, this will be a fine choice: https://github.com/asynkron/protoactor-go https://github.com/asynkron/protoactor-go.
Alifatisk 4y ago
>Ergo Framework almost 5 times outperforms the original Erlang network messaging This sounds like a breakthrough? Isn’t it?
Alifatisk 4y ago
Anything similar for Ruby? Update: Found this, https://ruby-concurrency.github.io/concurrent-ruby/master/Concurrent/Actor.html https://ruby-concurrency.github.io/concurrent-ruby/master/Co...
throwaway81523 4y ago
Ergo, heh. I wanted to do something like that and call it GoTP. But, some OTP features are hard to implement in Go. Particularly, Goroutines can clobber each others' data, unlike Erlang processes.

125 points by rapnie 4y ago

I originally thought this had something to due with the IRCd written in Go also named Ergo https://github.com/ergochat/ergo https://github.com/ergochat/ergo

kaycebasques 4y ago

I will forever think of Will Ferrell's parody of the Matrix whenever I hear "ergo": https://youtu.be/iRkiyy3EDl4?t=360 https://youtu.be/iRkiyy3EDl4?t=360

prophesi 4y ago

This is just a tongue-in-cheek response that I have no stock in, but I find it funny that the page says "You don't have to reinvent the wheel" after they rewrite Erlang processes in another language.

tete 4y ago
Being compatible with what's there is exactly the opposite of reinventing the wheel (mechanisms, protocols, etc.).
- prophesi 4y ago
  That's true, but it'd be even simpler to implement a node in x language that can communicate with Erlang/OTP, and let Erlang work its fault-tolerant and scalable magic.
  - gfodor 4y ago
    This is why analogies like the wheel analogy are silly - to stretch it closer to reality, sometimes you are eyeing a wheel in Go and a wheel in Erlang, so you’ve gotta pick which one to port.

lll-o-lll 4y ago

There was another recent Erlang post which posited the suggestion that the power of erlang was not in lightweight processes and message parsing, but in generic components erlang calls behaviours. These generic patterns are what allows for building reliable distributed systems, and Joe Armstrong wrote a PhD thesis about it. I gave an anecdote of having built a c++ framework that unwittingly followed the same principles for “big corp” long ago (convergent design), and posited that the same concepts could probably be built in any language. Here it is for go. So is it a good thing? Previously I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure. The problem I have is that as you start looking at these higher level patterns involving distributed computing, the landscape of frameworks seems to explode. Is this because the erlang/otp concepts are insufficient or outdated?

digilypse 4y ago
Maybe it’s because the frameworks themselves are less important than the *human language used to describe them. To someone who knows what a Genserver is, or has used one before, what does it matter which of the many frameworks they’re using?
- lll-o-lll 4y ago
  It just seems like we reinvent the same things again and again. Maybe that’s because at this level you can’t have a standard and things need to be bespoke. It could, on the other hand, be an indication that there is a huge amount of wasted effort in revisiting long ago solved problems.
  - digilypse 4y ago
    I agree, I just think it’s part of the evolutionary process. We reinvent them because the core concepts are right, but there isn’t enough connective tissue to re-use the implementations directly. I don’t see it as a bad thing. The concepts persist and gain momentum and with each iteration we get closer to something that can actually be relied on consistently without replication. There’s wasted effort, but something about the original was missing that would have made it the obvious choice. I think it feels wasted mostly because it isn’t yet obvious what the missing pieces were.

jerf 4y ago

On the one hand, this framework's front page almost undersells the biggest feature it has, which is that I believe it can function as an actual Erlang node. You can use this to join a Go system to your Erlang cluster directly. This is pretty impressive because that's nontrivial functionality. On the other hand... this is a pure an example of a transliteration as an idiomatic port as I can think of. There is simply no need, in pure Go at least, to blindly copy every last detail of the Erlang implementation for gen_server for general usage, and more than you should copy every last such thing for Javascript or Python or any other language. If you are joining to an Erlang node and working in the Erlang ecosystem, you have no real choice, but if you aren't doing that, this library is going to add a lot of friction to a lot of things in Go, all over the place. It's a very niche thing. If you need it it's probably worth all the money, but I'd pay to not get it put into a greenfield Go project.

Idk__Throwaway 4y ago
As someone without a lot of knowledge of GO and with only a trivial understanding of OTP/the gen_server impl, how would this add friction? What exactly is wrong with it and what would be a better alternative?
- jerf 4y ago
  I go over this fairly deeply in https://www.jerf.org/iri/post/2930/ https://www.jerf.org/iri/post/2930/ . The Big Idea is that it's rarely a good idea to transliterate a library. You should translate a library. When I was done translating into Go, I found that what was idiomatic in Go was much simpler than Erlang. To some extent this is because Go simply can't do some things Erlang can do (in particular Erlang is one of the very short list of languages that can safely use "asynchronous exceptions", or to put it another way, can let one thread simply reach out and nuke another thread, so safely it can be incorporated into the core design of systems rather than treated as a dangerous special case)... but if that is the case, why port over the aspects of the Erlang design that are now dangling free in space, unconnected to anything?
  - lll-o-lll 4y ago

moomoo11 4y ago

dumbo here, what are some use cases for this?

waynesonfire 4y ago
probably none for you, but a bullet item on the authors resume.
pejrich 4y ago
I imagine similar to the use cases for Erlang. Erlang can be a polarizing language, but what it's designed to do, it does exceptionally well, in fact, maybe the best. And that's not concurrency, multicore programming, or distribution. That was all a consequence of the main goal, which was fault tolerance. If you're a Swedish telecom company, you don't want a little bug in the code, that happens during someone's phone call, to knock out a switch, which takes down 50k phone calls, only for some guy to have to trek out into the middle of the forest to restart the computer running everything. So the model they came up with was high isolation, via processes(no OS processes, but super tiny, like a few hundred bytes processes, all managed by the VM), and distribution(can't have reliability if one node can knock out the system). The other unique part is the supervision trees that manage those processes. Let's say you have a chat server. You may have a process per device connected to the server, a supervisor for each chat room(group of processes), and a supervisor for all the chat rooms(supervisor of supervisors). Supervisors are themselves processes, but with a specific job. If your phone's connection/process on the server has some issue, it'll just crash, that's the erlang philosphy "let it crash". You as the user likely can't fix it. The code can't magically fix itself. But presumably through testing we know that initial connections with a clean state are stable, so let it crash, your phone reconnects. Then it crashes again. Reconnects. Crashes again. (This isn't the ideal state obviously, but a worst case for demonstration). Eventually that'll trigger some limit, where the chat group supervisor will notice, somethings wrong here, there's some state that's in a mess, it's time to crash so we can get a fresh try at this. So it'll crash. Usually the first phone reconnection would have been enough to fix it and the group supervisor wouldn't need to restart, but for demonstration purposes we'll say it does, and let's pretend when it comes back up it crashes again. This will propagate up, until enough restarts have fixed the state of the system to the point where it's stable again. I believe Erlang was the first system to achieve nine 9s of reliability, which is 31.56 milliseconds of downtime a year. Not saying this lib will get you that, but that's the spiel for Erlang.

gfodor 4y ago

Anything like this for Python?

rcarmo 4y ago
Not for the patterns, but Thespian is a pretty decent actor library.
- metadat 4y ago
  Thanks, I'd never heard of Thespian for Python: https://thespianpy.com/doc/ https://thespianpy.com/doc/
beagle3 4y ago
This included the concurrency primitives (and a little bit more) almost 20 years ago. Probably requires porting to Python 3. https://candygram.sourceforge.net/ https://candygram.sourceforge.net/

yumraj 4y ago

Does anyone know what tool was used to make these diagrams.. I really like the style.

Existenceblinks 4y ago
I wasn't impressed but I look at it again .. it could be more pretty with right colors. Anyone has answer?

cutler 4y ago

So that's it for Elixir then.

Alifatisk 4y ago
No?

rad_gruchalski 4y ago

Looks cool. However, since this is a paid product… if one wants an actor framework for go without the need to connect to Erlang nodes, this will be a fine choice: https://github.com/asynkron/protoactor-go https://github.com/asynkron/protoactor-go.

Alifatisk 4y ago

>Ergo Framework almost 5 times outperforms the original Erlang network messaging This sounds like a breakthrough? Isn’t it?

Alifatisk 4y ago

Anything similar for Ruby? Update: Found this, https://ruby-concurrency.github.io/concurrent-ruby/master/Concurrent/Actor.html https://ruby-concurrency.github.io/concurrent-ruby/master/Co...

throwaway81523 4y ago

Ergo, heh. I wanted to do something like that and call it GoTP. But, some OTP features are hard to implement in Go. Particularly, Goroutines can clobber each others' data, unlike Erlang processes.