5 ms·

Suture – Supervisor Trees for Go

149 points by leef 12y ago

wyc 12y ago
Great article. I especially enjoyed the clear descriptive problem framing and discussion about Erlang processes versus goroutines.
- ckluis 12y ago
  Agreed - I think this was one of the best written descriptions of a problem and then a solution set. Enough so that even someone only quasi-technical could make sense of a good portion of it.
wcummings 12y ago
Why not just use Erlang?
- mratzloff 12y ago
  It seems he wanted a static type system, for starters.
  - rubiquity 12y ago
    There's a running joke in the Erlang community that every Erlang programmer will at some point try and re-implement Supervisors and do it horribly wrong. This isn't because the person is a bad programmer, it's just easy to underestimate the amount of time and incredible programming that has gone into Erlang's supervisors. By sticking with Go, the author gets to keep the static types but likely gets faux-Supervisors at best.
    - jerf 12y ago
      I describe exactly what I get. Whether they're "faux" depends on your definition. No, seriously, that's really profoundly true. Erlang's definition isn't the only one. But you're free to take it. If you can bend a bit, though, these may not be "faux". One thing that's easier when it comes to implementing supervisors in not-Erlang though is that the vast majority of languages have something better than "behaviors" to work with, which are bizarrely klunky things. I actually implemented something more like a "gen_server" initially, but it wasn't a win. Some of that "incredible programming" is essential complexity (in Fred Brooks' terminology), but some of it is accidental complexity. Plus Erlang had to build it from true, total scratch; Go is a pre-existing runtime, and "defer" is simple, yet powerful.
      - rvirding 12y ago
        Did you try to implement your own behaviours? Or processes that aren't behaviours as such but still function correctly in supervision trees and in applications?
        jerf 12y ago
        What I found is that if you implement Serve() and Stop(), honestly, you're done, in the world of Go. If you want a state machine, just write it. Many of the things I'm supervising are goroutines that wrap an API around a channel that implements a server.
  - polskibus 12y ago
    While not exactly static typing, you can add type annotations in Erlang and use dialyzer as a step in compilation to catch typing bugs.
    - tel 12y ago
      It's exactly static typing, just a very loose system which allows a lot of holes.
    - jerf 12y ago
      In the spirit of rubiquity's post, I'll take a faux supervisor tree and static typing over a "real" supervisor tree (sorry, gotta scare quote it) and faux static typing. It should be pointed out specifically that Erlang's type system is incredibly, annoyingly weak, more so than merely "weak typing". It's a language that doesn't permit any sort of user-created type whatsoever. There are good reasons for this which would be its own interesting blog post, mostly related to the need for a clear serialization for cluster communication, but I think those good reasons have been superceded by later work in the general programming world on serialization of objects and the tradeoff is wrong in 2014. (But, no sarcasm, a good one when it was created. It is no real criticism of a language to observe that it couldn't use lessons learned only a decade after it was created.)
      - dragonwriter 12y ago
        > It's a language that doesn't permit any sort of user-created object whatsoever. I think you are misusing "object" to mean something like "class" or "type" (the two not really being the same thing except in some statically-typed, class-based, OOPLs); Erlang quite obviously allows user-created objects.
        jerf 12y ago
        Yes, it should be "type". I've edited it, so let the record show that I did say that at first. And of course you can use what exists to build "types", such as a dict. But it's still just a record of a certain format rather than a distinct "type"; there's no way in the core system to assert that you have such a "dict", beyond having a record.
- jerf 12y ago
  I'm comfortable with it, but none of my team wants to use it. Even after 7 years of experience, it is very klunky to program in, and in reimplementing the Erlang program I had in hand, I immediately added some features that were very hard to implement in Erlang. And note the architecture of the code didn't actually change, this isn't one of those cases where the second version was also radically different than the first so no comparison is possible, it was actually hard to implement in Erlang. Erlang is a very klunky language to work in. Raw Java might be klunkier, but it's easily the klunkiest of the languages I use. Erlang's clustering, which my code heavily depends on, just sort of breaks sometimes, on machines that are on the same switch, with every bit of hardware switched out over time. It is very opaque to figure out why it isn't working, or how to make it work again. Mnesia breaks even more often on loads that shouldn't be a problem. After 7 years of working with it, at this point even if the solution was just to turn on this one flag somewhere and all my problems would go away, I still would not consider that a positive. I don't "hate" it, but I've been waiting for years for a more conventional language that had pervasive microthreading in it that I could use for work. Go is not quite it... I'd still like the goroutines to be completely isolated, and I've got a handful of other quibbles... but it's close enough that I might be able to get out of the hell of trying to multiprocess in most other languages, while being something I might actually be able to get adopted. In general, the company I work for has not been overwhelmed by the effectiveness of Erlang. In practice, the biggest loss of fundamental capability is loss of true asynchronous exceptions, and, well... in practice that's only an annoyance, not an OH MY GOSH I WILL NEVER USE A LANGUAGE WITHOUT ASYNCHRONOUS EXCEPTIONS sort of thing. In most cases if you're throwing an async exception you've really already lost and now we're just discussing exactly how you're losing. Every language student should study Erlang, and learn from it, and possibly use it. It's a solid language. And there's nothing wrong with trying to port some of that solidity elsewhere. (I'd pick a real DB over Mnesia, though.)
  - tormeh 12y ago
    Have you taken a look at Akka (with Scala)? You sacrifice code hot-swapping and I think little else, but you get JVM performance and one of the most extensible non-Lisp and safe non-Haskell languages in existence.
mratzloff 12y ago
This was interesting simply for the explanation of Erlang systems, but as for the project itself I currently use Supervisor (http://supervisord.org http://supervisord.org) for this purpose, and I've been pretty happy with it. Any reasons to make the switch?
- leef 12y ago
  For when you want to supervise goroutines instead of processes (Presumably because processes are consuming too many resources).
  - lucaspiller 12y ago
    Exactly. One of the big ideas of Erlang is to have many lightweight processes (processes in the VM, not OS). As an example say you have a chat system, you will have a process for each user [0]. If you have a million users, you will end up with a million processes. The supervisor tree watches these processes, so if one crashes a single user may get logged out but everybody else will be unaffected. [0] In reality you'll probably have it even more fine grained than that, so users will end up with multiple processes.
tel 12y ago
It would be interesting to see what it takes to get this functionality (at least locally) into Haskell given that there are asynchronous exceptions... even ones with arguably greater safety than their Erlang compatriots. A similar effort is happening with the Cloud Haskell program, but my understanding is that they're pouring a lot of effort into transmitting arbitrary Haskell functions over the wire between computers. This is pretty unnecessary for supervision and "Let it Crash"-style error handling alone.
- jeremyjh 12y ago
  That is a core primitive in Cloud Haskell but not the only one - you also send and receive data as in Erlang. distributed-process-platform provides Erlang style supervisors, gen_server, etc. It's all there and very much captures the features and spirit of OTP in a powerful static type system.
  - tel 12y ago
    I'll definitely give it another look. I may have just misunderstood where the effort was being spent.
    - coolsunglasses 12y ago
      You may be thinking of the statics stuff being done with GHC 7.10.
      - tel 12y ago
        I think I just missoveremphasized the importance of a particular paper on it that I read.
- thinkpad20 12y ago
  Oh, that's a bit disappointing. I thought Cloud Haskell was focused on developing actor-pattern abstractions like lightweight, concurrent threads and fault tolerance. Serializing functions seems like a real red herring.
spenrose 12y ago
I did something similar in Python, using Redis for the "operating system" and Linux processes. The system managed wide sets of parallel I/O operations with second-level latency. Worked great, albeit orders of magnitude slower than something like this. If you keep the supervisor on the same machine, you can do async exceptions via kill.
dman 12y ago
Any pointers on what blogging setup is being used at jerf.org for writing this post? I found the page pretty tasteful.
- buro9 12y ago
  It has all the signatures of a static site generator.

149 points by leef 12y ago

wyc 12y ago

Great article. I especially enjoyed the clear descriptive problem framing and discussion about Erlang processes versus goroutines.

ckluis 12y ago
Agreed - I think this was one of the best written descriptions of a problem and then a solution set. Enough so that even someone only quasi-technical could make sense of a good portion of it.

wcummings 12y ago

Why not just use Erlang?

mratzloff 12y ago
It seems he wanted a static type system, for starters.
- rubiquity 12y ago
  There's a running joke in the Erlang community that every Erlang programmer will at some point try and re-implement Supervisors and do it horribly wrong. This isn't because the person is a bad programmer, it's just easy to underestimate the amount of time and incredible programming that has gone into Erlang's supervisors. By sticking with Go, the author gets to keep the static types but likely gets faux-Supervisors at best.
  - jerf 12y ago
    I describe exactly what I get. Whether they're "faux" depends on your definition. No, seriously, that's really profoundly true. Erlang's definition isn't the only one. But you're free to take it. If you can bend a bit, though, these may not be "faux". One thing that's easier when it comes to implementing supervisors in not-Erlang though is that the vast majority of languages have something better than "behaviors" to work with, which are bizarrely klunky things. I actually implemented something more like a "gen_server" initially, but it wasn't a win. Some of that "incredible programming" is essential complexity (in Fred Brooks' terminology), but some of it is accidental complexity. Plus Erlang had to build it from true, total scratch; Go is a pre-existing runtime, and "defer" is simple, yet powerful.
    - rvirding 12y ago
      Did you try to implement your own behaviours? Or processes that aren't behaviours as such but still function correctly in supervision trees and in applications?
      - jerf 12y ago
        What I found is that if you implement Serve() and Stop(), honestly, you're done, in the world of Go. If you want a state machine, just write it. Many of the things I'm supervising are goroutines that wrap an API around a channel that implements a server.
- polskibus 12y ago
  While not exactly static typing, you can add type annotations in Erlang and use dialyzer as a step in compilation to catch typing bugs.
  - tel 12y ago
    It's exactly static typing, just a very loose system which allows a lot of holes.
  - jerf 12y ago
    In the spirit of rubiquity's post, I'll take a faux supervisor tree and static typing over a "real" supervisor tree (sorry, gotta scare quote it) and faux static typing. It should be pointed out specifically that Erlang's type system is incredibly, annoyingly weak, more so than merely "weak typing". It's a language that doesn't permit any sort of user-created type whatsoever. There are good reasons for this which would be its own interesting blog post, mostly related to the need for a clear serialization for cluster communication, but I think those good reasons have been superceded by later work in the general programming world on serialization of objects and the tradeoff is wrong in 2014. (But, no sarcasm, a good one when it was created. It is no real criticism of a language to observe that it couldn't use lessons learned only a decade after it was created.)
    - dragonwriter 12y ago
      > It's a language that doesn't permit any sort of user-created object whatsoever. I think you are misusing "object" to mean something like "class" or "type" (the two not really being the same thing except in some statically-typed, class-based, OOPLs); Erlang quite obviously allows user-created objects.
      - jerf 12y ago
        Yes, it should be "type". I've edited it, so let the record show that I did say that at first. And of course you can use what exists to build "types", such as a dict. But it's still just a record of a certain format rather than a distinct "type"; there's no way in the core system to assert that you have such a "dict", beyond having a record.
jerf 12y ago
I'm comfortable with it, but none of my team wants to use it. Even after 7 years of experience, it is very klunky to program in, and in reimplementing the Erlang program I had in hand, I immediately added some features that were very hard to implement in Erlang. And note the architecture of the code didn't actually change, this isn't one of those cases where the second version was also radically different than the first so no comparison is possible, it was actually hard to implement in Erlang. Erlang is a very klunky language to work in. Raw Java might be klunkier, but it's easily the klunkiest of the languages I use. Erlang's clustering, which my code heavily depends on, just sort of breaks sometimes, on machines that are on the same switch, with every bit of hardware switched out over time. It is very opaque to figure out why it isn't working, or how to make it work again. Mnesia breaks even more often on loads that shouldn't be a problem. After 7 years of working with it, at this point even if the solution was just to turn on this one flag somewhere and all my problems would go away, I still would not consider that a positive. I don't "hate" it, but I've been waiting for years for a more conventional language that had pervasive microthreading in it that I could use for work. Go is not quite it... I'd still like the goroutines to be completely isolated, and I've got a handful of other quibbles... but it's close enough that I might be able to get out of the hell of trying to multiprocess in most other languages, while being something I might actually be able to get adopted. In general, the company I work for has not been overwhelmed by the effectiveness of Erlang. In practice, the biggest loss of fundamental capability is loss of true asynchronous exceptions, and, well... in practice that's only an annoyance, not an OH MY GOSH I WILL NEVER USE A LANGUAGE WITHOUT ASYNCHRONOUS EXCEPTIONS sort of thing. In most cases if you're throwing an async exception you've really already lost and now we're just discussing exactly how you're losing. Every language student should study Erlang, and learn from it, and possibly use it. It's a solid language. And there's nothing wrong with trying to port some of that solidity elsewhere. (I'd pick a real DB over Mnesia, though.)
- tormeh 12y ago
  Have you taken a look at Akka (with Scala)? You sacrifice code hot-swapping and I think little else, but you get JVM performance and one of the most extensible non-Lisp and safe non-Haskell languages in existence.

mratzloff 12y ago

This was interesting simply for the explanation of Erlang systems, but as for the project itself I currently use Supervisor (http://supervisord.org http://supervisord.org) for this purpose, and I've been pretty happy with it. Any reasons to make the switch?

leef 12y ago
For when you want to supervise goroutines instead of processes (Presumably because processes are consuming too many resources).
- lucaspiller 12y ago
  Exactly. One of the big ideas of Erlang is to have many lightweight processes (processes in the VM, not OS). As an example say you have a chat system, you will have a process for each user [0]. If you have a million users, you will end up with a million processes. The supervisor tree watches these processes, so if one crashes a single user may get logged out but everybody else will be unaffected. [0] In reality you'll probably have it even more fine grained than that, so users will end up with multiple processes.

tel 12y ago

It would be interesting to see what it takes to get this functionality (at least locally) into Haskell given that there are asynchronous exceptions... even ones with arguably greater safety than their Erlang compatriots. A similar effort is happening with the Cloud Haskell program, but my understanding is that they're pouring a lot of effort into transmitting arbitrary Haskell functions over the wire between computers. This is pretty unnecessary for supervision and "Let it Crash"-style error handling alone.

jeremyjh 12y ago
That is a core primitive in Cloud Haskell but not the only one - you also send and receive data as in Erlang. distributed-process-platform provides Erlang style supervisors, gen_server, etc. It's all there and very much captures the features and spirit of OTP in a powerful static type system.
- tel 12y ago
  I'll definitely give it another look. I may have just misunderstood where the effort was being spent.
  - coolsunglasses 12y ago
    You may be thinking of the statics stuff being done with GHC 7.10.
    - tel 12y ago
      I think I just missoveremphasized the importance of a particular paper on it that I read.
thinkpad20 12y ago
Oh, that's a bit disappointing. I thought Cloud Haskell was focused on developing actor-pattern abstractions like lightweight, concurrent threads and fault tolerance. Serializing functions seems like a real red herring.

spenrose 12y ago

I did something similar in Python, using Redis for the "operating system" and Linux processes. The system managed wide sets of parallel I/O operations with second-level latency. Worked great, albeit orders of magnitude slower than something like this. If you keep the supervisor on the same machine, you can do async exceptions via kill.

dman 12y ago

Any pointers on what blogging setup is being used at jerf.org for writing this post? I found the page pretty tasteful.

buro9 12y ago
It has all the signatures of a static site generator.