4 ms·

There was another recent Erlang post which posited the suggestion that the power of erlang was not in lightweight processes and message parsing, but in generic

There was another recent Erlang post which posited the suggestion that the power of erlang was not in lightweight processes and message parsing, but in generic components erlang calls behaviours. These generic patterns are what allows for building reliable distributed systems, and Joe Armstrong wrote a PhD thesis about it. I gave an anecdote of having built a c++ framework that unwittingly followed the same principles for “big corp” long ago (convergent design), and posited that the same concepts could probably be built in any language. Here it is for go. So is it a good thing? Previously I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure. The problem I have is that as you start looking at these higher level patterns involving distributed computing, the landscape of frameworks seems to explode. Is this because the erlang/otp concepts are insufficient or outdated?

digilypse 4y ago
Maybe it’s because the frameworks themselves are less important than the *human language used to describe them. To someone who knows what a Genserver is, or has used one before, what does it matter which of the many frameworks they’re using?
- lll-o-lll 4y ago
  It just seems like we reinvent the same things again and again. Maybe that’s because at this level you can’t have a standard and things need to be bespoke. It could, on the other hand, be an indication that there is a huge amount of wasted effort in revisiting long ago solved problems.
  - digilypse 4y ago
    I agree, I just think it’s part of the evolutionary process. We reinvent them because the core concepts are right, but there isn’t enough connective tissue to re-use the implementations directly. I don’t see it as a bad thing. The concepts persist and gain momentum and with each iteration we get closer to something that can actually be relied on consistently without replication. There’s wasted effort, but something about the original was missing that would have made it the obvious choice. I think it feels wasted mostly because it isn’t yet obvious what the missing pieces were.
    - lll-o-lll 4y ago
      I just don’t think this is true. Erlang has existed since the early 80s, open source in the late 90s. Joe Armstrong wrote his thesis in 2003. It’s been 2 decades, and yet reliable distributed systems are still something companies fail at consistently. The latest trend with kubernetes and web interfaces everywhere is, in my opinion, an example of these ideas never gaining popularity so we outsourced it to infrastructure (then realised it needed code and made a whole new class of network/developer, devops). Erlang basically posited that “process isolation” as a building block was fundamental to reliable distributed systems in the presence of software errors. I don’t think that’s ever been really challenged, but the final solution we have today seems almost ludicrously inefficient in comparison. I just hope lunatic gets traction!
jerf 4y ago
"Is this because the erlang/otp concepts are insufficient or outdated?" Insufficient or outdated is a pretty loaded term. What they are is really, really tied to the Erlang worldview, of immutable functions, mailbox APIs that can receive messages out of order, the way they have "behaviors" as a really limited kind of class system, and other details of Erlang. This isn't a criticism, it's really a good thing. An Erlang library should harmonize with Erlang, after all. But when you port them into environments that don't have all those characteristics, the optimal library that harmonizes with that environment looks different. In Go, for instance, rather than the whole gen_server thing, the idiomatic approach to this sort of thing tends to look a lot more like: for { select { case msg1 := me.Chan1: me.handleMsg1(msg1) case msg2 := me.Chan2: me.handleMsg2(msg2) // etc. } } Trying to jam Go code into the gen_server framework is very klunky. Another reason is that while Go and Erlang both have fairly weak type systems, they are weak in very different ways. It is idiomatic in Erlang to label what kind of message something is by making it the first atom in a tuple. Then you can use Erlang's pattern matching in functions to implement many different message types based on what you send to the gen_server. This ports into Go terribly. Go does it in very different ways. "I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure." No, it really doesn't work. Even two languages in the same basic family can end up with significant differences in patterns (witness Ruby versus Python, for instance, virtually the same language, significant difference in what is idiomatic). Trying to stretch a library from a language that is based on a super weak type system (on purpose; it is part of the Erlang solution to cross-version communication), immutability, pattern matching, and mailboxes into a language with a weak type system in a different way, mutability, conventional function calls, channels (very different than mailboxes), and all the other relevant differences may be theoretically possible, but it will always result in foreign environments in the language. You can actually see this in the real world by the way that C is the de facto cross-language in-process bus on Linux. It bends everything that has to use it. Personally I consider it one of the biggest impediments to a new language getting off the ground because of the design decisions it forces on a new language at a young age. We are, slowly, getting past this in a variety of ways, but it's hard.
- pdimitar 4y ago
h0l0cube 4y ago
> it would be good to capture these core concepts in standardized libraries A lot of Erlang's performance and robustness characteristics are enabled at both the language level and BEAM VM level (e.g., immutability, memory model, processes, supervision trees, lightweight pre-emption). Whole classes of problems like deadlocks, memory management, and GC pauses go away, enabling an ergonomic, expressive language and runtime for concurrent systems.

digilypse 4y ago

Maybe it’s because the frameworks themselves are less important than the *human language used to describe them. To someone who knows what a Genserver is, or has used one before, what does it matter which of the many frameworks they’re using?

lll-o-lll 4y ago
It just seems like we reinvent the same things again and again. Maybe that’s because at this level you can’t have a standard and things need to be bespoke. It could, on the other hand, be an indication that there is a huge amount of wasted effort in revisiting long ago solved problems.
- digilypse 4y ago
  I agree, I just think it’s part of the evolutionary process. We reinvent them because the core concepts are right, but there isn’t enough connective tissue to re-use the implementations directly. I don’t see it as a bad thing. The concepts persist and gain momentum and with each iteration we get closer to something that can actually be relied on consistently without replication. There’s wasted effort, but something about the original was missing that would have made it the obvious choice. I think it feels wasted mostly because it isn’t yet obvious what the missing pieces were.
  - lll-o-lll 4y ago
    I just don’t think this is true. Erlang has existed since the early 80s, open source in the late 90s. Joe Armstrong wrote his thesis in 2003. It’s been 2 decades, and yet reliable distributed systems are still something companies fail at consistently. The latest trend with kubernetes and web interfaces everywhere is, in my opinion, an example of these ideas never gaining popularity so we outsourced it to infrastructure (then realised it needed code and made a whole new class of network/developer, devops). Erlang basically posited that “process isolation” as a building block was fundamental to reliable distributed systems in the presence of software errors. I don’t think that’s ever been really challenged, but the final solution we have today seems almost ludicrously inefficient in comparison. I just hope lunatic gets traction!

jerf 4y ago

"Is this because the erlang/otp concepts are insufficient or outdated?" Insufficient or outdated is a pretty loaded term. What they are is really, really tied to the Erlang worldview, of immutable functions, mailbox APIs that can receive messages out of order, the way they have "behaviors" as a really limited kind of class system, and other details of Erlang. This isn't a criticism, it's really a good thing. An Erlang library should harmonize with Erlang, after all. But when you port them into environments that don't have all those characteristics, the optimal library that harmonizes with that environment looks different. In Go, for instance, rather than the whole gen_server thing, the idiomatic approach to this sort of thing tends to look a lot more like: for { select { case msg1 := me.Chan1: me.handleMsg1(msg1) case msg2 := me.Chan2: me.handleMsg2(msg2) // etc. } } Trying to jam Go code into the gen_server framework is very klunky. Another reason is that while Go and Erlang both have fairly weak type systems, they are weak in very different ways. It is idiomatic in Erlang to label what kind of message something is by making it the first atom in a tuple. Then you can use Erlang's pattern matching in functions to implement many different message types based on what you send to the gen_server. This ports into Go terribly. Go does it in very different ways. "I suggested it would be good to capture these core concepts in standardized libraries for a range of languages, but now I am not so sure." No, it really doesn't work. Even two languages in the same basic family can end up with significant differences in patterns (witness Ruby versus Python, for instance, virtually the same language, significant difference in what is idiomatic). Trying to stretch a library from a language that is based on a super weak type system (on purpose; it is part of the Erlang solution to cross-version communication), immutability, pattern matching, and mailboxes into a language with a weak type system in a different way, mutability, conventional function calls, channels (very different than mailboxes), and all the other relevant differences may be theoretically possible, but it will always result in foreign environments in the language. You can actually see this in the real world by the way that C is the de facto cross-language in-process bus on Linux. It bends everything that has to use it. Personally I consider it one of the biggest impediments to a new language getting off the ground because of the design decisions it forces on a new language at a young age. We are, slowly, getting past this in a variety of ways, but it's hard.

pdimitar 4y ago

h0l0cube 4y ago

> it would be good to capture these core concepts in standardized libraries A lot of Erlang's performance and robustness characteristics are enabled at both the language level and BEAM VM level (e.g., immutability, memory model, processes, supervision trees, lightweight pre-emption). Whole classes of problems like deadlocks, memory management, and GC pauses go away, enabling an ergonomic, expressive language and runtime for concurrent systems.