3 ms·

If I remember correctly I found a few edge cases, but they weren't ever hit by OTP, just by partially implemented VMs like mine :) It was kind of interesting e

If I remember correctly I found a few edge cases, but they weren't ever hit by OTP, just by partially implemented VMs like mine :) It was kind of interesting exploring the OTP internals, especially some of the parts that haven't changed in a long time. One example is the PAM: I think it stood for "patrick's abstract machine" and it would compile erlang terms into bytecode for pattern matches (intended for fast ETS lookups). It's all there in one file and it took a fair bit of digging to figure out how it works since it's been static for a long while and nothing on the internet really documented it.

d4mi3n 6y ago
Hah! Great find! If this is still the case you should definitely consider contributing to the documentation of those files. Odds are they'll be used by the next person to try something similar. :)
callamdelaney 6y ago
The pattern matching algorithm was originally based on the algorithm described in `The implementation of Functional Programming Languages`, the 1987 edition (there are two versions, one is more basic). Edit: this book is available for free here: https://www.microsoft.com/en-us/research/publication/the-implementation-of-functional-programming-languages https://www.microsoft.com/en-us/research/publication/the-imp...
- foota 6y ago
  I was really hoping this comment was going to be by patrick in that weird synchronicity that is demonstrated on hn :-)
bogomipz 6y ago
>"It was kind of interesting exploring the OTP internals, especially some of the parts that haven't changed in a long time." I have heard similar before at an Erlang meetup. Are there elements of the VM you encountered that were also static and lacking sufficient documentation? I'm guessing much of this is just tribal knowledge deep inside Ericsson then? It would be great if there were a public repository for these things.
- bitwalker 6y ago
  There are really two major pieces of the BEAM, the parts implemented in Erlang (e.g. the compiler, OTP), and the parts implemented in C (the VM, or emulator as it is called in the codebase). Virtually all of the C code is undocumented in any meaningful way, short of some internal documentation on a handful of topics, as well as a those parts of the code which have thorough comments that explain some tricky aspect of the implementation. From my own experience, those parts that are commented or documented tend to clarify some specific design constraints (for example, why processes have multiple locks on different parts, and why they are locked in a specific order, or the rationale of the carrier design); but you never really get a clear picture of why things overall are architected the way they are overall, what designs were considered and discarded due to some deficiency, what tradeoffs were made, etc. I think much of the actual content like that which may exist, is either buried in the minds of the original engineers, or in some internal documentation at Ericsson that has never been released. My suspicion is that you'd need to dig through mountains of emails and such to piece together a more complete picture of how things where put together over time. It's also the fact that the BEAM just has a lot of really complex pieces built in to it after all this time. Everything from binary pattern matching and construction, to garbage collection and memory management, ETS, Mnesia, etc. Each one of those things is not only non-trivial, but have evolved significantly over time, through the hands of many engineers. It also doesn't help that large portions of the C implementation are written in an extremely macro heavy style, which makes it quite hard to read without knowing what all the macros do and how they play together. Projects like Enigma, or Lumen, have a lot to give back to the community in the form of documenting how these pieces are built. Unfortunately, the lack of a specification for the Erlang language and its runtime, means it is very much a grind to work out how things are currently implemented, and why.
  - bogomipz 6y ago
    Thanks for the comprehensive insight. This does kind of invite the question though that how they themselves maintain these code bases if it's steeped in such arcana. Is there not the danger that this becomes similar to the situation with mainframes and Cobol for them?

d4mi3n 6y ago

Hah! Great find! If this is still the case you should definitely consider contributing to the documentation of those files. Odds are they'll be used by the next person to try something similar. :)

callamdelaney 6y ago

The pattern matching algorithm was originally based on the algorithm described in `The implementation of Functional Programming Languages`, the 1987 edition (there are two versions, one is more basic). Edit: this book is available for free here: https://www.microsoft.com/en-us/research/publication/the-implementation-of-functional-programming-languages https://www.microsoft.com/en-us/research/publication/the-imp...

foota 6y ago
I was really hoping this comment was going to be by patrick in that weird synchronicity that is demonstrated on hn :-)

bogomipz 6y ago

>"It was kind of interesting exploring the OTP internals, especially some of the parts that haven't changed in a long time." I have heard similar before at an Erlang meetup. Are there elements of the VM you encountered that were also static and lacking sufficient documentation? I'm guessing much of this is just tribal knowledge deep inside Ericsson then? It would be great if there were a public repository for these things.

bitwalker 6y ago
There are really two major pieces of the BEAM, the parts implemented in Erlang (e.g. the compiler, OTP), and the parts implemented in C (the VM, or emulator as it is called in the codebase). Virtually all of the C code is undocumented in any meaningful way, short of some internal documentation on a handful of topics, as well as a those parts of the code which have thorough comments that explain some tricky aspect of the implementation. From my own experience, those parts that are commented or documented tend to clarify some specific design constraints (for example, why processes have multiple locks on different parts, and why they are locked in a specific order, or the rationale of the carrier design); but you never really get a clear picture of why things overall are architected the way they are overall, what designs were considered and discarded due to some deficiency, what tradeoffs were made, etc. I think much of the actual content like that which may exist, is either buried in the minds of the original engineers, or in some internal documentation at Ericsson that has never been released. My suspicion is that you'd need to dig through mountains of emails and such to piece together a more complete picture of how things where put together over time. It's also the fact that the BEAM just has a lot of really complex pieces built in to it after all this time. Everything from binary pattern matching and construction, to garbage collection and memory management, ETS, Mnesia, etc. Each one of those things is not only non-trivial, but have evolved significantly over time, through the hands of many engineers. It also doesn't help that large portions of the C implementation are written in an extremely macro heavy style, which makes it quite hard to read without knowing what all the macros do and how they play together. Projects like Enigma, or Lumen, have a lot to give back to the community in the form of documenting how these pieces are built. Unfortunately, the lack of a specification for the Erlang language and its runtime, means it is very much a grind to work out how things are currently implemented, and why.
- bogomipz 6y ago
  Thanks for the comprehensive insight. This does kind of invite the question though that how they themselves maintain these code bases if it's steeped in such arcana. Is there not the danger that this becomes similar to the situation with mainframes and Cobol for them?