5 ms·

This article got me to go figure out precisely what Erlang processes are. Tl;dr - they aren't OS processes. So it is still conceivable that an error in Erlang c

This article got me to go figure out precisely what Erlang processes are. Tl;dr - they aren't OS processes. So it is still conceivable that an error in Erlang can bring down all your web servers. http://stackoverflow.com/questions/2708033/technically-why-are-processes-in-erlang-more-efficient-than-os-threads http://stackoverflow.com/questions/2708033/technically-why-a...

ghayes 11y ago
Well, not if you distribute your erlang processes across several physical nodes.
- jbardnz 11y ago
  You mean just like you can with Apache or Nginx?
  - felixgallo 11y ago
    No. Not at all like those.
    - niij 11y ago
      Can you please expand on why?
      - vertex-four 11y ago
        The processes can communicate with processes on other servers in exactly the same way that they can communicate with processes on the same server. If you have one traditional application server and want to do some form of cross-user interaction - let's say chat - you can do that trivially, put it in a queue for that user in a global map. Now when you outgrow that server, you need to rewrite all your code to understand the concept of users being on other servers or use an external message queueing system. In Erlang, all of this is built in by default - if you write code for the OTP framework (the standard library for dealing with messaging, process supervision, etc), all you need to do is connect the two servers together and point them at the same shared user->process mapping process (which you have to build whether you're dealing with one server or 20, as there's no global data otherwise). Of course, if you have absolutely no direct interaction between users, it's trivial to scale anything - fire up a new server and direct some portion of your traffic at it. Erlang's trick is to make it that easy even when you do have direct interaction between users. And of course that works for even backend workloads - if you have a backend server that your frontend servers talk to and need to scale it, if you've coded in Erlang and put as much logic as possible in per-connection processes, you're probably a significant chunk of the way there.
        ww520 11y ago
        What does any of these have to do with the equivalence between Apache/Ngix and Erlang when running multiple servers of them to avoid crashing all?
        vertex-four 11y ago
        You can't use traditional servers to scale in lots of cases without a lot of additional development work. You can use Erlang servers to do so. Therefore, they're not the same - Erlang covers a broader range of use cases.
    - reinhardt1053 11y ago
      A constructive comment would be appreciated.
  - statictype 11y ago
    Yes - it's exactly like that. The Erlang VM itself has to be rock solid just like you expect Apache and Nginx to be rock solid. Fortunately this more or less seems to be the case.
  - phamilton 11y ago
    I would say a better comparison is the Linux kernel. If the kernel crashes, everything on that box goes.
- ww520 11y ago
  The Erlang error in code could be replicated to every single physical node and all of them would crash.
  - felixgallo 11y ago
    That is not what happens.
  - rusanu 11y ago
    Presumably one would catch such a blatant error early in testing. The problem is the odd case, the code that only crashes in a rare state, never seen in test. When the odd state does occur the effect is mild, as only one actor dies as opposed to the entire OS process.
    - ww520 11y ago
      I was responding to the assertion that distributing code to multiple "processes" would avoid crashing all of them, which is simply not true. Even the rare state you mentioned would crash all the processes. The rare state would be an input that testing failed to catch. The code in any of the processes encountering the rare input could crash.
tvon 11y ago
I don't follow, how does Erlang processes not being OS processes relate to what could happen if an Erlang error occurred?
signa11 11y ago
> ... So it is still conceivable that an error in Erlang... are you talking about erlang-runtime here ? it is not very clear what you are implying. can you please elaborate ? if i take your statement to mean 'erlang runtime' then this is no different than saying, if there is a bug in glibc which gets tickled when i do something, then my process will crash, or if there is a bug in jvm, then the application hosted on the jvm will crash etc. etc.
- akkartik 11y ago
  From OP: "In Erlang we create very lightweight processes, one per connection and within that process spin up a web server. So we might end up with a few million web-servers with one user each." I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang, so I was curious if the word 'process' implied some extra level of insulation. Reading all the comments here it's still not clear to me what benefit is being obtained by having "2 million web servers" rather than 2 million connections.
  - phamilton 11y ago
    Shared memory space is one thing. A process has isolated memory. An error in a spawned thread could mess up shared memory, crashing multiple threads. An error in a process can't mess with the memory space of other processes. Since there's no shared memory, performance characteristics change. Garbage collection is done per process and so "stop the world" doesn't happen, meaning each individual "web server" isn't at the mercy of other requests for performance. It's lots of little things like that.
  - rdtsc 11y ago
    > I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang, That is a huge difference. Otherwise by now, why would Erlang not just run pthread calls and just spawn a thread. Why bother doing all that work? The reason is because Erlang processes do not share heap memory. That is one of the most important features of Erlang runtime. A crashing process can just crash on its own without leaving all the other 1999999 processes in an undetermined state.
toast0 11y ago
An error in the beam VM (or a nif) could lead to crash of the OS process, and related termination of all of the erlang processes within. However, the beam VM is pretty robust, because of the design, it turns out to be quite small and relatively simple[1], so you don't tend to end up with things were weird edge cases cause terrible crashes that are hard to find. For nifs you write yourself, it's up to you :) Process isolation isn't complete though, one erlang process can use all of many of the limited resources and end up with the whole vm shutting down. A single process can't use all the CPU, but it can use all the memory, all the atoms, all the filedescriptors, all the ets tables, etc; basically everything in the system limits[2] [1] Calling it simple is a compliment -- a normal person can look at the vm code and understand it, if they need to. [2] http://erlang.org/doc/efficiency_guide/advanced.html#id70200 http://erlang.org/doc/efficiency_guide/advanced.html#id70200
- akkartik 11y ago
  Thanks, yours was finally the comment I was hoping for. Could you point me at the right place to read (or read about) the beam sources? I'm looking at https://github.com/erlang/otp/wiki/Routemap-source-tree https://github.com/erlang/otp/wiki/Routemap-source-tree -- is there something better?
  - toast0 11y ago
    You're in the right place; unfortunately there's not a lot of good documentation on the internals (although do check out erts/emulator/internal_doc ). It's much easier to dive into the source if you have something specific on your mind (like debugging a performance problem, or if you find an beam crash...). erts/emulator/beam has the most basic parts of the virtual machine -- processes, memory allocation, process switching, message passing, interfaces to the operating system (although a lot of that is in erts/emulator/drivers), etc.
rdtsc 11y ago
> So it is still conceivable that an error in Erlang can bring down all your web servers. No it won't. Not if you mean an error in an Erlang process. It almost sounds impossible, but that is the beauty of the BEAM virtual machine. It is really a marvel of engineering. Now if you mean an error in Erlang VM itself, then yeah, if that crashes it will bring down the all the connections. But even that there is a answer -- distributed Erlang. Erlang VM can talk to processes on other Erlang VMs, even if those are running on another server on other side of the planet.

ghayes 11y ago

Well, not if you distribute your erlang processes across several physical nodes.

jbardnz 11y ago
You mean just like you can with Apache or Nginx?
- felixgallo 11y ago
  No. Not at all like those.
  - niij 11y ago
    Can you please expand on why?
    - vertex-four 11y ago
      The processes can communicate with processes on other servers in exactly the same way that they can communicate with processes on the same server. If you have one traditional application server and want to do some form of cross-user interaction - let's say chat - you can do that trivially, put it in a queue for that user in a global map. Now when you outgrow that server, you need to rewrite all your code to understand the concept of users being on other servers or use an external message queueing system. In Erlang, all of this is built in by default - if you write code for the OTP framework (the standard library for dealing with messaging, process supervision, etc), all you need to do is connect the two servers together and point them at the same shared user->process mapping process (which you have to build whether you're dealing with one server or 20, as there's no global data otherwise). Of course, if you have absolutely no direct interaction between users, it's trivial to scale anything - fire up a new server and direct some portion of your traffic at it. Erlang's trick is to make it that easy even when you do have direct interaction between users. And of course that works for even backend workloads - if you have a backend server that your frontend servers talk to and need to scale it, if you've coded in Erlang and put as much logic as possible in per-connection processes, you're probably a significant chunk of the way there.
      - ww520 11y ago
        What does any of these have to do with the equivalence between Apache/Ngix and Erlang when running multiple servers of them to avoid crashing all?
        vertex-four 11y ago
        You can't use traditional servers to scale in lots of cases without a lot of additional development work. You can use Erlang servers to do so. Therefore, they're not the same - Erlang covers a broader range of use cases.
  - reinhardt1053 11y ago
    A constructive comment would be appreciated.
- statictype 11y ago
  Yes - it's exactly like that. The Erlang VM itself has to be rock solid just like you expect Apache and Nginx to be rock solid. Fortunately this more or less seems to be the case.
- phamilton 11y ago
  I would say a better comparison is the Linux kernel. If the kernel crashes, everything on that box goes.
ww520 11y ago
The Erlang error in code could be replicated to every single physical node and all of them would crash.
- felixgallo 11y ago
  That is not what happens.
- rusanu 11y ago
  Presumably one would catch such a blatant error early in testing. The problem is the odd case, the code that only crashes in a rare state, never seen in test. When the odd state does occur the effect is mild, as only one actor dies as opposed to the entire OS process.
  - ww520 11y ago
    I was responding to the assertion that distributing code to multiple "processes" would avoid crashing all of them, which is simply not true. Even the rare state you mentioned would crash all the processes. The rare state would be an input that testing failed to catch. The code in any of the processes encountering the rare input could crash.

tvon 11y ago

I don't follow, how does Erlang processes not being OS processes relate to what could happen if an Erlang error occurred?

signa11 11y ago

> ... So it is still conceivable that an error in Erlang... are you talking about erlang-runtime here ? it is not very clear what you are implying. can you please elaborate ? if i take your statement to mean 'erlang runtime' then this is no different than saying, if there is a bug in glibc which gets tickled when i do something, then my process will crash, or if there is a bug in jvm, then the application hosted on the jvm will crash etc. etc.

akkartik 11y ago
From OP: "In Erlang we create very lightweight processes, one per connection and within that process spin up a web server. So we might end up with a few million web-servers with one user each." I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang, so I was curious if the word 'process' implied some extra level of insulation. Reading all the comments here it's still not clear to me what benefit is being obtained by having "2 million web servers" rather than 2 million connections.
- phamilton 11y ago
  Shared memory space is one thing. A process has isolated memory. An error in a spawned thread could mess up shared memory, crashing multiple threads. An error in a process can't mess with the memory space of other processes. Since there's no shared memory, performance characteristics change. Garbage collection is done per process and so "stop the world" doesn't happen, meaning each individual "web server" isn't at the mercy of other requests for performance. It's lots of little things like that.
- rdtsc 11y ago
  > I don't see the difference between spawning a thread for each connection in other languages and spawning a process for each connection in Erlang, That is a huge difference. Otherwise by now, why would Erlang not just run pthread calls and just spawn a thread. Why bother doing all that work? The reason is because Erlang processes do not share heap memory. That is one of the most important features of Erlang runtime. A crashing process can just crash on its own without leaving all the other 1999999 processes in an undetermined state.

toast0 11y ago

An error in the beam VM (or a nif) could lead to crash of the OS process, and related termination of all of the erlang processes within. However, the beam VM is pretty robust, because of the design, it turns out to be quite small and relatively simple[1], so you don't tend to end up with things were weird edge cases cause terrible crashes that are hard to find. For nifs you write yourself, it's up to you :) Process isolation isn't complete though, one erlang process can use all of many of the limited resources and end up with the whole vm shutting down. A single process can't use all the CPU, but it can use all the memory, all the atoms, all the filedescriptors, all the ets tables, etc; basically everything in the system limits[2] [1] Calling it simple is a compliment -- a normal person can look at the vm code and understand it, if they need to. [2] http://erlang.org/doc/efficiency_guide/advanced.html#id70200 http://erlang.org/doc/efficiency_guide/advanced.html#id70200

akkartik 11y ago
Thanks, yours was finally the comment I was hoping for. Could you point me at the right place to read (or read about) the beam sources? I'm looking at https://github.com/erlang/otp/wiki/Routemap-source-tree https://github.com/erlang/otp/wiki/Routemap-source-tree -- is there something better?
- toast0 11y ago
  You're in the right place; unfortunately there's not a lot of good documentation on the internals (although do check out erts/emulator/internal_doc ). It's much easier to dive into the source if you have something specific on your mind (like debugging a performance problem, or if you find an beam crash...). erts/emulator/beam has the most basic parts of the virtual machine -- processes, memory allocation, process switching, message passing, interfaces to the operating system (although a lot of that is in erts/emulator/drivers), etc.

rdtsc 11y ago

> So it is still conceivable that an error in Erlang can bring down all your web servers. No it won't. Not if you mean an error in an Erlang process. It almost sounds impossible, but that is the beauty of the BEAM virtual machine. It is really a marvel of engineering. Now if you mean an error in Erlang VM itself, then yeah, if that crashes it will bring down the all the connections. But even that there is a answer -- distributed Erlang. Erlang VM can talk to processes on other Erlang VMs, even if those are running on another server on other side of the planet.