8 ms·

The Erlang Shell

127 points by strmpnk 13y ago

brickcap 13y ago
> As an Erlang programmer I often claim that “You can’t pick parts of Erlang and then claim you have the same functionality. It all comes together as a whole”. I am guilty of this. I am relatively new to erlang programming and I have skipped on some of the features that make the language great. For instance I have neglected learning both supervision and hot code reloading.
- banachtarski 13y ago
  Well you're relatively new :). It takes time to absorb an entire language's ecosystem. As long as you take it upon yourself to acquaint yourself with those features, things will be fine. The biggest thing to get on your initial survey should be minimal awareness so you know that that feature exists when you encounter the problem it was originally built to solve. Supervision though, is fundamental to OTP and Erlang. It's certainly easy to learn so I would consider that a low hanging fruit for you. Hot code reloading is a consequence of the continuation passing style that is prevalent in every Erlang process and easy to understand too (although employing it in production can be a bit trickier).
- lostcolony 13y ago
  Hot code reloading I'd personally grant you as an exception; it's a maintenance tool, not a development tool, and you can largely ignore it until you want it, if you're adhering to OTP principles. Ignoring supervision, however, is...not wise, if you have -any- sort of reliability considerations (even ones as weak as "I'd prefer it to remain up"; if it's just a one off thing that you'll manually execute again if it fails, meh, fine). Supervision is really, really easy to use, at least for the base use case (creating a good hierarchy, so that processes restart the other processes that should be restarted, is an architectural consideration that can be a bit hairy), and it's truly heartening to grep a log for something you've been running a while, find crashes that indicate issues you want to fix (even simple stuff, like sanitizing inputs), but the system is still running as though nothing happened.
- brickcap 13y ago
  @lostcolony @banachtarski thanks guys. I will start with supervision first then. See if I can make use of it in some of my existing code.
flibertgibit 13y ago
FUCK erlang
rurban 13y ago
Oh nice. They have now almost a Common Lisp environment. Getting closer to Greenspun's tenth rule.
- bitwalker 13y ago
  Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang.
  - devgutt 13y ago
    An interesting example of Erlang hot code reloading in embedded system: http://www.youtube.com/watch?v=96UzSHyp0F8 http://www.youtube.com/watch?v=96UzSHyp0F8
  - agentultra 13y ago
    > Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? Yes. > I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang. It's not. iIRC there's even a version of the JVM with code reloading. What's interesting about Common Lisp is that not only can code be dynamically compiled into the running image but class definitions can be changed and live instances will be updated without stopping the program or reloading anything. There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software.
    - simula67 13y ago
      > there's even a version of the JVM with code reloading. Yes, because of Java's extensible classloading mechanism, all you have to do is drop an war/ear to the 'deploy' folder of your application server. New files are deployed, rewritten files are redeployed and deleted files are undeployed.
      - phamilton 13y ago
tel 13y ago
I find it funny that the author willingly gave up static typing to have Erlang process control—I made the opposite switch as I grew tired of the overhead and attrition of modeling a complex domain in Erlang. I think the Erlang VM is easily my favorite place to live as a programmer, but I do with Erlang-the-language gave more.
- lostcolony 13y ago
  What were your thoughts re: Dialyzer?
  - tel 13y ago
    It's nice and once I learned how to use it I tried to be very consistent with it. The success types provide good documentation, but I found that the types still felt very flat. The fundamental component of Erlang abstraction is the process and the process tree, so your types reflect that and don't end up composing well.
    - tel 13y ago
      (To clarify: rather your types end up composing much like messages between actors. That's a very fine way to compose things, but it's still just one way)
rdtsc 13y ago
> When systems have faults due to concurrency and distribution, debuggers will not work. Those are very tricky indeed, mix in threads with pointers and a system becomes haunted. "This one customer noticed a crash, on this hardware, after running for 10 weeks, but ... we couldn't reproduce it". "Oh I suspect there is a memory overwrite or use after free that corrupted the data". People start doubting their sanity -- "Oh I swear I saw it crash, that one time, or ...did I, maybe I was tired...". Someone (could have even been Jesper Andersen, the author) said that the biggest performance gain an application gets is when it goes from not working or being crashed to working. And the biggest slowdown is also when it goes from working to crashing unexpectedly. There was talk of 60 hour weeks here before, one of the things that happens at 8pm at night is people huddled over a keyboard debugging some of these hard to track bugs. Managers and some programmers see it as great heroism, pats on the back for everyone, but, others see it as reaping the fruits of previous bad decisions and it is a rather sad sight for some. It all depends on the perspective. I guess the point is one of the main qualities of Erlang is not concurrency but fault tolerance. Many systems copy the actor concurrency patterns and claim "we have Erlang now but it is also faster!", and that is a good thing, but I haven't heard too many claim "We copied Erlang's fault tolerance, and it is even better than Erlang's!". [+] [+] you can do the same pattern up to a point using OS processes, LXC containers, separate servers, having watchdog processes restart workers, for example.
- chubot 13y ago
  Nitpick: mixing threads with pointers isn't really the problem; it's mixing threads with shared mutable state. A concurrency idiom I've gotten a lot of mileage out of is to use threads, without shared state, but with pointers. Instead of shared state, you use a message passing style where you pass pointers over queues. Threads only mutate data that they are explicitly handed via a queue. After putting an object on a queue, they NULL out their local reference. It's basically like Go's goroutine idiom, but with C/C++/Python, real OS threads, and a thread safe queue (which is all a channel is). There is also some relation to the SEDA architecture. The logical structure of such a program is not really that different than an Erlang program, I imagine. It just works a little better with existing C/C++ codebases, and is potentially faster in some cases. It has a completely different character than spaghetti written using threads and shared state, and can be made extremely reliable. Especially because the explicit channels provide a hook for testing black box behavior.
phamilton 13y ago
I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). Attaching to a process is useless, but the Rails console essentially gives you an isolated process to inspect production code and data. I don't see how extending that to an existing process would be useful. So, challenging my assumptions, perhaps short-lived processes aren't the best approach for a web service. Perhaps that's the paradigm in Rails because Ruby lacks the tooling to make a long-lived process easy to maintain. If a long-lived process is as easy to maintain as short-lived processes, then suddenly new architectures open up. How are these architectures different? What are the benefits? What are some examples?
- diminish 13y ago
  curious, which type of projects do you plan to develop in Erlang? especially after rails.
  - phamilton 13y ago
    That's the question I can't answer. I have some problems I want to solve. If I build them in Erlang the way I would on Rails, then there's not much gain. I'm trying to figure out how Erlang will allow me to solve those problems differently. One problem is how to handle authentication across our private and public APIs. With Rails, I've got a db with credentials and tons of memcached running to minimize latency (since every endpoint needs to check auth). How would a long running application in Erlang handle that differently? How is that better than what we currently have?
- biokoda 13y ago
  When you are running in production and something is not right, you can attach to the console and see what is going on. This is of course not something that you should do often, but can be a real life saver. Simulating something that is designed to work with thousands or even millions of users at the same time is pretty much impossible. Hot code loading and attaching to console gives you the option to see what is going on in real time and even fix it if you have to.
_pmf_ 13y ago
> And all this without service interruption. And you get all this for free, just because you picked Erlang as the implementation language. GDB or even VS's debugger can do a lot of magic even if you use C. Until absolutely necessary, leave debugging information reasonably high and optimization reasonably low or off and you'll get an amazing, dynamic environment to inspect a live C system.
skywhopper 13y ago
Reading about "correct software" as if it were an realistic goal makes me itch. Sure, we should be striving for as few bugs as possible, but any non-trivial application will never be made "correct", if only because it'd be impossible to write the full specification to compare it to.

127 points by strmpnk 13y ago

> As an Erlang programmer I often claim that “You can’t pick parts of Erlang and then claim you have the same functionality. It all comes together as a whole”. I am guilty of this. I am relatively new to erlang programming and I have skipped on some of the features that make the language great. For instance I have neglected learning both supervision and hot code reloading.

banachtarski 13y ago
Well you're relatively new :). It takes time to absorb an entire language's ecosystem. As long as you take it upon yourself to acquaint yourself with those features, things will be fine. The biggest thing to get on your initial survey should be minimal awareness so you know that that feature exists when you encounter the problem it was originally built to solve. Supervision though, is fundamental to OTP and Erlang. It's certainly easy to learn so I would consider that a low hanging fruit for you. Hot code reloading is a consequence of the continuation passing style that is prevalent in every Erlang process and easy to understand too (although employing it in production can be a bit trickier).
lostcolony 13y ago
Hot code reloading I'd personally grant you as an exception; it's a maintenance tool, not a development tool, and you can largely ignore it until you want it, if you're adhering to OTP principles. Ignoring supervision, however, is...not wise, if you have -any- sort of reliability considerations (even ones as weak as "I'd prefer it to remain up"; if it's just a one off thing that you'll manually execute again if it fails, meh, fine). Supervision is really, really easy to use, at least for the base use case (creating a good hierarchy, so that processes restart the other processes that should be restarted, is an architectural consideration that can be a bit hairy), and it's truly heartening to grep a log for something you've been running a while, find crashes that indicate issues you want to fix (even simple stuff, like sanitizing inputs), but the system is still running as though nothing happened.
brickcap 13y ago
@lostcolony @banachtarski thanks guys. I will start with supervision first then. See if I can make use of it in some of my existing code.

flibertgibit 13y ago

FUCK erlang

rurban 13y ago

Oh nice. They have now almost a Common Lisp environment. Getting closer to Greenspun's tenth rule.

bitwalker 13y ago
Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang.
- devgutt 13y ago
  An interesting example of Erlang hot code reloading in embedded system: http://www.youtube.com/watch?v=96UzSHyp0F8 http://www.youtube.com/watch?v=96UzSHyp0F8
- agentultra 13y ago
  > Is it actually possible to do hot code reloading while preserving the state of the running system in Common Lisp? Yes. > I'm genuinely curious, I kind of had this impression that it was a feature fairly unique to Erlang. It's not. iIRC there's even a version of the JVM with code reloading. What's interesting about Common Lisp is that not only can code be dynamically compiled into the running image but class definitions can be changed and live instances will be updated without stopping the program or reloading anything. There are a tonne of features in Common Lisp like this that are geared towards robust, maintainable software.
  - simula67 13y ago
    > there's even a version of the JVM with code reloading. Yes, because of Java's extensible classloading mechanism, all you have to do is drop an war/ear to the 'deploy' folder of your application server. New files are deployed, rewritten files are redeployed and deleted files are undeployed.
    - phamilton 13y ago

tel 13y ago

I find it funny that the author willingly gave up static typing to have Erlang process control—I made the opposite switch as I grew tired of the overhead and attrition of modeling a complex domain in Erlang. I think the Erlang VM is easily my favorite place to live as a programmer, but I do with Erlang-the-language gave more.

lostcolony 13y ago
What were your thoughts re: Dialyzer?
- tel 13y ago
  It's nice and once I learned how to use it I tried to be very consistent with it. The success types provide good documentation, but I found that the types still felt very flat. The fundamental component of Erlang abstraction is the process and the process tree, so your types reflect that and don't end up composing well.
  - tel 13y ago
    (To clarify: rather your types end up composing much like messages between actors. That's a very fine way to compose things, but it's still just one way)

rdtsc 13y ago

> When systems have faults due to concurrency and distribution, debuggers will not work. Those are very tricky indeed, mix in threads with pointers and a system becomes haunted. "This one customer noticed a crash, on this hardware, after running for 10 weeks, but ... we couldn't reproduce it". "Oh I suspect there is a memory overwrite or use after free that corrupted the data". People start doubting their sanity -- "Oh I swear I saw it crash, that one time, or ...did I, maybe I was tired...". Someone (could have even been Jesper Andersen, the author) said that the biggest performance gain an application gets is when it goes from not working or being crashed to working. And the biggest slowdown is also when it goes from working to crashing unexpectedly. There was talk of 60 hour weeks here before, one of the things that happens at 8pm at night is people huddled over a keyboard debugging some of these hard to track bugs. Managers and some programmers see it as great heroism, pats on the back for everyone, but, others see it as reaping the fruits of previous bad decisions and it is a rather sad sight for some. It all depends on the perspective. I guess the point is one of the main qualities of Erlang is not concurrency but fault tolerance. Many systems copy the actor concurrency patterns and claim "we have Erlang now but it is also faster!", and that is a good thing, but I haven't heard too many claim "We copied Erlang's fault tolerance, and it is even better than Erlang's!". [+] [+] you can do the same pattern up to a point using OS processes, LXC containers, separate servers, having watchdog processes restart workers, for example.

chubot 13y ago
Nitpick: mixing threads with pointers isn't really the problem; it's mixing threads with shared mutable state. A concurrency idiom I've gotten a lot of mileage out of is to use threads, without shared state, but with pointers. Instead of shared state, you use a message passing style where you pass pointers over queues. Threads only mutate data that they are explicitly handed via a queue. After putting an object on a queue, they NULL out their local reference. It's basically like Go's goroutine idiom, but with C/C++/Python, real OS threads, and a thread safe queue (which is all a channel is). There is also some relation to the SEDA architecture. The logical structure of such a program is not really that different than an Erlang program, I imagine. It just works a little better with existing C/C++ codebases, and is potentially faster in some cases. It has a completely different character than spaghetti written using threads and shared state, and can be made extremely reliable. Especially because the explicit channels provide a hook for testing black box behavior.

phamilton 13y ago

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). Attaching to a process is useless, but the Rails console essentially gives you an isolated process to inspect production code and data. I don't see how extending that to an existing process would be useful. So, challenging my assumptions, perhaps short-lived processes aren't the best approach for a web service. Perhaps that's the paradigm in Rails because Ruby lacks the tooling to make a long-lived process easy to maintain. If a long-lived process is as easy to maintain as short-lived processes, then suddenly new architectures open up. How are these architectures different? What are the benefits? What are some examples?

diminish 13y ago
curious, which type of projects do you plan to develop in Erlang? especially after rails.
- phamilton 13y ago
  That's the question I can't answer. I have some problems I want to solve. If I build them in Erlang the way I would on Rails, then there's not much gain. I'm trying to figure out how Erlang will allow me to solve those problems differently. One problem is how to handle authentication across our private and public APIs. With Rails, I've got a db with credentials and tons of memcached running to minimize latency (since every endpoint needs to check auth). How would a long running application in Erlang handle that differently? How is that better than what we currently have?
biokoda 13y ago
When you are running in production and something is not right, you can attach to the console and see what is going on. This is of course not something that you should do often, but can be a real life saver. Simulating something that is designed to work with thousands or even millions of users at the same time is pretty much impossible. Hot code loading and attaching to console gives you the option to see what is going on in real time and even fix it if you have to.

_pmf_ 13y ago

> And all this without service interruption. And you get all this for free, just because you picked Erlang as the implementation language. GDB or even VS's debugger can do a lot of magic even if you use C. Until absolutely necessary, leave debugging information reasonably high and optimization reasonably low or off and you'll get an amazing, dynamic environment to inspect a live C system.

skywhopper 13y ago

Reading about "correct software" as if it were an realistic goal makes me itch. Sure, we should be striving for as few bugs as possible, but any non-trivial application will never be made "correct", if only because it'd be impossible to write the full specification to compare it to.