4 ms·

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for m

I'm another recent Erlang convert (though I'm still trying to get used to the paradigm and ecosystem). There are a few concepts that haven't quite clicked for me. I come from a Rails background. While Erlang's hot code reloading and the ability to attach to running nodes is cool, I am not quite able to connect the dots on how this is useful. In Rails, a process is short-lived by design (the lifecycle of a request). Attaching to a process is useless, but the Rails console essentially gives you an isolated process to inspect production code and data. I don't see how extending that to an existing process would be useful. So, challenging my assumptions, perhaps short-lived processes aren't the best approach for a web service. Perhaps that's the paradigm in Rails because Ruby lacks the tooling to make a long-lived process easy to maintain. If a long-lived process is as easy to maintain as short-lived processes, then suddenly new architectures open up. How are these architectures different? What are the benefits? What are some examples?

diminish 13y ago
curious, which type of projects do you plan to develop in Erlang? especially after rails.
- phamilton 13y ago
  That's the question I can't answer. I have some problems I want to solve. If I build them in Erlang the way I would on Rails, then there's not much gain. I'm trying to figure out how Erlang will allow me to solve those problems differently. One problem is how to handle authentication across our private and public APIs. With Rails, I've got a db with credentials and tons of memcached running to minimize latency (since every endpoint needs to check auth). How would a long running application in Erlang handle that differently? How is that better than what we currently have?
biokoda 13y ago
When you are running in production and something is not right, you can attach to the console and see what is going on. This is of course not something that you should do often, but can be a real life saver. Simulating something that is designed to work with thousands or even millions of users at the same time is pretty much impossible. Hot code loading and attaching to console gives you the option to see what is going on in real time and even fix it if you have to.
- phamilton 13y ago
  I get that. For persistent connections, it's a life saver. But for a synchronous web service, every connection comes and goes quickly. What do you gain from attaching to the console in that situation that you wouldn't get from the rails console?
  - rdtsc 13y ago
    If you only have a socket dispatcher and a quick request processor that servers a page and dies, then that code path is quite minimal. Now you should probably have a supervisor. Your connections could be handled by a connection or socket pool. You could have a database driver. There could be a background processing queue. Now things get interesting, not all those things get torn down and recreated on each request. Even for short lived requests you could attached to a live system and trace one of the short lived requests or set a debug point to see what it does in "slow motion" to so speak. One can use Erlang for quite a bit without touching the distribution (as in distributed nodes across many systems forming a single cluster). Then the shell can help you log in to any other node, transparently. The ability to reload code at runtime is not something many systems can boast. Because that is one case you might touch once you have found the problem, but that is again one thing you can do. You can diagnose loading and system resource utilization as well. If you have a fully-featured system (say downloaded from Erlang solutions) open the shell and type observer:start(). to start the observers. It will launch a GUI window (make sure to have X forwarding if running via SSH). There you can see runtime stats, the application tree, process list, ETS tables. For each application in the tree you can click on the process and see its state.
    - phamilton 13y ago
      Thanks! That makes sense. I see it now. For the benefit of other Rails devs looking in: think about debugging issues with unicorn or passenger on a live system without interrupting service. Then think about being able to fix the code without interrupting service. That's powerful. That's not something I do every day, but I can understand why that's powerful.
- 13y ago
e_proxus 13y ago
If you want to develop web pages, don't do Erlang. If you want to do web services, do Erlang. I've used both Ruby (Rails) and Python (trying to do concurrency with tornado, greenlets, gevent) and they just fell flat. You can do it, but it is incredibly messy and unstable (because there's so much work involved). In Erlang, you would typically answer a web request in a new actor every time. Just as in Rails, running for example N number of threads serving one request each. Of course, in Erlang that number of actors can vary, which means your system will be more responsive and you'll have lower average latency. However, what differs mostly, is that in Erlang all those actors live inside one VM and you can have central actors managing things, collecting statistics, keeping global state, talking to databases / other services. This, I think, is the core strength of Erlang compared conventional single threaded / GIL systems. A more concrete example would be to dynamically show the current request rate on a web page. You can just create an actor that gets a message from any other actor once they handle a request, and keeps track of how many such messages per second arrive. Then when rendering that web page, you just ask that actor for it's current value. Simple, beautiful, dynamic. No need for a database or any other central storage.
- klibertp 13y ago
  > If you want to develop web pages, don't do Erlang. I'm not sure I agree. There's https://github.com/ChicagoBoss/ChicagoBoss https://github.com/ChicagoBoss/ChicagoBoss and quite a few other frameworks (https://github.com/ChicagoBoss/ChicagoBoss/wiki/Comparison-of-Erlang-Web-Frameworks https://github.com/ChicagoBoss/ChicagoBoss/wiki/Comparison-o...) and there's also Elixir and some frameworks for it (https://github.com/dynamo/dynamo https://github.com/dynamo/dynamo). It's of course nowhere near the ease of development of simple web pages in Django or Rails, but it's not like you need to reinvent everything from scratch either. And you gain ability to scale almost without any effort. It's a tradeoff, of course, and using Erlang for every web page you build from now on would be probably an overkill. But there are web sites which are not services yet, which could still benefit from Erlang greatly. And I think (and hope) that the initial overhead of Erlang will get smaller and smaller over time as new frameworks emerge and that someday deploying your blog on top of Erlang will become viable choice :)
  - 13y ago
davidw 13y ago
> In Rails, a process is short-lived by design (the lifecycle of a request) Yes and no. Rails processes are so huge that they are loaded up and kept around between requests. Erlang processes (which are not unix processes!) are much smaller and lighter. One Erlang program can handle tons of concurrent, always open requests, whereas Rails just isn't cut out for that. As to what Erlang is good for, I think its sweet spot with web stuff is where you need a lot of always open connections, such as with web sockets. I'm a fan of Erlang, but for most SaaS kinds of startuppy things, am not sure it's a good fit: you get so many more tools packaged up and easy to use for you with Rails that unless you know Rails does not work for you, I would almost always choose Rails.
Jacqued 13y ago
Since you recently started out, may I ask you what good resources you found to get started with Erlang ?
- davidw 13y ago
  This is a good book to start with: http://learnyousomeerlang.com/ http://learnyousomeerlang.com/ Free on the web, but buy a copy - it's worth it.
- Veratyr 13y ago
  http://learnyousomeerlang.com/ http://learnyousomeerlang.com/ seems to be the standard reference. I find it most amusing in both style and name (Learn You Some Erlang -> LYSE -> Lies) as well as informative.
- Luc 13y ago
  Besides 'Learn You Some Erlang', Joe Armstrong's new 'Programming Erlang' is very good, too: www.amazon.com/dp/B00I9GR4TW/
pessimizer 13y ago
>In Rails, a process is short-lived by design (the lifecycle of a request). I'm not sure if this is by design, or a limitation of Ruby. >short-lived processes aren't the best approach for a web service. Not for me - I'm building all of my web services in Erlang these days, and there's little to no relationship between the life of most processes in the service and the life of requests. Although each request is usually serviced by its own process, that process may interact with other processes that may have been alive for as long as the application has been running. For example, my RAM caching strategies are usually written as part of the application, not put in front of them. Makes pseudo-ESI easy to hand-roll. Also, short lived processes may have been spawned by long lived processes, so changing how they're created can be done without taking anything down. I find this style so much more performant than the one thread per request thing from Ruby, Perl, PHP et al. It's very easy and natural to minimize the amount of time spent in the db (which is where I'm usually bound.)
- phamilton 13y ago
  This is exactly the type of response I was looking for. A few questions and comments: 1. RAM caching - I can see a big benefit to keeping cache in Erlang. No marshaling of data to and from memcached, for example. How do you handle eviction? 2. Avoiding the db - I'm going to guess this ties in with RAM caching. Are you doing a write through cache or something like that? Thanks for your response, it's helping it all to come together for me.

diminish 13y ago

curious, which type of projects do you plan to develop in Erlang? especially after rails.

phamilton 13y ago
That's the question I can't answer. I have some problems I want to solve. If I build them in Erlang the way I would on Rails, then there's not much gain. I'm trying to figure out how Erlang will allow me to solve those problems differently. One problem is how to handle authentication across our private and public APIs. With Rails, I've got a db with credentials and tons of memcached running to minimize latency (since every endpoint needs to check auth). How would a long running application in Erlang handle that differently? How is that better than what we currently have?

biokoda 13y ago

When you are running in production and something is not right, you can attach to the console and see what is going on. This is of course not something that you should do often, but can be a real life saver. Simulating something that is designed to work with thousands or even millions of users at the same time is pretty much impossible. Hot code loading and attaching to console gives you the option to see what is going on in real time and even fix it if you have to.

phamilton 13y ago
I get that. For persistent connections, it's a life saver. But for a synchronous web service, every connection comes and goes quickly. What do you gain from attaching to the console in that situation that you wouldn't get from the rails console?
- rdtsc 13y ago
  If you only have a socket dispatcher and a quick request processor that servers a page and dies, then that code path is quite minimal. Now you should probably have a supervisor. Your connections could be handled by a connection or socket pool. You could have a database driver. There could be a background processing queue. Now things get interesting, not all those things get torn down and recreated on each request. Even for short lived requests you could attached to a live system and trace one of the short lived requests or set a debug point to see what it does in "slow motion" to so speak. One can use Erlang for quite a bit without touching the distribution (as in distributed nodes across many systems forming a single cluster). Then the shell can help you log in to any other node, transparently. The ability to reload code at runtime is not something many systems can boast. Because that is one case you might touch once you have found the problem, but that is again one thing you can do. You can diagnose loading and system resource utilization as well. If you have a fully-featured system (say downloaded from Erlang solutions) open the shell and type observer:start(). to start the observers. It will launch a GUI window (make sure to have X forwarding if running via SSH). There you can see runtime stats, the application tree, process list, ETS tables. For each application in the tree you can click on the process and see its state.
  - phamilton 13y ago
    Thanks! That makes sense. I see it now. For the benefit of other Rails devs looking in: think about debugging issues with unicorn or passenger on a live system without interrupting service. Then think about being able to fix the code without interrupting service. That's powerful. That's not something I do every day, but I can understand why that's powerful.
13y ago

e_proxus 13y ago

If you want to develop web pages, don't do Erlang. If you want to do web services, do Erlang. I've used both Ruby (Rails) and Python (trying to do concurrency with tornado, greenlets, gevent) and they just fell flat. You can do it, but it is incredibly messy and unstable (because there's so much work involved). In Erlang, you would typically answer a web request in a new actor every time. Just as in Rails, running for example N number of threads serving one request each. Of course, in Erlang that number of actors can vary, which means your system will be more responsive and you'll have lower average latency. However, what differs mostly, is that in Erlang all those actors live inside one VM and you can have central actors managing things, collecting statistics, keeping global state, talking to databases / other services. This, I think, is the core strength of Erlang compared conventional single threaded / GIL systems. A more concrete example would be to dynamically show the current request rate on a web page. You can just create an actor that gets a message from any other actor once they handle a request, and keeps track of how many such messages per second arrive. Then when rendering that web page, you just ask that actor for it's current value. Simple, beautiful, dynamic. No need for a database or any other central storage.

klibertp 13y ago
> If you want to develop web pages, don't do Erlang. I'm not sure I agree. There's https://github.com/ChicagoBoss/ChicagoBoss https://github.com/ChicagoBoss/ChicagoBoss and quite a few other frameworks (https://github.com/ChicagoBoss/ChicagoBoss/wiki/Comparison-of-Erlang-Web-Frameworks https://github.com/ChicagoBoss/ChicagoBoss/wiki/Comparison-o...) and there's also Elixir and some frameworks for it (https://github.com/dynamo/dynamo https://github.com/dynamo/dynamo). It's of course nowhere near the ease of development of simple web pages in Django or Rails, but it's not like you need to reinvent everything from scratch either. And you gain ability to scale almost without any effort. It's a tradeoff, of course, and using Erlang for every web page you build from now on would be probably an overkill. But there are web sites which are not services yet, which could still benefit from Erlang greatly. And I think (and hope) that the initial overhead of Erlang will get smaller and smaller over time as new frameworks emerge and that someday deploying your blog on top of Erlang will become viable choice :)
- 13y ago

davidw 13y ago

> In Rails, a process is short-lived by design (the lifecycle of a request) Yes and no. Rails processes are so huge that they are loaded up and kept around between requests. Erlang processes (which are not unix processes!) are much smaller and lighter. One Erlang program can handle tons of concurrent, always open requests, whereas Rails just isn't cut out for that. As to what Erlang is good for, I think its sweet spot with web stuff is where you need a lot of always open connections, such as with web sockets. I'm a fan of Erlang, but for most SaaS kinds of startuppy things, am not sure it's a good fit: you get so many more tools packaged up and easy to use for you with Rails that unless you know Rails does not work for you, I would almost always choose Rails.

Jacqued 13y ago

Since you recently started out, may I ask you what good resources you found to get started with Erlang ?

davidw 13y ago
This is a good book to start with: http://learnyousomeerlang.com/ http://learnyousomeerlang.com/ Free on the web, but buy a copy - it's worth it.
Veratyr 13y ago
http://learnyousomeerlang.com/ http://learnyousomeerlang.com/ seems to be the standard reference. I find it most amusing in both style and name (Learn You Some Erlang -> LYSE -> Lies) as well as informative.
Luc 13y ago
Besides 'Learn You Some Erlang', Joe Armstrong's new 'Programming Erlang' is very good, too: www.amazon.com/dp/B00I9GR4TW/

pessimizer 13y ago

>In Rails, a process is short-lived by design (the lifecycle of a request). I'm not sure if this is by design, or a limitation of Ruby. >short-lived processes aren't the best approach for a web service. Not for me - I'm building all of my web services in Erlang these days, and there's little to no relationship between the life of most processes in the service and the life of requests. Although each request is usually serviced by its own process, that process may interact with other processes that may have been alive for as long as the application has been running. For example, my RAM caching strategies are usually written as part of the application, not put in front of them. Makes pseudo-ESI easy to hand-roll. Also, short lived processes may have been spawned by long lived processes, so changing how they're created can be done without taking anything down. I find this style so much more performant than the one thread per request thing from Ruby, Perl, PHP et al. It's very easy and natural to minimize the amount of time spent in the db (which is where I'm usually bound.)

phamilton 13y ago
This is exactly the type of response I was looking for. A few questions and comments: 1. RAM caching - I can see a big benefit to keeping cache in Erlang. No marshaling of data to and from memcached, for example. How do you handle eviction? 2. Avoiding the db - I'm going to guess this ties in with RAM caching. Are you doing a write through cache or something like that? Thanks for your response, it's helping it all to come together for me.