6 ms·

Erlang Scheduler Details and Why They Matter

188 points by hamidreza-s 11y ago

markatkinson 11y ago
Lots of Erlang articles making it to the front page... I like.
wheaties 11y ago
Is there a resurgence of interest in Erlang because of Elixir? Or has the Actor model really taken a hold of developers because of it's direct and easy to understand model of concurrency?
- lmm 11y ago
  I think there's a growing appreciation for functional style more generally (e.g. pattern matching) and Erlang's just caught up in it. Compare OCaml.
- deleted 11y ago
  [deleted]
- hamidreza-s 11y ago
  There are lots of demands for making things concurrent, fault-tolerant and distributed nowadays, not only in big companies but also among startups. Erlang already has those features as well as standard tools (OTP) which lets everyone to create something simply. However among all the languages which run on top of Erlang virtual machine (BEAM) like LFE, Elixir, Efene, Luerl, Erlog, and such the Elixir has an active and bigger community, more interesting frameworks and wider adoption. So it could be claimed that it helped to introduce Erlang values to a wider range of developers.
- Everhusk 11y ago
  A lot of the rails community is migrating to Phoenix so I would imagine it has something to do with it.
- davidw 11y ago
  The short version: * 'event driven' things like Node.js got popular because they use fewer resources to serve the same amount of data (generally). But for a lot of people, JS is not really their idea of a good time. Erlang (and Go) fill this niche pretty well. * Elixir finally made Erlang more palatable to more people. Jose has done a superb job with it, because it's not just a nicer syntax, there is a bunch of nice stuff he's built into it. It doesn't hurt that as a former(?) member of the Rails core team, he has a deep understanding of web programming and its requirements. * It is solid, solid tech. There's lots of new development happening on top of, say, Elixir, like Phoenix, but the underlying system is pretty hardened.
bryanjos 11y ago
The Erlang scheduler is very interesting. I attempted to reimplement it in JavaScript with some success. It works like the first version of the scheduler but I could see web workers being used to make it more like the current version. https://github.com/bryanjos/processes https://github.com/bryanjos/processes
kitd 11y ago
in case of high level platforms, languages or libraries it can be claimed that Erlang virtual machine is almost unique because JVM threads depend on operating system schedulers, CAF which is a C++ actor library uses cooperative scheduling, Golang is not fully preemptive and it also applies to Python’s Twisted, Ruby’s Event Machine and Nodejs. Most languages/runtimes that are taking concurrency seriously do so using libraries or frameworks (eg Akka on the JVM) that look a lot like Erlang's under the hood. Erlang is not quite as unique as it was, but obviously baking it into the language is v useful.
- phamilton 11y ago
  Akka is not preemptive. That's the problem with the library solutions, it's still a squarish peg in a round hole.
  - strmpnk 11y ago
    As far as I understand, you can make an actor reentrant which frees the thread. It's definitely a leaky concern compared to Erlang's process implementation.
    - phamilton 11y ago
      My understanding is that isn't really preemption, rather it's cooperative scheduling. As soon as the scheduler hand execution to the actor, everything else in the system has to hope there isn't a bug in the actor that permanently ties up that thread.
      - strmpnk 11y ago
        Good point. I guess I find it like a hybrid where you don't benefit from the cooperative side and don't achieve true preemption. I think either end would be better than keeping the middle ground.
  - felixgallo 11y ago
    to be fair, the original article is fundamentally incorrect and Erlang is not truly preemptive either. It uses a reduction-counting based cooperative multitasking system which yields at function calls, but a badly programmed NIF can still ruin your day if you don't put it on a dirty scheduler.
bad_login 11y ago
FYI racket and gambit VM use preemtive green thread, but they have nothing to do with SMP.
bnchrch 11y ago
I am enjoying the steam that both Erlang and Elixir are gaining here on Hacker News. They're both interesting, well designed languages built atop a rock solid VM with some great abstractions for concurrency. I'm hoping that this trend continues because out of all the noise of new languages and frameworks I feel like Erlang/Elixir are very deserving of becoming the new way forward for developing anything that runs on a network.
- dfischer 11y ago
  Absolutely this. I've enjoyed JavaScript the past few years. (Dare I admit that). And the only light I see is something like Elixir that has a fundamentally superior foundation for scale and concurrency. I'm intrigued and am hoping elixir / erlang / Phoenix keep growing.
- dfischer 11y ago
  Absolutely this. I've enjoyed JavaScript the past few years. (Dare I admit that). And the only light I see is something like Elixir that has a fundamentally superior foundation for scale and concurrency. I'm intrigued and am hoping elixir / erlang / Phoenix keep growing.
- markatkinson 11y ago
  I stumbled onto an Elixir article a few months back (on HN) and was enchanted by it, ever since then I have been banging away in my free time at Elixir books and tutorials, finished intro to Elixir last night and half way through Programming Elixir. It is a super exciting language to learn and those moments where the ball finally drops on concepts like pattern matching are glorious, it has reignited that magical feeling of being some sort of tech wizard that I first got when I started programming as a child! Anyway, enough of the gospel of Elixir, I too look forward to seeing how it/Erlang develop and feel like it is already getting some noteworthy traction. P.S. I also scan HN daily for anything Erlang/Elixir, and frequency of posts seem to be increasing.
cvik 11y ago
To my knowledge, Erlang uses a cooperative scheduler. The process itself counts downs its reductions and yields when it is time. The programmer can call erlang:yield() to do this before time if he wants. Calling something like receive will also yield. I guess it comes down to definitions, but since Erlang is said to have only soft real-time (as opposed to hard real-time) properties, this makes more sense. The schedulers doesn't give any guarantees as to when a process gets to run and the scheduler never really preempts a process. Again, to my knowledge. Great article though!
- hyperpape 11y ago
  I was also confused by this phrasing. The only thing I could think of was that from the programmer's perspective, it's not cooperative (you don't have to call yield), but that's also true of threads. I see people down thread repeating this claim. Can anyone explain how the scheduler is preemptive?
  - sbalea 11y ago
    You are absolutely correct, the schedulers never preempt a process in the classical sense, but from an Erlang code perspective, the system behaves as if it would be preemptive. The schedulers are actually running a byte code interpreter and the interpreter will switch processes once a certain number of reductions is reached (or the code yields). This breaks down when running native code in the form of NIFs - it is entirely possible to block a scheduler indefinitely while running native code.
  - sasa555 11y ago
    I like to think of it as preemptive because (putting NIFs aside) it's not possible for a single process to completely block the entire scheduler, even if it ends up in an infinite CPU bound loop. So in other words, a task (i.e. process) doesn't have to cooperate with others. It can do whatever it wants, be it I/O or CPU bound, and doesn't have to yield explicitly. Due to aggressive "preemption" it won't block other pending tasks significantly. In contrast, if this is not guaranteed (which is not in most other Erlang "alternatives" I've looked at), then a task might accidentally paralyse large portion of the system.
BuckRogers 11y ago
I've been into Erlang for about 5 years now and once I found it, I never looked back. They really nailed it and many competitive technologies like Go don't have a preemptive scheduler yet. I wish Elixir had a Pythonic syntax, but I'm all for anything BEAM gaining steam and have enjoyed using Elixir. I personally feel that if someone made a Pythonic version of Elixir, say "Hydra", it would take over the world. If anyone starts such a project, look me up!
- eggy 11y ago
  I second that. Elixir is great, but I was not a Ruby programmer, and prefer other syntax. I've tried LFE, and I love Lisp, but it has not gained traction. Perhaps I'll look into Luerl. Both LFE and Luerl were created by Robert Virding one of the original workers on Erlang with Joe Armstrong. I just might go with straight Erlang. I don't think the syntax is that bad.

188 points by hamidreza-s 11y ago

Lots of Erlang articles making it to the front page... I like.

Is there a resurgence of interest in Erlang because of Elixir? Or has the Actor model really taken a hold of developers because of it's direct and easy to understand model of concurrency?

lmm 11y ago
I think there's a growing appreciation for functional style more generally (e.g. pattern matching) and Erlang's just caught up in it. Compare OCaml.
deleted 11y ago
[deleted]
hamidreza-s 11y ago
There are lots of demands for making things concurrent, fault-tolerant and distributed nowadays, not only in big companies but also among startups. Erlang already has those features as well as standard tools (OTP) which lets everyone to create something simply. However among all the languages which run on top of Erlang virtual machine (BEAM) like LFE, Elixir, Efene, Luerl, Erlog, and such the Elixir has an active and bigger community, more interesting frameworks and wider adoption. So it could be claimed that it helped to introduce Erlang values to a wider range of developers.
Everhusk 11y ago
A lot of the rails community is migrating to Phoenix so I would imagine it has something to do with it.
davidw 11y ago
The short version: * 'event driven' things like Node.js got popular because they use fewer resources to serve the same amount of data (generally). But for a lot of people, JS is not really their idea of a good time. Erlang (and Go) fill this niche pretty well. * Elixir finally made Erlang more palatable to more people. Jose has done a superb job with it, because it's not just a nicer syntax, there is a bunch of nice stuff he's built into it. It doesn't hurt that as a former(?) member of the Rails core team, he has a deep understanding of web programming and its requirements. * It is solid, solid tech. There's lots of new development happening on top of, say, Elixir, like Phoenix, but the underlying system is pretty hardened.

bryanjos 11y ago

The Erlang scheduler is very interesting. I attempted to reimplement it in JavaScript with some success. It works like the first version of the scheduler but I could see web workers being used to make it more like the current version. https://github.com/bryanjos/processes https://github.com/bryanjos/processes

kitd 11y ago

in case of high level platforms, languages or libraries it can be claimed that Erlang virtual machine is almost unique because JVM threads depend on operating system schedulers, CAF which is a C++ actor library uses cooperative scheduling, Golang is not fully preemptive and it also applies to Python’s Twisted, Ruby’s Event Machine and Nodejs. Most languages/runtimes that are taking concurrency seriously do so using libraries or frameworks (eg Akka on the JVM) that look a lot like Erlang's under the hood. Erlang is not quite as unique as it was, but obviously baking it into the language is v useful.

phamilton 11y ago
Akka is not preemptive. That's the problem with the library solutions, it's still a squarish peg in a round hole.
- strmpnk 11y ago
  As far as I understand, you can make an actor reentrant which frees the thread. It's definitely a leaky concern compared to Erlang's process implementation.
  - phamilton 11y ago
    My understanding is that isn't really preemption, rather it's cooperative scheduling. As soon as the scheduler hand execution to the actor, everything else in the system has to hope there isn't a bug in the actor that permanently ties up that thread.
    - strmpnk 11y ago
      Good point. I guess I find it like a hybrid where you don't benefit from the cooperative side and don't achieve true preemption. I think either end would be better than keeping the middle ground.
- felixgallo 11y ago
  to be fair, the original article is fundamentally incorrect and Erlang is not truly preemptive either. It uses a reduction-counting based cooperative multitasking system which yields at function calls, but a badly programmed NIF can still ruin your day if you don't put it on a dirty scheduler.

bad_login 11y ago

FYI racket and gambit VM use preemtive green thread, but they have nothing to do with SMP.

bnchrch 11y ago

I am enjoying the steam that both Erlang and Elixir are gaining here on Hacker News. They're both interesting, well designed languages built atop a rock solid VM with some great abstractions for concurrency. I'm hoping that this trend continues because out of all the noise of new languages and frameworks I feel like Erlang/Elixir are very deserving of becoming the new way forward for developing anything that runs on a network.

dfischer 11y ago
Absolutely this. I've enjoyed JavaScript the past few years. (Dare I admit that). And the only light I see is something like Elixir that has a fundamentally superior foundation for scale and concurrency. I'm intrigued and am hoping elixir / erlang / Phoenix keep growing.
dfischer 11y ago
Absolutely this. I've enjoyed JavaScript the past few years. (Dare I admit that). And the only light I see is something like Elixir that has a fundamentally superior foundation for scale and concurrency. I'm intrigued and am hoping elixir / erlang / Phoenix keep growing.
markatkinson 11y ago
I stumbled onto an Elixir article a few months back (on HN) and was enchanted by it, ever since then I have been banging away in my free time at Elixir books and tutorials, finished intro to Elixir last night and half way through Programming Elixir. It is a super exciting language to learn and those moments where the ball finally drops on concepts like pattern matching are glorious, it has reignited that magical feeling of being some sort of tech wizard that I first got when I started programming as a child! Anyway, enough of the gospel of Elixir, I too look forward to seeing how it/Erlang develop and feel like it is already getting some noteworthy traction. P.S. I also scan HN daily for anything Erlang/Elixir, and frequency of posts seem to be increasing.

cvik 11y ago

To my knowledge, Erlang uses a cooperative scheduler. The process itself counts downs its reductions and yields when it is time. The programmer can call erlang:yield() to do this before time if he wants. Calling something like receive will also yield. I guess it comes down to definitions, but since Erlang is said to have only soft real-time (as opposed to hard real-time) properties, this makes more sense. The schedulers doesn't give any guarantees as to when a process gets to run and the scheduler never really preempts a process. Again, to my knowledge. Great article though!

hyperpape 11y ago
I was also confused by this phrasing. The only thing I could think of was that from the programmer's perspective, it's not cooperative (you don't have to call yield), but that's also true of threads. I see people down thread repeating this claim. Can anyone explain how the scheduler is preemptive?
- sbalea 11y ago
  You are absolutely correct, the schedulers never preempt a process in the classical sense, but from an Erlang code perspective, the system behaves as if it would be preemptive. The schedulers are actually running a byte code interpreter and the interpreter will switch processes once a certain number of reductions is reached (or the code yields). This breaks down when running native code in the form of NIFs - it is entirely possible to block a scheduler indefinitely while running native code.
- sasa555 11y ago
  I like to think of it as preemptive because (putting NIFs aside) it's not possible for a single process to completely block the entire scheduler, even if it ends up in an infinite CPU bound loop. So in other words, a task (i.e. process) doesn't have to cooperate with others. It can do whatever it wants, be it I/O or CPU bound, and doesn't have to yield explicitly. Due to aggressive "preemption" it won't block other pending tasks significantly. In contrast, if this is not guaranteed (which is not in most other Erlang "alternatives" I've looked at), then a task might accidentally paralyse large portion of the system.

BuckRogers 11y ago

I've been into Erlang for about 5 years now and once I found it, I never looked back. They really nailed it and many competitive technologies like Go don't have a preemptive scheduler yet. I wish Elixir had a Pythonic syntax, but I'm all for anything BEAM gaining steam and have enjoyed using Elixir. I personally feel that if someone made a Pythonic version of Elixir, say "Hydra", it would take over the world. If anyone starts such a project, look me up!

eggy 11y ago
I second that. Elixir is great, but I was not a Ruby programmer, and prefer other syntax. I've tried LFE, and I love Lisp, but it has not gained traction. Perhaps I'll look into Luerl. Both LFE and Luerl were created by Robert Virding one of the original workers on Erlang with Joe Armstrong. I just might go with straight Erlang. I don't think the syntax is that bad.