9 ms·
AFAIK, Erlang is still (as of 2016) the only distributed actor model implementation with preemptive scheduler. All other major implementations (including Akka)
by atemerev 11y ago
AFAIK, Erlang is still (as of 2016) the only distributed actor model implementation with preemptive scheduler.
All other major implementations (including Akka) have cooperative scheduling, i.e. forbidding blocking code in actors. Erlang allows it. This is huge.
And actor supervision is the best way to write reliable systems. I have wrote some code in Akka without much effort and testing (streaming market data aggregation), and it is still running with a few years uptime.
- dewiz 11y agoSo is preemptive or cooperative better in your opinion ? I couldn't figure it out from your comment.
- findjashua 11y agopreemptive. Cooperative scheduling runs the risk that a long-running actor might starve the other actors
- davidw 11y agoNote that "better" means better for some kinds of things. Preemptive scheduling has a cost, just like running a program on top of, say, Linux costs more than if you coded it specifically to run on the chip itself with no OS.
- strmpnk 11y agoExactly. In terms of fault tolerance, a buggy actor is isolated in regards to scheduling others. Bad behavior has a more limited scope. This isn't to say that one can't create a process that eventually has harmful consequences to the entire system, it just means it's much less likely. These little points of isolation add up. Scheduling, memory management, lifecycle, crashes, &c... It's why it's a bit funny when I hear Akka compared. Akka is quite impressive but I'd never consider it a true alternative to an Erlang or BEAM style runtime.
- atemerev 11y agoAkka is quite good in terms of reliability (I developed in both Akka and Erlang). All actors in Akka are supervised by default, and restarted if any exception happens. This allows skipping most of error handling in Akka code. If something's wrong, your actor will be restarted. Works great!
- strmpnk 11y agoThis assumes failure is clean. Some failures are tougher, for example, imagine code going into an infinite loop. It either system, you have waste but the scale of the disruption in Erlang is kept isolated where Akka would fail to release the thread. Otherwise, I'd definitely agree that Akka handles vanilla failures just fine by using supervision trees.
- Blackthorn 11y agoCooperative scheduling can work fine so long as the scheduler can detect blocking and pull them out of the run queue. This still counts as cooperative.
- im_down_w_otp 11y ago+1. I think a preemptive scheduling model is really the only way to implement that model effectively. Any other scheduling method eventually causes the consistency of the programming model to break down as it's now pushing the cognitive burden about what to run, where, and when onto me as the developer again.
- angersock 11y agoIt is specifically because of the preemptive scheduling that I find it worthwhile to write code that would otherwise be better suited to a different language in Elixir. Numerical and statistical stuff, for example, is never going to be performant--but that's okay, because I'd much rather have a whole bunch of slow actors making incremental progress safely than a couple of blazing fireballs that could take down my system using NIFs or whatever.
- devbug 11y agoThis unfortunately breaks down with (bad acting) NIFs. Thankfully you can mark 'em as the dirty evil little things that they are (with negligible overhead): ERL_NIF_DIRTY_JOB_CPU_BOUND. [1] I implore anyone interested in Erlang or its surrounding languages, to read its source code. [2] More specifically, the BEAM. I'll warn you that it's very 80's hackeresque, but in a good way. Incredibly pragmatic. The way they achieve their world-class scheduling? Tip: not via a perfect theoretical design cooked up in some comp-sci lab. [3] If you've been following game engine design within the last 9 years or so, you'll probably have a good idea of how they do it. [4][5] [1]: https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7#.sakhllz5r https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea... [2]: https://github.com/erlang/otp/tree/maint/erts/emulator/beam https://github.com/erlang/otp/tree/maint/erts/emulator/beam [3]: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [4]: http://blog.molecular-matters.com/2012/04/05/building-a-load-balanced-task-scheduler-part-1-basics/ http://blog.molecular-matters.com/2012/04/05/building-a-load... [5]: There's a hell of a lot of depth to this problem domain. Charles Bloom's guiding principles for Oodle are spot, and well worth a look.
- davidw 11y ago> This unfortunately breaks down with (bad acting) NIFs. Where possible, I think the Erlangy thing to do would be to just have a separate executable and communicate with that. Also ensures that things keep running if there's a segfault or something.
- devbug 11y agoExactly! Well, kinda. You can always throw the NIFs onto other nodes. You can also write ports that are glorified external processes. Other times, you just need low-latency and high throughput. In that case, you expect failure and design accordingly.
- rdtsc 11y ago> I'll warn you that it's very 80's hackeresque, but in a good way. Incredibly pragmatic. I consider the BEAM VM as one of the marvels of software engineering. You know it is good, when you explain to other programmers that you can have something like an isolated memory process just like an OS process, with preemption and only a few Ks of memory, with a low latency GC, with distribution across machines built in -- and they don't believe me. Even experienced senior developers are skeptical saying stuff like "well yeah but those are green threads then and they have to yield explicitly -- nope, they don't", "so you have callbacks then somehow, no, not callbacks", "what do you mean separate memory spaces, it is a single OS process right?" and so on. It sounds like magic -- this stuff shouldn't exist, right, but the awesome thing is, it does. Moreover, it is not a hacked up version from a lab some place or a PhD disertation proof of concept -- this is what powers banking, databases, messaging, and probably more than 50% of smartphone access to the internet today.
- jeremyjh 11y agoThere is one other. Haskell has green threads, a preemptive scheduler, and it has a pretty decent implementation of Erlang-inspired multi-node concurrency primitives and higher-level framework including supervisors, gen_server equivalent etc in the Cloud Haskell project. It does NOT have Erlangs deployment base and track record but it is still a very promising framework and very appealing if you like Haskell's type system. http://haskell-distributed.github.io/ http://haskell-distributed.github.io/
- mistercow 11y agoI've been confused for some time as to why people get excited about green threads. From what I've read, the main advantage seems to be that you can have threads on hardware that doesn't support threads natively, which is cool if you're on that kind of hardware. There's also some spin-up advantages I guess? But they don't get load-balanced across cores, right? I feel like I'm missing something important.
- logicchains 11y agoThe advantage is you can have many more green threads than you could have OS threads (hundreds of thousands vs thousands), due to green threads being more lightweight. This allows a programming model based on message passing between green threads, which many people consider nicer to reason about. E.g. for a web server you could have a goroutine/Erlang process for each client with 50k clients and still have excellent performance, whereas if you had 50k OS threads you'd likely suffer performance issues and use a heap more ram.
- EdHominem 11y agoAnd this is a huge benefit in the ease of coding. If you only have two or four or sixteen threads you write overtly threaded code. But when you have millions you don't. Your program looks single-threaded and yet operates better and more safely.
- EdHominem 11y agoI think it's not as much green threads as very many threads. And then they're not as much threads as independently threadable parts of the code. You don't have to intentionally write a work queue and balance the number of readers vs writers, etc. You just let the runtime make things go as fast as possible. They do get balanced across cores in almost all cases. Erlang is a functional language which really lends itself to this.
- incepted 11y ago> And actor supervision is the best way to write reliable systems. Need some citation there, because we've been writing extremely reliable systems (multiple nines) for decades in various languages (mostly C, C++ and Java). All these languages (including Erlang) come with pros and cons to write such systems, but so far, Erlang has failed to deliver on most of the promises that its advocates repeatedly make.
- daveguy 11y ago> but so far, Erlang has failed to deliver on most of the promises that its advocates repeatedly make. Speaking of citations...
- incepted 11y agoWell, Erlang is a niche programming language, isn't that evidence enough that all these claims that Erlang will save us from the multithreaded hell are vastly overblown? We're doing multithreaded programming just fine in a lot of very varied languages.
- rdtsc 11y ago> Erlang has failed to deliver on most of the promises that its advocates repeatedly make. Need some citation there,
- DasIch 11y agoErlang was developed for developing telecommunication systems and is still used heavily in that area today. When was the last time you heard of someone unable to make a phone call because a phone network went down?
- incepted 11y agoThat would be a powerful argument if you could prove the code path enabling this uses Erlang. According to Wikipedia, shortly after Armstrong was let go of Ericsson, the company quickly ripped out Erlang from all its products and replaced it with C and C++.
- wocram 11y agoCare to elaborate on 'streaming market data aggregation'?
- atemerev 11y agoNothing too complicated — connecting to various FX liquidity providers (LMAX, Currenex, CFH etc.) via FIX protocol and merging their individual offers using VWAP (Volume weighted average price).
- pron 11y agoQuasar has preemptive scheduling for fibers/actors on the JVM.
- logicchains 11y agoCould a function with a body like just for(;;){ } be preempted?
- pron 11y agoYou could, but we removed time-slice-based preemption a few versions ago b/c it doesn't make sense for fibers if you also have access to heavyweight threads, so now we only preempt on IO/sleep etc.. The thing about time-slice preemption is that it just doesn't help with lightweight threads anyway, as your machine can only support a tiny number of execution threads that require time-slice preemption while you can have hundreds of thousands, or even millions, of lightweight threads. So runtimes like Erlang that don't give you direct access to kernel threads have no choice but to support time slice preemption, but on the JVM we realized that it serves no purpose, so we took it out.
- jeremyjh 11y ago>now we only preempt on IO/sleep etc The IO you are talking about - that is I assume IO methods/libraries that are specifically written/adapted for use in Quasar, yes? If you just use an off-the-shelf JDBC provider, you would block the entire thread when one fiber calls out to the database right? If so, that is not preemption, that is cooperative multi-tasking.
- pron 11y agoIt's not cooperative because the fibers don't explicitly yield control; they are preempted as soon as they perform an operation that does not require use of the CPU. As to the choice of libraries, that is an artificial distinction. Runtimes like Erlang and Go also require libraries specifically written/adapted for the runtime; calling an off-the-shelf IO library from Erlang/Go would also block the entire thread. As integrating a library with Quasar does not require changing its interface, the question of whether to trap calls to specific implementations (which is trivial on the JVM) or require the use of fiber-friendly implementation (wrappers) is a question of design. So far, we've opted for the latter simply for the sake of "least surprise", but we may choose to do the former, too. Also, unlike Erlang/Go (I believe), any accidental blocking of the kernel thread is automatically detected by Quasar at run time, and reported with a stack-trace. Quasar operates in the exact same manner as Erlang/Go, only that we've disabled time-slice preemption once we realized it neither helps nor is it required on the JVM. The only real difference is one the nature of the ecosystem: while all pure-Erlang libraries are designed to work with fibers, most JVM libraries aren't, and so require a thin integration layer that is provided separately. OTOH, thanks to the size of the JVM ecosystem and the standard interface approach, I believe it is the case that today there are more IO libraries that support Quasar fibers (e.g. all servlet servers) than those supporting Erlang processes.
- amelius 11y ago> AFAIK, Erlang is still (as of 2016) the only distributed actor model implementation with preemptive scheduler. It is one of the open issues with Golang. https://github.com/golang/go/issues/11462 https://github.com/golang/go/issues/11462 User aclements notes: > @GoranP, you can think of the Go scheduler as being partially preemptive. It's by no means fully cooperative, since user code generally has no control over scheduling points, but it's also not able to preempt at arbitrary points. Currently, the general rule is that it can preempt at function calls, since at that point there's very little state to save or restore, which makes preemption very fast and dramatically simplifies some aspects of garbage collection. Your loop happens to contain none of these controlled preemption points, which is why the scheduler can't preempt it, but as @davecheney mentioned, this is fairly uncommon in real code (though by no means unheard of). We would like to detect loops like this and insert preemption points, but that work just hasn't happened yet.
- atemerev 11y agoAlso, coroutines (while also cool; I admire Go and its efforts to bring concurrency to the masses) are not actors. Actors can be persistent and have lifecycle.
- atemerev 11y agoAnd goroutines are not distributed, while actors can be transparently deployed across remote physical nodes, both in Akka and Erlang.
- amelius 11y agoHow does this work with state? Isn't it inefficient to transfer the state of the actor to another node compared to just letting the actor run on the same machine?
- atemerev 11y agoThe API only defines a way to start a new actor remotely. Now, each actor per actor model convention has to survive restarts. In distributed environment, the state can be persisted e.g. in distributed memory cache / storage (Hazelcast, Cassandra).
- thesz 11y agoCloud Haskell allows blocking code in actors.
- deleted 11y ago[deleted]
- crazydoggers 11y agoI'd like to mention Celluloid here as well. Generally I prefer Erlang/Elixir, but in Ruby centered apps, Celluloid (especially run under the JVM with JRuby) solves a lot of concurrency problems easily. > All other major implementations (including Akka) have cooperative scheduling, i.e. forbidding blocking code in actors. Erlang allows it. Curious as to what people's thoughts on Celluloid:IO are with respect to solving the problem of blocking code in actors. Celluloid:IO provides an event driven reactor that works in parallel with the actors. I haven't used it myself, but it seems an interesting approach.