2 ms·
Well, I wouldn't call these techniques snake oil, but, just like anything, you can't just throw them at a problem and expect that they will just work. You have
by dkersten 10y ago
Well, I wouldn't call these techniques snake oil, but, just like anything, you can't just throw them at a problem and expect that they will just work. You have to take all of the usual concurrency issues into account (eg contention).
STM might make things simpler, but if many threads are contending for the same data, then they will need to be serialised and while STM might make this serialisation easier for the programmer, it won't automatically make it perform better[1].
For parallel threads to be faster than a single thread, you need the threads to be as independent from each other as possible. You can use actors, STM and such to do this, but its not a silver bullet that just works. You need to apply the correct technique given your situation and requirements.
I think where Clojure largely failed (and I say this as a big-time Clojure enthusiast) is in its promise of making concurrency easy through these magical primitives. Hell, since its release, Clojure has gained more concurrency primitives (futures, core.async, transducers) because the original ones (threads, STM, agents, atoms, vars) weren't enough.
As for a game engine, I would make sure that each entity wrote to its own independent data, so that writes never block each other. That is, they can all read global world state and they can read the previous state of the other entities, but each only ever writes its own data to its own atom (or whatever; I'd have to think about it more to figure out the most appropriate place to store the state). Then I can update all of the atoms in parallel. This is just off the top of my head, so it may need a little refinement. Agents provide a nice async update, but if you have many in flight, they still need to serialise and take turns, losing much or all of the parallelism.
The point is, I don't believe we'll have a magic "throw cores at it" concurrency technique any time soon, we will continue having to think about it and structure our code to keep the cores as busy as possible and synchronisation as low as possible.
LMAX disruptor is indeed a fantastic piece of engineering and it works so well because it eliminates the need for synchronisation (because the updates always happen in a single thread). The awesome "concurrency" from disruptor comes from the design of their ring buffer, which allows many threads to quickly write to it, and the single reader to quickly read, all without synchronisation. Basically, the processing thread can be kept busy without ever having to wait on writers. Fantastic!
My favourite means of parallelism, I think, is still a work-stealing task-based system like what Intel Threading Building Blocks provides, because its relatively easy to think about and deal with, quite flexible (allows firing off tasks for async things, "going wide" for nicely data-parallel things, allows pipelined processing for sequential steps over a collection of data).
But... there's no concurrency/parallelism silver bullet.
[1] STM can of course make sure the locks are as fine-grained as possible, but at the end of the day, if two things write to the same place, then they need to happen serially.