4 ms·

I worry about and benchmark this sort of stuff on a daily basis (in the context of Manticore, a parallel dialect of Standard ML), so I have a few data points to

I worry about and benchmark this sort of stuff on a daily basis (in the context of Manticore, a parallel dialect of Standard ML), so I have a few data points to throw out: - Pinning to cores is not a good idea unless you have also pinned all other threads in the system to a separate core. Even with Very Smart schedulers, poor interrupts due to OS threads or even just intermittent services can end up stealing a core that a thread is pinned to, requiring the system to notice it and migrate. We have seen identical median execution times while removing all of the really bad outliers by pinning to a package rather than a core and relying on the OS scheduler to try not to blow out L1 cache by moving you around, both on Intel and AMD architectures. - I am not as familiar with Haskell, but in our implementation of ML we have interruption points the scheduler can hop in at both at allocation locations and at a variety of other points: any loop, a C library call, etc. By doing it at known points but introducing extra ones, we keep it down to 10 or so instructions before a check, with a check that's a single in-memory pointer check (the allocation pointer gets zero'd out, for the curious), _and_ we then produce all of the information that's also needed by the garbage collector if some series of actions leads to the need for a GC at that point. Which can happen even if you weren't allocating if the other threads in the system determine there's a need for a synchronized global collection, for example. - Not mentioned in this article is anything about the Erlang GC's interaction with the scheduler and NUMA memory. In systems with lots of threads, if you don't have a fairly concurrent GC at the per-thread level combined with global collections that maintain locality (remember each CPU package has a separate chunk of memory!) _and_ your scheduler isn't aware of where the code is running, it's easy to have a poor GC and scheduler decision lead to really bad performance outliers. Erlang is far more mature than Manticore, so I'm sure they've dealt with this, but it's a key aspect of engineering such a runtime system on modern multicore machines, so I was surprised not to see anything about it.

rdtsc 14y ago
> Not mentioned in this article is anything about the Erlang GC's interaction with the scheduler and NUMA memory. Great point. Erlang has one of the best GC implementations. Not because it is fancy or has sophisticated algorithms in it or clever trickery ,it is one of the best because of how the rest of the system was thought out and engineered. The main idea is this, since processes are isolated _and_ there is single assignment, and processes have their own private heap, GC now becomes something elegant, obvious and something that doesn't break the system under load. No need to coordinate a stop the world scenario among 100K running processes serving requests.
jerf 14y ago
"a fairly concurrent GC at the per-thread level combined with global collections that maintain locality" Each Erlang process gets its own independent little memory space. GC is trivial to run per-Erlang-process as a result. The downside is, no sharing between processes, even if you want it for some reason. There is some sharing of binaries with reference counting, though, with the usual problems of potentially accidentally holding on to megabytes of memory because one process is holding on to a four-byte slice. (Though you can make copies in advance if you know that's going to happen, and usually you have or can get a pretty good idea.) I don't know anything about specific NUMA support. A quick Google turns up little to nothing. It is possible that just by the way it is structured it tends to work out reasonably well; less well than perhaps it could with deep integration, but better than a program written in most languages with no thought given to it. Processes and the RAM they use to end up tightly bound together in small contiguous memory arenas just by the way Erlang works. (It is not uncommon to have a system with thousands and thousands of processes that each may only be holding on to a few hundred bytes of data that actually belongs to them.) I've been learning Go lately. I so badly want a decent Erlang replacement whose syntax and basic libraries are so much more friendly. And I respect Go and think it has a future, perhaps even a bright one. But I have to admit it has only deepened my respect for Erlang. Erlang is the shy librarian of computer languages; the outside may be superficially unappealing, socially awkward, not like the other glitzier girls, but at the end of the movie she takes her glasses off, lets her hair down, and dazzles the audience by being the best dancer there is... if you stuck with her long enough to get to the end of the movie.

rdtsc 14y ago

> Not mentioned in this article is anything about the Erlang GC's interaction with the scheduler and NUMA memory. Great point. Erlang has one of the best GC implementations. Not because it is fancy or has sophisticated algorithms in it or clever trickery ,it is one of the best because of how the rest of the system was thought out and engineered. The main idea is this, since processes are isolated _and_ there is single assignment, and processes have their own private heap, GC now becomes something elegant, obvious and something that doesn't break the system under load. No need to coordinate a stop the world scenario among 100K running processes serving requests.

jerf 14y ago

"a fairly concurrent GC at the per-thread level combined with global collections that maintain locality" Each Erlang process gets its own independent little memory space. GC is trivial to run per-Erlang-process as a result. The downside is, no sharing between processes, even if you want it for some reason. There is some sharing of binaries with reference counting, though, with the usual problems of potentially accidentally holding on to megabytes of memory because one process is holding on to a four-byte slice. (Though you can make copies in advance if you know that's going to happen, and usually you have or can get a pretty good idea.) I don't know anything about specific NUMA support. A quick Google turns up little to nothing. It is possible that just by the way it is structured it tends to work out reasonably well; less well than perhaps it could with deep integration, but better than a program written in most languages with no thought given to it. Processes and the RAM they use to end up tightly bound together in small contiguous memory arenas just by the way Erlang works. (It is not uncommon to have a system with thousands and thousands of processes that each may only be holding on to a few hundred bytes of data that actually belongs to them.) I've been learning Go lately. I so badly want a decent Erlang replacement whose syntax and basic libraries are so much more friendly. And I respect Go and think it has a future, perhaps even a bright one. But I have to admit it has only deepened my respect for Erlang. Erlang is the shy librarian of computer languages; the outside may be superficially unappealing, socially awkward, not like the other glitzier girls, but at the end of the movie she takes her glasses off, lets her hair down, and dazzles the audience by being the best dancer there is... if you stuck with her long enough to get to the end of the movie.