4 ms·

I disagree. I think the author is unjustifiably extrapolating conclusions from a C/kernel development world to the rest of software development. For example, a

I disagree. I think the author is unjustifiably extrapolating conclusions from a C/kernel development world to the rest of software development. For example, a bunch of systems use the "green threads" model. Erlang SMP is one. Erlang SMP is unarguably successful. The referenced article on STM confuses parallelism with concurrency[1]. They test on a bunch of benchmarks that are mostly 1) not concurrent and 2) for performance rely on fast numeric operations that have a vastly different level of granularity to typical business ops. [1] Parallelism = speeding things up by using multiple cores. Concurrency = modelling a system that naturally has different activities going on at once. These are not the same thing.

kscaldef 16y ago
Erlang SMP is successful on one axis (ease of writing correct and reliable concurrent problems), but it seems to me that the argument in the article was about the performance costs of the model. And, at least as of a year or two ago, Erlang SMP did _not_ have good performance. There was a minimum factor of 2 overhead in total CPU usage just by turning on SMP, and heavily concurrent programs could run massively slower (http://shootout.alioth.debian.org/u64q/benchmark.php?test=chameneosredux&lang=all http://shootout.alioth.debian.org/u64q/benchmark.php?test=ch...). At least part of the problem is that there's no affinity between Erlang processes and scheduler threads (and, therefore, CPU/cores), so you've got constant cache misses. There's probably other issues as well. As the author says, it's a hard problem.
- gtani 16y ago
  stackoverflow shows recent BEAM and secret incantation (I haven't tried this yet, maybe tonite) http://stackoverflow.com/questions/1996524/optimizing-performance-on-erlang-processes/1998048#1998048 http://stackoverflow.com/questions/1996524/optimizing-perfor...
  - kscaldef 16y ago
    There was definitely a lot of discussion of these issues going on the last time I was playing around with Erlang, so it's quite possible progress has been made. My recommendation is to take my comments with a grain of salt, and comments on stackoverflow, and micro-benchmarks; and do your own testing and tweaking for your workload.
    - igouy 16y ago
      "About Erlang/OTP and Multi-core performance in particular" http://www.erlang-factory.com/conference/London2009/speakers/kennethlundin http://www.erlang-factory.com/conference/London2009/speakers...
      - gtani 16y ago
        gory details in release notes: http://www.erlang.org/download/ http://www.erlang.org/download/ (files otp_src_R*.readme ) especially: http://www.erlang.org/download/otp_src_R13B01.readme http://www.erlang.org/download/otp_src_R13B01.readme
- igouy 16y ago
  > There was a minimum factor of 2 overhead in total CPU usage just by turning on SMP n-body without SMP 123.30s n-body without SMP 123.30s elapsed n-body with SMP 133.52s n-body with SMP 133.57s elapsed spectral-norm without SMP 46.68s spectral-norm without SMP 46.68s elapsed spectral-norm with SMP 47.71s spectral-norm with SMP 13.90s elapsed mandelbrot without SMP 230.07s mandelbrot without SMP 230.10s elapsed mandelbrot with SMP 253.66s mandelbrot with SMP 87.76s elapsed Maybe that Erlang chameneos-redux program is unusual?
  - kscaldef 16y ago
    "at least as of a year or two ago". So, maybe, as other people have pointed out and as I've already acknowledged, the situation may have improved.
kenjackson 16y ago
"I think the author is unjustifiably extrapolating conclusions from a C/kernel development world to the rest of software development. " Exactly. This is the core problem with threading in real systems. Not that it's not possible, but it's not likely going to yield good results with your non-kernel devs. And unfortunately, given that processors are now getting most of the perf benefit by adding cores, this is a problem. And his conclusions about two-level scheduling, IMO, should be tempered. For example, the lack of pre-emptability is something that I believe most user-level thread packages now deal with. Another issue he discusses is that under load, the kernel doesn't realize that there are 'm' degrees of parallelism. It only sees the LWP. But a good user-level threads package should work system-wide. He also says that detachment and reattachment is more expensive than one might think. I'm curious if they looked at some of their contemporary work, like Cilk, to see how they handled this? I actually think he did good work in his thesis. But it seems more like a snapshot of a particular implementation than a statement about the technology in general.
- grogers 16y ago
  I haven't read his thesis, but more and more languages/environments are moving to two-level scheduling being the dominant way to get performance. When I read through it later I'll be interested to see how his results are applicable to the systems used in eg. Go, erlang, .NET, etc.
  - grogers 16y ago
    Ug... I'm fairly disappointed... His conclusions: - Separating a cold thread from its underlying hot thread is not without performance overhead... Of course cache misses dominate performance. This is basically the first vector that work stealing queues attack. For eg. erlang tries to tie erlang processes to the same run queue (which maps to a kernel thread), for as long as it can as long as it maintains reasonably balanced run queues. - Lack of preemptibility based on time quanta creates performance problems... Well yeah, but either that is part of the model already (thread pool), or that is taken into account by the user level scheduler (ie. run X instructions or time then automatically yield to next runnable user level process). - Nondeterministic LWP creation can have an appreciable and unpredictable impact on performance. The nondeterministic nature of the SIGWAITING mechanism makes performance debugging difficult... Mainly because the SIGWAITING mechanism hes described (which sounds like it is based on solaris) is pretty bad for implementing user level scheduling. Windows 7 and MAC OS X Snow Leopard both seem to do significantly better (Maybe that's because they've had 10+ years to fix it...). In any case thread pools have generally worked around this by handling LWP creation based on the number of waiting threads and have seemed to be reasonable in performance. That he even brings up his 14 year old outdated thesis is pretty sad when talking about the state of the art today. STM is no panacea but it is not worthless either.

kscaldef 16y ago

Erlang SMP is successful on one axis (ease of writing correct and reliable concurrent problems), but it seems to me that the argument in the article was about the performance costs of the model. And, at least as of a year or two ago, Erlang SMP did _not_ have good performance. There was a minimum factor of 2 overhead in total CPU usage just by turning on SMP, and heavily concurrent programs could run massively slower (http://shootout.alioth.debian.org/u64q/benchmark.php?test=chameneosredux&lang=all http://shootout.alioth.debian.org/u64q/benchmark.php?test=ch...). At least part of the problem is that there's no affinity between Erlang processes and scheduler threads (and, therefore, CPU/cores), so you've got constant cache misses. There's probably other issues as well. As the author says, it's a hard problem.

gtani 16y ago
stackoverflow shows recent BEAM and secret incantation (I haven't tried this yet, maybe tonite) http://stackoverflow.com/questions/1996524/optimizing-performance-on-erlang-processes/1998048#1998048 http://stackoverflow.com/questions/1996524/optimizing-perfor...
- kscaldef 16y ago
  There was definitely a lot of discussion of these issues going on the last time I was playing around with Erlang, so it's quite possible progress has been made. My recommendation is to take my comments with a grain of salt, and comments on stackoverflow, and micro-benchmarks; and do your own testing and tweaking for your workload.
  - igouy 16y ago
    "About Erlang/OTP and Multi-core performance in particular" http://www.erlang-factory.com/conference/London2009/speakers/kennethlundin http://www.erlang-factory.com/conference/London2009/speakers...
    - gtani 16y ago
      gory details in release notes: http://www.erlang.org/download/ http://www.erlang.org/download/ (files otp_src_R*.readme ) especially: http://www.erlang.org/download/otp_src_R13B01.readme http://www.erlang.org/download/otp_src_R13B01.readme
igouy 16y ago
> There was a minimum factor of 2 overhead in total CPU usage just by turning on SMP n-body without SMP 123.30s n-body without SMP 123.30s elapsed n-body with SMP 133.52s n-body with SMP 133.57s elapsed spectral-norm without SMP 46.68s spectral-norm without SMP 46.68s elapsed spectral-norm with SMP 47.71s spectral-norm with SMP 13.90s elapsed mandelbrot without SMP 230.07s mandelbrot without SMP 230.10s elapsed mandelbrot with SMP 253.66s mandelbrot with SMP 87.76s elapsed Maybe that Erlang chameneos-redux program is unusual?
- kscaldef 16y ago
  "at least as of a year or two ago". So, maybe, as other people have pointed out and as I've already acknowledged, the situation may have improved.

kenjackson 16y ago

"I think the author is unjustifiably extrapolating conclusions from a C/kernel development world to the rest of software development. " Exactly. This is the core problem with threading in real systems. Not that it's not possible, but it's not likely going to yield good results with your non-kernel devs. And unfortunately, given that processors are now getting most of the perf benefit by adding cores, this is a problem. And his conclusions about two-level scheduling, IMO, should be tempered. For example, the lack of pre-emptability is something that I believe most user-level thread packages now deal with. Another issue he discusses is that under load, the kernel doesn't realize that there are 'm' degrees of parallelism. It only sees the LWP. But a good user-level threads package should work system-wide. He also says that detachment and reattachment is more expensive than one might think. I'm curious if they looked at some of their contemporary work, like Cilk, to see how they handled this? I actually think he did good work in his thesis. But it seems more like a snapshot of a particular implementation than a statement about the technology in general.

grogers 16y ago
I haven't read his thesis, but more and more languages/environments are moving to two-level scheduling being the dominant way to get performance. When I read through it later I'll be interested to see how his results are applicable to the systems used in eg. Go, erlang, .NET, etc.
- grogers 16y ago
  Ug... I'm fairly disappointed... His conclusions: - Separating a cold thread from its underlying hot thread is not without performance overhead... Of course cache misses dominate performance. This is basically the first vector that work stealing queues attack. For eg. erlang tries to tie erlang processes to the same run queue (which maps to a kernel thread), for as long as it can as long as it maintains reasonably balanced run queues. - Lack of preemptibility based on time quanta creates performance problems... Well yeah, but either that is part of the model already (thread pool), or that is taken into account by the user level scheduler (ie. run X instructions or time then automatically yield to next runnable user level process). - Nondeterministic LWP creation can have an appreciable and unpredictable impact on performance. The nondeterministic nature of the SIGWAITING mechanism makes performance debugging difficult... Mainly because the SIGWAITING mechanism hes described (which sounds like it is based on solaris) is pretty bad for implementing user level scheduling. Windows 7 and MAC OS X Snow Leopard both seem to do significantly better (Maybe that's because they've had 10+ years to fix it...). In any case thread pools have generally worked around this by handling LWP creation based on the number of waiting threads and have seemed to be reasonable in performance. That he even brings up his 14 year old outdated thesis is pretty sad when talking about the state of the art today. STM is no panacea but it is not worthless either.