4 ms·

How can Erlang give you "true parallelism" when it does not share memory between cores without copying it?

by bullen 5y ago

Jtsummers 5y ago
His "true parallelism" means two (or more) actual simultaneous executions (threads, processes, etc.) occurring at the same time (redundant with simultaneous, for emphasis). This is contrasted with concurrency (see Pike's talk Concurrency is not Parallelism [0] for an example of how this line is drawn), which may or may not be implemented with "true parallelism". Coroutines, the example Gosling gave, often do not actually have simultaneous execution. That Erlang forces some synchronization that other languages may not does not affect the ability to (with versions of Erlang since, what, 2007 or 2008?) make use of SMP for simultaneous execution of processes on separate cores or processors. See also OpenMPI for an example of a distributed programming system that also can't share memory without copying it. Though in its case it was across physical nodes (and so the need to communicate was more obvious, they physically didn't share memory anyways). Still true parallelism. [0] https://blog.golang.org/waza-talk https://blog.golang.org/waza-talk
- cowsandmilk 5y ago
  And while MPI was designed for multi nodes, I’ve found that in many cases where a code base supports both MPI and openMP (shared memory with multiple threads in one process), the MPI version runs faster because each single threaded process knows that it owns it’s memory and there’s no need for synchronization. Being explicit about the data being passed around while defaulting to “not shared” has benefits for making use of all your cores more efficiently.
capableweb 5y ago
Because anything described as "true X" is a fallacy and everyone has different views on what "true" is, what "parallelism" is and what "true parallelism" is. Sure, if you define "true parallelism" as "share memory between cores without copying it", then Erlang doesn't give you that. It seems that James Goslings view of "true parallelism" is "let you take advantage of multiple processors", so with that frame of mind, Erlang does give you "true parallelism". > Coroutines kind of magically sidestep some of the naughty issues in true parallelism. And for me, one of the problems with coroutines, which is why I haven't used them in a long time, is that they don't actually let you do or let you take advantage of multiple processors. You can't do true parallelism. > But as soon as you've got one of these coroutine-based languages and you try to exploit multiple processors, if you're doing a lot of coroutine-type operations and you don't have enough processors, you're just saturating one processor.
- Jtsummers 5y ago
  I'm not sure there's too much debate on what parallelism is (simultaneous execution of 2 or more processes/threads). The broader debate seems to be on what concurrency is. Is it a design feature (in which case coroutines in a single threaded program offer concurrency) or is it an execution model (in which case it intersects with or is the same as the common definition of parallelism). The other debate is on how parallelism (and concurrency) should be done. Shared-nothing or shared-memory? It'd be foolish to say that only one of them is parallelism, but it would be reasonable to debate the merits of each approach in different systems and circumstances.
  - bullen 5y ago
    I'm trying to name the version of parallelism that can share memory between threads, and so far I have tried "Joint Parallelism" and now lately "Atomic Parallelism" without much success. What would you suggest?
    - deleted 5y ago
      [deleted]
    - Jtsummers 5y ago
      I'm not sure it needs a special name. It's the default model that almost everyone learns first, shared-memory parallelism/concurrency. It's the whole reason that semaphores and mutexes were developed (to avoid contention for the same resource or resource set from multiple threads of execution).
      - bullen 5y ago
        Yes, but I also mean that parallelism to be atomic/non-locking. afaik only Java/C# can do this on a high-level (without going back to atomic arrays in C) but I still can't explain why, the best I found so far is: "Many lock-free structures offer atomic-free read paths, notably concurrent containers in garbage collected languages, such as ConcurrentHashMap in Java. Languages without garbage collection have fewer straightforward options, mostly because safe memory reclamation is a hard problem..." - Travis Downs https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html https://travisdowns.github.io/blog/2020/07/06/concurrency-co... But it seems there is also some benefit to have a VM in addition to a GC for this joint/atomic parallelism.

How can Erlang give you "true parallelism" when it does not share memory between cores without copying it?

by bullen 5y ago

How can Erlang give you "true parallelism" when it does not share memory between cores without copying it?

His "true parallelism" means two (or more) actual simultaneous executions (threads, processes, etc.) occurring at the same time (redundant with simultaneous, for emphasis). This is contrasted with concurrency (see Pike's talk Concurrency is not Parallelism [0] for an example of how this line is drawn), which may or may not be implemented with "true parallelism". Coroutines, the example Gosling gave, often do not actually have simultaneous execution. That Erlang forces some synchronization that other languages may not does not affect the ability to (with versions of Erlang since, what, 2007 or 2008?) make use of SMP for simultaneous execution of processes on separate cores or processors. See also OpenMPI for an example of a distributed programming system that also can't share memory without copying it. Though in its case it was across physical nodes (and so the need to communicate was more obvious, they physically didn't share memory anyways). Still true parallelism. [0] https://blog.golang.org/waza-talk https://blog.golang.org/waza-talk

cowsandmilk 5y ago
And while MPI was designed for multi nodes, I’ve found that in many cases where a code base supports both MPI and openMP (shared memory with multiple threads in one process), the MPI version runs faster because each single threaded process knows that it owns it’s memory and there’s no need for synchronization. Being explicit about the data being passed around while defaulting to “not shared” has benefits for making use of all your cores more efficiently.

capableweb 5y ago

Because anything described as "true X" is a fallacy and everyone has different views on what "true" is, what "parallelism" is and what "true parallelism" is. Sure, if you define "true parallelism" as "share memory between cores without copying it", then Erlang doesn't give you that. It seems that James Goslings view of "true parallelism" is "let you take advantage of multiple processors", so with that frame of mind, Erlang does give you "true parallelism". > Coroutines kind of magically sidestep some of the naughty issues in true parallelism. And for me, one of the problems with coroutines, which is why I haven't used them in a long time, is that they don't actually let you do or let you take advantage of multiple processors. You can't do true parallelism. > But as soon as you've got one of these coroutine-based languages and you try to exploit multiple processors, if you're doing a lot of coroutine-type operations and you don't have enough processors, you're just saturating one processor.

Jtsummers 5y ago
I'm not sure there's too much debate on what parallelism is (simultaneous execution of 2 or more processes/threads). The broader debate seems to be on what concurrency is. Is it a design feature (in which case coroutines in a single threaded program offer concurrency) or is it an execution model (in which case it intersects with or is the same as the common definition of parallelism). The other debate is on how parallelism (and concurrency) should be done. Shared-nothing or shared-memory? It'd be foolish to say that only one of them is parallelism, but it would be reasonable to debate the merits of each approach in different systems and circumstances.
- bullen 5y ago
  I'm trying to name the version of parallelism that can share memory between threads, and so far I have tried "Joint Parallelism" and now lately "Atomic Parallelism" without much success. What would you suggest?
  - deleted 5y ago
    [deleted]
  - Jtsummers 5y ago
    I'm not sure it needs a special name. It's the default model that almost everyone learns first, shared-memory parallelism/concurrency. It's the whole reason that semaphores and mutexes were developed (to avoid contention for the same resource or resource set from multiple threads of execution).
    - bullen 5y ago
      Yes, but I also mean that parallelism to be atomic/non-locking. afaik only Java/C# can do this on a high-level (without going back to atomic arrays in C) but I still can't explain why, the best I found so far is: "Many lock-free structures offer atomic-free read paths, notably concurrent containers in garbage collected languages, such as ConcurrentHashMap in Java. Languages without garbage collection have fewer straightforward options, mostly because safe memory reclamation is a hard problem..." - Travis Downs https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html https://travisdowns.github.io/blog/2020/07/06/concurrency-co... But it seems there is also some benefit to have a VM in addition to a GC for this joint/atomic parallelism.