7 ms·

People often get parallelism, concurrency, multi-thread and async concepts mixed in weird ways.

by oxff 4y ago

amelius 4y ago
I think that's because parallelism and concurrency are words that were originally very close in meaning and both could have been slapped on the same concept.
- CogitoCogito 4y ago
  I think it's best to be aware of the differences, but not overly pedantic about it. If you legitimately don't know what people mean, then ask, but otherwise just accept that people are sometimes a bit unclear about it. I know the difference between all these things, but I'll often mix up terminology as well.
- marcosdumay 4y ago
  Do they come from geometry or does the geometry terms come from them? On manual work, pretty much as on geometry, they are almost antonyms. If people can do a few tasks in parallel, that means different people can do them at the same time without going over each other. But if two people tasks as concurrent, that means only one can happen at a time. It's the mapping of them into computer terms that confused everybody, because concurrency on computers is the opposite of what it is for people. At least the modern meaning of "concurrency", because it's got changed when computers became multi-tasking.
- jjtheblunt 4y ago
  There was certainly no difference in their use the late 1980s working on early shared memory multiprocessors fulltime at a big academic/industrial collaboration. 20 years later the younger kids with PhDs would distinguish, and i realized the terms had specialized in a way I'd not noticed.
agumonkey 4y ago
IMO it's a matter of logical/functional dependency. Parallelism implies no deps, therefore full overlap is possible. Concurrency less so, so you need joins/rendez-vous.
deleted 4y ago
[deleted]
dragontamer 4y ago
I think I finally figured it out recently. Concurrency is an I/O parallelism technique. One core can make 10,000 TCP connections at the same time thanks to async, golang lightweight threads and whatnot. This is useful because many I/O operations are high latency. A TCP connection, like HTTPS, takes high latency but the I/O device supports many operations. ------- If you are I/O latency bound, you use concurrency to increase performance. Even with one CPU, concurrency or I/O parallelism helps a lot. But if you are CPU bound, you need to use totally different techniques (parallel processing)
- paskozdilar 4y ago
  You're on a right track - the most useful application of concurrency nowdays is I/O, because of the order-of-magnitude larger latency than CPU. However, concurrency is much more than IO. Concurrency can be best defined as a lack of ordering of computation. For example, you want to compute A, B, C and their values are independent of each other, the order you compute them does not matter - that is concurrency. If your language has means of expressing "compute these things, i dont care about the order", then your language supports concurrency. I suggest reading Communicating Sequential Processes by Tony Hoare. It is a very influential work on concurrency - and the proof of its influence is the design of the Go programming language.
- pengaru 4y ago
  Coroutines are a form of concurrency, and they may be executed in parallel if scheduled on separate cores/threads, but usually are just cooperatively scheduled serially. Concurrency is not an IO-specific concept.
  - dragontamer 4y ago
    The issue is that splitting up work into different blocks wrecks your L1 cache and locality of data. So if your code is: A(data) B(data) C(data) Doing A(data1), B(data1), C(data1) then A(data2), B(data2), C(data2) is much much faster than A(data1), A(data2), B(data1), B(data2), C(data1), C(data2). In most cases anyway (assuming data1 and data2 have a portion that fits somewhere in the cache hierarchy). So even if you are doing parallel execution, it ends up being a lot slower than sequential. -------- The goal of parallel processing is to actually get faster in terms of CPU time. Just grabbing concurrency techniques can be counterproductive.
paskozdilar 4y ago
Concepts such as "multi-threading" and "async/await" are particular implementations of concurrency and, potentially, parallelism. They are often used by people who are used to a particular implementation and don't see a reason to use any more abstract word. But mathematically speaking, "concurrency" and "parallelism" are very much related - by definition, in any concurrent algorithm, there are branches of computation that can be safely parallelized; and every parallel algorithm is, by definition, concurrent. The difference between the two, I think, is that parallelism colloquialy implies parallel execution (e.g. OpenMP, Numpy), while concurrency does not (e.g. NodeJS) - concurrency is just independence of computation. That's the reason functional languages generally excel at concurrency - by having explicit dependencies between computational objects, the compiler can just topologically sort the dependency graph, and trivially find which branches of computation can be parallelized.
- User23 4y ago
  As I understand it the unifying concept is nondeterminism. It’s a logical framework that can make sense of parallelism, concurrency, and interrupts. Nondeterministic doesn’t mean unpredictable. For example a pure functional language without side effects is free to evaluate function parameters in any order and still always return the same result. Other times a nondeterministic algorithm can produce different results each execution but still produce a result that satisfies some predicate. For example iterating over a Go hash makes no promises about the order elements will be visited in, but it does promise they will all be visited. So if your reduce function is commutative and associative you’ll get the same result regardless of ordering.
  - paskozdilar 4y ago
    I agree with your perspective. Nondeterminism == lack of ordering. As simple as that.
bob1029 4y ago
Whenever I find myself wrapped around a post on these topics, I force myself back to 1 fundamental question: > Will this action/design increase or decrease the amount of latency incurred on the hot path relative to the physical layout of my processor. So far, keeping an eye on this has guided me well. There are no silver bullets either. async/await is elegant, but it incurrs substantial latency overhead compared to busy waiting over batches of work (which itself has obvious caveats).

People often get parallelism, concurrency, multi-thread and async concepts mixed in weird ways.

by oxff 4y ago

People often get parallelism, concurrency, multi-thread and async concepts mixed in weird ways.

I think that's because parallelism and concurrency are words that were originally very close in meaning and both could have been slapped on the same concept.

CogitoCogito 4y ago
I think it's best to be aware of the differences, but not overly pedantic about it. If you legitimately don't know what people mean, then ask, but otherwise just accept that people are sometimes a bit unclear about it. I know the difference between all these things, but I'll often mix up terminology as well.
marcosdumay 4y ago
Do they come from geometry or does the geometry terms come from them? On manual work, pretty much as on geometry, they are almost antonyms. If people can do a few tasks in parallel, that means different people can do them at the same time without going over each other. But if two people tasks as concurrent, that means only one can happen at a time. It's the mapping of them into computer terms that confused everybody, because concurrency on computers is the opposite of what it is for people. At least the modern meaning of "concurrency", because it's got changed when computers became multi-tasking.
jjtheblunt 4y ago
There was certainly no difference in their use the late 1980s working on early shared memory multiprocessors fulltime at a big academic/industrial collaboration. 20 years later the younger kids with PhDs would distinguish, and i realized the terms had specialized in a way I'd not noticed.

agumonkey 4y ago

IMO it's a matter of logical/functional dependency. Parallelism implies no deps, therefore full overlap is possible. Concurrency less so, so you need joins/rendez-vous.

deleted 4y ago

[deleted]

dragontamer 4y ago

I think I finally figured it out recently. Concurrency is an I/O parallelism technique. One core can make 10,000 TCP connections at the same time thanks to async, golang lightweight threads and whatnot. This is useful because many I/O operations are high latency. A TCP connection, like HTTPS, takes high latency but the I/O device supports many operations. ------- If you are I/O latency bound, you use concurrency to increase performance. Even with one CPU, concurrency or I/O parallelism helps a lot. But if you are CPU bound, you need to use totally different techniques (parallel processing)

paskozdilar 4y ago
You're on a right track - the most useful application of concurrency nowdays is I/O, because of the order-of-magnitude larger latency than CPU. However, concurrency is much more than IO. Concurrency can be best defined as a lack of ordering of computation. For example, you want to compute A, B, C and their values are independent of each other, the order you compute them does not matter - that is concurrency. If your language has means of expressing "compute these things, i dont care about the order", then your language supports concurrency. I suggest reading Communicating Sequential Processes by Tony Hoare. It is a very influential work on concurrency - and the proof of its influence is the design of the Go programming language.
pengaru 4y ago
Coroutines are a form of concurrency, and they may be executed in parallel if scheduled on separate cores/threads, but usually are just cooperatively scheduled serially. Concurrency is not an IO-specific concept.
- dragontamer 4y ago
  The issue is that splitting up work into different blocks wrecks your L1 cache and locality of data. So if your code is: A(data) B(data) C(data) Doing A(data1), B(data1), C(data1) then A(data2), B(data2), C(data2) is much much faster than A(data1), A(data2), B(data1), B(data2), C(data1), C(data2). In most cases anyway (assuming data1 and data2 have a portion that fits somewhere in the cache hierarchy). So even if you are doing parallel execution, it ends up being a lot slower than sequential. -------- The goal of parallel processing is to actually get faster in terms of CPU time. Just grabbing concurrency techniques can be counterproductive.

paskozdilar 4y ago

Concepts such as "multi-threading" and "async/await" are particular implementations of concurrency and, potentially, parallelism. They are often used by people who are used to a particular implementation and don't see a reason to use any more abstract word. But mathematically speaking, "concurrency" and "parallelism" are very much related - by definition, in any concurrent algorithm, there are branches of computation that can be safely parallelized; and every parallel algorithm is, by definition, concurrent. The difference between the two, I think, is that parallelism colloquialy implies parallel execution (e.g. OpenMP, Numpy), while concurrency does not (e.g. NodeJS) - concurrency is just independence of computation. That's the reason functional languages generally excel at concurrency - by having explicit dependencies between computational objects, the compiler can just topologically sort the dependency graph, and trivially find which branches of computation can be parallelized.

User23 4y ago
As I understand it the unifying concept is nondeterminism. It’s a logical framework that can make sense of parallelism, concurrency, and interrupts. Nondeterministic doesn’t mean unpredictable. For example a pure functional language without side effects is free to evaluate function parameters in any order and still always return the same result. Other times a nondeterministic algorithm can produce different results each execution but still produce a result that satisfies some predicate. For example iterating over a Go hash makes no promises about the order elements will be visited in, but it does promise they will all be visited. So if your reduce function is commutative and associative you’ll get the same result regardless of ordering.
- paskozdilar 4y ago
  I agree with your perspective. Nondeterminism == lack of ordering. As simple as that.

bob1029 4y ago

Whenever I find myself wrapped around a post on these topics, I force myself back to 1 fundamental question: > Will this action/design increase or decrease the amount of latency incurred on the hot path relative to the physical layout of my processor. So far, keeping an eye on this has guided me well. There are no silver bullets either. async/await is elegant, but it incurrs substantial latency overhead compared to busy waiting over batches of work (which itself has obvious caveats).