4 ms·

Understanding Concurrency, Parallelism and JavaScript

100 points by kugurerdem 2y ago

necovek 2y ago
The one missed distinction is that concurrent tasks can be executing in parallel, it just doesn't imply they are or aren't. Basically, all parallel tasks are also concurrent, but there are concurrent tasks which are not executed in parallel.
- yazzku 2y ago
  That is how I learned it at uni too. Unfortunately, I have found these definitions are all over the place in some literature.
  - austin-cheney 2y ago
    Think about it not in terms of timing but in terms of instruction. Parallelism requires that different instructions can occur with resource independence. In JavaScript that requires something like web workers or Node clusters. Concurrency merely means one task does not block another, which is less restrictive and more commonly available through the event loop.
    - PROMISE_237 2y ago
      [dead]
  - jerf 2y ago
    Pragmatically it doesn't feel like there's a lot of difference to humans, because even single-threaded concurrent code feels like it's parallel because computers are just so, so much faster than humans that switching rapidly feels parallel. And you often are well advised to treat it as parallel anyhow, at least to some extent, for safety's sake. That is, you may be able to get away with less "locking" in concurrent-but-not-parallel code, but you still can't write it exactly the same as conventionally single-threaded code or you will still get into a lot of trouble very quickly. So pragmatically I don't find a lot of value in distinguishing between "concurrent" and "parallel". Some. But not a lot. There is a difference for sure. It just isn't useful in proportion to the way people like to jump up and correct people about the differences.
  - derefr 2y ago
    To sort through the "dialect of jargon" that a piece of educational CS/SWEng writing is using, it helps to put it in the context of its publication. Consider both when it was published, and what kind of work the author of the writing does. Why "when it was published"? Well, mainframes were single-core until the 70s, and then multi-core (really, massively multi-socket NUMA) thereafter. PCs were single-core until the late 90s, and then multi-core thereafter. Thus: • Anyone writing about "concurrency" in some old Comp Sci paper, is probably thinking about "concurrency" as it pertains to their local single-core Minix mainframe, or on their single-core Sparc/NeXT/SGI university workstation — and so is inherently thinking and talking about some form of cooperative or pre-emptive multi-tasking through context-switching on a single core, with or without hardware-assisted process address-space isolation. • Anyone writing about "concurrency" in some old "industry-sponsored" Software Engineering or (especially) Operational Research paper, was likely working with the early parallel batch-processing mainframes, or perhaps with HPC clusters — and so is much more loose/sloppy with their definitions. In their mental model, there is no context-switching — there is just a cluster-level workload scheduler (itself bound to a core) which assigns workloads to cores; where these workloads are essentially single-threaded or shared-nothing-multi-threaded virtual machines, which own the cores they run on until they finish. (Actually very similar to writing CUDA code for a GPU!) To them, the only kind of concurrency that exists is parallelism, so they just use the words interchangeably. And why "what kind of work the author does"? Well, anyone writing about "concurrency" today, is writing in a context where everything — even the tiniest little microcontrollers — have both multiple cores and the inherent hardware capability to do pre-emptive multitasking (if not perhaps the memory for doing so to make any sense — unless your "tasks" can be measured in kilobytes); and yet everything does it in a slightly different way. Which in turn means that: • If such a person is writing product software, not in control of the deploy environment for their software — then to them, "concurrency" and "parallelism" both just mean "using the multi-threading abstractions provided by the runtime, where these might become OS threads or green-threads, might pin their schedulers to a core during CPU work or not, might yield the core during IO or not, who knows." None of these things can be guaranteed — even core count can't be guaranteed — so their definition of what "concurrency" or even "parallelism" will do for them, has to be rather weak. To these people, "parallelism" is "nice if you can get it" — but not something they think about much, as they have to write code under the assumption that the software will inevitably get stuck running on a single core (e.g. on a heavily-overloaded system) at some point; and they must ensure it won't deadlock or livelock under those conditions. • Meanwhile, if such a person comes from a background of writing SaaS software (i.e. software that runs in a big knowable environment), then anything they say about concurrency / parallelism is likely founded on the assumption of having large distributed clusters of big highly-multicore servers, where the chief concern isn't actually in achieving higher throughput through parallelism (as that part is "easy"), but in resolving distributed data races through write-linearization via artificial concurrency-bottlenecking abstractions like channels, message queues, or actors that hold their own linear inboxes. For these types, "parallelism" puts them in mind of the "multi-threaded with shared state behind semaphores" model that they want to avoid at all costs to keep their software scalable and ensure distributed fault-tolerance. So this type prefers to talk about designing architectures made of little actors that are individually intentionally concurrent-but-not-parallel; and then hand-wavingly introducing shared-nothing instances or pools of these trees of little actors, that can live and move within greater parallel distributed clusters. • And if such a person comes from a background of writing embedded software (i.e. software that runs in a small knowable environment), then their assumption will likely be founded on a concern for achieving realtime dataflow semantics for at least some parts of the system — requiring some, but not all, of the cores to sit there bound to particular tasks and spin-waiting if they finish their processing step early; while other cores are free to be "application cores", executing arbitrarily-long instruction sequences. To these people, "concurrency" is mostly the frustrating low-level process of handing off data between the real-time and non-realtime worlds, using lockless abstractions like shared ring buffers; and "parallelism" is mostly just getting the single most expensive part of the application to schedule a pool of almost-identical expensive operations onto a pool of specialized identical cores. (This is the design perspective that made the PS3's Cell architecture seem like a good idea.)
- weinzierl 2y ago
  I like to use "interleaved concurrency" for the later category.
  - lioeters 2y ago
    That reminds me of the JavaScript interpreter written in JavaScript (which is used for Scratch, I think) - that supports stepping through multiple instances of the interpreter to achieve a kind of interleaved concurrency. https://neil.fraser.name/software/JS-Interpreter/docs.html#threading https://neil.fraser.name/software/JS-Interpreter/docs.html#t... > JavaScript is single-threaded, but the JS-Interpreter allows one to run multiple threads at the same time. Creating two or more completely independent threads that run separately from each other is trivial: just create two or more instances of the Interpreter, each with its own code, and alternate calling each interpreter's step function. They may communicate indirectly with each other through any external APIs that are provided. > A more complex case is where two or more threads should share the same global scope..
duped 2y ago
If you prefer to learn by video, here's an excellent talk on the same subject by Rob Pike that I link all the time to people https://www.youtube.com/watch?v=oV9rvDllKEg https://www.youtube.com/watch?v=oV9rvDllKEg
- donatj 2y ago
  I had posted this as well before realizing you had beaten me to it! For the uninitiated, this is a seminal and oft cited Go talk "Concurrency is not Parallelism" by Rob Pike - well worth your time for anyone, even outside the Go community. I'd encourage anyone who finds themselves in the comments here to check it out, it expands on the content of the article beautifully. Another link, with some added context https://go.dev/blog/waza-talk https://go.dev/blog/waza-talk As well as the slides for those who may not want to watch the video https://go.dev/talks/2012/waza.slide#1 https://go.dev/talks/2012/waza.slide#1
- kitd 2y ago
  Good video. (My paraphrase of) Rob Pike's tldr: Concurrency is a characteristic of the design that allows multiple tasks to execute, coordinate and be scaled. Parallelism is a characteristic of the runtime that allows simultaneous execution of those tasks.
rdtsc 2y ago
I like to think of them as different levels. Concurrency is at a higher abstraction level: steps that can execute without needing to wait on each other. Parallelism is a bit lower and reflects the ability to actually execute the steps at the same time. Sometimes you can have concurrent units like multiple threads, but a single CPU, so they won’t execute in parallel. In an environment with multiple CPU they might execute in parallel.
- PROMISE_237 2y ago
  [dead]
CalRobert 2y ago
I just learned something! I realize now I was talking about parallelism in a recent interview question about concurrency. Oh well.
deleted 2y ago
[deleted]
bugtodiffer 2y ago
Learn Go and you will understand concurrency
bryanrasmussen 2y ago
thinking about this - is there a term for tasks which are partially parallel, that is to say X starts at 0.1 and ends at 1 and Y starts at 0.2 and ends at 0.9 - X and Y are not parallel, but they are something that I'm not sure what the technical term for is. (this is assuming they are not executed concurrently either of course)
- meiraleal 2y ago
  If there are 2 processes/threads, they are parallel, if there is just one it is concurrent. For example, the event loop in JS is concurrent, offloading computation to workers is parallel.
  - bryanrasmussen 2y ago
    sure but parallel comes anyway from the English word, inheriting the meaning "occurring or existing at the same time or in a similar way; corresponding.", 2 processes that exist in the same time all time are, to coin a phrase, classically parallel (by the original English usage), but two processes where they exist only part of the time at the same time are not parallel in the original meaning although they may be for CS - but the question is if there is a special technical term for this condition that anyone knows of?
    - duped 2y ago
      > 2 processes that exist in the same time all time are, to coin a phrase, classically parallel This is not physically realizable. > but two processes where they exist only part of the time at the same time are not parallel in the original meaning It doesn't, both formally and colloquially one would still refer to them as "in parallel." --- In a very loose way you're talking about structured vs unstructured parallelism but it has nothing to do with time, and you can have parallel processes that start at the same point but do not join at the same point and it's still considered "structured." Structured parallelism is where the distinction of where a process starts/joins is meaningful semantically, but there is not really a distinction between "these N things live for the same amount of time" and "N things have different lifetimes" in terms of vocabulary. They're all still parallel or concurrent processes.
dragontamer 2y ago
Concurrency often is about running your I/O routines in parallel, achieving higher bandwidth. For example, one computer handling 50 concurrent HTTP requests simultaneously. No single HTTP request uses all the CPU power or even your Ethernet bandwidth. The bulk of your waiting is latency issues. So while one task is waiting on Ethernet responses under the hood, the system should do something else. Hard Drives are another: you can have random I/O bandwidths of 5MB/s or so, but every request always takes 4ms on the average for a 7200 RPM drive (aka: 120 rotations per second, or about 8 miliseconds for a complete rotation. So 4ms on the average for any request to complete). So while waiting for the HDD to respond, your OS can schedule other reads or writes for the head to move to which improves average performance (ex: if 8 requests all are within the path of the head, you'll still wait 4ms on the average, but maybe each will be read per 1ms). ---------- Parallelism is often about CPU limited situations where you use a 2nd CPU (today called a core). For example, if one CPU core is too slow, you can use a 2nd, or 8 or even 128 cores simultaneously. ------------ Hyperthreads is the CPU designer (Intel and AMD) that the above concurrency technique can apply to modern RAM because a single RAM read is like 50ns or 200 clock ticks. Any RAM-latency problem (ex: linked list traversals) would benefit from the CPU core doing something else while waiting for the RAM latency to respond. ----- Different programming languages have different patterns to make these situations easier to program.

100 points by kugurerdem 2y ago

The one missed distinction is that concurrent tasks can be executing in parallel, it just doesn't imply they are or aren't. Basically, all parallel tasks are also concurrent, but there are concurrent tasks which are not executed in parallel.

yazzku 2y ago
That is how I learned it at uni too. Unfortunately, I have found these definitions are all over the place in some literature.
- austin-cheney 2y ago
  Think about it not in terms of timing but in terms of instruction. Parallelism requires that different instructions can occur with resource independence. In JavaScript that requires something like web workers or Node clusters. Concurrency merely means one task does not block another, which is less restrictive and more commonly available through the event loop.
  - PROMISE_237 2y ago
    [dead]
- jerf 2y ago
  Pragmatically it doesn't feel like there's a lot of difference to humans, because even single-threaded concurrent code feels like it's parallel because computers are just so, so much faster than humans that switching rapidly feels parallel. And you often are well advised to treat it as parallel anyhow, at least to some extent, for safety's sake. That is, you may be able to get away with less "locking" in concurrent-but-not-parallel code, but you still can't write it exactly the same as conventionally single-threaded code or you will still get into a lot of trouble very quickly. So pragmatically I don't find a lot of value in distinguishing between "concurrent" and "parallel". Some. But not a lot. There is a difference for sure. It just isn't useful in proportion to the way people like to jump up and correct people about the differences.
- derefr 2y ago
  To sort through the "dialect of jargon" that a piece of educational CS/SWEng writing is using, it helps to put it in the context of its publication. Consider both when it was published, and what kind of work the author of the writing does. Why "when it was published"? Well, mainframes were single-core until the 70s, and then multi-core (really, massively multi-socket NUMA) thereafter. PCs were single-core until the late 90s, and then multi-core thereafter. Thus: • Anyone writing about "concurrency" in some old Comp Sci paper, is probably thinking about "concurrency" as it pertains to their local single-core Minix mainframe, or on their single-core Sparc/NeXT/SGI university workstation — and so is inherently thinking and talking about some form of cooperative or pre-emptive multi-tasking through context-switching on a single core, with or without hardware-assisted process address-space isolation. • Anyone writing about "concurrency" in some old "industry-sponsored" Software Engineering or (especially) Operational Research paper, was likely working with the early parallel batch-processing mainframes, or perhaps with HPC clusters — and so is much more loose/sloppy with their definitions. In their mental model, there is no context-switching — there is just a cluster-level workload scheduler (itself bound to a core) which assigns workloads to cores; where these workloads are essentially single-threaded or shared-nothing-multi-threaded virtual machines, which own the cores they run on until they finish. (Actually very similar to writing CUDA code for a GPU!) To them, the only kind of concurrency that exists is parallelism, so they just use the words interchangeably. And why "what kind of work the author does"? Well, anyone writing about "concurrency" today, is writing in a context where everything — even the tiniest little microcontrollers — have both multiple cores and the inherent hardware capability to do pre-emptive multitasking (if not perhaps the memory for doing so to make any sense — unless your "tasks" can be measured in kilobytes); and yet everything does it in a slightly different way. Which in turn means that: • If such a person is writing product software, not in control of the deploy environment for their software — then to them, "concurrency" and "parallelism" both just mean "using the multi-threading abstractions provided by the runtime, where these might become OS threads or green-threads, might pin their schedulers to a core during CPU work or not, might yield the core during IO or not, who knows." None of these things can be guaranteed — even core count can't be guaranteed — so their definition of what "concurrency" or even "parallelism" will do for them, has to be rather weak. To these people, "parallelism" is "nice if you can get it" — but not something they think about much, as they have to write code under the assumption that the software will inevitably get stuck running on a single core (e.g. on a heavily-overloaded system) at some point; and they must ensure it won't deadlock or livelock under those conditions. • Meanwhile, if such a person comes from a background of writing SaaS software (i.e. software that runs in a big knowable environment), then anything they say about concurrency / parallelism is likely founded on the assumption of having large distributed clusters of big highly-multicore servers, where the chief concern isn't actually in achieving higher throughput through parallelism (as that part is "easy"), but in resolving distributed data races through write-linearization via artificial concurrency-bottlenecking abstractions like channels, message queues, or actors that hold their own linear inboxes. For these types, "parallelism" puts them in mind of the "multi-threaded with shared state behind semaphores" model that they want to avoid at all costs to keep their software scalable and ensure distributed fault-tolerance. So this type prefers to talk about designing architectures made of little actors that are individually intentionally concurrent-but-not-parallel; and then hand-wavingly introducing shared-nothing instances or pools of these trees of little actors, that can live and move within greater parallel distributed clusters. • And if such a person comes from a background of writing embedded software (i.e. software that runs in a small knowable environment), then their assumption will likely be founded on a concern for achieving realtime dataflow semantics for at least some parts of the system — requiring some, but not all, of the cores to sit there bound to particular tasks and spin-waiting if they finish their processing step early; while other cores are free to be "application cores", executing arbitrarily-long instruction sequences. To these people, "concurrency" is mostly the frustrating low-level process of handing off data between the real-time and non-realtime worlds, using lockless abstractions like shared ring buffers; and "parallelism" is mostly just getting the single most expensive part of the application to schedule a pool of almost-identical expensive operations onto a pool of specialized identical cores. (This is the design perspective that made the PS3's Cell architecture seem like a good idea.)
weinzierl 2y ago
I like to use "interleaved concurrency" for the later category.
- lioeters 2y ago
  That reminds me of the JavaScript interpreter written in JavaScript (which is used for Scratch, I think) - that supports stepping through multiple instances of the interpreter to achieve a kind of interleaved concurrency. https://neil.fraser.name/software/JS-Interpreter/docs.html#threading https://neil.fraser.name/software/JS-Interpreter/docs.html#t... > JavaScript is single-threaded, but the JS-Interpreter allows one to run multiple threads at the same time. Creating two or more completely independent threads that run separately from each other is trivial: just create two or more instances of the Interpreter, each with its own code, and alternate calling each interpreter's step function. They may communicate indirectly with each other through any external APIs that are provided. > A more complex case is where two or more threads should share the same global scope..

duped 2y ago

If you prefer to learn by video, here's an excellent talk on the same subject by Rob Pike that I link all the time to people https://www.youtube.com/watch?v=oV9rvDllKEg https://www.youtube.com/watch?v=oV9rvDllKEg

donatj 2y ago
I had posted this as well before realizing you had beaten me to it! For the uninitiated, this is a seminal and oft cited Go talk "Concurrency is not Parallelism" by Rob Pike - well worth your time for anyone, even outside the Go community. I'd encourage anyone who finds themselves in the comments here to check it out, it expands on the content of the article beautifully. Another link, with some added context https://go.dev/blog/waza-talk https://go.dev/blog/waza-talk As well as the slides for those who may not want to watch the video https://go.dev/talks/2012/waza.slide#1 https://go.dev/talks/2012/waza.slide#1
kitd 2y ago
Good video. (My paraphrase of) Rob Pike's tldr: Concurrency is a characteristic of the design that allows multiple tasks to execute, coordinate and be scaled. Parallelism is a characteristic of the runtime that allows simultaneous execution of those tasks.

rdtsc 2y ago

I like to think of them as different levels. Concurrency is at a higher abstraction level: steps that can execute without needing to wait on each other. Parallelism is a bit lower and reflects the ability to actually execute the steps at the same time. Sometimes you can have concurrent units like multiple threads, but a single CPU, so they won’t execute in parallel. In an environment with multiple CPU they might execute in parallel.

PROMISE_237 2y ago
[dead]

CalRobert 2y ago

I just learned something! I realize now I was talking about parallelism in a recent interview question about concurrency. Oh well.

deleted 2y ago

[deleted]

bugtodiffer 2y ago

Learn Go and you will understand concurrency

bryanrasmussen 2y ago

thinking about this - is there a term for tasks which are partially parallel, that is to say X starts at 0.1 and ends at 1 and Y starts at 0.2 and ends at 0.9 - X and Y are not parallel, but they are something that I'm not sure what the technical term for is. (this is assuming they are not executed concurrently either of course)

meiraleal 2y ago
If there are 2 processes/threads, they are parallel, if there is just one it is concurrent. For example, the event loop in JS is concurrent, offloading computation to workers is parallel.
- bryanrasmussen 2y ago
  sure but parallel comes anyway from the English word, inheriting the meaning "occurring or existing at the same time or in a similar way; corresponding.", 2 processes that exist in the same time all time are, to coin a phrase, classically parallel (by the original English usage), but two processes where they exist only part of the time at the same time are not parallel in the original meaning although they may be for CS - but the question is if there is a special technical term for this condition that anyone knows of?
  - duped 2y ago
    > 2 processes that exist in the same time all time are, to coin a phrase, classically parallel This is not physically realizable. > but two processes where they exist only part of the time at the same time are not parallel in the original meaning It doesn't, both formally and colloquially one would still refer to them as "in parallel." --- In a very loose way you're talking about structured vs unstructured parallelism but it has nothing to do with time, and you can have parallel processes that start at the same point but do not join at the same point and it's still considered "structured." Structured parallelism is where the distinction of where a process starts/joins is meaningful semantically, but there is not really a distinction between "these N things live for the same amount of time" and "N things have different lifetimes" in terms of vocabulary. They're all still parallel or concurrent processes.

dragontamer 2y ago

Concurrency often is about running your I/O routines in parallel, achieving higher bandwidth. For example, one computer handling 50 concurrent HTTP requests simultaneously. No single HTTP request uses all the CPU power or even your Ethernet bandwidth. The bulk of your waiting is latency issues. So while one task is waiting on Ethernet responses under the hood, the system should do something else. Hard Drives are another: you can have random I/O bandwidths of 5MB/s or so, but every request always takes 4ms on the average for a 7200 RPM drive (aka: 120 rotations per second, or about 8 miliseconds for a complete rotation. So 4ms on the average for any request to complete). So while waiting for the HDD to respond, your OS can schedule other reads or writes for the head to move to which improves average performance (ex: if 8 requests all are within the path of the head, you'll still wait 4ms on the average, but maybe each will be read per 1ms). ---------- Parallelism is often about CPU limited situations where you use a 2nd CPU (today called a core). For example, if one CPU core is too slow, you can use a 2nd, or 8 or even 128 cores simultaneously. ------------ Hyperthreads is the CPU designer (Intel and AMD) that the above concurrency technique can apply to modern RAM because a single RAM read is like 50ns or 200 clock ticks. Any RAM-latency problem (ex: linked list traversals) would benefit from the CPU core doing something else while waiting for the RAM latency to respond. ----- Different programming languages have different patterns to make these situations easier to program.