7 ms·

Concurrency in Julia

181 points by leephillips 5y ago

skybrian 5y ago
This article uses very inefficient mathematical computations to illustrate concurrency. That’s fine, they are just examples. But you might also be interested in learning how it’s really done, for example by reading the Primes package. [1] [1] https://github.com/JuliaMath/Primes.jl/blob/master/src/Primes.jl https://github.com/JuliaMath/Primes.jl/blob/master/src/Prime...
- leephillips 5y ago
  Yeah, as I wrote there, “As with all of the examples in the article, isprime() leaves many potential optimizations on the table in the interests of simplicity.” The source you link to is longer than my entire article. But it is good to see an example of a “real” program.
  - WhatIsDukkha 5y ago
    The care you took in showing the usefulness of the task migration was appreciated, thanks.
alhirzel 5y ago
I really like how simply Julia unifies many computational concepts. For example, MPI is a classic solution to some of the same problems, but it is only for coarse parallelism. OS-specific threading libraries are available for fine-grained parallelism, but the ligature between the two is typically messy. I have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models?
- Sukera 5y ago
  According to this[1], Julias threading/task model is in part inspired by Go. I don't know about any detailed analysis on similarities and/or differences though. [1] https://julialang.org/blog/2019/07/multithreading/ https://julialang.org/blog/2019/07/multithreading/
- gbrown 5y ago
  I’ve been playing with GPGPU on Julia lately also, and it really seems like things have come a long way in the last few years. Check out the Juliacon 2021 talk on GPU compute if you’re interested.
  - e12e 5y ago
    This one? https://live.juliacon.org/talk/P8KJSW https://live.juliacon.org/talk/P8KJSW
    - vchuravy 5y ago
      Probably more this talk about CUDA 3.0 https://live.juliacon.org/talk/UGX8YR https://live.juliacon.org/talk/UGX8YR or the workshop https://www.youtube.com/watch?v=Hz9IMJuW5hU https://www.youtube.com/watch?v=Hz9IMJuW5hU
      - gbrown 5y ago
        Yeah, I was thinking of the workshop.
  - alhirzel 5y ago
    Are AMD and NVidia GPUs on par nowadays? Or is it still an NVidia-first world when it comes to compute support.
    - eigenspace 5y ago
      There was actually a question about this earlier today: https://discourse.julialang.org/t/amdgpu-jl-status/71191 https://discourse.julialang.org/t/amdgpu-jl-status/71191 TLDR: The entire GPUArrays.jl test suite now passes with AMDGPU.jl. There are still some missing features and it is not as mature as the nvidia version, but this space is progressing rapidly, and benefited from the generic GPU compilation pipeline that was initially built for CUDA.jl
      - snicker7 5y ago
- adgjlsfhk1 5y ago
  It's probably worth mentioning that Julia has MPI.jl which creates an array type that works over MPI transparently to the user.
  - antoine-levitt 5y ago
    MPI.jl does not have an array wrapper type, it just exposes the bare mpi calls. Some other packages do use it to build array types.
- masklinn 5y ago
  > I have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models? There's nothing fundamentally special to Go's "parallelism". It uses near-preempted (and now actually preempted I think, since a few versions back? before that they'd have implicit yield points at function calls but without function calls you could lock out the scheduler) userland threads with small stacks. The parallelism comes from the m:n scheduling. Go has no constructs for massive parallelisation, where you hand it a loop and it efficiently runs that on a hundred cores, at least not built in.
  - adgjlsfhk1 5y ago
    I don't think there's anything too special with Go's constructs, it's just that they are very well optimized and widely used.
  - initplus 5y ago
    Having lightweight userland threads fundamentally changes the way that you write concurrent programs. There is no need to worry about creating too many threads or of thread creation being too heavy. Old school patterns like process/thread/goroutine per connection are very natural to write in Go and result in dead simple code that performs well under load. Reasoning about threads & basic blocking calls is a simpler mental model compared to async/await style concurrency in my opinion. Go is fundamentally designed for developing backend applications, not high perf mathematical/scientific computing. Features like unrolling for-loops across hundreds of cores or offloading to GPU's would be out of place in the language.
    - dragontamer 5y ago
      > Having lightweight userland threads fundamentally changes the way that you write concurrent programs. There is no need to worry about creating too many threads or of thread creation being too heavy. Old school patterns like process/thread/goroutine per connection are very natural to write in Go and result in dead simple code that performs well under load. Well... there's still a problem in the tens-of-millions of Goroutines. But the issue is that OS-threads generally had issues at the ~100,000 of OS-threads. Making something "more lightweight" than pthreads makes sense, because 10,000,000 coroutines is a fundamentally different program design than 100,000 pthreads.
voorwerpjes 5y ago
For a job interview I did the same parallelization of a`isPrime` in Rust, and it the basically as simple as the one for Julia, replacing a call of `into_iter()` with `into_par_iter()`. I also show the strong scaling relationship, which i wish all things talking about concurrency did. Link for the curious: https://github.com/pcrumley/parallel_primes_rs https://github.com/pcrumley/parallel_primes_rs
cbkeller 5y ago
The Folds.jl package [1] mentioned in the article is very nicely written, IMHO. For another alternative to Julia's built-in `Threads.@threads` macro, folks may also be interested in checking out `@batch` from Polyester.jl [2] (formerly CheapThreads.jl), which features particularly low-overhead threading. [1] https://github.com/JuliaFolds/Folds.jl https://github.com/JuliaFolds/Folds.jl [2] https://github.com/JuliaSIMD/Polyester.jl https://github.com/JuliaSIMD/Polyester.jl
foerbert 5y ago
Every time Julia pops up I can't help but get disappointed all over again. I was so intrigued by multiple aspects of the language, and even the type system seemed promising at first. However the inability to subtype concrete types is a total dealbreaker for me. Everything else - even much of the type system - is extremely interesting. However the inability to subtype concrete types means you are essentially stuck with the unfortunately common inability to trivially specify how the heck your program is modeling your problem. I just want the ability to specify an Apple_Count is not an Orange_Count without having to define my own damn operators. Why do so many languages make this so hard? And Julia was so close, until one little sentence moved it so far away.
- leephillips 5y ago
  I don’t get it. Aren’t apple counts and orange counts both integers? Can you give an example of why you would want to subtype concrete types, and how this deficiency would prevent you from writing the kind of program that you want to write?
  - the_svd_doctor 5y ago
    I guess he does not want to be able to add Apples to Oranges.
    - ted_dunning 5y ago
      That's interesting. But Julia really does that already. Check out Unitful.jl
      - foerbert 5y ago
        The approach might work but I don't think in general it actually solves the same problem. While I used integer-like things in the example, it wasn't actually important. It could be strings (think Apple_Description and Orange_Description) or even more complex type from a library (maybe something like Apple_Throw_Plot and Orange_Throw_Plot).
  - 5y ago
leephillips 5y ago
All day on the front page and not a single complaint about 1-based indexing nor Unicode identifiers. It’s a new day.
- galangalalgol 5y ago
  Those both are fine. I do wish the dependency management system was friendlier to either airgapped or nexus/solarcube development environments.
  - leephillips 5y ago
    Have you asked about this on the discussion board? (https://discourse.julialang.org/ https://discourse.julialang.org/) I don’t know anything about it myself, but I’ve heard that some people have had success setting up private package servers. I see there’s some discussion at https://discourse.julialang.org/t/pkg-private-registries-and-corporate-firewalls/63065/8 https://discourse.julialang.org/t/pkg-private-registries-and...

181 points by leephillips 5y ago

This article uses very inefficient mathematical computations to illustrate concurrency. That’s fine, they are just examples. But you might also be interested in learning how it’s really done, for example by reading the Primes package. [1] [1] https://github.com/JuliaMath/Primes.jl/blob/master/src/Primes.jl https://github.com/JuliaMath/Primes.jl/blob/master/src/Prime...

leephillips 5y ago
Yeah, as I wrote there, “As with all of the examples in the article, isprime() leaves many potential optimizations on the table in the interests of simplicity.” The source you link to is longer than my entire article. But it is good to see an example of a “real” program.
- WhatIsDukkha 5y ago
  The care you took in showing the usefulness of the task migration was appreciated, thanks.

alhirzel 5y ago

I really like how simply Julia unifies many computational concepts. For example, MPI is a classic solution to some of the same problems, but it is only for coarse parallelism. OS-specific threading libraries are available for fine-grained parallelism, but the ligature between the two is typically messy. I have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models?

Sukera 5y ago
According to this[1], Julias threading/task model is in part inspired by Go. I don't know about any detailed analysis on similarities and/or differences though. [1] https://julialang.org/blog/2019/07/multithreading/ https://julialang.org/blog/2019/07/multithreading/
gbrown 5y ago
I’ve been playing with GPGPU on Julia lately also, and it really seems like things have come a long way in the last few years. Check out the Juliacon 2021 talk on GPU compute if you’re interested.
- e12e 5y ago
  This one? https://live.juliacon.org/talk/P8KJSW https://live.juliacon.org/talk/P8KJSW
  - vchuravy 5y ago
    Probably more this talk about CUDA 3.0 https://live.juliacon.org/talk/UGX8YR https://live.juliacon.org/talk/UGX8YR or the workshop https://www.youtube.com/watch?v=Hz9IMJuW5hU https://www.youtube.com/watch?v=Hz9IMJuW5hU
    - gbrown 5y ago
      Yeah, I was thinking of the workshop.
- alhirzel 5y ago
  Are AMD and NVidia GPUs on par nowadays? Or is it still an NVidia-first world when it comes to compute support.
  - eigenspace 5y ago
    There was actually a question about this earlier today: https://discourse.julialang.org/t/amdgpu-jl-status/71191 https://discourse.julialang.org/t/amdgpu-jl-status/71191 TLDR: The entire GPUArrays.jl test suite now passes with AMDGPU.jl. There are still some missing features and it is not as mature as the nvidia version, but this space is progressing rapidly, and benefited from the generic GPU compilation pipeline that was initially built for CUDA.jl
    - snicker7 5y ago
adgjlsfhk1 5y ago
It's probably worth mentioning that Julia has MPI.jl which creates an array type that works over MPI transparently to the user.
- antoine-levitt 5y ago
  MPI.jl does not have an array wrapper type, it just exposes the bare mpi calls. Some other packages do use it to build array types.
masklinn 5y ago
> I have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models? There's nothing fundamentally special to Go's "parallelism". It uses near-preempted (and now actually preempted I think, since a few versions back? before that they'd have implicit yield points at function calls but without function calls you could lock out the scheduler) userland threads with small stacks. The parallelism comes from the m:n scheduling. Go has no constructs for massive parallelisation, where you hand it a loop and it efficiently runs that on a hundred cores, at least not built in.
- adgjlsfhk1 5y ago
  I don't think there's anything too special with Go's constructs, it's just that they are very well optimized and widely used.
- initplus 5y ago
  Having lightweight userland threads fundamentally changes the way that you write concurrent programs. There is no need to worry about creating too many threads or of thread creation being too heavy. Old school patterns like process/thread/goroutine per connection are very natural to write in Go and result in dead simple code that performs well under load. Reasoning about threads & basic blocking calls is a simpler mental model compared to async/await style concurrency in my opinion. Go is fundamentally designed for developing backend applications, not high perf mathematical/scientific computing. Features like unrolling for-loops across hundreds of cores or offloading to GPU's would be out of place in the language.
  - dragontamer 5y ago
    > Having lightweight userland threads fundamentally changes the way that you write concurrent programs. There is no need to worry about creating too many threads or of thread creation being too heavy. Old school patterns like process/thread/goroutine per connection are very natural to write in Go and result in dead simple code that performs well under load. Well... there's still a problem in the tens-of-millions of Goroutines. But the issue is that OS-threads generally had issues at the ~100,000 of OS-threads. Making something "more lightweight" than pthreads makes sense, because 10,000,000 coroutines is a fundamentally different program design than 100,000 pthreads.

voorwerpjes 5y ago

For a job interview I did the same parallelization of a`isPrime` in Rust, and it the basically as simple as the one for Julia, replacing a call of `into_iter()` with `into_par_iter()`. I also show the strong scaling relationship, which i wish all things talking about concurrency did. Link for the curious: https://github.com/pcrumley/parallel_primes_rs https://github.com/pcrumley/parallel_primes_rs

cbkeller 5y ago

The Folds.jl package [1] mentioned in the article is very nicely written, IMHO. For another alternative to Julia's built-in `Threads.@threads` macro, folks may also be interested in checking out `@batch` from Polyester.jl [2] (formerly CheapThreads.jl), which features particularly low-overhead threading. [1] https://github.com/JuliaFolds/Folds.jl https://github.com/JuliaFolds/Folds.jl [2] https://github.com/JuliaSIMD/Polyester.jl https://github.com/JuliaSIMD/Polyester.jl

foerbert 5y ago

Every time Julia pops up I can't help but get disappointed all over again. I was so intrigued by multiple aspects of the language, and even the type system seemed promising at first. However the inability to subtype concrete types is a total dealbreaker for me. Everything else - even much of the type system - is extremely interesting. However the inability to subtype concrete types means you are essentially stuck with the unfortunately common inability to trivially specify how the heck your program is modeling your problem. I just want the ability to specify an Apple_Count is not an Orange_Count without having to define my own damn operators. Why do so many languages make this so hard? And Julia was so close, until one little sentence moved it so far away.

leephillips 5y ago
I don’t get it. Aren’t apple counts and orange counts both integers? Can you give an example of why you would want to subtype concrete types, and how this deficiency would prevent you from writing the kind of program that you want to write?
- the_svd_doctor 5y ago
  I guess he does not want to be able to add Apples to Oranges.
  - ted_dunning 5y ago
    That's interesting. But Julia really does that already. Check out Unitful.jl
    - foerbert 5y ago
      The approach might work but I don't think in general it actually solves the same problem. While I used integer-like things in the example, it wasn't actually important. It could be strings (think Apple_Description and Orange_Description) or even more complex type from a library (maybe something like Apple_Throw_Plot and Orange_Throw_Plot).
- 5y ago

leephillips 5y ago

All day on the front page and not a single complaint about 1-based indexing nor Unicode identifiers. It’s a new day.

galangalalgol 5y ago
Those both are fine. I do wish the dependency management system was friendlier to either airgapped or nexus/solarcube development environments.
- leephillips 5y ago
  Have you asked about this on the discussion board? (https://discourse.julialang.org/ https://discourse.julialang.org/) I don’t know anything about it myself, but I’ve heard that some people have had success setting up private package servers. I see there’s some discussion at https://discourse.julialang.org/t/pkg-private-registries-and-corporate-firewalls/63065/8 https://discourse.julialang.org/t/pkg-private-registries-and...