8 ms·
We're probably a programming language revolution away from these kinds of CPUs being practical. Isn't parallelism complicated and practically dangerous in most
by oblio 6y ago
We're probably a programming language revolution away from these kinds of CPUs being practical. Isn't parallelism complicated and practically dangerous in most mainstream languages?
Is there any language where you can just go:
parallel-for:
do-my-stuff
and have the programming language Do The Right Thing with no further hassle? The right thing usually being blocking execution until every parallel do-my-stuff finishes execution, in my experience.
In most programming languages I've seen this is a 20+ line boiler plate full of caveats and risks.
- foerbert 6y agoI'm not sure this is the best link, but I don't have anything better in mind and it was the first thing that popped up, so... Anyway, there's a language called ParaSail[0] that's doing stuff in that direction. A little bit back the guy behind it got hired by AdaCore (Ada compiler folks) and they seem to have adopted it and continuing with it. I haven't heard too much about it though. [0]https://adacore.github.io/ParaSail/ https://adacore.github.io/ParaSail/
- 4gotunameagain 6y agoMATLAB has parfor, but that's barely a language..
- jackpeterfletch 6y agoOccam has exactly this. SEQ executes statements sequentially. PAR executes them in parallel, though the language is based on CSP and Cooperative Multitasking, so the idea of it doing 'the right thing' with a PAR is a little different. Think of Occam programs as big Factorio factories. https://en.wikipedia.org/wiki/Occam_(programming_language) https://en.wikipedia.org/wiki/Occam_(programming_language) Alot the ideas that came out of Occam and CSP have been brought into various coroutine libraries though. Kotlin Coroutines, Goroutines, etc. So its definitely something that is explored.
- jacquesm 6y agoI was looking for this comment before posting. Thank you.
- yoshuaw 6y agoMSVC's C++ Parallel algorithms library gets close [1], and so does Rust's Rayon crate [2]. C# has parallel for loops [3], and apparently so does PowerShell [4] Though none of these are quite as convenient as I think they could be. I suspect there is a path for Rust to add parallel iteration syntax in the future [5]. There are quite a few steps needed to get there, but the result could look something like this: let mut listener = TcpListener::bind("127.0.0.1:8080")?; println!("Listening on {}", listener.local_addr()?); par for stream? in listener.incoming() { println!("Accepting from: {}", stream.peer_addr()?); io::copy(&stream, &stream)?; } [1]: https://devblogs.microsoft.com/cppblog/using-c17-parallel-algorithms-for-better-performance/ https://devblogs.microsoft.com/cppblog/using-c17-parallel-al... [2]: https://docs.rs/rayon/1.5.0/rayon/ https://docs.rs/rayon/1.5.0/rayon/ [3]: https://dotnettutorials.net/lesson/parallel-for-method-csharp/ https://dotnettutorials.net/lesson/parallel-for-method-cshar... [4]: https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/ https://devblogs.microsoft.com/powershell/powershell-foreach... [5]: https://blog.yoshuawuyts.com/parallel-stream/#future-directions https://blog.yoshuawuyts.com/parallel-stream/#future-directi...
- eptcyka 6y agoWhat'd be the concurrency primitive used in the parallel block? Is scheduling done per loop or globally? All of these questions are trivial if there's a runtime and become very cumbersome if the language has ambitions to be usable without any standard libraries.
- volta83 6y agoIf you use the standard library, its parallel run-time gets automatically linked. This is often the right choice when building binaries for a platform with an operating system, etc. If you do not include the standard library _AND_ if you use these features, then you'd need to provide your own parallel run-time or your application won't compile. There are many valid implementations of these run-times tuned for different applications, these run-times can be as simple or complex as you'd like, in some cases some run-times only work on particular bare metal targets, and in most cases you just pull a library with the implementation that you'd want to use. Since a valid implementation of the run-time is to just run all tasks sequentially, you can provide a tiny (often "zero-size") run-time for those environments on which code-size trumps everything. You can also provide the standard parallel run-time, but this might require operating-system like functionality, like thread, mutex, etc. support. If your embedded target does not provide those, you'd need to provide implementations of these yourself. The important thing is that you don't need to change any code that uses the run-time, e.g., all the app code can still use fork-join parallelism, but you control how that's executed by swapping the modular runtime component.
- PartiallyTyped 6y agoIn python, with numpy, you can use `np.vectorize`, there exists a jax equivalent that can enable accelerators to do your parallel stuff. Cuda exists, but it isn't as simple as `parallel-for`. For trivial for-loops, you can use OMP.
- rand_r 6y agoUnfortunately np.vectorize doesn’t do anything smart. From the docs: The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop.
- PartiallyTyped 6y agoAh right, yes I remember. Perhaps .vmap from pytorch?
- spiritplumber 6y agoParallax's Spin lets you do this.
- flohofwoe 6y agoYou can get that without a programming language revolution by reformulating the problem a bit. For instance 3D-API shading languages do just that under the hood, just without the "parallel-for". Instead you tell a 3D-API what code should run per-vertex or per-pixel in (usually) a traditional sequential C-style language, and the GPU driver and hardware care about the parallelization and scheduling.
- bipson 6y agoADA is quite good at this. But not every problem (i.e. the Algorithm for the problem) can be divided easily without any coordination (i.e. communication) between the chunks. This overhead can quickly become significant. There is a reason why desktop software (particularly the computationally intensive) is having a hard time exploiting multi-core benefits (besides the programming language and habits in general).
- leprechaun1066 6y agoq has this out the box: https://code.kx.com/q/basics/peach/ https://code.kx.com/q/basics/peach/
- prennert 6y agoDont have many languages already some sort of map implementation that parallelizes pure function calls? Even python has ThreadPoolExecutor.map and ProcessPoolExecutor.map. It gets difficult once you have side effects and/or synchronization (join) of the results are required.
- mhh__ 6y agoD does this as a library i.e. just add parallel to your foreach aggregate. As to languages built around this, there are quite a few, e.g. Intel have Data Parallel C++
- gpderetta 6y ago#pragma omp parallel for
- oblio 6y agoWhat's the operational cost of doing that?
- gpderetta 6y agoWhat do you mean exactly? You need a C/C++/fortran compiler that supports openmp, which is most of the mainstream ones (I believe MSVC still only support an older standard, still more than enough for parallel for). Compilers that do not support openmp are supposed to ignore the pragma and the code will still run fine (although serially) and be fully compatible. Other than that and possibly passing the correct command line flag, openmp is fairly low friction. A parallel for can provide significant speedup for embassingly parallel problems and these days it even support custom random access iterators in C++. Of course it is more involved to express less embarrassingly parallelizable problems, but it is still doable.
- Faaak 6y agoIt's builtin into prolog's logic and its derivates, alas they are rarely used in the industry :'(
- jiofih 6y agoPony? https://www.ponylang.io/ https://www.ponylang.io/
- hvidgaard 6y agoIt's simple to do what you write, multiple languages already do just that. It's the class of problems that is called "embarrassingly parallel" (https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel). However, it turns out that it places some rather serious restrictions on the kind of calculations you can perform this way. In general for two pieces of code to be executed in parallel, they must be independent of each other. I.e for the following 1: tmpA := A(input) 2: tmpB := B(tmpA) 3: tmpC := C(tmpB) 4: result := D(tmpC) it's impossible to calculate a line before the previous line has finished. You cannot do this calculation concurrently. On the other hand if it was like this: 1: resA := A(input) 2: resB := B(input) 3: resC := C(input) 4: resD := D(input) You can calculate all 4 concurrently and achieve a great speed-up if you have resources to do it. Except if B, C, or D somehow access the result of a previous line. For mathematical notation this is simple, but for general programming languages there are plenty of ways to do this. We know that an optimal optimization, i.e. finding the fastest way a program can be executed is known uncomputable. Not difficult, not brute forceable, but impossible to compute with traditional computers. The best we can do is identify _some_ concurrency optimizations, but so far it has not been a fruitful adventure for general programming languages. So what we can do is use patterns and constructs that rely on dividing the problems into concurrent pieces and use proper guarding on shared state. This is however notoriously easy to get wrong.
- chmod775 6y ago> We know that an optimal optimization, i.e. finding the fastest way a program can be executed is known uncomputable. Not difficult, not brute forceable, but impossible to compute with traditional computers. I'm really curious about the proof of this, since there's only a finite number of ways you can re-arrange instructions in a program. Likewise there's only a finite number of ways you could shard them across threads. For this to be true you'd clearly need to phrase the problem in a way that allows you to have an infinite number of possibilities, then prove you can't arrive at the optimal solution through some means that doesn't involve checking an infinite number of them. (Edit: Or prove that it is impossible to decide on which one is the fastest at the time the optimizer runs. When does the optimizer run?). Was the assumption that you'd also be (infinitely) unrolling loops or maybe 'rewriting' the program into something equivalent but faster? What does "finding the fastest way a program can be executed" mean? Just re-ordering instructions and distributing them across threads, or something more?
- nicoburns 6y agoRust's Rayon library is pretty good for these use-cases https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon use rayon::prelude::*; fn sum_of_squares(input: &[i32]) -> i32 { input.par_iter() // <-- just change that! .map(|&i| i * i) .sum() }
- tonyedgecombe 6y agoC# has it: https://docs.microsoft.com/en-us/dotnet/api/system.threading.tasks.parallel.for?view=net-5.0 https://docs.microsoft.com/en-us/dotnet/api/system.threading... Of course when you start doing things in parallel there are automatically a number of caveats and risks.
- cfstras 6y agoJava has this since 8: Arrays.asList("hello", "world").parallelStream() .forEach(s -> System.out.println(s)); If you want to calculate stuff, you can List<String> suffixed = Arrays.asList("hello", "world").parallelStream() .map(s -> s + "_suffix") .collect(Collectors.toList())
- legulere 6y agoThis architecture is called a manycore processor. There are several programming paradigms that are researched for it, among others actor languages, where the idea is to run parts of your program on different cores that communicate with each other. https://ne.wikipedia.org/wiki/Manycore_processor https://ne.wikipedia.org/wiki/Manycore_processor
- gens 6y agoWhile reading the article i thought of Erlang. Multiple independent programs sending messages to each other fits well into multiple independent cores sending messages to each other.
- Someone 6y agoIn scala (https://docs.scala-lang.org/overviews/parallel-collections/overview.html https://docs.scala-lang.org/overviews/parallel-collections/o...): Serial: list.map(_ + 42) Parallel: list.par.map(_ + 42) In C# (https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-write-a-simple-parallel-for-loop https://docs.microsoft.com/en-us/dotnet/standard/parallel-pr...) Serial: for (int i = 0; i < matARows; i++) Parallel: Parallel.For(0, matARows, i => (Function takes a lambda that gets called for each value) Swift has DispatchQueue.concurrentPerform (https://developer.apple.com/documentation/dispatch/dispatchqueue/2016088-concurrentperform https://developer.apple.com/documentation/dispatch/dispatchq...). That is a bit less clear and less flexible (won’t iterate over an array’s values, only over its indexes) but also fairly easy to use. There probably are many more languages with similar constructs.
- snak 6y agoYes, I do agree C# does have many constructs (Parallel For/ForEach/Invoke...) but there's many things to consider that are not designed with thread safety in mind (e.g. generic collections, List<T>, Dictionary<T>, etc... must be replaced by their System.Collections.Concurrent respectives) and many other risks that parallel computing poses. int x = 0; Parallel.For(0, 99999, z => { x++; }); Console.WriteLine(x); Something as simple as an integer sum won't produce the expected output unless you're using Interlocked increments. And that definetely does not match what oblio means with "have the programming language Do The Right Thing with no further hassle".
- api 6y agoRust is much safer for parallelism and has a lot of safety rules around closures, so this would probably be achievable in Rust without much pain via a clever template library. The problem is that not all problems are easy to parallelism at the algorithmic level, and for some it may be impossible. Alongside massively multi core chips I would also love to see more efforts to push the envelope on single threaded performance.
- rbanffy 6y ago> Is there any language where you can just go: Nothing prevents a map function to run in parallel. There are other things that can happen in parallel, however - the whole instruction flow is reordered in the CPU before it is executed and if you have enough execution ports and enough in-flight instructions to keep them fed, you can accomplish a lot in a single clock cycle before your source code even needs to acknowledge that instructions don't happen exactly in the sequence they are written.
- GTP 6y ago>Isn't parallelism complicated and practically dangerous in most mainstream languages? Yes, but functional languages solve this problem.
- Kuinox 6y agoYes, SQL. Write your query and it will be magically parallelised without you knowing it.
- kevin_thibedeau 6y agoHardware description languages are parallelized by default.