4 ms·

I want to see python with automatic and transparent concurrency. Ie. The programmer writes: for file in glob.glob('*.jpg'): data.append(load_file(fi

I want to see python with automatic and transparent concurrency. Ie. The programmer writes: for file in glob.glob('*.jpg'): data.append(load_file(file)) Then python itself parallelizes that work. It should do that by identifying loops of pure functions and pushing them off to other threads. One should also be able to mark a function as pure. Python itself would be responsible for making everything appear as if the code ran serially - for example by buffering/reordering log statements. Rationale: Python is designed to be easy and simple. Concurrency in general is not that. Python should do only do concurrency in a way that can be made easy and simple.

lmeyerov 3y ago
Not sure if intentional: That's a super frustrating program for a traditional automatic parallelization compiler, e.g., polyhedral: * glob is an i/o operation: impure * load_file is an i/o operation: impure * data.append serializes part of the execution order --- Interestingly, GPT4 does pretty ok: files = glob.glob('*.jpg') with ThreadPoolExecutor() as executor: data = list(executor.map(load_file, files)) --- Separately, if just going for concurrency for i/o, async/await is pretty amazing: tasks = [load_file(file) for file in glob.glob('*.jpg')] data = await asyncio.gather(*tasks) A lot less program transformation needed as it's essentially sugar over promises, and avoids much of the need to restructure code beyond the coloring problem. So I'm not as much about the case for a compiler here..
- londons_explore 3y ago
  > not sure if intentional. It sure was! I believe any automatic parallelization should be able to deal with all these things. With a combination of opportunistic execution (ie. guess which bits can be parallelized, and roll back if wrong), and clever heuristics to avoid rollbacks, it should be pretty do-able. CPU manufacturers do similar stuff at a smaller scale with parallel execution of single threaded code with Tomasulo's algorithm, and they get massive gains compared to non-superscalar CPU's. They even do similar stuff at medium scale with Hardware Lock Eliding.
- starlevel003 3y ago
  > Separately, if just going for concurrency for i/o, async/await is pretty amazing: That's not async or concurrent. You're running a synchronous function in an asynchronous task which cancels out.
kortex 3y ago
Terrible idea. Not only is it completely counter to the python zen koan of "explicit is better than implicit", you really don't want implicit concurrency without some sort of zero-cost abstraction and really strong guarantees against side effects. > Rationale: Python is designed to be easy and simple. Concurrency in general is not that. Python is way too dynamic for that. Furthermore, you have a syscall at the start of the loop, and one syscall per loop. Even if you did this in Rust, it probably would not give you the true results you wanted, if that file tree were manipulated in any sort of way during the loop, was a network FS, or any of a number of other assumptions were violated. That's exactly why automatic concurrency is a bad idea. Now you have no idea if your code is executing in "simple mode" or "complex mode".
- fl0ki 3y ago
  [Foreword: I agree that it would make no sense for any language to automatically parallelize loops, especially over filesystem mutations. However I am very happy manually doing it with one function so it's explicit when it happens and maintainers know to take it into it account.] > Even if you did this in Rust, it probably would not give you the true results you wanted, if that file tree were manipulated in any sort of way during the loop, was a network FS, or any of a number of other assumptions were violated. That's factually true but not really relevant. Introducing your own parallelism barely adds to this problem. Your program cannot assume it is not parallel with another program affecting the same file tree in arbitrary ways. The parallelism you add to your own code (say, with Rayon) doesn't change the set of operations that can occur to those files. If you design your own operations to be non-overlapping then parallelism is no concern, and if you didn't then that's a bug and you might get non-deterministic results even without parallelism. For what it's worth, even for writing that kind of code, Rust's standard library has taken the recent wave of TOCTOU vulnerabilities much more seriously than contemporaries like C++ implementations. Good writeup here: https://www.reddit.com/r/cpp/comments/151cnlc/a_safety_culture_and_c_we_need_to_talk_about/ https://www.reddit.com/r/cpp/comments/151cnlc/a_safety_cultu... This of course only makes safer operations possible in more cases. Code still has to do what it can with the constraint that filesystem contents may change in arbitrary ways at arbitrary times. Adding your own parallelism does not add significantly to that. And if one is to add parallelism, it may as well be with something as high-level as Rayon and with all of the usual data race protection Rust provides, so that you do get to focus on filesystem concerns and not also on thread-safety in general.

lmeyerov 3y ago

Not sure if intentional: That's a super frustrating program for a traditional automatic parallelization compiler, e.g., polyhedral: * glob is an i/o operation: impure * load_file is an i/o operation: impure * data.append serializes part of the execution order --- Interestingly, GPT4 does pretty ok: files = glob.glob('*.jpg') with ThreadPoolExecutor() as executor: data = list(executor.map(load_file, files)) --- Separately, if just going for concurrency for i/o, async/await is pretty amazing: tasks = [load_file(file) for file in glob.glob('*.jpg')] data = await asyncio.gather(*tasks) A lot less program transformation needed as it's essentially sugar over promises, and avoids much of the need to restructure code beyond the coloring problem. So I'm not as much about the case for a compiler here..

londons_explore 3y ago
> not sure if intentional. It sure was! I believe any automatic parallelization should be able to deal with all these things. With a combination of opportunistic execution (ie. guess which bits can be parallelized, and roll back if wrong), and clever heuristics to avoid rollbacks, it should be pretty do-able. CPU manufacturers do similar stuff at a smaller scale with parallel execution of single threaded code with Tomasulo's algorithm, and they get massive gains compared to non-superscalar CPU's. They even do similar stuff at medium scale with Hardware Lock Eliding.
starlevel003 3y ago
> Separately, if just going for concurrency for i/o, async/await is pretty amazing: That's not async or concurrent. You're running a synchronous function in an asynchronous task which cancels out.

kortex 3y ago

Terrible idea. Not only is it completely counter to the python zen koan of "explicit is better than implicit", you really don't want implicit concurrency without some sort of zero-cost abstraction and really strong guarantees against side effects. > Rationale: Python is designed to be easy and simple. Concurrency in general is not that. Python is way too dynamic for that. Furthermore, you have a syscall at the start of the loop, and one syscall per loop. Even if you did this in Rust, it probably would not give you the true results you wanted, if that file tree were manipulated in any sort of way during the loop, was a network FS, or any of a number of other assumptions were violated. That's exactly why automatic concurrency is a bad idea. Now you have no idea if your code is executing in "simple mode" or "complex mode".

fl0ki 3y ago
[Foreword: I agree that it would make no sense for any language to automatically parallelize loops, especially over filesystem mutations. However I am very happy manually doing it with one function so it's explicit when it happens and maintainers know to take it into it account.] > Even if you did this in Rust, it probably would not give you the true results you wanted, if that file tree were manipulated in any sort of way during the loop, was a network FS, or any of a number of other assumptions were violated. That's factually true but not really relevant. Introducing your own parallelism barely adds to this problem. Your program cannot assume it is not parallel with another program affecting the same file tree in arbitrary ways. The parallelism you add to your own code (say, with Rayon) doesn't change the set of operations that can occur to those files. If you design your own operations to be non-overlapping then parallelism is no concern, and if you didn't then that's a bug and you might get non-deterministic results even without parallelism. For what it's worth, even for writing that kind of code, Rust's standard library has taken the recent wave of TOCTOU vulnerabilities much more seriously than contemporaries like C++ implementations. Good writeup here: https://www.reddit.com/r/cpp/comments/151cnlc/a_safety_culture_and_c_we_need_to_talk_about/ https://www.reddit.com/r/cpp/comments/151cnlc/a_safety_cultu... This of course only makes safer operations possible in more cases. Code still has to do what it can with the constraint that filesystem contents may change in arbitrary ways at arbitrary times. Adding your own parallelism does not add significantly to that. And if one is to add parallelism, it may as well be with something as high-level as Rayon and with all of the usual data race protection Rust provides, so that you do get to focus on filesystem concerns and not also on thread-safety in general.