3 ms·

Not sure if intentional: That's a super frustrating program for a traditional automatic parallelization compiler, e.g., polyhedral: * glob is an i/o operation:

Not sure if intentional: That's a super frustrating program for a traditional automatic parallelization compiler, e.g., polyhedral: * glob is an i/o operation: impure * load_file is an i/o operation: impure * data.append serializes part of the execution order --- Interestingly, GPT4 does pretty ok: files = glob.glob('*.jpg') with ThreadPoolExecutor() as executor: data = list(executor.map(load_file, files)) --- Separately, if just going for concurrency for i/o, async/await is pretty amazing: tasks = [load_file(file) for file in glob.glob('*.jpg')] data = await asyncio.gather(*tasks) A lot less program transformation needed as it's essentially sugar over promises, and avoids much of the need to restructure code beyond the coloring problem. So I'm not as much about the case for a compiler here..

londons_explore 3y ago
> not sure if intentional. It sure was! I believe any automatic parallelization should be able to deal with all these things. With a combination of opportunistic execution (ie. guess which bits can be parallelized, and roll back if wrong), and clever heuristics to avoid rollbacks, it should be pretty do-able. CPU manufacturers do similar stuff at a smaller scale with parallel execution of single threaded code with Tomasulo's algorithm, and they get massive gains compared to non-superscalar CPU's. They even do similar stuff at medium scale with Hardware Lock Eliding.
starlevel003 3y ago
> Separately, if just going for concurrency for i/o, async/await is pretty amazing: That's not async or concurrent. You're running a synchronous function in an asynchronous task which cancels out.