4 ms·
In F#, it would just be let results = files |> Array.Parralel.map processFile Literally that easy. Earlier this week, I used a ProcessPoolExecutor to run
by bmitc 2y ago
In F#, it would just be
let results = files |> Array.Parralel.map processFile
Literally that easy.
Earlier this week, I used a ProcessPoolExecutor to run some things in their own process. I needed a bare minimum of synchronization, so I needed a queue. Well, multiprocessing has its own queue. But that queue is not joinable. So I chose the multiprocessing JoinableQueue. Well, it turns out that that queue can't be used across processes. For that, you need to get a queue from the launching process' manager. That Queue is the regular Python queue.
It is a gigantic mess. And yes, asyncio also has its own queue class. So in Python, you literally have a half a dozen or so queue classes that are all incompatible, have different interfaces, and have different limitations that are rarely documented.
That's just one highlight of the mess between threading, asyncio, and multiprocessing.
- 0x000xca0xfe 2y agoWell I'm not here to debate the API cleanliness, I just wanted to point out to OP that Python can utilize multicore processors without threads ;) Here is the part of multiprocessing I used: with Pool() as p: results = p.map(calc_func, file_paths) So, pretty easy too IMO.