4 ms·

Not really anything new in there. Been dealing with python concurrency a lot and i dont find it great compared to other languages (eg kotlin). One thing I am s

by c-fe 2y ago

Not really anything new in there. Been dealing with python concurrency a lot and i dont find it great compared to other languages (eg kotlin). One thing I am struggling with right now is how do I handle a function that its both I/O intensive and CPU-bound? To give more context, I am processing data which on paper is easy to parallelise. Say for 1000 lines of data, I have to execute my function f for each line, in any order. However f using the cpu a lot, but also doing up to 4 network requests. My current approach is to divide 1000/n_cores, then launch n_cores processes and on each of them run the function f asynchronoulsy on all inputs of that process, async to handle switching on I/O. I wonder if my approach could be improved.

VagabundoP 2y ago
Interested in seeing if you have tried 3.13 free threading. Your usage case might be worth a test there if moving from a process to threading model isn't too much work. Where does your implementation bottleneck? Python concurrency does suffer from being relatively new and being bolted on to a decades old language. I'd expect the state of the art of python to be much cleaner once no-Gil is hammered on for a few release cycles. As always I suggest Core.py podcast as it has a bunch of background details[1]. There are no-Gil updates throughout the series. [1] https://podcasts.apple.com/us/podcast/core-py/id1712665877 https://podcasts.apple.com/us/podcast/core-py/id1712665877
- c-fe 2y ago
  the no-GIL threading indeed looks very promising, thanks for mentioning it. Unfortunately I think its just a bit too new today to use it in production - at least thats the feeling I get reading the docs.
  - VagabundoP 2y ago
    Oh definitely. It will need a cycle or two to mature.
numba888 2y ago
> I wonder if my approach could be improved. Yes. When you use N batches by the number of cores the total time is defined by the slowest batch. At the end it will be just one job running. If you make batches smaller, like 1000/n_cores/k then you may get better CPU utilization and start-to-end total time. Making k too big will add overhead. Assuming n_cores==10 then k==5 may be a good compromise. Depends on start/stop time per job.
- c-fe 2y ago
  the numba in your name is convincing me to give this a try! But good idea - I see the point, If I understand you correctly you say with n_cores==10, doing 50 batches that a pool schedules on any free core will reduce the chance of one core taking much longer. I will try this out.

by c-fe 2y ago

VagabundoP 2y ago

Interested in seeing if you have tried 3.13 free threading. Your usage case might be worth a test there if moving from a process to threading model isn't too much work. Where does your implementation bottleneck? Python concurrency does suffer from being relatively new and being bolted on to a decades old language. I'd expect the state of the art of python to be much cleaner once no-Gil is hammered on for a few release cycles. As always I suggest Core.py podcast as it has a bunch of background details[1]. There are no-Gil updates throughout the series. [1] https://podcasts.apple.com/us/podcast/core-py/id1712665877 https://podcasts.apple.com/us/podcast/core-py/id1712665877

c-fe 2y ago
the no-GIL threading indeed looks very promising, thanks for mentioning it. Unfortunately I think its just a bit too new today to use it in production - at least thats the feeling I get reading the docs.
- VagabundoP 2y ago
  Oh definitely. It will need a cycle or two to mature.

numba888 2y ago

> I wonder if my approach could be improved. Yes. When you use N batches by the number of cores the total time is defined by the slowest batch. At the end it will be just one job running. If you make batches smaller, like 1000/n_cores/k then you may get better CPU utilization and start-to-end total time. Making k too big will add overhead. Assuming n_cores==10 then k==5 may be a good compromise. Depends on start/stop time per job.

c-fe 2y ago
the numba in your name is convincing me to give this a try! But good idea - I see the point, If I understand you correctly you say with n_cores==10, doing 50 batches that a pool schedules on any free core will reduce the chance of one core taking much longer. I will try this out.