3 ms·
> I wonder if my approach could be improved. Yes. When you use N batches by the number of cores the total time is defined by the slowest batch. At the end it
by numba888 2y ago
> I wonder if my approach could be improved.
Yes. When you use N batches by the number of cores the total time is defined by the slowest batch. At the end it will be just one job running. If you make batches smaller, like 1000/n_cores/k then you may get better CPU utilization and start-to-end total time. Making k too big will add overhead. Assuming n_cores==10 then k==5 may be a good compromise. Depends on start/stop time per job.
- c-fe 2y agothe numba in your name is convincing me to give this a try! But good idea - I see the point, If I understand you correctly you say with n_cores==10, doing 50 batches that a pool schedules on any free core will reduce the chance of one core taking much longer. I will try this out.