4 ms·

See my other replay in this thread. > Parallelizing your number crunching is probably not going to work very well. [...] The question is, what exactly does "n

See my other replay in this thread. > Parallelizing your number crunching is probably not going to work very well. [...] The question is, what exactly does "number crunching" mean? We do aerial imagery analysis, so image processing in essence, which I would classify as a "number crunching" problem. A common thing e.g. is to do a time-series analysis and you can simply start multiple (2, 4, ..., N with clusters, etc.) processes for each problem. Obviously this works because most methods are computation and/or memory heavy - the additional memory requirements and "overhead" of Python itself (IMHO people overestimate the weight of starting new processes instead of threads) is completely dwarfed by the requirements (memory and CPU) of the method itself.

kerkeslager 9y ago
...which is true, but doesn't mean you can just ignore it. Interpreter state is among the most frequently accessed memory in many applications, meaning it's ideal to have it in cache. The difference between two interpreter states and one might not be big compared to the data being processed, but it's big enough to bump a lot of interpreter state out of cache, which for many programs can have drastic performance implications. If you don't think cache locality is important, look at radix sort versus quicksort. Radix sort has a much lower O, but performs worse in most cases because of its poor cache locality. Look, I get that there are fairly easy ways to work around these problems, but let's not just blithely pretend they aren't problems.
- zeptomu 9y ago
  Agreed, but there is a lot of misinformation about the topic. I met developers that thought the GIL prevents you from running your program in multiple instances at the same time on one machine - which is obviously not the case. Sure, it's a problem for specific workloads, and Python will get there eventually - I just don't think it is a deal breaker.

kerkeslager 9y ago

...which is true, but doesn't mean you can just ignore it. Interpreter state is among the most frequently accessed memory in many applications, meaning it's ideal to have it in cache. The difference between two interpreter states and one might not be big compared to the data being processed, but it's big enough to bump a lot of interpreter state out of cache, which for many programs can have drastic performance implications. If you don't think cache locality is important, look at radix sort versus quicksort. Radix sort has a much lower O, but performs worse in most cases because of its poor cache locality. Look, I get that there are fairly easy ways to work around these problems, but let's not just blithely pretend they aren't problems.

zeptomu 9y ago
Agreed, but there is a lot of misinformation about the topic. I met developers that thought the GIL prevents you from running your program in multiple instances at the same time on one machine - which is obviously not the case. Sure, it's a problem for specific workloads, and Python will get there eventually - I just don't think it is a deal breaker.