3 ms·
Not a single one of our data scientists knows how to use multiprocessing. Their jobs are all 1 CPU core, hundreds of gigabytes of RAM. The EC2 instances they ru
by electroly 3y ago
Not a single one of our data scientists knows how to use multiprocessing. Their jobs are all 1 CPU core, hundreds of gigabytes of RAM. The EC2 instances they run on end up wasting tons of cores, even with high-memory instances. This isn't necessarily Python's fault, but rather because Python has a lot of users who are not computer programmers.
One thing we can blame Python for is that the data scientists can't ever loop over their data with a regular loop. Simply impossible; it's too slow and would immediately wreck performance. They're more Pandas programmers than they are Python programmers; everything has to be done with Pandas operations for performance to be acceptable.
This limits our (the computer programmers) ability to help out, because the languages we use are not slow like that, so we may give suggestions from our world that simply can't be used in Python because it's too slow. "Can't you just loop over the data and do XYZ?" Nope, they can't.
We still use Python because it's all the data scientists know, and it's easier/cheaper to just write a bigger check to AWS than to hire people with both data science and computer programming backgrounds.