4 ms·

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Pyt

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via direct SQL or a REST API to whatever service owns the database). All these tasks were heavily network-bound. The amount of time the CPU was doing any work was minuscule compared to the amount of time it was waiting to get data back from the network. I doubt Nim or any other language would have been a significant performance improvement in this case. For what it's worth, that made these scripts excellent candidates for multithreading. I'd run them with 20+ threads, and it was glorious. At first I did multiprocessing, because of all the GIL horror stories, but multiprocessing made it very difficult to cache data, so eventually I said "well, all this is network-bound so the GIL doesn't even apply" and switched over to multiprocessing.dummy (which implements pools using the same API as multiprocessing but with threads instead of processes), and I never looked back. Edit: For what it's worth, Nim sounds like a really cool language, and it's right up my alley in several ways, I just don't think Python is particularly slow at network-bound tasks that use very little CPU.

otherme123 5y ago
You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again). But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.
- baq 5y ago
  > Just it's huge ecosystem. self-contradiction at its best. kindly be reminded the same advantage was the only thing that kept Java alive for so long until it finally started to enter the XXI century a couple years ago. you just can't discount an ecosystem, especially if its huge.
  - curioussavage 5y ago
    Exactly. Also I’m not impressed with a lot of the ecosystem. After getting more acquainted with many very important libraries I can see the poor quality and it’s scary. Grateful for the free tools but IMO there other Languages like go where more essential libraries and tooling are written by better programmers. Like actual professionals instead of hobby tier
m_mueller 5y ago
I agree with your entire post but, and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that your ETL process could save it into a new table. And boom you‘re into CPU bound lands now. Often it‘s fine, you can wait those 1-2min for a daily occuring process. But what if for example you put this whole component behind a REST API that is behind a load balancer that is set with a certain timeout? There are even strict upper limits if for example you chose AWS Lambda for your stuff. And suddenly you need to introduce quite a bit more technical complexity into this story that‘s gonna be hard to explain to management - all they see is that you now can insert a couple of millions of DB rows and their Big Data consultants[TM] told them that this is nowadays not even worth thinking about. Point being: If your performance ceiling is low, you‘re gonna hit it sooner.
- DeathArrow 5y ago
  >Point being: If your performance ceiling is low, you‘re gonna hit it sooner. And that isn't the only culprit. Large code bases will be hard to structure, maintain and organize. And you will spend more time writing tests than writing productive code. Because you don't have many defenses against errors and because once a bug will hit production it will be very difficult to debug.
  - m_mueller 5y ago
    I somewhat disagree there. IMO you can structure a large application just as well in python as in Java, but it requires more architecture guidance as your degree of freedom is higher. Regarding testing, mypy and later python versions have been a big help there, I don't think it's much of a difference anymore and I'd take it over Node.js for that reason because the core language IMO is more consistent and strict and requires way way less outside dependencies. In fact the standard lib for me is the main reason to choose python today. So for me it's really performance being the main thing to think about with python. If one anticipates tighter latency or throughput requirements I'd go for something else.
csmpltn 5y ago
> "I'll also add that Python is decent at tasks that aren't CPU-bound" IO-bound tasks are almost by definition outside of your Python application's control. You yield control to the system to execute the actual task, and from that point on - you're no longer in control of how long the task will take to complete. In other words, Python "being fast" by waiting on a Socket to complete receiving data isn't a particularily impressive feat.
- bmicraft 5y ago
  The main point (I think) was that python is a viable language for many use cases that are not processing intensive, while also being very easy and quick to write which is often the most important thing.
yellow_lead 5y ago
As other commenters point out, how can a language be fast in a way besides something CPU bound? You are saying it is fast when it's not doing anything. Not sure I understand.
- baq 5y ago
  it's a shorthand for 'fast enough'. is it slow if nobody would notice if it was 10x faster?
gpderetta 5y ago
In my experience, Python is so slow that it will make CPU bound tasks that have no business being CPU bound.
nerdponx 5y ago
You might want to look into async (asyncio or anyio) instead of or in addition to threads for network-heavy code. Async coroutines I find can be much easier to debug and develop than OS-threaded code.

otherme123 5y ago

You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again). But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.

baq 5y ago
> Just it's huge ecosystem. self-contradiction at its best. kindly be reminded the same advantage was the only thing that kept Java alive for so long until it finally started to enter the XXI century a couple years ago. you just can't discount an ecosystem, especially if its huge.
- curioussavage 5y ago
  Exactly. Also I’m not impressed with a lot of the ecosystem. After getting more acquainted with many very important libraries I can see the poor quality and it’s scary. Grateful for the free tools but IMO there other Languages like go where more essential libraries and tooling are written by better programmers. Like actual professionals instead of hobby tier

m_mueller 5y ago

I agree with your entire post but, and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that your ETL process could save it into a new table. And boom you‘re into CPU bound lands now. Often it‘s fine, you can wait those 1-2min for a daily occuring process. But what if for example you put this whole component behind a REST API that is behind a load balancer that is set with a certain timeout? There are even strict upper limits if for example you chose AWS Lambda for your stuff. And suddenly you need to introduce quite a bit more technical complexity into this story that‘s gonna be hard to explain to management - all they see is that you now can insert a couple of millions of DB rows and their Big Data consultants[TM] told them that this is nowadays not even worth thinking about. Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

DeathArrow 5y ago
>Point being: If your performance ceiling is low, you‘re gonna hit it sooner. And that isn't the only culprit. Large code bases will be hard to structure, maintain and organize. And you will spend more time writing tests than writing productive code. Because you don't have many defenses against errors and because once a bug will hit production it will be very difficult to debug.
- m_mueller 5y ago
  I somewhat disagree there. IMO you can structure a large application just as well in python as in Java, but it requires more architecture guidance as your degree of freedom is higher. Regarding testing, mypy and later python versions have been a big help there, I don't think it's much of a difference anymore and I'd take it over Node.js for that reason because the core language IMO is more consistent and strict and requires way way less outside dependencies. In fact the standard lib for me is the main reason to choose python today. So for me it's really performance being the main thing to think about with python. If one anticipates tighter latency or throughput requirements I'd go for something else.

csmpltn 5y ago

> "I'll also add that Python is decent at tasks that aren't CPU-bound" IO-bound tasks are almost by definition outside of your Python application's control. You yield control to the system to execute the actual task, and from that point on - you're no longer in control of how long the task will take to complete. In other words, Python "being fast" by waiting on a Socket to complete receiving data isn't a particularily impressive feat.

bmicraft 5y ago
The main point (I think) was that python is a viable language for many use cases that are not processing intensive, while also being very easy and quick to write which is often the most important thing.

yellow_lead 5y ago

As other commenters point out, how can a language be fast in a way besides something CPU bound? You are saying it is fast when it's not doing anything. Not sure I understand.

baq 5y ago
it's a shorthand for 'fast enough'. is it slow if nobody would notice if it was 10x faster?

gpderetta 5y ago

In my experience, Python is so slow that it will make CPU bound tasks that have no business being CPU bound.

nerdponx 5y ago

You might want to look into async (asyncio or anyio) instead of or in addition to threads for network-heavy code. Async coroutines I find can be much easier to debug and develop than OS-threaded code.