7 ms·

Does Python still have performance issues? That's always been my concern when considering its adoption.

by bicx 7y ago

pytester 7y ago
It's always prioritized delivery speed and quality over raw performance. If you want an ultra optimized web framework it will always be the wrong choice - it was never designed for that.
heinrichhartman 7y ago
nothing significant changed.
ageitgey 7y ago
The are some new-ish features like async and there some new parallelization options in 3.8 that help in some very specific cases, but generally you choose Python because you value development speed more than execution speed. You can use Python (or Ruby, etc) to build fast websites that scale well if you are smart about how you use it and how you design your application, but it still isn't a particularly fast language compared to others.
sojmq 7y ago
Yes. If you can afford the CPU power, by all means go with Python. Otherwise just don't. Instagram can afford it so it works fine for them. Reddit can't, so their site is always slow or down.
- traverseda 7y ago
  I think reddits issues are more related to database IO than cpu time.
- kashug 7y ago
  And you know it is becouse of python reddit is often slow or down? Or are you just blaming pythin without any actual data on the underlying reasons?
- mattbillenstein 7y ago
  You don't know what you're talking about.
traverseda 7y ago
I mean, what kind of performance issues were you running into? You're not going to get maximum performance out of a python program, generally. But unless your python program is using 100% cpu that's not really an issue, and there are very few cases where I'm using python and it's CPU that's slowing down the program. Mostly it ends up being some kind of IO. If you're looking to do complicated mathematical operations, you can call out to an external library like `numpy` of tensorflow to take better advantage of cpu/gpu compute. You can also use a faster python interpreter, like `pypy`, although then you start getting slower startup times and more memory usage. The performance is closer to something like java but drawbacks are generally enough that they don't get used, since it's very rare that you need that extra performance. There's also micropython, which lets you run python on a microcontroller and makes it pretty easy to optimize things for your hardware, although you lose some useful debugging and introspection features. But for the most part, I haven't run into anything I'd call "performance issues", just the knowledge that python is a scripting language and it's never going to be the fastest thing around. Python's performance just isn't an issue very often.
- sojmq 7y ago
  In my experience most people creating CRUD websites with Django will hit a CPU bottleneck.
  - onepointsixC 7y ago
    Would Django going async ease that a bit or not really?
    - sojmq 7y ago
      async doesn't decrease the amount of CPU cycles needed to do work, in any case it only increases it.
  - traverseda 7y ago
    I've got more than a decade of on-and-off web-dev experience, a lot of integrating data-science projects with a web-interface, and I've never seen anything like that. You have to work pretty hard to hit a CPU bottleneck, and I can't imagine how you'd do that building a simple CRUD website. Can you explain a bit more about how the people you know are hitting that bottleneck? I mean I've literally built CRUD apps on an esp32 microcontroller using python, and they work performantly. If real python with a real web-framework can't do the same then something is going horrible wrong.
pas 7y ago
Yes and no. The Python interpreter/bytecode-VM is not as fast as it could be, because only a few people work on it. PyPy is great, but not drop in. Lack of funding is the proximate cause, of course, but that's just masking the ultimate cause of lack of interest. Because people use Python as a glue, just like PHP, or Node. To do the heavy lifting you have many options. (Even in Python, you can just use Cython, use FFI to call out to anything with A C interface, like Rust, or C++, or C. Many Python packages do exactly that, wrap a battle tested old native lib.) That said the Python VM is a well optimized beast, it's very dynamic, you can monkey patch anything and everything at runtime on-the-fly. This is of course what makes it hard to speed up, too many things to "check for overloadedness" when doing anything. A JIT could help, but that's a lot of work. (Hence why PyPy took a decade, and the whole Python C API compatibility problem is still there - PyPy can only JIT the Py parts.) Anyway. With that much dynamic stuff and with the VM and the language already well tuned for concurrent memory safety its 'async' story is great and performant. (Like NodeJS'.) So Python is great for orchestrating whatever you need to handle requests. And if you want to scale up, just spin up more instances. (Just like with Java, but you don't have to think much about the overhead garbage collection, because CPython uses simple reference counting.) Finally, my concern with Python was the lack of static typing, which made any large project hard to manage. (And resulted in a lot of boilerplate tests and excessive defensive coding.) But mypy is great, and with that it's a joy to work in Python. (Just like TypeScript made the JS world exponentially much more saner.)
- zielmicha 7y ago
  > PyPy is great, but not drop in. Nowadays it works correctly with almost all C modules. Mostly you just need to use `pypy3 -m pip install foo` instead of `python3 -m pip install foo`.
  - pas 7y ago
    Last year I had to fight with it quite a bit to get it to work with uwsgi. It seems that any new/greenfield Python project should target PyPy though, just in case, and the performance never hurts.
- thekingofh 7y ago
takeda 7y ago
Yes it does, and almost everything else will have performance issues except assembly. With web services, chances are that you'll hit other bottlenecks before you hit it. In python you have multiple ways to optimize performance, for example use PyPy, or various libraries like Cython that compile your performance sensitive code. That's how asyncpg can outperform[1] some libraries that you normally would think was not possible. [1] https://github.com/MagicStack/asyncpg https://github.com/MagicStack/asyncpg

Does Python still have performance issues? That's always been my concern when considering its adoption.

by bicx 7y ago

Does Python still have performance issues? That's always been my concern when considering its adoption.

It's always prioritized delivery speed and quality over raw performance. If you want an ultra optimized web framework it will always be the wrong choice - it was never designed for that.

heinrichhartman 7y ago

nothing significant changed.

ageitgey 7y ago

The are some new-ish features like async and there some new parallelization options in 3.8 that help in some very specific cases, but generally you choose Python because you value development speed more than execution speed. You can use Python (or Ruby, etc) to build fast websites that scale well if you are smart about how you use it and how you design your application, but it still isn't a particularly fast language compared to others.

sojmq 7y ago

Yes. If you can afford the CPU power, by all means go with Python. Otherwise just don't. Instagram can afford it so it works fine for them. Reddit can't, so their site is always slow or down.

traverseda 7y ago
I think reddits issues are more related to database IO than cpu time.
kashug 7y ago
And you know it is becouse of python reddit is often slow or down? Or are you just blaming pythin without any actual data on the underlying reasons?
mattbillenstein 7y ago
You don't know what you're talking about.

traverseda 7y ago

I mean, what kind of performance issues were you running into? You're not going to get maximum performance out of a python program, generally. But unless your python program is using 100% cpu that's not really an issue, and there are very few cases where I'm using python and it's CPU that's slowing down the program. Mostly it ends up being some kind of IO. If you're looking to do complicated mathematical operations, you can call out to an external library like `numpy` of tensorflow to take better advantage of cpu/gpu compute. You can also use a faster python interpreter, like `pypy`, although then you start getting slower startup times and more memory usage. The performance is closer to something like java but drawbacks are generally enough that they don't get used, since it's very rare that you need that extra performance. There's also micropython, which lets you run python on a microcontroller and makes it pretty easy to optimize things for your hardware, although you lose some useful debugging and introspection features. But for the most part, I haven't run into anything I'd call "performance issues", just the knowledge that python is a scripting language and it's never going to be the fastest thing around. Python's performance just isn't an issue very often.

sojmq 7y ago
In my experience most people creating CRUD websites with Django will hit a CPU bottleneck.
- onepointsixC 7y ago
  Would Django going async ease that a bit or not really?
  - sojmq 7y ago
    async doesn't decrease the amount of CPU cycles needed to do work, in any case it only increases it.
- traverseda 7y ago
  I've got more than a decade of on-and-off web-dev experience, a lot of integrating data-science projects with a web-interface, and I've never seen anything like that. You have to work pretty hard to hit a CPU bottleneck, and I can't imagine how you'd do that building a simple CRUD website. Can you explain a bit more about how the people you know are hitting that bottleneck? I mean I've literally built CRUD apps on an esp32 microcontroller using python, and they work performantly. If real python with a real web-framework can't do the same then something is going horrible wrong.

pas 7y ago

Yes and no. The Python interpreter/bytecode-VM is not as fast as it could be, because only a few people work on it. PyPy is great, but not drop in. Lack of funding is the proximate cause, of course, but that's just masking the ultimate cause of lack of interest. Because people use Python as a glue, just like PHP, or Node. To do the heavy lifting you have many options. (Even in Python, you can just use Cython, use FFI to call out to anything with A C interface, like Rust, or C++, or C. Many Python packages do exactly that, wrap a battle tested old native lib.) That said the Python VM is a well optimized beast, it's very dynamic, you can monkey patch anything and everything at runtime on-the-fly. This is of course what makes it hard to speed up, too many things to "check for overloadedness" when doing anything. A JIT could help, but that's a lot of work. (Hence why PyPy took a decade, and the whole Python C API compatibility problem is still there - PyPy can only JIT the Py parts.) Anyway. With that much dynamic stuff and with the VM and the language already well tuned for concurrent memory safety its 'async' story is great and performant. (Like NodeJS'.) So Python is great for orchestrating whatever you need to handle requests. And if you want to scale up, just spin up more instances. (Just like with Java, but you don't have to think much about the overhead garbage collection, because CPython uses simple reference counting.) Finally, my concern with Python was the lack of static typing, which made any large project hard to manage. (And resulted in a lot of boilerplate tests and excessive defensive coding.) But mypy is great, and with that it's a joy to work in Python. (Just like TypeScript made the JS world exponentially much more saner.)

zielmicha 7y ago
> PyPy is great, but not drop in. Nowadays it works correctly with almost all C modules. Mostly you just need to use `pypy3 -m pip install foo` instead of `python3 -m pip install foo`.
- pas 7y ago
  Last year I had to fight with it quite a bit to get it to work with uwsgi. It seems that any new/greenfield Python project should target PyPy though, just in case, and the performance never hurts.
thekingofh 7y ago

takeda 7y ago

Yes it does, and almost everything else will have performance issues except assembly. With web services, chances are that you'll hit other bottlenecks before you hit it. In python you have multiple ways to optimize performance, for example use PyPy, or various libraries like Cython that compile your performance sensitive code. That's how asyncpg can outperform[1] some libraries that you normally would think was not possible. [1] https://github.com/MagicStack/asyncpg https://github.com/MagicStack/asyncpg