10 ms·
Python 3.11 Performance Benchmarks Are Looking Fantastic
- mountainriver 4y agoSo glad they are doing this and it’s just the start from what I understand. We also may have actual concurrency in python soon.
- gshulegaard 4y agoCan you elaborate on what you mean by "actual concurrency in python soon"?
- AlphaSite 4y agoThere was an experimental branch where someone reimplemented Python 3.9 sans the Gil and managed to maintain performance parity with the Gil’ed version. It’s likely this performance uplift is through the various optimisations that PR implemented.
- bratao 4y agoThe majority of the improvements come from the Faster-CPython (https://github.com/faster-cpython/ideas https://github.com/faster-cpython/ideas) project, sponsored by Microsoft (Thanks!!)
- samwillis 4y agoI believe some of these improvements were part of the same work to eventually remove the GIL (global interpreter lock), albeit not tied to it. However, the final decision on removing the GIL hasn't been made.
- gshulegaard 4y agoAh yes, this is what I thought was the current state. I was curious if a GIL removal decision had been made that I had missed, but seems like it's still a soft "maybe": https://lukasz.langa.pl/5d044f91-49c1-4170-aed1-62b6763e6ad0/ https://lukasz.langa.pl/5d044f91-49c1-4170-aed1-62b6763e6ad0... Edit: And to be fair, you can do "true" concurrency in Python you just have to eject to Multi-processing (with the tradeoffs that implies), so I didn't want to assume the comment was about GIL removal.
- axblount 4y agoSam Gross has been working on a branch that removes the GIL and has a few other performance improvements. More details are here on the python mailing list: https://mail.python.org/archives/list/python-dev@python.org/thread/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/ https://mail.python.org/archives/list/python-dev@python.org/... There's a link to his design overview document in the thread.
- onphonenow 4y agoA lot of credit for these improvements come from that gil removal effort. I very very much hope we get it or a well supported branch with those changes.
- dekhn 4y agoI think that's actually "actual parallelism", not concurrency. Concurrency is multiple processes making forward progress using a single processor and some form of blocking/switching, while parallelism takes advantage of multiple cores. Personally, I consider concurrency a degenerate form of parallelism limited to multiplexing processes on a single core.
- pertymcpert 4y agoConcurrency is really a higher level of abstraction than parallelism. It's more of a program structuring concept.
- samwillis 4y agoThere is a breakdown of the test implementations here: https://pyperformance.readthedocs.io/benchmarks.html https://pyperformance.readthedocs.io/benchmarks.html I'm particularly happy to see the django_template test showing so much improvement. The implementation of Django templates has been a hot topic for speed improvements for along time (and why many people jump to Jinja). Its a good example of how "complex" python will be improved by these optimisations. Two interesting ones to compare are pickle_pure_python and json_loads (both doing (de)serialisation activities), the former shows a large improvement, the latter much less so. Again it's showing how the improvements are coming to complex pure python code, json.load is mostly c whereas the pickle_pure_python test is run against the pure python Pickle implementation. It would be interesting to see a benchmark based on the Django test suite, that would probably be quite indicative of the impairments we can expect in real world web server Python.
- butterisgood 4y agoPython for Netgroups
- EdSchouten 4y ago*Workgroups
- butterisgood 4y agoD'Oh!
- pjmlp 4y agoOne day it will finally be an Algol based Lisp, with a proper JIT on the reference implementation. https://norvig.com/python-lisp.html https://norvig.com/python-lisp.html Yeah, I know, at least my UNIX scripting gets a bit of performance boost.
- cjalmeida 4y agoLike Julia :)
- UmbertoNoEco 4y agoWithout the fanboyism and the excessive corporate slant
- amkkma 4y agoYou think python is free of those things?
- pjmlp 4y agoYes, big Julia fan, although there are domains where Julia might never come, like sysadmin.
- NeverFade 4y agoUnfortunately, there are no signs that CPython is moving towards integrating a JIT after all these years and despite its massive popularity. There's therefore no clear path towards substantially improved CPython performance, and it remains among the slowest mainstream languages, really the slowest among its top group of 4-5 most popular languages.
- kzrdude 4y agoJIT has been mentioned in the faster-cpython work though. But you're right to be skeptical, they are putting it off too ("it's not 3.11 and maybe not for 3.12"), and it's not something that can be added easily between two releases.
- pid-1 4y agoI just formatted my laptop to try Ubuntu 22.04 out. Thanks for remind me I need to install an ad blocker. Seriously, I counted ten banners for the whole article.
- kzrdude 4y agohttps://ublockorigin.com https://ublockorigin.com is the one to recommend
- tyingq 4y agoHow this happened with PHP was interesting to watch from the side. The competition that came from Facebook's Hack/HHVM seemed to be the catalyst, and helped set expectations for some benchmarks.
- axblount 4y agoI hope Sam Gross's no-GIL changes are adopted. It would be amazing to have full multithreaded concurrency in python. https://mail.python.org/archives/list/python-dev@python.org/thread/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/ https://mail.python.org/archives/list/python-dev@python.org/...
- bratao 4y agoI really wish this project didn't die, this would open to so many use-cases. I hate to use multi-process to scale my projects.
- matsemann 4y agoYeah, while it in some regards is nice having multiple small deployments of our python app than can be scaled up and down independently, it's sometimes such an hurdle to something "extra" in a smaller app. Have a simple webserver and want to do some things out of band? In jvm I would use a concurrent queue and add tasks to it, and spin up some threads consuming that. In python you either spin up a new process and deal with the communication problems, become a master of greenlet and python internals, or most likely default to celery which need it's own deployment. It sometimes boggles my mind how python is considered the quick and easy way for startups, while at the same time doing trivial things become such a hurdle.
- ledauphin 4y agoit's a huge hurdle in any scenario where you have large, static dataset from which you want to derive computation-heavy results. Multiprocessing generally requires making copies of that data, which can be impossible. The workarounds (e.g. mmap) take an order of magnitude more engineering effort.
- dekhn 4y agoI agree 100% (I come from a "use all the cores with threads and shared memory to get linear speedups on many workloads" world). But I think at this point the python leadership has committed to the "every time the GIL-removal looks plausible, find some more single-core speedups to stave off the change". Put another way: I've given up on the hope that python will ever multicore the way I think it should multicore.
- Sin2x 4y agoThat's only testing beta 1, so there's going to be a lot more presents.
- kzrdude 4y agoI think there won't be any radical changes during betas, they probably end up in the next release.
- Sin2x 4y agoRadical no, that would require to add JIT which is planned for 3.12 AFAIK. But there are two performance improvements after release of first beta already: https://speed.python.org/ https://speed.python.org/
- kzrdude 4y agoI've been following what PRs the faster-cpython people merge and it's been nothing on the beta branch, I think. For example this PR is not getting merged, it seems. https://github.com/python/cpython/pull/93379 https://github.com/python/cpython/pull/93379
- Sin2x 4y agoThat's only 7 days ago. I might be wrong, but I think Python betas are not like RCs -- they are changing things, not just polishing.
- m_ke 4y agoAnother interesting thing coming soon is PEP 690: Lazy Imports, which should speed up start time and lower memory usage. See https://www.youtube.com/watch?v=ohTPzi9Lry0 https://www.youtube.com/watch?v=ohTPzi9Lry0
- dash2 4y agoI remember Perl 6 (now Raku) used to advertise itself as "the first 100 year programming language". I wonder if actually that could be Python. It seems like the update process has a lot of "taste", in that it adds features that are really useful and yet the syntax still looks clean and intuitive.
- samwillis 4y ago"the first 100 year programming language" is definitely JavaScript. (If thinking in terms of Perl/Python like scripting and versatility, otherwise clearly C)
- ska 4y agoLisp has a 20 (25?) year head start; it's not more dead than usual.
- qsort 4y agoDepends on what you mean by "100-year language". If it's in the pg sense (i.e. a language that's on the main branch of the evolutionary path of programming languages) I think the answer is clearly the ML family. If you mean "what language is most likely to be around in 2100 AD", realistically we'll have C for as long as we'll have computers.
- chucky_z 4y agoSQL!
- rat87 4y agoI was thinking probably Cobol will be first since it has at least 63 years already but apparently LISP is technically a year older. Fortran is a year older but I'm not sure it will last as long as Cobol/Lisp (it's losing out to python for simplicity and is likely to bleed a lot to c/c++/rust for science libraries/things that require faster glue code)
- m463 4y agoA C G T :)
- TekMol 4y agoIs there an easy way to use Python 3.11 on Debian?
- whalesalad 4y agoInstall dependencies with apt (a few popular pages on google for this). Download tarball. Configure. Final step is ‘make altinstall’ and then you can use it standalone. If you already use virtual environments you would then recreate your venv and the rest of your experience would remain the same. I’d share actual syntax but I’m mobile right now. Alternatively a tool like pyenv makes this much easier.
- TekMol 4y agoThis might be above my paygrade, but let me try: >>> Install dependencies with apt (a few popular pages on google for this). I don't know what this means :) So let's go to step 2: >>> Download tarball. Me tries this: wget 'https://www.python.org/ftp/python/3.11.0/Python-3.11.0a1.tgz' >>> Configure. What does this mean? >>> make altinstall I guess I somehow need to untar the tar thing first? Me tries this: tar -xf Python-3.11.0a1.tgz Hurray, I have a directory :) Let's cd into it: cd Python-3.11.0a1 And now the make altinstall? make altinstall Gives me: bash: make: command not found Hm... apt install make make altinstall Gives me: make: *** No rule to make target 'altinstall'. Stop. Hm... yeah, looks like it is beyond my paygrade.
- teddyh 4y ago> >>> Configure. > What does this mean? It means to run the “./configure” script first, which is normally present in the unpacked tar file. Also, on Debian, to compile programs, you usually first have to install the “build-essential” package.
- TingPing 4y ago> > Install dependencies with apt (a few popular pages on google for this). > I don't know what this means :) So let's go to step 2: `apt-get build-dep python3`
- your_challenger 4y agoLooks like with "PEP 659: Specializing Adaptive Interpreter"[1] and typing[2] we're slowly building a faster language [1] https://docs.python.org/3.11/whatsnew/3.11.html#pep-659-specializing-adaptive-interpreter https://docs.python.org/3.11/whatsnew/3.11.html#pep-659-spec... [2] https://docs.python.org/3/library/typing.html https://docs.python.org/3/library/typing.html
- kzrdude 4y agoHow is typing making Python faster?
- funny_falcon 4y agoTyping allows reduce runtime type-checking. It allows to use faster specialized byte-code. It will help JIT to build faster code.
- your_challenger 4y agoHahah the age-old py question "Is there a JIT compiler?"[1]. No [1] https://docs.python.org/3.11/whatsnew/3.11.html#faq https://docs.python.org/3.11/whatsnew/3.11.html#faq
- rbanffy 4y agoI don't think it's that much of a problem. Python fills a niche where performance is not a huge issue. When you compare languages, you look for a balance between how much development costs, what's your time to market, and how much money you'll spend in hosting. Time to market and cost of development usually are more important than hosting costs unless you know you'll be deploying to hundreds or thousands of machines. Most apps never grow that large.