5 ms·

A Team at Microsoft Is Helping Make Python Faster

41 points by pieterr 4y ago

scaredginger 4y ago
I think it's fairly well understood at this point that a big reason for Python's slow performance is the desire to keep the interpreter simple so that people can more easily contribute to it. I'd add that this has evidently been a great strategy; Python is beloved and its ecosystem is among the best of all languages. However, approaches should evolve over time and Python is at the point of maturity where people probably don't need to be hacking on the interpreter, and the inefficiencies are causing problems; for instance, in server costs. Some groups have even forked Python to implement their own optimisations, which fractures the ecosystem. Hence, the time has come to focus on performance, for the health of the language's community, as well as the social benefits. All this is to say that in my opinion, Python continues to be a well-managed project and it is still growing in a positive direction
- roflyear 4y ago
  Yeah but coming from someone who has read a lot of MS produced code I'm a little frightened lol. But if there is one thing MS knows it is compilers.
  - klyrs 4y ago
    Microsoft is a giant collection of individuals, each with their own programming abilities. Specifically, Guido is a Microsoft employee, he's at the head of this team. And my understanding is that they're going to go the traditional PEP route to mainline their contributions.
    - t-vi 4y ago
      Also Mark Shannon who drew up the original Faster Python plan that spurred this work, joined Microsoft to work on this.
ngcc_hk 4y ago
It is a long way from feeling immediately Halloween or extend/… to now listening to a group hired by Microsoft working on such a major open source project without too much worrying. Cheers.
rwalle 4y ago
I guess of course I should not expect to see IronPython mentioned under the "Microsoft’s Commitment to Python" section.
- lloydatkinson 4y ago
  Doesn't surprise me unfortunately, any languages that target .NET that aren't C# or VB are ignored.
29athrowaway 4y ago
I hope it does not become like node-chakracore and the "VM neutrality" and other farces intended to 3E node, but now with Python. Or Microsoft R Open and MRAN, the 3E of R and CRAN.
webmobdev 4y ago
Embrace, extend, extinguish - https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
- kkielhofner 4y ago
  I remember that era quite well, especially as I naturally gravitated towards Linux and open source. It was a genuine concern at the time. Seeing as that was over 25 years ago I have to wonder how much it holds true today. Looking at MS moves over the past decade (the .NET ecosystem, Mono, VS Code, Github, Azure, etc) I’m not that concerned.
  - traverseda 4y ago
    Really? This is pretty clearly the "embrace" step. I can't help but see things like the inability to use passwords with github unless you use the github-specific cli tool as an attempt to extend. See also, directx for wsl. Sure, we're mostly in the happy embrace stage now, and wr may even be able to use some of the extentions, but pretty soon the barriers to entry will start going up more and more.
    - UncleEntity 4y ago
      I guess in the same way google is extending with all these electron apps? Google isn’t a convicted monopolist so I guess there’s that.
    - kkielhofner 4y ago
      Or maybe, just maybe, after 25 years and multiple changes of leadership it's not the same company. I'd be curious to see how many employees from the mid 1990s are left. There are murderers walking around today that had shorter sentences. Note that I'm not defending them specifically, I just think it's strange the EEE stuff gets reposted every once in a while without a 1996 tag...
      - Qem 4y ago
eska 4y ago
Frankly speaking the target 5x improvement, if everything works out as desired, doesn’t change too much overall. Python is often magnitudes slower than other languages, oftentimes even other scripting languages. So for me at least it won’t change when I use python and when I decide to use/rewrite in another language.
- Qem 4y ago
  IIRC, that 5x figure was assuming they could improve about 50% across four consecutive versions, as 1.5^4 ~= 5. I think the first stage improved things by 25% on average, so I wonder if that initial 5x figure should be updated to 2.5x, as 1.25^4 ~= 2.5.
- nicolaslem 4y ago
  Python is really good at gluing code together. I predict that Python will continue to see tremendous success if it can be fast enough that using it as glue is a no-brainer. As an example, I maintain a SaaS with a backend written in Python. Only one third of the time in the backend is spent executing Python code, the rest is spent talking with remote services (database, third-parties...). Switching to a language an order of magnitude faster would barely change the perceived responsiveness of the backend. On the other hand, I welcome any speed improvements in the interpreter because from my perspective it really is free performance.
antman 4y ago
At this point all optimizations that make the codebase more complicated should be postponed. Finally someone made a good enough GIL-less patch a couple of versions back, this should be analysed and integrated ASAP so that we take advantage of today's multicore systems. MS's 25% speed improvement is nice but I an not sure it doesn't move things in a wrong direction.
bfrog 4y ago
At this point isn’t pypy the peak effort on this front? If things need to go faster it’s probably easier to either write it in go or rust, or write a small extension in C or Rust I’d think?

41 points by pieterr 4y ago

I think it's fairly well understood at this point that a big reason for Python's slow performance is the desire to keep the interpreter simple so that people can more easily contribute to it. I'd add that this has evidently been a great strategy; Python is beloved and its ecosystem is among the best of all languages. However, approaches should evolve over time and Python is at the point of maturity where people probably don't need to be hacking on the interpreter, and the inefficiencies are causing problems; for instance, in server costs. Some groups have even forked Python to implement their own optimisations, which fractures the ecosystem. Hence, the time has come to focus on performance, for the health of the language's community, as well as the social benefits. All this is to say that in my opinion, Python continues to be a well-managed project and it is still growing in a positive direction

roflyear 4y ago
Yeah but coming from someone who has read a lot of MS produced code I'm a little frightened lol. But if there is one thing MS knows it is compilers.
- klyrs 4y ago
  Microsoft is a giant collection of individuals, each with their own programming abilities. Specifically, Guido is a Microsoft employee, he's at the head of this team. And my understanding is that they're going to go the traditional PEP route to mainline their contributions.
  - t-vi 4y ago
    Also Mark Shannon who drew up the original Faster Python plan that spurred this work, joined Microsoft to work on this.

ngcc_hk 4y ago

It is a long way from feeling immediately Halloween or extend/… to now listening to a group hired by Microsoft working on such a major open source project without too much worrying. Cheers.

rwalle 4y ago

I guess of course I should not expect to see IronPython mentioned under the "Microsoft’s Commitment to Python" section.

lloydatkinson 4y ago
Doesn't surprise me unfortunately, any languages that target .NET that aren't C# or VB are ignored.

29athrowaway 4y ago

I hope it does not become like node-chakracore and the "VM neutrality" and other farces intended to 3E node, but now with Python. Or Microsoft R Open and MRAN, the 3E of R and CRAN.

webmobdev 4y ago

Embrace, extend, extinguish - https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

kkielhofner 4y ago
I remember that era quite well, especially as I naturally gravitated towards Linux and open source. It was a genuine concern at the time. Seeing as that was over 25 years ago I have to wonder how much it holds true today. Looking at MS moves over the past decade (the .NET ecosystem, Mono, VS Code, Github, Azure, etc) I’m not that concerned.
- traverseda 4y ago
  Really? This is pretty clearly the "embrace" step. I can't help but see things like the inability to use passwords with github unless you use the github-specific cli tool as an attempt to extend. See also, directx for wsl. Sure, we're mostly in the happy embrace stage now, and wr may even be able to use some of the extentions, but pretty soon the barriers to entry will start going up more and more.
  - UncleEntity 4y ago
    I guess in the same way google is extending with all these electron apps? Google isn’t a convicted monopolist so I guess there’s that.
  - kkielhofner 4y ago
    Or maybe, just maybe, after 25 years and multiple changes of leadership it's not the same company. I'd be curious to see how many employees from the mid 1990s are left. There are murderers walking around today that had shorter sentences. Note that I'm not defending them specifically, I just think it's strange the EEE stuff gets reposted every once in a while without a 1996 tag...
    - Qem 4y ago

eska 4y ago

Frankly speaking the target 5x improvement, if everything works out as desired, doesn’t change too much overall. Python is often magnitudes slower than other languages, oftentimes even other scripting languages. So for me at least it won’t change when I use python and when I decide to use/rewrite in another language.

Qem 4y ago
IIRC, that 5x figure was assuming they could improve about 50% across four consecutive versions, as 1.5^4 ~= 5. I think the first stage improved things by 25% on average, so I wonder if that initial 5x figure should be updated to 2.5x, as 1.25^4 ~= 2.5.
nicolaslem 4y ago
Python is really good at gluing code together. I predict that Python will continue to see tremendous success if it can be fast enough that using it as glue is a no-brainer. As an example, I maintain a SaaS with a backend written in Python. Only one third of the time in the backend is spent executing Python code, the rest is spent talking with remote services (database, third-parties...). Switching to a language an order of magnitude faster would barely change the perceived responsiveness of the backend. On the other hand, I welcome any speed improvements in the interpreter because from my perspective it really is free performance.

antman 4y ago

At this point all optimizations that make the codebase more complicated should be postponed. Finally someone made a good enough GIL-less patch a couple of versions back, this should be analysed and integrated ASAP so that we take advantage of today's multicore systems. MS's 25% speed improvement is nice but I an not sure it doesn't move things in a wrong direction.

bfrog 4y ago

At this point isn’t pypy the peak effort on this front? If things need to go faster it’s probably easier to either write it in go or rust, or write a small extension in C or Rust I’d think?