6 ms·

Pyjion – A JIT for Python based upon CoreCLR

72 points by Lofkin 11y ago

andor 11y ago
Here's their list of passing and failing CPython tests: https://github.com/Microsoft/Pyjion/blob/master/Tests/python_tests.txt https://github.com/Microsoft/Pyjion/blob/master/Tests/python...
- brettcannon 11y ago
  One thing to note about the list of failing tests is the vast majority of them are due to a single bug (or two): https://github.com/Microsoft/Pyjion/issues/103 https://github.com/Microsoft/Pyjion/issues/103 . And we currently only have 11 open bugs so the test failures make it look worse than it is.
sametmax 11y ago
One of the most promissing projects : * it keeps CPython as is. * hence retains compat with extensions. * but still has the potential to improve performance like pypy. * and yet can be shipped just as an additional compiled lib inside the Python dist. If it ever works, it will be wonderful.
- mozumder 11y ago
  It would also be nice if it worked outside of Windows.
  - jsherer 11y ago
    I had the same though and looked into it. It seems that CoreCLR isn't the limitation there. It supports a variety of platforms: https://github.com/dotnet/coreclr#build-status https://github.com/dotnet/coreclr#build-status Looks like the authors are focused on Windows for their initial prototype development.
    - sametmax 11y ago
      Yep. They are MS ingenieers, of course Windows is easier for them. Plus it's good publicity. Given that they are paying for it, it's fair they start like that.
      - brettcannon 11y ago
        Exactly, we started on Windows out of ease-of-start/momentum. There are absolutely no plans to make this a Windows-exclusive. Basically we want to show this work pans out before working before putting in the work to make it cross-platform (we have an open issue to move to CMake to help solve the cross-platform problem: https://github.com/Microsoft/Pyjion/issues/76 https://github.com/Microsoft/Pyjion/issues/76).
        Lofkin 11y ago
        Exciting work! Is pyjion exploratory or is MS determined to see this through?
        brettcannon 11y ago
        Yes. :) It's exploratory in the sense that we are seeing if this general approach works for CPython. But if this exploratory work shows promise, then we will see it through. And since I'm also a member of the Python development team I always want to see Python be faster and succeed more. :)
        Lofkin 11y ago
        Sweet, new Microsoft is awesome! Pyston seems to be catching up to PyPy fairly quickly ...not sure if there is anything transferable to pyjion, but if you haven't done so might be worth checking out. Also for getting around the GIL, PyParallel seems to have a really interesting (and seemingly successful approach: http://pyparallel.org/ http://pyparallel.org/ Its being worked on by one of the Guy's at continuum...and It's definitely in continuum's best interest to help python's future so I'm sure they wouldn't mind collab.
  - smortaz 11y ago
    if the speedups pan out, that is the plan! the work on performance has not been started yet however.
    - Lofkin 11y ago
      Awesome! A while back there was some talk of native excel-python integration. Any chance that is in the pipes, or is MS sticking to R in that sense?
      - brettcannon 11y ago
        You can vote for that feature at https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10549005-python-as-an-excel-scripting-language https://excel.uservoice.com/forums/304921-excel-for-windows-...
    - mozumder 11y ago
      Sounds great. Looking forward to testing it out.
the_gigi 11y ago
How does Pygion compare with IronPython?
- ericfrederich 11y ago
  From reading the description, and from what little I know about IronPython... IronPython is a re-implementation of Python in C# which allows it to play nice with .NET. This is similar to Jython which is a re-implementation of Python in Java that plays nice with the JVM. This project on the other had is not a re-implementation at all, but aims to augment the existing CPython implementation with JIT APIs and is using CoreCLR for the JIT.
  - brettcannon 11y ago
    What ericfrederich said is all accurate. We are also considering playing with ChakraCore as a JIT back-end as well once we are sure that our abstract interpreter code that drives the JIT emission code is compatible with Python semantics.
    - tadlan 11y ago
      Woah!!! Cool
ericfrederich 11y ago
Is there a PEP for this, or is it too early, more of a proof of concept?
- brettcannon 11y ago
  It's too early. We want to prove to ourselves that we can get the performance we want before proposing the changes we think we want to CPython's C API.
brettcannon 11y ago
Thanks to all of the great questions we got, we have now written an FAQ for the project: https://github.com/Microsoft/Pyjion#faq https://github.com/Microsoft/Pyjion#faq

72 points by Lofkin 11y ago

andor 11y ago

Here's their list of passing and failing CPython tests: https://github.com/Microsoft/Pyjion/blob/master/Tests/python_tests.txt https://github.com/Microsoft/Pyjion/blob/master/Tests/python...

brettcannon 11y ago
One thing to note about the list of failing tests is the vast majority of them are due to a single bug (or two): https://github.com/Microsoft/Pyjion/issues/103 https://github.com/Microsoft/Pyjion/issues/103 . And we currently only have 11 open bugs so the test failures make it look worse than it is.

sametmax 11y ago

One of the most promissing projects : * it keeps CPython as is. * hence retains compat with extensions. * but still has the potential to improve performance like pypy. * and yet can be shipped just as an additional compiled lib inside the Python dist. If it ever works, it will be wonderful.

mozumder 11y ago
It would also be nice if it worked outside of Windows.
- jsherer 11y ago
  I had the same though and looked into it. It seems that CoreCLR isn't the limitation there. It supports a variety of platforms: https://github.com/dotnet/coreclr#build-status https://github.com/dotnet/coreclr#build-status Looks like the authors are focused on Windows for their initial prototype development.
  - sametmax 11y ago
    Yep. They are MS ingenieers, of course Windows is easier for them. Plus it's good publicity. Given that they are paying for it, it's fair they start like that.
    - brettcannon 11y ago
      Exactly, we started on Windows out of ease-of-start/momentum. There are absolutely no plans to make this a Windows-exclusive. Basically we want to show this work pans out before working before putting in the work to make it cross-platform (we have an open issue to move to CMake to help solve the cross-platform problem: https://github.com/Microsoft/Pyjion/issues/76 https://github.com/Microsoft/Pyjion/issues/76).
      - Lofkin 11y ago
        Exciting work! Is pyjion exploratory or is MS determined to see this through?
        brettcannon 11y ago
        Yes. :) It's exploratory in the sense that we are seeing if this general approach works for CPython. But if this exploratory work shows promise, then we will see it through. And since I'm also a member of the Python development team I always want to see Python be faster and succeed more. :)
        Lofkin 11y ago
        Sweet, new Microsoft is awesome! Pyston seems to be catching up to PyPy fairly quickly ...not sure if there is anything transferable to pyjion, but if you haven't done so might be worth checking out. Also for getting around the GIL, PyParallel seems to have a really interesting (and seemingly successful approach: http://pyparallel.org/ http://pyparallel.org/ Its being worked on by one of the Guy's at continuum...and It's definitely in continuum's best interest to help python's future so I'm sure they wouldn't mind collab.
- smortaz 11y ago
  if the speedups pan out, that is the plan! the work on performance has not been started yet however.
  - Lofkin 11y ago
    Awesome! A while back there was some talk of native excel-python integration. Any chance that is in the pipes, or is MS sticking to R in that sense?
    - brettcannon 11y ago
      You can vote for that feature at https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10549005-python-as-an-excel-scripting-language https://excel.uservoice.com/forums/304921-excel-for-windows-...
  - mozumder 11y ago
    Sounds great. Looking forward to testing it out.

the_gigi 11y ago

How does Pygion compare with IronPython?

ericfrederich 11y ago
From reading the description, and from what little I know about IronPython... IronPython is a re-implementation of Python in C# which allows it to play nice with .NET. This is similar to Jython which is a re-implementation of Python in Java that plays nice with the JVM. This project on the other had is not a re-implementation at all, but aims to augment the existing CPython implementation with JIT APIs and is using CoreCLR for the JIT.
- brettcannon 11y ago
  What ericfrederich said is all accurate. We are also considering playing with ChakraCore as a JIT back-end as well once we are sure that our abstract interpreter code that drives the JIT emission code is compatible with Python semantics.
  - tadlan 11y ago
    Woah!!! Cool

ericfrederich 11y ago

Is there a PEP for this, or is it too early, more of a proof of concept?

brettcannon 11y ago
It's too early. We want to prove to ourselves that we can get the performance we want before proposing the changes we think we want to CPython's C API.

brettcannon 11y ago

Thanks to all of the great questions we got, we have now written an FAQ for the project: https://github.com/Microsoft/Pyjion#faq https://github.com/Microsoft/Pyjion#faq