44 ms·

Pyston 0.3: Self-hosting Sufficiency

127 points by kmod 12y ago

andrewchambers 12y ago
Thanks kmod, keep up the good work.
rdtsc 12y ago
Great work! I remember when Unladen Swallow project was active, they were encountering some issues with LLVM which slowed their progress down. They often had to stop to fix LLVM. Wonder if Pyston team got a chance to look or learn from Unladen Swallow's idea or even use any of the code?
- boulos 12y ago
  A lot of the barriers that Unladen Swallow / Reid and friends ran into were fixed as part of their efforts (e.g. MB limits on emitted code, no gdb support, etc.). Building a dynamic language runtime on top of LLVM is much much easier today than it was when Unladen Swallow tried, and I'd say it's partly thanks to their initial effort.
  - acdha 12y ago
    I'd bet they've also benefited considerably from the work done with the WebKit FTLJIT project, too: https://trac.webkit.org/wiki/FTLJIT https://trac.webkit.org/wiki/FTLJIT
    - boulos 12y ago
      Right, I should have mentioned that. Though as the WebKit-related patchpoint and other bits happened more recently, I had (still?) assumed that Pyston hasn't gotten to leverage those much yet. Turns out I'm wrong: https://github.com/dropbox/pyston/blob/master/src/codegen/patchpoints.cpp https://github.com/dropbox/pyston/blob/master/src/codegen/pa...
phkahler 12y ago
Does Pyston work with any of the GUI toolkits? If not, is it expected to at some point?
montecarl 12y ago
Is it expected that Pyston play well with numpy/scipy? This is the biggest barrier that I see towards adoption of PyPy for numerical computing. If Pyston can work with numpy and scipy then it would be a huge accomplishment.
- ngoldbaum 12y ago
  Their focus is on getting the dropbox application working under pyston. It would probably take contributions from someone outside of dropbox to make this a reality. Presumably it will be easier on pyston than with pypy since pyston has designed in better compatibility with the CPython C API.
- dagw 12y ago
  If speeding up numpy/scipy code is important to you (and it's certainly important to me), then numba is probably the project on the horizon to follow.
cpr 12y ago
Is Guido still at Dropbox? Contributing to this effort? If he is, that would certainly make the effort carry a lot more weight.
- techdragon 12y ago
  Why? Not downplaying how important our esteemed BDFL Guido is to Python, but I'll be flat out honest, aiming their sights on Python 2 just further entrenches its eventual place as the COBOL of 2050. Python 2 is DEAD to me. It should be dead to you. It's Dead with a capital D. Node.js wound up with all this fork nonsense in HALF the time Python 2.7 has been stagnating the Python ecosystem. Python 2 only code is nuclear waste grade technical debt. I haven't written a line of Python 2 in 6 months after 2 years of slow decline. I now use Python 2 vs 3 as an interview question. I run my tests on 3.4.x, 3.5-dev, and versions of HEAD that pass the Python test suite. Why does Guido working on Pyston do anything but hurt the future of Python by legitimising the position that it's ok as a community of Python programmers, to keep accruing this technical debt ?
  - shadowmint 12y ago
    Pyston isnt python 3 compatible.
  - svisser 12y ago
    Python 3 has no benefits if your Python 2 code works and many dependencies are not Python 3 compatible yet. Why would a company even port to Python 3?
    - jsmeaton 12y ago
      There aren't that many dependencies without py3 support anymore. Of course, if your project uses any then being stuck on py2 is a given. I still don't think projects should port to python 3 unless they need to be moved forward or continually developed. Greenfield projects should most definitely start on py3 though.
    - duaneb 12y ago
codexon 12y ago
How is this different from pypy?
- chrisseaton 12y ago
  One difference that was originally highlighted by the Pyston team as being significant is that this is a method based JIT, where PyPy is a tracing JIT. I think it also prioritises support for C extensions, which PyPy does not.
Scarbutt 12y ago
But what if condoleza sneaks a trojan somewhere in the pyston code?
eiopa 12y ago
What this actually does right now is run some internal Dropbox scripts that exercises an unknown subset of Python + python stdlib, plus the perf benchmark. Unless I am missing something, the stuff you guys achieved have absolutely nothing to do with "self-hosting". I am curious - how does this compare to pypy, performance wise? Is there a big win with this approach over pypy?
- ctoshok 12y ago
  The use of "self-hosting" was addressed in post. I think its use is actually right on the mark, but will grant you that it's slightly different than its use in the context of a static compiler written in the target language. It basically means "able to host its own development". This means something different in different contexts, e.g. JS JITs "self-host" when some part of their builtin code is written in JS. We share that property with JS JITs, but in this particular case it means we can host our own build/test infrastructure. So we're using pyston to develop pyston. ergo, "self-hosting (qualified)." --- We're definitely slower than pypy at present - we're only a small fraction faster than cpython after all on our benchmarks, and pypy is scary fast :) I think we're still in the part of our work where the answer to the "big win with this approach" question is "we don't know." We're confident that we can/will be much faster than we are now, but we have different constraints than pypy, so it will always be something of an apples/oranges comparison.
  - eiopa 12y ago
    I see. I think it's a huge stretch :) That's like Go claiming they are self-hosting because some of their tools are written in Go. On a related note, I have to say that I still don't understand the point of this project. Were you guys unhappy with pypy's performance or development? I understand that in theory, the approach you guys are taking is incompatible with pypy's, but I can't help but wonder how thing would play out if you Dropbox put their weight behind PyPy. Maybe it could've been the start of every Python programmer's wet dream: Python 2.8.
    - rian 12y ago
shadowmint 12y ago
Unless I've deeply misunderstood the code on github, this appears to be a python 2 compatible implementation. I have mixed feelings about that.
- coldtea 12y ago
  Python 2 isn't going away any time soon. Less than 10% of the community uses 3 according to pip version usages...
  - shadowmint 12y ago
    I know, but the project page says: Currently, Pyston targets Python 2.7, only runs on x86_64 platforms, and only has been tested on Ubuntu. Support for more platforms -- along with Python 3 compatibility -- is planned for the future, but this is the initial target due to prioritization constraints. ...but if pypy has taught us anything, surely it's that implementing python3 after you have a working python2 implementation is a seriously huge piece of work. Especially if you're wholesale copying unicode naive cpython code into your code base. I'm deeply skeptical python 3 support will ever land for this. Basically, this is 'modernize python 2 and make it faster'; there's been a lot of talk about no one being willing to pickup and maintain a 2.8 version, but this is it, effectively. Major backer, major new features. So, lots of good things here, but there no doubt that its going to be divisive in the community, and I'm not sure I really support that. ...promising py3 support nebulously at some point in the future doesn't fix anything. tldr; If you ever plan on supporting python3, do it already. Otherwise don't make fake promises.
    - BuckRogers 12y ago
      It seems the main feature to Python3 over 2.x is that it enables you to beg projects to port to Python3. But Python3 people- it's opensource. Isn't that wonderful? Python3 patches welcome.
      - shadowmint 12y ago
        Don't be ridiculous. There's a reason the python 2.x line is 'patch only' by the developers. Python 3 made fundamental changes to underlying string operations internally for UFT16, which you can easily argue, was a huge mistake, but there you go. You can't just 'patch python3 support' in. You literally have to rip out anything that uses char * in the code base and replace it with a unicode supported alternative, which is both more complex, and breaks python 2 backwards compatibility. Pypy is very clever in how they handle this through rpython, which is why they can kind of support both; but randomly dropping cpython 2.x code into the project is completely not forward looking. I'd be happy with: "We never intend to support python 3, sorry". If that's the path you want to walk for all the complicated reasons you choose it, fair enough.
kasabali 12y ago
Does anybody know if they considered a similar thing to what HotPy (2) [0] did? I'm not anywhere near to understanding the implementation details, but HotPy stood up most when I looked into different Python implementations. Basically, it was a fork of an early development version of CPython 3.3 code, with more bytecodes added (which were easier to optimize than the originals), a tracing engine and some other things I don't understand. The best thing is you get language compatibilitity for free and it is also binary compatible with CPython extensions. Unfortunately its development looks to be ceased. [0] https://sites.google.com/site/makingcpythonfast/ https://sites.google.com/site/makingcpythonfast/
- IanOzsvald 12y ago
  HotPy is Mark Shannon's research project, he's not making a 'new better Python for the masses', he's using it to test his assumptions about ways CPython could be improved. Ask him about it if you bump into him at a python conference!
  - kasabali 12y ago
    Thanks, it's more clear now. I misunderstood Pyston's motives and thought it was just making Python faster rather than writing it from stratch.
rogerbinns 12y ago
Does the C extension API remain the same? ie can I take an existing extension and just recompile it, or does the code have to support a different API. I ported my main extension to PyPy but had to leave bits of functionality out because of missing parts of the API that CPython had that they didn't.

127 points by kmod 12y ago

Thanks kmod, keep up the good work.

rdtsc 12y ago

Great work! I remember when Unladen Swallow project was active, they were encountering some issues with LLVM which slowed their progress down. They often had to stop to fix LLVM. Wonder if Pyston team got a chance to look or learn from Unladen Swallow's idea or even use any of the code?

boulos 12y ago
A lot of the barriers that Unladen Swallow / Reid and friends ran into were fixed as part of their efforts (e.g. MB limits on emitted code, no gdb support, etc.). Building a dynamic language runtime on top of LLVM is much much easier today than it was when Unladen Swallow tried, and I'd say it's partly thanks to their initial effort.
- acdha 12y ago
  I'd bet they've also benefited considerably from the work done with the WebKit FTLJIT project, too: https://trac.webkit.org/wiki/FTLJIT https://trac.webkit.org/wiki/FTLJIT
  - boulos 12y ago
    Right, I should have mentioned that. Though as the WebKit-related patchpoint and other bits happened more recently, I had (still?) assumed that Pyston hasn't gotten to leverage those much yet. Turns out I'm wrong: https://github.com/dropbox/pyston/blob/master/src/codegen/patchpoints.cpp https://github.com/dropbox/pyston/blob/master/src/codegen/pa...

phkahler 12y ago

Does Pyston work with any of the GUI toolkits? If not, is it expected to at some point?

montecarl 12y ago

Is it expected that Pyston play well with numpy/scipy? This is the biggest barrier that I see towards adoption of PyPy for numerical computing. If Pyston can work with numpy and scipy then it would be a huge accomplishment.

ngoldbaum 12y ago
Their focus is on getting the dropbox application working under pyston. It would probably take contributions from someone outside of dropbox to make this a reality. Presumably it will be easier on pyston than with pypy since pyston has designed in better compatibility with the CPython C API.
dagw 12y ago
If speeding up numpy/scipy code is important to you (and it's certainly important to me), then numba is probably the project on the horizon to follow.

cpr 12y ago

Is Guido still at Dropbox? Contributing to this effort? If he is, that would certainly make the effort carry a lot more weight.

techdragon 12y ago
Why? Not downplaying how important our esteemed BDFL Guido is to Python, but I'll be flat out honest, aiming their sights on Python 2 just further entrenches its eventual place as the COBOL of 2050. Python 2 is DEAD to me. It should be dead to you. It's Dead with a capital D. Node.js wound up with all this fork nonsense in HALF the time Python 2.7 has been stagnating the Python ecosystem. Python 2 only code is nuclear waste grade technical debt. I haven't written a line of Python 2 in 6 months after 2 years of slow decline. I now use Python 2 vs 3 as an interview question. I run my tests on 3.4.x, 3.5-dev, and versions of HEAD that pass the Python test suite. Why does Guido working on Pyston do anything but hurt the future of Python by legitimising the position that it's ok as a community of Python programmers, to keep accruing this technical debt ?
- shadowmint 12y ago
  Pyston isnt python 3 compatible.
- svisser 12y ago
  Python 3 has no benefits if your Python 2 code works and many dependencies are not Python 3 compatible yet. Why would a company even port to Python 3?
  - jsmeaton 12y ago
    There aren't that many dependencies without py3 support anymore. Of course, if your project uses any then being stuck on py2 is a given. I still don't think projects should port to python 3 unless they need to be moved forward or continually developed. Greenfield projects should most definitely start on py3 though.
  - duaneb 12y ago

codexon 12y ago

How is this different from pypy?

chrisseaton 12y ago
One difference that was originally highlighted by the Pyston team as being significant is that this is a method based JIT, where PyPy is a tracing JIT. I think it also prioritises support for C extensions, which PyPy does not.

Scarbutt 12y ago

But what if condoleza sneaks a trojan somewhere in the pyston code?

eiopa 12y ago

What this actually does right now is run some internal Dropbox scripts that exercises an unknown subset of Python + python stdlib, plus the perf benchmark. Unless I am missing something, the stuff you guys achieved have absolutely nothing to do with "self-hosting". I am curious - how does this compare to pypy, performance wise? Is there a big win with this approach over pypy?

ctoshok 12y ago
The use of "self-hosting" was addressed in post. I think its use is actually right on the mark, but will grant you that it's slightly different than its use in the context of a static compiler written in the target language. It basically means "able to host its own development". This means something different in different contexts, e.g. JS JITs "self-host" when some part of their builtin code is written in JS. We share that property with JS JITs, but in this particular case it means we can host our own build/test infrastructure. So we're using pyston to develop pyston. ergo, "self-hosting (qualified)." --- We're definitely slower than pypy at present - we're only a small fraction faster than cpython after all on our benchmarks, and pypy is scary fast :) I think we're still in the part of our work where the answer to the "big win with this approach" question is "we don't know." We're confident that we can/will be much faster than we are now, but we have different constraints than pypy, so it will always be something of an apples/oranges comparison.
- eiopa 12y ago
  I see. I think it's a huge stretch :) That's like Go claiming they are self-hosting because some of their tools are written in Go. On a related note, I have to say that I still don't understand the point of this project. Were you guys unhappy with pypy's performance or development? I understand that in theory, the approach you guys are taking is incompatible with pypy's, but I can't help but wonder how thing would play out if you Dropbox put their weight behind PyPy. Maybe it could've been the start of every Python programmer's wet dream: Python 2.8.
  - rian 12y ago

shadowmint 12y ago

Unless I've deeply misunderstood the code on github, this appears to be a python 2 compatible implementation. I have mixed feelings about that.

coldtea 12y ago
Python 2 isn't going away any time soon. Less than 10% of the community uses 3 according to pip version usages...
- shadowmint 12y ago
  I know, but the project page says: Currently, Pyston targets Python 2.7, only runs on x86_64 platforms, and only has been tested on Ubuntu. Support for more platforms -- along with Python 3 compatibility -- is planned for the future, but this is the initial target due to prioritization constraints. ...but if pypy has taught us anything, surely it's that implementing python3 after you have a working python2 implementation is a seriously huge piece of work. Especially if you're wholesale copying unicode naive cpython code into your code base. I'm deeply skeptical python 3 support will ever land for this. Basically, this is 'modernize python 2 and make it faster'; there's been a lot of talk about no one being willing to pickup and maintain a 2.8 version, but this is it, effectively. Major backer, major new features. So, lots of good things here, but there no doubt that its going to be divisive in the community, and I'm not sure I really support that. ...promising py3 support nebulously at some point in the future doesn't fix anything. tldr; If you ever plan on supporting python3, do it already. Otherwise don't make fake promises.
  - BuckRogers 12y ago
    It seems the main feature to Python3 over 2.x is that it enables you to beg projects to port to Python3. But Python3 people- it's opensource. Isn't that wonderful? Python3 patches welcome.
    - shadowmint 12y ago
      Don't be ridiculous. There's a reason the python 2.x line is 'patch only' by the developers. Python 3 made fundamental changes to underlying string operations internally for UFT16, which you can easily argue, was a huge mistake, but there you go. You can't just 'patch python3 support' in. You literally have to rip out anything that uses char * in the code base and replace it with a unicode supported alternative, which is both more complex, and breaks python 2 backwards compatibility. Pypy is very clever in how they handle this through rpython, which is why they can kind of support both; but randomly dropping cpython 2.x code into the project is completely not forward looking. I'd be happy with: "We never intend to support python 3, sorry". If that's the path you want to walk for all the complicated reasons you choose it, fair enough.

kasabali 12y ago

Does anybody know if they considered a similar thing to what HotPy (2) [0] did? I'm not anywhere near to understanding the implementation details, but HotPy stood up most when I looked into different Python implementations. Basically, it was a fork of an early development version of CPython 3.3 code, with more bytecodes added (which were easier to optimize than the originals), a tracing engine and some other things I don't understand. The best thing is you get language compatibilitity for free and it is also binary compatible with CPython extensions. Unfortunately its development looks to be ceased. [0] https://sites.google.com/site/makingcpythonfast/ https://sites.google.com/site/makingcpythonfast/

IanOzsvald 12y ago
HotPy is Mark Shannon's research project, he's not making a 'new better Python for the masses', he's using it to test his assumptions about ways CPython could be improved. Ask him about it if you bump into him at a python conference!
- kasabali 12y ago
  Thanks, it's more clear now. I misunderstood Pyston's motives and thought it was just making Python faster rather than writing it from stratch.

rogerbinns 12y ago

Does the C extension API remain the same? ie can I take an existing extension and just recompile it, or does the code have to support a different API. I ported my main extension to PyPy but had to leave bits of functionality out because of missing parts of the API that CPython had that they didn't.