7 ms·

Pyston 0.6.1 released, and future plans

53 points by uluyol 10y ago

MikusR 10y ago
In short. Project dead. Dropbox moves to Go.
- avbor 10y ago
  Moving from Python to Go seems to be the thing that happens once you need your Python to be more performant.
  - rurban 10y ago
    Same thing for ruby and perl also. I would have advocated to move to pony instead, since it is a much cleaner and much faster language than Go, and has guaranteed deadlock and race safety, unlike Go or rust. It also needs much less memory.
    - jankiel 10y ago
      Pony is in alpha and seems to be overcomplicated at first glance
      - shakna 10y ago
        Pony doesn't feel alpha. I've spoken a little about it before [0]. The Actor model might feel verbose, but it gives you amazing concurrency, and with Pony's compiler, safety. Pony is fast, concurrent by default, and type safe. That's amazing. [0] https://news.ycombinator.com/item?id=13233607 https://news.ycombinator.com/item?id=13233607
      - rurban 10y ago
        Overcomplicated? Compare that to manual locking required for rust concurrency or c++ concurrency. Or the copying overhead you get with Go, which can still deadlock. pony does not need any locks, just proper types. And it can copy or share state in threads. And it has nominal or structural types. Old-style inherited or new mixin OO. It is far advanced over rust. Just distributed actors as in Erlang are not there yet.
  - crdoconnor 10y ago
    This is what happens when you follow the fashion driven development merry go round: https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5B6EGFZ8XTOTNJ7Hd40WOUHo/mobilepresent?pli=1#slide=id.g70b0035b2_1_168 https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5... Re-implementing the hot path (which is likely to be a small piece of code) in a non-garbage collected language (C or ideally Rust) is a much more sane approach than the big bang rewrite: https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html https://blog.sentry.io/2016/10/19/fixing-python-performance-...
kasabali 10y ago
> We spent much more time than we expected on compatibility We similarly had to spend more time on memory usage due to it being a bigger concern than expected What did they expect anyway for starting from scratch instead of improving on existing solid implementations.
- mhneu 10y ago
  When they say compatibility they largely mean C API compatibility. PyPy doesn't even try for this, and yet it's key for any real world adoption. Building on top of PyPy wouldn't have helped with C API compatibility. I really hope PyPy can organize funds to hire these guys to continue working. They have accumulated massive experience on JITting the Python runtime.
zellyn 10y ago
Parallel post from Unladen Swallow… http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospective.html http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospect...
- zellyn 10y ago
  Also, can I just take a moment to whine about saying "95% faster"? Does that mean it takes just 5% of the time to run the same benchmark? Or does it mean it takes 95% of the time to run the same benchmark? Or does it mean it can run 195 iterations of a routine in the same time that 100 used to take?
- mhneu 10y ago
  Not really the same - Unladen Swallow gave up because llvm JIT was immature. Pyston gave up because Dropbox is moving to Go. That's a bad sign.
  - zellyn 10y ago
    Sure, the approach is different. But the "large company allows a couple of devs to work on a faster Python impl for a while until they realize it's harder than expected" story seems very similar.
wiz21c 10y ago
FTA : In the end, we are happy that we attempted this, are excited about the many potential ways that our work on Pyston could still be useful, and are happy to refocus ourselves on domains with more immediate needs. was this written by the HR department ? They're basically happy to work on something dead, happy their employer (most probably) stopped supporting them, happy to change job. They're always happy. I'd love to work at dropbox :-)
jerven 10y ago
PyPy seems to be the one real contender for fast Python. ZipPy was super interesting academic work but unfortunately does not seem to be going anywhere. I am not surprised that a silicon valley company stopped sponsoring this. 3 years of silicon valley salary is quite an investment in small gains of CPU perf. Look at the Ruby sphere, where competitive implementations have taken millions in man hours and still most run on MRI. The Oracle Labs Truffle approach and the PyPy RPython ones seem to be the only way to deliver a compatible, fast language implementation at a cost that a community and or enterprise can bear. Both Truffle and PyPy are long running projects, but both offered more than just faster CPU times. i.e. see related work on accelerating SQL using RPython and most likely behind closed doors for Truffle. Truffle also has a sane compatibility story for Ruby and similar ZipPy would have offered it for Python.
beagle3 10y ago
There are quite a few python JITs over the years - Psyco -> PyPy, unladen swallow, pyston, numba ; that's in addition to Python2C, ShedSkin and Nuitka (and even Cython) which are AOT by way of C. And yet, none emerges as a general clear choice. I have the feeling this is because python is too dynamic making this a sysiphean effort. I urge designers of future dynamic languages to study Lua and LuaJIT if you want your language to not inhibit JIT development.
- mhneu 10y ago
  You are right, the difficulties are in Python's design. See the Pyston blog "Why is Python slow". http://blog.kevmod.com/2016/07/why-is-python-slow/ http://blog.kevmod.com/2016/07/why-is-python-slow/ That said, the Pyston guys knew what the problems were (dynamism in C runtime) and were making progress. This was the future of fast Python - JIT experts optimizing the runtime and feeding code and ideas back to CPython. I think we will look back at this Pyston announcement as the turning point where people gave up on Python speed.
  - coldtea 10y ago
    >You are right, the difficulties are in Python's design. Not anymore than there were in Javascript's design -- which is as, if not more, dynamic. It's just lack of funding. PyPy managed to get much faster than CPython, but doesn't have the funding or the blessing of being official (Javascript doesn't need it, since there were always multiple implementations).
    - beagle3 10y ago
      Javascript does have it easier by virtue of being single threaded, with no overloading and with a single numeric type. In the python version of this loop: for x in range(1000): sum = sum + math.sin(x) The global "math" and "math.sin" can be changed by another thread at any point. Javascript guarantees that this cannot happen. Also, what constructs of Javascript are more dynamic than python? I can only think of "eval" and "with", both of which the JavaScript JIT decline to compile and interpret instead.
mhneu 10y ago
It is really bad news for Python that Dropbox is mostly done with them. (Announcement says most new code is in Go, so further Pyston development couldn't be justified). PyPy is not feasible for most workloads. Pyston was making incremental improvements and their developers demonstrated deep knowledge about both Python and JITs (see their blog, "Why is Python slow"). Accelerating numpy is not important, and rewriting it in Python is not useful because the key numerics will never be as fast as Atlas. Only the Pyston approach of JITting the C runtime is likely to work in the wild. And Dropbox just gave up on it. Unless another company like Facebook funds these two guys or another JIT/VM expert to do full-time work on Pyston, I believe Python's future as a competitor to Go and JS is dead.
- z1mm32m4n 10y ago
  Is that a bad thing? People used Python even when it was slow and there were faster alternatives. People will still use it going forward.
  - seanp2k2 10y ago
    Yeah...for a lot of scripts I've written in Python, it doesn't make a lick of difference if they run in 1 second or 10 minutes. They're glue-type things holding some internal systems together which run in response to events or on crontabs, using widely-used battle-tested libraries, and there clearly-written + decently documented. If someone needs to mess with them some day, they shouldn't have much trouble. As much as I would personally rather write ruby or bash over python (key error will be on my tombstone), for AWS automation stuff, load CSV, mess with it, stuff it into a DB stuff, simple REST APIs, and countless other small-ish automations, Python works just fine at the current speed.

53 points by uluyol 10y ago

MikusR 10y ago

In short. Project dead. Dropbox moves to Go.

avbor 10y ago
Moving from Python to Go seems to be the thing that happens once you need your Python to be more performant.
- rurban 10y ago
  Same thing for ruby and perl also. I would have advocated to move to pony instead, since it is a much cleaner and much faster language than Go, and has guaranteed deadlock and race safety, unlike Go or rust. It also needs much less memory.
  - jankiel 10y ago
    Pony is in alpha and seems to be overcomplicated at first glance
    - shakna 10y ago
      Pony doesn't feel alpha. I've spoken a little about it before [0]. The Actor model might feel verbose, but it gives you amazing concurrency, and with Pony's compiler, safety. Pony is fast, concurrent by default, and type safe. That's amazing. [0] https://news.ycombinator.com/item?id=13233607 https://news.ycombinator.com/item?id=13233607
    - rurban 10y ago
      Overcomplicated? Compare that to manual locking required for rust concurrency or c++ concurrency. Or the copying overhead you get with Go, which can still deadlock. pony does not need any locks, just proper types. And it can copy or share state in threads. And it has nominal or structural types. Old-style inherited or new mixin OO. It is far advanced over rust. Just distributed actors as in Erlang are not there yet.
- crdoconnor 10y ago
  This is what happens when you follow the fashion driven development merry go round: https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5B6EGFZ8XTOTNJ7Hd40WOUHo/mobilepresent?pli=1#slide=id.g70b0035b2_1_168 https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5... Re-implementing the hot path (which is likely to be a small piece of code) in a non-garbage collected language (C or ideally Rust) is a much more sane approach than the big bang rewrite: https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html https://blog.sentry.io/2016/10/19/fixing-python-performance-...

kasabali 10y ago

> We spent much more time than we expected on compatibility We similarly had to spend more time on memory usage due to it being a bigger concern than expected What did they expect anyway for starting from scratch instead of improving on existing solid implementations.

mhneu 10y ago
When they say compatibility they largely mean C API compatibility. PyPy doesn't even try for this, and yet it's key for any real world adoption. Building on top of PyPy wouldn't have helped with C API compatibility. I really hope PyPy can organize funds to hire these guys to continue working. They have accumulated massive experience on JITting the Python runtime.

zellyn 10y ago

Parallel post from Unladen Swallow… http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospective.html http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospect...

zellyn 10y ago
Also, can I just take a moment to whine about saying "95% faster"? Does that mean it takes just 5% of the time to run the same benchmark? Or does it mean it takes 95% of the time to run the same benchmark? Or does it mean it can run 195 iterations of a routine in the same time that 100 used to take?
mhneu 10y ago
Not really the same - Unladen Swallow gave up because llvm JIT was immature. Pyston gave up because Dropbox is moving to Go. That's a bad sign.
- zellyn 10y ago
  Sure, the approach is different. But the "large company allows a couple of devs to work on a faster Python impl for a while until they realize it's harder than expected" story seems very similar.

wiz21c 10y ago

FTA : In the end, we are happy that we attempted this, are excited about the many potential ways that our work on Pyston could still be useful, and are happy to refocus ourselves on domains with more immediate needs. was this written by the HR department ? They're basically happy to work on something dead, happy their employer (most probably) stopped supporting them, happy to change job. They're always happy. I'd love to work at dropbox :-)

jerven 10y ago

PyPy seems to be the one real contender for fast Python. ZipPy was super interesting academic work but unfortunately does not seem to be going anywhere. I am not surprised that a silicon valley company stopped sponsoring this. 3 years of silicon valley salary is quite an investment in small gains of CPU perf. Look at the Ruby sphere, where competitive implementations have taken millions in man hours and still most run on MRI. The Oracle Labs Truffle approach and the PyPy RPython ones seem to be the only way to deliver a compatible, fast language implementation at a cost that a community and or enterprise can bear. Both Truffle and PyPy are long running projects, but both offered more than just faster CPU times. i.e. see related work on accelerating SQL using RPython and most likely behind closed doors for Truffle. Truffle also has a sane compatibility story for Ruby and similar ZipPy would have offered it for Python.

beagle3 10y ago

There are quite a few python JITs over the years - Psyco -> PyPy, unladen swallow, pyston, numba ; that's in addition to Python2C, ShedSkin and Nuitka (and even Cython) which are AOT by way of C. And yet, none emerges as a general clear choice. I have the feeling this is because python is too dynamic making this a sysiphean effort. I urge designers of future dynamic languages to study Lua and LuaJIT if you want your language to not inhibit JIT development.

mhneu 10y ago
You are right, the difficulties are in Python's design. See the Pyston blog "Why is Python slow". http://blog.kevmod.com/2016/07/why-is-python-slow/ http://blog.kevmod.com/2016/07/why-is-python-slow/ That said, the Pyston guys knew what the problems were (dynamism in C runtime) and were making progress. This was the future of fast Python - JIT experts optimizing the runtime and feeding code and ideas back to CPython. I think we will look back at this Pyston announcement as the turning point where people gave up on Python speed.
- coldtea 10y ago
  >You are right, the difficulties are in Python's design. Not anymore than there were in Javascript's design -- which is as, if not more, dynamic. It's just lack of funding. PyPy managed to get much faster than CPython, but doesn't have the funding or the blessing of being official (Javascript doesn't need it, since there were always multiple implementations).
  - beagle3 10y ago
    Javascript does have it easier by virtue of being single threaded, with no overloading and with a single numeric type. In the python version of this loop: for x in range(1000): sum = sum + math.sin(x) The global "math" and "math.sin" can be changed by another thread at any point. Javascript guarantees that this cannot happen. Also, what constructs of Javascript are more dynamic than python? I can only think of "eval" and "with", both of which the JavaScript JIT decline to compile and interpret instead.

mhneu 10y ago

It is really bad news for Python that Dropbox is mostly done with them. (Announcement says most new code is in Go, so further Pyston development couldn't be justified). PyPy is not feasible for most workloads. Pyston was making incremental improvements and their developers demonstrated deep knowledge about both Python and JITs (see their blog, "Why is Python slow"). Accelerating numpy is not important, and rewriting it in Python is not useful because the key numerics will never be as fast as Atlas. Only the Pyston approach of JITting the C runtime is likely to work in the wild. And Dropbox just gave up on it. Unless another company like Facebook funds these two guys or another JIT/VM expert to do full-time work on Pyston, I believe Python's future as a competitor to Go and JS is dead.

z1mm32m4n 10y ago
Is that a bad thing? People used Python even when it was slow and there were faster alternatives. People will still use it going forward.
- seanp2k2 10y ago
  Yeah...for a lot of scripts I've written in Python, it doesn't make a lick of difference if they run in 1 second or 10 minutes. They're glue-type things holding some internal systems together which run in response to events or on crontabs, using widely-used battle-tested libraries, and there clearly-written + decently documented. If someone needs to mess with them some day, they shouldn't have much trouble. As much as I would personally rather write ruby or bash over python (key error will be on my tombstone), for AWS automation stuff, load CSV, mess with it, stuff it into a DB stuff, simple REST APIs, and countless other small-ish automations, Python works just fine at the current speed.