9 ms·
I'm not versed in the details of the politics of CPython, but why did this project fork instead of just contributing to CPython? Is CPython really slow in inte
by jedberg 6y ago
I'm not versed in the details of the politics of CPython, but why did this project fork instead of just contributing to CPython? Is CPython really slow in integrating community contributions?
Edit: I read the blog post closer, and found this: "Our plan is to open-source the code in the future, but since compiler projects are expensive and we no longer have benevolent corporate sponsorship, it is currently closed-source while we iron out our business model."
- simonh 6y agoThey intend to monetise it.
- deleted 6y ago[deleted]
- dwheeler 6y agoAs far as I know CPython is quite welcoming. However, per the blog post, this version of Pyston is closed source. So CPython won't be interested, and many others won't be interested either.
- rrdharan 6y agoPython is a welcoming community, but they are decidedly not welcoming of contributions that significantly increase the complexity of the CPython reference implementation. This has been discussed extensively: https://news.ycombinator.com/item?id=11125769 https://news.ycombinator.com/item?id=11125769
- toolslive 6y agoit's _still_ essentially the same switch(opcode) based interpreter it was 20 years ago. no threading, no super instructions, no jitting, nothing. https://github.com/python/cpython/blob/master/Python/ceval.c#L1568 https://github.com/python/cpython/blob/master/Python/ceval.c...
- jashmatthews 6y agoDirect threading is in here https://github.com/python/cpython/blob/0564aafb71a153dd0aca4b9266dfae9336a4f2cb/Python/ceval.c#L990 https://github.com/python/cpython/blob/0564aafb71a153dd0aca4...
- toolslive 6y agoha! you're right. so some optimizations did happen while I wasn't looking ;)
- FartyMcFarter 6y ago> no threading, CPython does use threads (on Linux, it spawns one pthread per Python thread). However it also has a lock (the infamous GIL) that prevents those threads from interpreting Python code simultaneously.
- PeCaN 6y agoNot that. https://en.wikipedia.org/wiki/Threaded_code https://en.wikipedia.org/wiki/Threaded_code It's an interpreter implementation technique.
- throwaway894345 6y agoThis is why I see little hope for Python, which is to say that while I'm sure it will continue to have a large following for many years a la C, C++, etc, I don't have hope for it being an exciting language or one that is particularly productive. Python already has performance and packaging problems which don't seem to be easily divorced from CPython, since virtually the whole reference implementation is depended upon directly by much of the ecosystem due to the sprawling C-extension interface. Pypy has done yeoman's work in improving performance while maintaining compatibility with an impressive amount of the ecosystem, and even still there are many important packages which aren't compatible with Pypy and for which Pypy-compatible analogs don't exist (or aren't supported/maintained). For example, the only Pypy-compatible Postgres drivers were unsupported last I checked. Moreover, the Python community (or at least its leadership) seems to have very little energy around tackling these longstanding problems. Meanwhile, there are many other languages which are not only performant, but which are rapidly encroaching on Python's historically unique(ish) "easiness" (in the sense that Python is considered "easy", which is to say for people who don't have to manage build/deploy/packaging/etc or otherwise have performance issues). Further, many of these languages continue to improve at a remarkable pace, while Python is content to rest on the laurels of its scientific computing mindshare--and given the rather poor nature of the numeric computing package APIs and their somewhat low performance ceiling, I don't expect Python to be so dominant in this domain in another 5-10 years, especially as more companies need to figure out how to productionize scientific workloads.
- saurik 6y ago100% agree. To a very real extent, Go probably only exists at all because Google poured an infinite amount of effort into Unladen Swallow--a project designed to remove the GIL from Python 2 to make it more usable for concurrent programs and add an LLVM-based JIT compiler to improve performance--and Python's response was to not only not merge it, but to break the entire Python ecosystem for a decade by forking the language... while somehow leaving "concurrency" and "performance" off the list of goals while making their backwards incompatible change (which of course also destroyed all the work on Unladen Swallow); hell: early versions of Python 3 were benchmarking universally slower than Python 2 :/. Google seems to have "gotten the hint", and around the same time (2009) hired Rob Pike to do Go, and has since "moved on" from Python and taken a massive part of the ecosystem of users Python used to have with it (the rest having bailed for node.js; remember when Python was the future of web development, competing with Ruby thanks to Django? those were the days): people still use Python, but it is now an entirely different crop of data scientists and AI people... all of whom are starting to run into the performance issues in even their glue layer, and so if the Swift tensorflow efforts ever work out, Python is done.
- Jasper_ 6y agoIt's a shame how the core maintainers see CPython as a "reference implementation", rather than an opportunity to make a world-class programming language implementation. That the CPython maintainers have, time and time again, decided for a "simple implementation" has pushed away many professional VM engineers and researchers who would be more than willing to help maintain a JIT. I also will say that a lot of the complexity of making a Python JIT is all the weird edge cases and special interpreter functionality. To a VM engineer, CPython is a very bizarre codebase; all the complexity is tucked away in corners other than its C codebase. Your complexity has to go somewhere.
- coldtea 6y ago>It's a shame how the core maintainers see CPython as a "reference implementation" Ironically this also makes CPython less interesting as a reference language - which keeps PyPy and co insignificant in adoption...
- orf 6y agoI might have to eat those old words as there is some form of runtime code specialisation being considered for inclusion in CPython. It doesn’t sound like a fully-fledged JIT, but that might be a distinction without a difference.
- ak217 6y agoLink?
- orf 6y agohttps://github.com/markshannon/faster-cpython/blob/master/plan.md https://github.com/markshannon/faster-cpython/blob/master/pl... And some discussion: https://news.ycombinator.com/item?id=24848318 https://news.ycombinator.com/item?id=24848318
- ak217 6y agoThanks - Mark Shannon is a core committer, right? So hopefully he has standing to get this done. The plan looks very high level at this point, but it looks like Mark is an expert in interpreter VM and JIT technologies. All I can hope for is that he doesn't get blocked by the "keep cpython simple" obstructionism.
- mrtranscendence 6y agoHe thinks (they think?) the project will need to be funded to the tune of $2M. That seems like a hefty sum to me, though maybe it's doable.
- ak217 6y agoIt's not that hefty once they pitch this improvement to the many wealthy VC-funded companies whose business and data science divisions depend on Python.
- jnwatson 6y agoThat post was 4 years ago. Even though I haven't heard explicitly, certainly some of the recent comment from core developer talk about performance techniques that would definitely make things more complicated. There's new boss(es) over the Python project, and things might be changing in the near future.
- N1H1L 6y agoI read the thread, it's from 2016. But hasn't numba made some big progress in JITting CPython?
- kmod 6y agoI would say there's a small minority of our changes that we can upstream, and eventually we'd like to upstream them. For the rest, I would say the different set of priorities for the two projects means they probably will stay separate: - In the blog post I linked to an issue of someone trying to upstream a quickening implementation and meeting resistance due to the added complexity - CPython prefers portability over performance, and we added a number of big build dependencies that may not work on all the platforms that CPython supports, though from our perspective it works on all the important ones - CPython has included a number of performance-degrading features over the years and we plan to start cutting out some that are only used for debugging but still hurt release mode. I don't know for sure but I would expect resistance to backing out those changes. There are more, but the general idea is that the two projects will make different tradeoffs. Maybe there is a world where the code all lives in the same repository but is gated between different behaviors, but the CPython maintainers have already rejected a proposal like that.
- samcolvin 6y agoVictor Stinner (Python core developer focused on performance) has a great speech about this question: There are a lot of reasons actually - Performance limited by old CPython design. If you fork it you have to deal with all the legacy code. - CPython is limited to 1 thread because of the GIL. - Specific memory allocators, C structures, reference counting, specific garbage collector etc. You can find that video in here: https://youtu.be/TXRPCZ7Nmh4 https://youtu.be/TXRPCZ7Nmh4