4 ms·
This is why I see little hope for Python, which is to say that while I'm sure it will continue to have a large following for many years a la C, C++, etc, I don'
by throwaway894345 6y ago
This is why I see little hope for Python, which is to say that while I'm sure it will continue to have a large following for many years a la C, C++, etc, I don't have hope for it being an exciting language or one that is particularly productive. Python already has performance and packaging problems which don't seem to be easily divorced from CPython, since virtually the whole reference implementation is depended upon directly by much of the ecosystem due to the sprawling C-extension interface.
Pypy has done yeoman's work in improving performance while maintaining compatibility with an impressive amount of the ecosystem, and even still there are many important packages which aren't compatible with Pypy and for which Pypy-compatible analogs don't exist (or aren't supported/maintained). For example, the only Pypy-compatible Postgres drivers were unsupported last I checked.
Moreover, the Python community (or at least its leadership) seems to have very little energy around tackling these longstanding problems. Meanwhile, there are many other languages which are not only performant, but which are rapidly encroaching on Python's historically unique(ish) "easiness" (in the sense that Python is considered "easy", which is to say for people who don't have to manage build/deploy/packaging/etc or otherwise have performance issues). Further, many of these languages continue to improve at a remarkable pace, while Python is content to rest on the laurels of its scientific computing mindshare--and given the rather poor nature of the numeric computing package APIs and their somewhat low performance ceiling, I don't expect Python to be so dominant in this domain in another 5-10 years, especially as more companies need to figure out how to productionize scientific workloads.
- saurik 6y ago100% agree. To a very real extent, Go probably only exists at all because Google poured an infinite amount of effort into Unladen Swallow--a project designed to remove the GIL from Python 2 to make it more usable for concurrent programs and add an LLVM-based JIT compiler to improve performance--and Python's response was to not only not merge it, but to break the entire Python ecosystem for a decade by forking the language... while somehow leaving "concurrency" and "performance" off the list of goals while making their backwards incompatible change (which of course also destroyed all the work on Unladen Swallow); hell: early versions of Python 3 were benchmarking universally slower than Python 2 :/. Google seems to have "gotten the hint", and around the same time (2009) hired Rob Pike to do Go, and has since "moved on" from Python and taken a massive part of the ecosystem of users Python used to have with it (the rest having bailed for node.js; remember when Python was the future of web development, competing with Ruby thanks to Django? those were the days): people still use Python, but it is now an entirely different crop of data scientists and AI people... all of whom are starting to run into the performance issues in even their glue layer, and so if the Swift tensorflow efforts ever work out, Python is done.
- uranusjr 6y agoDidn’t the original Go creators mention on multiple occasions they intended to build a replacement to C, only accidentally produced an application language? Assuming that’s true, Go would have happened regardless. Google couldn’t have pulled out of Unladen Swallow before Go is usable, since they could not have known it’s actually an alternative at the time.
- pjmlp 6y agoC++ actually, but most of us did not care. https://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html https://commandcenter.blogspot.com/2012/06/less-is-exponenti... If this version of generics finally makes it, then I care, otherwise only when dealing with Docker and Kubernetes eco-system.
- hitekker 6y agoHuh, I was wondering why Unladen Swallow faded rapidly since 2009 and why Python seems secondary at Google compared to Java and Go. Silence and distance seems to be a common echo of failed projects.
- rhencke 6y agoGoogle did not hire Rob Pike with Go in mind. Go did not even come around as an idea until Pike had worked at Google for a while. [1] Don't forget Rob Pike created Sawzall at Google first (2005). [2] [1] https://golang.org/doc/faq#history https://golang.org/doc/faq#history [2] https://research.google/pubs/pub61/ https://research.google/pubs/pub61/
- fnord123 6y agoThe early Go compiler reused the code generator from Limbo. To the point that many comments referred to Limbo. So while Rob Pike might not have joined Google to work on Go or had Go in mind, the idea and direction for Go was probably preordained. It doesn't really support or attack your claims. It's just a thing I wanted to share.
- sk2020 6y agoScipy has a poor performance ceiling? Numpy has a poor API? Compared to what? Eigen? Whatever the Scala guys use? That sounds kind of silly to me, especially when hardly anyone is actually CPU-bound, anyway.
- throwaway894345 6y agoWRT performance ceiling, I'm mostly talking about things like Pandas which eagerly evaluate and which aren't amenable to a parallel execution model (multiple threads operating on the same data frame with minimal contention). WRT poor APIs, I'm talking about things like matplotlib or pandas or etc that take a whole slew of arguments and try to guess the caller's intent by inspecting the types of the arguments. The referent isn't "some other scientific computing API" (although I'm sure there are some sane scientific computing APIs), but rather "other APIs in general" since there's nothing inherent to any particular domain that demands this kind of 'magical' API. WRT 'hardly anyone is CPU bound'--the context is numeric computing; what are people bound by if not CPU? I've seen several projects where web endpoints were timing out while grinding in Pandas, largely because there weren't good options for taking advantage of multiple processors. Based on prototypes I did, I'm confident that other languages could serve those requests in single-digit seconds if not sub-second.
- willseth 6y ago"Based on prototypes I [spent a limited amount of time on and didn't research better methods], I'm confident..."
- ihnorton 6y agoIn the context of pandas, 3 GB of (raw, uncompressed) data could easily require 30 GB of RAM, and that kind of overhead adds up quickly.
- willseth 6y agoPandas is not some mysterious black box. If you need predictable runtime performance or bounded memory usage, you have to figure it out. Pandas doesn't inherently have a staggering or unpredictable amount of overhead, given that it's a statistical analysis package. There are ways to mitigate Pandas memory usage (10x is a sign that something has gone very horribly wrong), and sometimes Pandas is simply the wrong tool for the job.
- mrtranscendence 6y ago> Meanwhile, there are many other languages which are not only performant, but which are rapidly encroaching on Python's historically unique(ish) "easiness" What are those languages? I may have a blindspot, but the languages that get enough buzz for me to notice are either not competing with Python in important dimensions (e.g. Rust) or have a narrower focus (e.g. Julia). Elixir maybe? JavaScript and its derivatives? But these lack the scientific programming ecosystem that's helped drive Python recently. I have no doubt that languages exist that might fit the bill of being both as easy as and more performant than Python -- there are a lot of languages out there. But I'm unaware of any whose mindshare has been sufficiently growing that it threatens Python in the mid-term. If I'm off base please let me know. I'd love to find a viable competitor to Python that's strictly better than it.
- throwaway894345 6y ago> I'd love to find a viable competitor to Python that's strictly better than it. I strongly recommend Go as a better Python. Personally, I think it's easier to write than Python (although people who care very little about correctness will be bothered a bit by the type checker), and the tooling is many times better (single-binary deployments, great dependency management, etc are awesome). Also, the performance is about 100-1000 times better for serial execution, and Go's goroutines allow you to take advantage of multiple cores much more easily than with Python. JavaScript and TypeScript are similarly easy-to-use, performant languages with a better-than-Python tooling story. I've also heard similar things about Elixir, Closure, and Kotlin.
- vosper 6y agoWhat's Go like for REPL-driven / exploratory development? That's mostly how I use Python.
- bywaterstreet 6y agoGo has nothing on Python in this regard. I write Go every day, and come from a Python background. I often describe Go as the strongly-typed, more performant version of Python. I say this mostly because my Go code isn't too dissimilar from my Python code (structure, naming, packages). But I still drop into Python if I want to do something quickly. I don't really know why. Maybe it's the Go tooling, e.g. unused variables cause compilation errors, so things like this slow me down. Or maybe it's because Python just offers _so_ much out of the box, e.g. all the data structures you'll ever need (list, set, dict, tuples), and all small things you take for granted (like writing "is a in list", which would require a function in Go). The REPL is Python's killer feature.
- willseth 6y agoA different perspective is that Python leadership has been overwhelmed addressing the concerns of the enormous and growing Python community, for whom generally performance is not yet the primary concern - believe it or not. It's probably fair to suggest PSF has stumbled in executing some of their goals, most notably and publicly the transition to v3, but overall it seems like the general Python community is most interested in language and ecosystem features, while performance-critical workloads are already being addressed in a number of projects (not just PyPy, but Numba, Cython, and others). Data science may be at the heart of Python's strengths, but it's simply not accurate to suggest Python (whoever that is!) is resting on its laurels, as evidenced by the very active, albeit sprawling, package ecosystem. And while the CPython devs seem to have found their stride in 3.x releases. It's also inaccurate to suggest numeric computing in Python has a "low performance ceiling," considering you can get near-C performance via JIT or AOT using a package like Numba, and in most cases it's not even necessary because of Numpy and the many other highly optimized compute packages that can do most of the heavy lifting. I think the main draw of Python is not just that the syntax and language features are approachable, but that the package ecosystem is so broad and active, you are likely to make lighter work of the same job done in another language. I think to displace Python you would have to displace the package ecosystem, which seems as big and broad as it's ever been.
- visarga 6y agoNot just the package ecosystem, but you also have to have a large number of developers and jobs, so you can do hiring. Developing a project in a non-mainstream language means more difficult hiring.
- ak217 6y agoI agree with your overall characterization. As someone who works a lot with Python and is very familiar with its package ecosystem, I find the lack of leadership from PSF to be discouraging and upsetting. The past few Python releases have what I would characterize as cosmetic improvements while repeatedly missing opportunities to improve interpreter, packaging, and interface fundamentals. The position that CPython has to be kept simple as a reference implementation is untenable in the absence of an active collaboration or effort to produce a performance oriented implementation. As far as I'm concerned, CPython as an interpreter technology has not advanced in the past decade - and Python has grown thanks to the data science community efforts and excellent library ecosystem you mention, not due to PSF. PSF can only miss so many opportunities before something comprehensively better starts to eclipse it.
- woeirua 6y agoI agree that packaging and dependency resolution can be a total nightmare at times. Anaconda helps to an extent, but it's far from ideal especially compared to the default options for other languages. The main problem I have with Python is its maintainability, especially when you have multiple developers working in the same code base. I would never choose a dynamically typed language again to build something that is going to be more than several thousand lines of code, especially because of how it cripples your IDE which is essential for helping junior developers understand the existing codebase. Type hints help to an extent, but they're just a small bandaid over an oozing sore. Languages like Go are significantly better if you're going to build large codebases that need to survive for a long period of time.
- travisgriggs 6y agoI'd be curious which languages you feel are closing the gap, and which gaps are getting closer by them. For example, I do a bit of Swift in addition to Python, and sometimes I've heard people try to compare the two. But I vastly prefer Python to Swift when I can afford to.