7 ms·
https://blog.pyston.org/ https://blog.pyston.org/ You can look at the last release of Pyston for why companies are not funding this work. They're just moving t
by staticassertion 6y ago
https://blog.pyston.org/ https://blog.pyston.org/
You can look at the last release of Pyston for why companies are not funding this work. They're just moving to other languages.
Consider that Dropbox, Google, and Instragram likely spend much more than 1M on optimizing/ improving Python, just to have it still be a relatively slow language.
At some point it becomes way cheaper to just move to other languages that don't require that constant level of effort. Think about how much time and money is spent on things like:
* Adding types to Python
* Improving performance
when you could just move performance critical code to Go/ Rust as Dropbox has done, for a fraction of the cost, with far less maintenance burden.
"Make Python faster" is just a losing game imo. It is fundamentally never going to be as fast as other languages, it's far too dynamic (and that's a huge appeal of the language). Just look at the optimizations done here - moving `str` to local scope to avoid a global lookup? And can you even avoid that? Not without changing semantics - what if I want to change `str` globally?
Still, I was surprised by some of the sorta nutty wins that were achieved here. There's clearly some perf left on the table, and I'm not an expert on interpreters, it just seems really hard to build a semantically equivalent Python ("I can change global functions from a background thread") that can automatically optimize these things.
- disgruntledphd2 6y agoRewrites are the devil though, at least that's what Joel Spolsky taught me. Additionally, spending time at a really large tech company taught me that rewrites apparently never work (or at least it's better to build compilers for your shitty slow code rather than rewrite it in a better language). The linked post concerns Dropbox, who to be fair are an infrastructure company with low margins selling a commodity product. It doesn't surprise me that they would find such a move more important than say, Instagram who have a very large margin selling a differentiated product. I do really appreciate python types (and things like monkeypatch from IG for adding them) as they are an incremental improvement to a widely used language.
- staticassertion 6y agoRewrites have probably saved Dropbox 10's of millions of dollars. They just have to be done well. They've rewritten many critical components in the last few years (magic pocket is a big one, nucleus) to great success. Where rewrites are bad are when someone comes to a codebase, goes "wow this sucks I Can do this better", and starts from scratch. This is worth calling out because it's quite common, but it's very easy to avoid. I'm a fan of Python for what it's good for. I use it professionally, and built a product around it. I'd be happy to see Python perf wins, but I'm never going to expect it to compete with other languages.
- ngcc_hk 6y agoAgreed. Only optimize those part Ned optimize.
- gsnedders 6y ago> Rewrites are the devil though, at least that's what Joel Spolsky taught me. From scratch rewrites of anything large are, absolutely. But it's entirely possible to rewrite code component-by-component in many cases, and that's what most of the successful moves have done.
- strunz 6y agoWhere are you getting this information that they're moving to other languages? Isn't Instagram almost completely Python? I talk to them at PyCon every year and they spend a lot of time talking about it.
- staticassertion 6y agoThe post I linked talks about Dropbox moving to other languages. I didn't mean to imply that I knew Instagram was also moving away from it, sorry, I see how I misworded that.
- sandGorgon 6y agoEven assuming you're right - that all of these companies should move away from Python because it's a losing game - it's STILL going to be efficient to spend a million on a drop in replacement that gives a boost for free....while the moving away is happening
- staticassertion 6y agoAnd they do spend that money, much more than a million I'm sure.
- fastball 6y agoThe issue is that pypy is not a drop-in replacement, otherwise you'd be right. Many highly optimized libs are C extensions, which only run under CPython.
- jashmatthews 6y agoPyPy supports C extensions as does GraalPython.
- fastball 6y agoNot out of the box. For example, the fastest (and my favorite) JSON (de)serializer available for Python, orjson[1], does not support pypy. [1] https://github.com/ijl/orjson https://github.com/ijl/orjson
- catblast 6y ago> "Make Python faster" is just a losing game imo. It is fundamentally never going to be as fast as other languages, it's far too dynamic (and that's a huge appeal of the language). This cannot be overstated. Unfortunately python is especially perversely dynamic. After all Javascript and Ruby are highly dynamic, but for a number of reasons have avenues that allow more meaningful optimization gains without changing language semantics. (And although it is true that Google had tremendous resources to pour into V8, it's not like this point doesn't stand - luajit as an example). Python imho took the performance/productivity trade off way too far. You can get some very effective dynamism with some minimal restrictions that won't so badly shoot yourself in the foot for performance opportunities later. Frankly, for mainstream development, Kotlin or C# gives very pleasant, productive languages with strong ecosystems without paying such a penalty. Swift is good. Go isn't personally my cup of tea, but sure. Python sort of got a foothold in science.. but that's never been known for being a field of quality software engineering.
- Demiurge 6y agoWhat exactly is known for being a field of quality software engineering? 100% of Python users find it to be fast enough for their use case. Let's stop beating this horse into a black hole, it doesn't need to be fast at converting integers to unicode 100,000 times, to allow quality software engineering. To compare speed of CPython with quality of anything is such a narrow view, it is self-defeating.
- staticassertion 6y ago> 100% of Python users find it to be fast enough for their use case. This is absolutely false. Lots of people outgrow Python and use another language.
- Demiurge 6y agoThat's the intended humor, they stop being Python users if they outgrow it. The language and ecosystem is growing, regardless. I've been using it professionally since 2007, and some tasks I know it is not good at, and thats when I know to use another tool. The point is that it is a tool that has its uses, and I consider it the best tool for certain categories. It doesn't need to be the best tool for all categories. That seems pretty "absolutist".
- jashmatthews 6y agoI think only Dropbox actually put any significant resources into it? Google has donated a few tens of thousands of dollars to PyPy and didn't actually support Unladen Swallow it was just a side project by a few employees. Instagram did contribute the GC.freeze work but that's all I know of? Do they also donate? It's nothing like the effort these companies put into building V8 or HHVM. PyPy was all research funding and tiny compared to those investments.
- staticassertion 6y agoWell, Google has their own Python type system and employed Guido for years. I'm sure they've spent millions on improving Python or their usage of Python. I suspect Instragram has spent > 1M dollars of eng time on Python. Agree, nowhere near v8/js.
- jashmatthews 6y agoYeah, for sure! I meant focused work on improving Python performance.
- staticassertion 6y agoWhat I'm trying to say is that the companies spend money on the problem of Python performance. I doubt much of it is upstreaming improvements to Python, though I'm sure is (at least indirectly, funding Guido and other major Python developers), but most is probably just improving their own internal libraries, or, more likely, finding ways to FFI.
- gsnedders 6y agoGoogle don't invest heavily in Python performance because the vast majority of their Python isn't performance sensitive, AIUI.
- entha_saava 6y agoGoogle had a project called unladen swallow which was about having a JIT in CPython. Google has also created grumpy, python to Go translator.