8 ms·
This isn't an attitude of "no" - it's an attitude of "yes" to other things. The arguments are that making Python startup fast makes other things worse, and we c
by geofft 9y ago
This isn't an attitude of "no" - it's an attitude of "yes" to other things. The arguments are that making Python startup fast makes other things worse, and we care about those other things.
Here are some other things we can say "yes" to:
- Rewrite as much of Mercurial in Rust as possible, which will provide performance improvements well beyond what Python can possibly offer. https://www.mercurial-scm.org/wiki/OxidationPlan https://www.mercurial-scm.org/wiki/OxidationPlan
- Spend resources on developing PyPy, which (being a JIT) has relatively slow startup but much faster performance in general, for people who want fast performance.
- Write compilers from well-typed Python to native code.
- Keep CPython easy to hack on, so that more people with a "can do" spirit can successfully contribute to CPython instead of it being a mess of special cases in Guido's head.
Will you join me in saying "yes" to these things and not convincing ourselves to accept mediocrity?
- x404 9y agoI agree with you. As the limitation of developing resources, say "no" is difficult but important.
- MadWombat 9y agoI am slightly afraid to ask, but what is a "well typed python"?
- geofft 9y agoHave you seen MyPy, static type annotations / checking for Python? http://mypy-lang.org/ http://mypy-lang.org/ In context, what I'm really getting at "a sufficiently non-dynamic subset of Python that it can be compiled statically, but also a sufficiently large one that real Python programs can have a chance of being in the subset." PyPy has a thing called RPython that fits the former but not really the latter (I don't know of any non-PyPy-related codebases that work in RPython). In general, adding complete type annotations to a codebase is pretty correlated with making it static enough to do meta-level things on like compiling and optimizing it - for instance if you have a variable that changes types as the program runs, at least now you've enumerated its possible types. It's not the only way of doing so, but it seems to work well in practice and there seems to be a correlation between compiled vs. interpreted languages and static vs. dynamic typing.
- always_good 9y agoIt's funny when developers themselves think effort is so fungible. Like if you spent 1 hour on A, then you would've also made 1 hour of progress on B, C, or D, and that it would've been worthwhile. To the point of fallacy in your post. I would think developers have the experience to realize this isn't true but I see it all the time on these forums.
- geofft 9y agoI think I'm making the opposite claim - effort isn't fungible (and availability of effort isn't fungible). You can't necessarily spend 1 hour that would otherwise go into, say, rewriting Mercurial into a compiled language and instead spend it on making CPython faster and get the same results. One of these is more likely to work, and also the two problems are going to attract interest from different people. And one of the things that affects how productive one hour of work will be - and also whether random volunteers will even show up with one hour of work - is the likelihood of getting a change accepted and shipped to users. This is influenced by both the maintainers' fundamental openness to that sort of change, and any standards (influenced by the maintainers, who are in turn influenced by their users) about how careful a change must be to not make the project worse on other interesting standards of evaluation. It's also influenced by the number of people working on the project (network effects) because a more vibrant project is more likely to review your code promptly, finish a release, and get it into the hands of more users. So I'm claiming that it's better to spend time on rewriting Mercurial in Rust than to spend time on getting CPython startup faster, because the Mercurial folks are actively interested in such contributions and the CPython folks are actively uninterested, and because there are fewer external constraints in making Mercurial startup faster than in making CPython startup faster. And I'm saying that the more we encourage folks to help with rewriting Mercurial in Rust, the more likely additional folks are to show up and help with the same project, thereby making 1 hour of effort even more productive.
- quotemstr 9y agoI have to note that none of the projects you suggested, all of which are good and useful, will do anything to address cpython startup latency problem under discussion. Why shouldn't cypthon be better? There's also no reason to believe that startup improvements would make the interpreter incomprehensible; the unstated assumption that improvements in this area must hurt hackability is interesting. IME, optimizations frequently boost both simplicity and performance, usually by unifying disparate code paths and making logic orthogonal.
- taneq 9y ago> Why shouldn't cypthon be better? The point is that "better" is almost never a well-defined direction unless you only consider a single use-case. It's almost always a tradeoff, especially in a widely used project. A language is a point on a landscape of possible language variants, and "better" is a different direction on that landscape for every user.
- edanm 9y agoI think you misunderstood the point. These weren't things that would address the cpython startup problem - these were other priorities that can be worked on, instead of (or in addition to) the latency problems under discussion. Saying yes to fixing one thing usually means saying no to all the other things you can be doing with your time instead. Unless you're lucky and can "kill 2 birds with 1 stone".
- bb88 9y ago> - Write compilers from well-typed Python to native code. That is one thing I really want to happen, because I think it begins to open up python to the embedded space. Micropython is nice, but it still needs an interpreter embedded into it.
- yorwba 9y agoThere have been plenty of attempts to compile Python to faster code (usually by translating to C). Cython can use type annotations and type inference to unbox numbers for faster numerical code, but uses ordinary Python objects otherwise. http://cython.org http://cython.org ShedSkin translates a restricted (statically typeable) subset of Python to C++. https://shedskin.github.io https://shedskin.github.io RPython has a multi-stage approach where you can use full dynamic Python for setup, but starting from the specified entry point, it is statically typed. Since it was created for PyPy, it comes with support for writing JIT compilers. https://rpython.readthedocs.io https://rpython.readthedocs.io Pythran translates a subset of Python with additional type annotations to C++. http://pythran.readthedocs.io http://pythran.readthedocs.io In general, the major hurdle for all attempts to compile Python to native code is that Python code is dynamically typed by default.
- turbinerneiter 9y agoI'm so longing for a Python(like) compiler. MicroPython put together a Python in 250kb. Why the hell can't we make an LLVM frontend for Python that can use type hints for optimization? Sure, you lose some dynamic features as you optimize for speed, but that's the dream. Quickly write a prototype, not caring about types, optimize later with adding types and removing dynamicism. I'm currently learning Racket and LLVM and I have about 70 more years to live. I'm gonna try make Python fast on slow weekends 'til I die.
- crististm 9y agoYou might be interested in Matthew Might's course on compilers. In one of his courses he targeted a Python compiler written in his weapon of choice, Racket.
- quotemstr 9y agoUnless your Python compiler can use cpython modules without a massive performance penalty, it's going to see very limited adoption. The ecosystem matters.
- Rotareti 9y agoI don't think it would be hard to add type annotations to existing projects, though. Something similar happened in the JS world with the transition of JS projects to TypeScript and it wasn't a big deal, IIRC.
- sitkack 9y agoWe had a chance with ctypes and now CFFI to move away from the platform calcifying cpython module interface that is overly coupled to the cpython runtime. I am very disappointed in the lack of affordances that cpython gives to alternative pythons to support their work. The stdlib is a crufty mess that is overly coupled to cpython as well. The batteries are corroded and need to be swapped out for a modular pack.
- Thrymr 9y ago> I'm so longing for a Python(like) compiler. There is Cython, of course, which is a great python (+) compiler. I assume you mean a just-in-time compiler as opposed to a static compiler.
- endymi0n 9y agoThis hits the nail on the head. Optimization is very, very rarely completely „free“ - and usually a concious trade of some property for another trait that‘s deemed more important in a specific case. Simplicity for performance. Code size for compilation speed. Startup time for architectural complexity. UX for security. For a great product, you need to say „no“ much more often than not. Do one thing and do it well. Be Redis, not JBoss. I love how this article gets down to the essence of it: https://blog.intercom.com/product-strategy-means-saying-no/ https://blog.intercom.com/product-strategy-means-saying-no/
- neillyons 9y agoAgree. I recently found that trying to optimise code can make it a lot more complex.
- drb91 9y agoNone of these address, for instance, the issue raised about the firefox build invoking python many times. This seems both an accepted use case of cpython and an area where traditionally cpython has a huge edge on the JVM and PyPy. If scripts are not a priority, what is the expected use case of cpython? I would like to note, the cpython ties to the PyObject C abi seem to stymie rather than encourage “hacking”. Cpython seems to have traditionally valued stability over all else.... see the issues pypy has had chasing compatibility with c and retaining speed. So: normally i’m with you and a language should lean into its strengths, but i’ve always listed startup time as a primary strength of python!
- geofft 9y agoIn my experience, "script" is usually well-correlated with "a bit of inefficiency is okay." There's a reason that, say, many UNIX commands that could be implemented as scripts (true, false, yes) are actually implemented as binaries. There's a reason that most commercial UNIXes/clones (Solaris, macOS, Ubuntu, RHEL, etc.) switched from a script-based startup mechanism to a C-program-based one. I certainly write and continue to write Python scripts where even an extra half second won't matter. It's doing some manipulation of data where the cost of what it's doing is dominated by loading the data (e.g., grabbing it from some web service), and even if the script is small and quick, it's not so small and quick that I'll notice 50-100 ms being shaved off of it. Use cases where CPython continues to make sense to me are non-CGI web applications and things like Ansible, where load time isn't sensitive to milliseconds and runtime performance is pretty good. (Although if you believe the PyPy folks, perhaps everything that's PyPy-compatible should be running on PyPy.)
- deleted 9y ago[deleted]
- jVinc 9y ago> This isn't an attitude of "no" - it's an attitude of "yes" to other things. You are literally bringing an attitude of "no" to the question of whether you are being an attitude of "no" to the discussion....
- catach 9y agoIf those who complain about no-attitudes are insisting that the only acceptable response to anything is "yes", I doubt they'll get far.
- nhumrich 9y agocorrect. Every time you say yes, your saying no to something else. Its important to realize what your saying no to, before you say yes.
- dom96 9y ago> Rewrite as much of Mercurial in Rust as possible, which will provide performance improvements well beyond what Python can possibly offer. https://www.mercurial-scm.org/wiki/OxidationPlan https://www.mercurial-scm.org/wiki/OxidationPlan I read that article and I'm still wondering: why Rust?
- RantyDave 9y agoBecause he works for Mozilla.
- dingo_bat 9y agoThat is probably the only reason to use rust over go or cpp.
- catamorphismic 9y agoReally? The only reason?
- dingo_bat 9y agoOnly one I can see. Want speed and performance and safety? C++. Want easy multithreading? Go. Rust's only claim to fame is that Mozilla is dogfooding it.
- apta 9y ago> Want speed and performance and safety? C++. Speed and performance sure, but safety automatically rules out C++. > Want easy multithreading? Go Want speed, performance, safety, and easy multithreading? Rust.
- dingo_bat 9y agoC++ still does multi-threading better than rust, just that go does it better. Similarly go does perf better than rust, just that cpp is even better. So yeah, if you want the worst of all worlds coupled with the pains associated with a brand new language (try compiling rust for a armv5 soc) rust all the way!
- patrec 9y ago> PyPy, which (being a JIT) has relatively slow startup > time pypy -c 'print "Hello World"' Hello World pypy -c 'print "Hello World"' 0.08s user 0.04s system 96% cpu 0.120 total > time luajit -e 'io.write("Hello World!\n")' Hello World! luajit -e 'io.write("Hello World!\n")' 0.00s user 0.00s system 0% cpu 0.002 total
- JackCh 9y agoSometimes I wonder why we're not all using Lua instead of Python. Lua seems to get a strange amount of hate in some circles, but I've found both Lua and Python to be reasonably pleasant languages to work with.
- zeugmasyllepsis 9y agoI had a similar feeling a while back, but when I actually picked up Lua for a project I was shocked by how limited the standard library is. Third party libraries aside (which Python clearly has in spades), even just the standard library is pretty sparse. It makes sense, since Lua is at least partly motivated by being lean and embeddable, but it puts Lua in a completely different space for me.
- make3 9y agono batteries
- civility 9y agoOther folks complain about the lack of libraries. Personally, the two things which turn me off on Lua (luajit) are 1-based arrays and the conflation of hash-tables and array-lists into a single thing.
- enriquto 9y ago> the conflation of hash-tables and array-lists into a single thing. Precisely, one of the things that turns me off on Python (coming from Lua), is the unnatural proliferation of different container types.