9 ms·
Why Python Is Slow: Looking Under the Hood (2014)
- cm2187 10y agoWhat's so great with dynamic languages that justifies the performance hit? Most statically typed languages have type inferrence, which reduce the syntax to something very close to a dynamic language. And is it such a good idea to only catch mistyping at runtime, and require to write twice as much code with testing to compensate for the lack of typing?
- aldanor 10y agoEasy of prototyping would be one thing
- cm2187 10y agoIf the syntax is about the same, how is that easier?
- sourceless 10y agoThe syntax doesn't make too much difference. With a dynamic or duck type system, though, you can avoid a lot of writing out interfaces or making class hierarchies. You end up with something less structurally uniform, but with a much shorter time to get results. If you care about type mismatches, use a statically typed language. If you just want to get something working, use a dynamically typed one.
- danudey 10y agoWell, there are a few responses to that. First, there aren't really any compiled languages where the syntax is similar in ease of use to Python (or any other popular dynamic language I'm familiar with). All the compiled languages with which I'm familiar require a significant amount more work to accomplish the same thing; C apps are full of boilerplate to detect error cases, allocate memory, check buffer lengths, etc.; dynamic languages handle this for you so the total amount of code tends to be lower. Writing a simple program to test an algorithm or proof of concept is generally significantly less code, the code/test/debug loop tends to be significantly shorter, and you can often get to a solution significantly quicker while avoiding the typical bugs that come with lower-level languages. Python also has a pretty substantial standard library which is easy to learn and use and very consistent, meaning much less copypasta code or external dependencies. On top of that, a great deal of use-cases for languages like Python aren't CPU-bound, but rather I/O bound (disk or network). In cases where your app is CPU-dependent, it can often be trivial to replace that inner loop with optimized native code written in C or other languages, while still keeping the rest of your application easy to read and maintain. Lastly, for complex algorithmic work, there's a lot of packages which implement standard data structures and algorithms in native code and provide usable, idiomatic interfaces in Python, which means that for a lot of things you can work in native code with the ease of Python, at a minuscule performance hit but a dramatic increase in programming efficiency.
- cm2187 10y agoThe thing is I can understand if we are comparing C and Python, but if you compare C# or modern VB.net to Python, I am not so sure the syntax is much heavier. Now if you are not doing something that requires performance, I think there is still a good case for static typing: advanced tooling. Dynamic language limit a lot the ability of the IDE to help you finding bug, making the language and the API self discoverable, auto completing the code, providing advanced analysis (who calls what) or refactoring (rename a property). Some people do not need that, but to those who rely on that, dynamic languages must really be easier to write by an order of magnitude to end up being faster to prototype. If I was suggesting a language to a beginner, I would push for a language with a strong IDE.
- jerf 10y ago"Most statically typed languages have type inferrence, which reduce the syntax to something very close to a dynamic language." To understand the appeal of dynamic languages, it's critical to consider the era they came from. Your statement is not true of the general programming landscape ten years ago. It's not really true of the general programming landscape today, for that matter, which is still by sheer mass C++ and Java. But 10 years ago, there weren't hardly any choices of statically-typed languages that weren't a bit of a bear to use. 1990s C++ is a monstrous nightmare. A lot of this monstrous nightmare was arguably the programming environment (Windows, Xwindows, UNIX, everything was a bear to program in back then), but when you took battle-scarred C++ Windows programmers and showed them print "Hello world!" they weren't in a mood to analyze exactly which differences between Python/Perl and C++ were essential and which were accidental; they were too busy drinking in the joy. I think there's a general trend towards languages that either make the static typing easy enough to use that you don't feel like you're losing out on much, or make the static typing pay for itself better (Rust), and I think that will continue to be the story for the next ten years on the developing language front. I think it may be the case that if they weren't already entrenched, what I call the 1990s-style dynamic languages (because it's not just that they're "dynamic", Perl, Python, Javascript, and Ruby are almost the same language, compare with the "dynamic" Erlang to see a major difference) wouldn't succeed if someone wrote one for the first time this year. In fact they'd probably take a beating for the performance issues it would have. But they are entrenched and have over a decade of refined libraries, support documentation, tutorials, and of course, the big hitter, existing code. So they aren't going anywhere any time soon.
- cm2187 10y agoFrom that point of view I agree. The other thing is tooling. There are lots of hardcore developers to whom even syntax highlighting is too much distraction, let alone error checking, auto completion, etc. Well... To say the least I am not one of them. In fact to me, a large part of the simplicity of a language resides in the interaction with the IDE, the feedback it gives you on what's legal and what is not, where you may have a bug or an ambiguity, what can I do from there, etc. Dynamic languages defeat advanced and proactive tooling. And to me it makes them actually quite hard to work with, not self-discoverable, and all-in slower to develop. The time not spent writing a class or a return type ends up being spent on the reference page of the library (after you checked this is the correct version) checking what are the arguments of a function, etc.
- Scarbutt 10y agoWhat statically typed language with great type inference has the library ecosystem of python or javascript? I can't think of one, libraries matter a lot. Swift looks promising in this regard.
- cm2187 10y agoAny .net language (C#, VB.net, F#). In term of libraries they will be hard to beat. What they will not have is cross-platform though. Microsoft just kicked off a cross platform version but surely it will take at least 5 years before there is a large cross platform ecosystem.
- module0000 10y agoThanks for taking the time to publish this, I've always been interested in tearing at CPython, but this pushed me over the edge. Looking forward to new and exciting ways to crash the runtime :)
- foota 10y agoThere was someone a while ago in a group I was in making the claim that python was no inherently slower than other languages that were more static like Java. I might have to point them at this now. Great read! Reminds me of: http://blog.kevmod.com/2016/07/why-is-python-slow/ http://blog.kevmod.com/2016/07/why-is-python-slow/ which was discussed a few days ago at https://news.ycombinator.com/item?id=12025309 https://news.ycombinator.com/item?id=12025309
- qznc 10y agoI remember another article, but cannot find it. It had very different arguments what is slow about Python and would have disagreed with those two articles. Python is slow because it is dynamically typed rather than statically typed? No. In most cases, type inference is good enough. See Javascript. Python is slow because it is interpreted rather than compiled? That is not really a limit in the long run. Insert a JIT compiler. Java was interpreted as well initially. Python's object model can lead to inefficient memory access? No we are getting somewhere. Scripting languages in general tend to overuse hashmaps. That gives you a lot of flexibility and monkey patching. It is also a lot harder to optimize than dynamic dispatch. Another problem of scripting languages is the inability to control your memory. Python forces you to make a lot of memory allocations and there is little control over the garbage collector. Even Java suffers from this, but the JVM engineers invested a lot to fix it. There are no value record types (struct) in Java.
- telotortium 10y agoWhy Python, Ruby, and Javascript Are Slow: https://news.ycombinator.com/item?id=5304873 https://news.ycombinator.com/item?id=5304873
- qznc 10y agoThat might be it. I remember it as an article. Maybe http://lukauskas.co.uk/articles/2014/02/13/why-your-python-runs-slow-part-1-data-structures/ http://lukauskas.co.uk/articles/2014/02/13/why-your-python-r...
- 10y ago
- jstimpfle 10y agoIs there a reason why Python must be slower than Javascript? I.e. given economic and cultural incentives, couldn't there be something like V8, for python?
- chrisseaton 10y agoNo, I don't know of anything in Python that could not be made as efficient as Javascript, using techniques that we already know about (and given the time and money that you've already acknowledged are also barriers). I'm not considering Python's C extensions there. I don't know if you'd consider that part of the 'language' or 'ecosystem'. Pyston is trying to make Python fast, including the C extensions, which is not as easy.
- the_mitsuhiko 10y agoJavaScript has fewer problematic features than Python which makes it possible to write efficient implementations. Also it has a standard which Python does not.
- fovc 10y agoWhat are the problematic features in Python that are not in JS? I'm not as familiar with JS
- optimuspaul 10y agoWhat do you mean Python doesn't have a standard? There is a very mature standard and process for managing that standard.
- chrisseaton 10y agoThere isn't a document describing the semantics of Python is there? That's what people mean by a 'standard'. Something at least trying to be somewhat semi-formal. If you mean that the implementation of CPython is a standard, well then I suppose that's arguable, but it's really not what anyone means.
- WestCoastJustin 10y agoNot because I want to limit discussions here, but this has been posted before on HN and there are many great comments in these threads. So, this comment is more of a useful suggestion, that if you are interested, check these comments out too (both 2+ years old). https://news.ycombinator.com/item?id=8625280 https://news.ycombinator.com/item?id=8625280 https://news.ycombinator.com/item?id=7721096 https://news.ycombinator.com/item?id=7721096
- deleted 10y ago[deleted]
- blaze33 10y agoAlso, 4 pays ago there was a "why is python slow" post with many interesting comments: https://news.ycombinator.com/item?id=12025309 https://news.ycombinator.com/item?id=12025309
- 0xmohit 10y agoAnd https://news.ycombinator.com/item?id=12025529 https://news.ycombinator.com/item?id=12025529 therein specified these: The following may also be of interest: - Why Python is Slow: Looking Under the Hood [0] - Fast Python, Slow Python by Alex Gaynor [1] [0] https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow.. https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow.... [1] https://www.youtube.com/watch?v=7eeEf_rAJds https://www.youtube.com/watch?v=7eeEf_rAJds
- amyjess 10y agoI have to wonder how much this is CPython specific and how much applies to all implementations of Python (including PyPy, which is noticeably faster than CPython).
- sevensor 10y agoI've found that PyPy is very fast in most cases. While I've seen hand-optimized C code beat PyPy by a mile, I've seen more cases where the gains were in the 10-20% range. And PyPy keeps getting faster.
- deleted 10y ago[deleted]
- Symmetry 10y agoGenerally good but it makes it sound like what it calls the dynamic typing penalty is a matter of more instructions. If that was true Python would only be a factor of 2 slower or something like that. Instead the problem is that the cost of getting the value of an integer is first of all that after reading the type of the variable Python has to execute a fairly high entropy branch which will probably cost the branch delay penalty of 20 cycles or so. And then the extra memory lookup will cause a slowdown as well.
- amenod 10y agoIt is easy to miss the point with such articles. OP touched upon it in chapter "So Why Use Python?". Raw speed, while sexy, is not a deciding factor in most cases - not everybody is working on HFT. And when it is important to be really fast, you can always write an extension in C (or Rust I imagine?) just for that single tight loop and get raw speed without sacrificing maintainability. Seriously, most bottlenecks (in other people's software of course ;) I came across were either due to bad architecture or unreadable code, which lead to not noticing that too much work was being done. So yeah, Python might be slower than alternatives (though I have a hard time believing that it is slower than Java), but it is fast enough.
- cwyers 10y agoOh, the crippling sense of irony that comes from reading this comment after reading the front page post about Azul and the JVM, with a lot of technical details about how HFT firms use Java.
- amelius 10y agoThis might certainly be true for single threaded programs. For multithreaded programs, Python has a bottleneck known as the "Global Interpreter Lock" (or GIL), which is not so easily optimized away, unfortunately.
- amenod 10y agoTrue, though this is not the kind of speed this article was talking about. You can get around that limitation somewhat by using multiprocessing module. I'm not saying Python is the best language of all, but dissecting Python bytecode is kind of missing the point. Python's strong point is its ease of use.
- d0mine 10y agoGIL may and does get released in the interpreter (on I/O) and C extensions such as regex, lxml, numpy. Performance sensitive extensions may even use multiple CPU cores implicitly e.g., as numpy.dot() can do for large enough input without changing the code in any way.
- BuckRogers 10y agoPython is slow because the core dev team has long concerned with Python3. Priority is not on speed. CPython actually got slower with 3, and it still is. They say it's "a wash" but I haven't seen that in anything I've done. Since Microsoft got its claws into Python through the core dev team, we've seen a shift from the Linux (POSIX) text model in Python2 to the Windows text model in Python3. It now matches how the .Net CLR handles unicode. Which is a shame since Linux is kind of increasingly popular. There are some recent developments where Microsoft is investing through the core dev surrogates into Python3 and speed with Pyjion. That is promising but it's a framework where it requires CPython3 to use a JIT, which strikes me as a way to enforce control- a response to PyPy and alternatives taking the mantle when they abdicated the throne with Python3. But it remains to be seen how it works out, and PyPy already is likely to remain the leader on faster-Python.
- bshanks 10y agoThank you 0xmohit for this link (via https://news.ycombinator.com/item?id=12025529 https://news.ycombinator.com/item?id=12025529 )