14 ms·

Why Python is Slow: Looking Under the Hood

208 points by underyx 12y ago

PythonicAlpha 12y ago
Yes, Python is slower in execution than some other languages, but: ... efficient use of development time ... That is the reason, why Python counts (not only references). Python has many very good libraries, is a good OOP language, easy to learn, but still very, very powerful. You can express in Python some things in a single line, where you need hundreds of lines in C++ or other languages. The few percent of running speed that you might loose, are neglect-able in most cases against the win in development speed. In many applications you don't need the full CPU power, but often times you are hindered by e.g. the disk speed or other factors ... and than you don't lose anything when you are a little bit slower in some minor tasks.
- zo1 12y ago
  And if you really need to have a piece of your code execute really quickly, you can just make C extension that python can easily invoke.
  - zmmmmm 12y ago
    Python's support for this is awesome, but there is a gap: you lost cross platform. You now have to figure out how to make it compile on every platform that anybody could every possibly want to run it on, including ones not yet invented in the future ... Jython / Java (or other JVM languages) are quite a nice option in that space.
    - PythonicAlpha 12y ago
      Of course you loose a little platform independence, but that is something you don't have anyway when you program in C++ or that kind. Also you get rather rich support for makefiles that are cross platform. I had no problems till now with C code I wrote that integrates into the Python environment. When you don't need OS specifics and have knowledge how to implement things CPU independent (given a common >=32bit architecture) in C, you will have little to no trouble at all. Also one little hint: Use Cython for enhancements. Cython is great for implementing small enhancements when you don't want to hassle with refcounts and the normal Python API. Also your enhancements are automatically compatible to Python 2 + Python 3. You can also call C functions from Cython very easily for full speed. Cython is really great, when you don't need to optimize to the last nanosecond.
- nly 12y ago
  > You can express in Python some things in a single line, where you need hundreds of lines in C++ or other languages. citation needed.
  - PythonicAlpha 12y ago
    I have none, it is my personal experience, since I programmed in C++ (and other languages) before. So cite me, if necessary.
  - Borogove 12y ago
    Example: x = {'foo':34,'bar':[1,2,3],'baz':'quux'} The headers alone for STL maps are hundreds of LOC.
    - GFK_of_xmaspast 12y ago
      I agree with the upstream comment about expressiveness, but this isn't a fair example.
    - zmmmmm 12y ago
      Counting headers is rather unfair. C++11: map<int, char> x = {{1, 'a'}, {3, 'b'}, {5, 'c'}, {7, 'd'}}; Any not-ancient C++ using boost Assign: map<int, char> x = map_list_of (1, 'a') (3, 'b') (5, 'c') (7, 'd'); Shamelessly stolen from here: http://stackoverflow.com/questions/138600/initializing-a-static-stdmapint-int-in-c http://stackoverflow.com/questions/138600/initializing-a-sta...
    - nly 12y ago
      That's tricky, because C++ is statically typed and you've selected a varying value type, but here's the closest thing in real terms. #include <map> #include <vector> #include <string> #include <boost/any.hpp> using boost::any; using std::map; using std::string; using std::vector; int main() { map<string,any> x = {{"foo", 34}, {"bar", vector<int>{1,2,3}}, {"bar", "quux"}}; } Syntax overhead amortizes to constant factor ;)
- deleted 12y ago
  [deleted]
ape4 12y ago
This is hardly breakthru stuff.
- phorese 12y ago
  True. It also never claims to be. The very first part says that this is only a writeup for people who are not intimately familiar with why "dynamically typed" might slow down Python.
  - falcolas 12y ago
    Based off my own experience, dynamic typing has much less to do with slowness than the plethora of string copying which occurs in your typical python script.
    - blt 12y ago
      but for scientific computing with numbers - the author's target audience - dynamic typing is the main bottleneck.
      - freyrs3 12y ago
        Boxing and memory locality are the the main bottlenecks in scientific computations with Python, dynamic typing is orthogonal.
        andreasvc 12y ago
        If a language is dynamically typed, it is very likely that it will use boxing, so they do not seem orthogonal to me.
    - dragonwriter 12y ago
      There's ways to avoid that, where it matters, though its a pretty common thing in naive Python/Ruby, true.
rpearl 12y ago
> binary_add<int, int>(a, b) no... there is simply not function call overhead for adding two integers in C.
- zhemao 12y ago
  It's just notation, to make it clear that two integers (and not some other type of object) are being added.
  - rpearl 12y ago
    It's just not good notation at all, though. It strongly implies "call". And "object". This is misleading.
- Rusky 12y ago
  It's inlined ;)
kghose 12y ago
The 'slow' part wasn't so new to me, but the `id` command and the attendant understanding that smply typing 110 in the interpreter creates an OBJECT and when you assign a=110 `a` points at that object (and when you reassign a=30, a new object is created and a points at that), blew my mind. Thanks for this! Previously I had thought that doing a=110 creates an object 'a' that stores the value 110 (and when we do b=a, b simply points to a). I had no idea there is a third object in play.
- taejo 12y ago
  `a=110; b=a` involves only one object: 110. a and b are just names.
  - dekhn 12y ago
    And 110 is probably an interned int, so a and b are likely referencing an object that was compiled into the interpreter.
    - icebraining 12y ago
      Yeap: >>> a = 110 >>> b = 110 >>> id(a) == id(b) True
      - deleted 12y ago
        [deleted]
        icebraining 12y ago
        Uh, I didn't use assignment between variables; my code is the same as your second example (both variables independently assigned to 110).
        taejo 12y ago
        Indeed; I was confused by the dekhn's aside.
Fede_V 12y ago
Jake's blog is pretty great. I really suggest reading more stuff in there if you care about scientific computing.
williamstein 12y ago
This is convenient, since it's almost exactly what I plan to talk about in the undergrad class I'm teaching today (https://github.com/williamstein/sage2014 https://github.com/williamstein/sage2014). I view this sort of article as good motivation for Cython, and the value of this article is merely that it aims at typical undergraduates not in computer science.
pnathan 12y ago
One thing that bothers me - and has for a long time - is why Python (Perl, Ruby, etc), never have really leveraged the work Common Lisp systems have done (CMUCL, SBCL, etc), which provide very good performance without sacrificing dynamic typing or the REPL.
- agentultra 12y ago
  It bothers me too. Dynamically typed does not imply no types. A good CL implementation takes that to heart and produces really nice code out of the box. And the language spec provides for an API to allow program developers to specify types and hints to the compiler to optimize certain functions and code paths where it's needed. The benefit from this is that you can rapidly prototype your application and harden it to the platform you deploy on when performance becomes an issue... and with built-in conditional compilation you can do that in a cross-platform way.
  - mikeklaas 12y ago
    Python has types; I would even describe it as strongly-typed. It is thus relatively JIT-able (as pypy and psyco have shown). The chief problem is the high level of dynamic typing going, plus the huge amount of globally-changable assumptions (you can add methods to any instance of a class, you can globally change the definition of language built-ins, etc.)
    - agentultra 12y ago
      Some charity, please; I know Python is strongly-typed. It just doesn't take advantage of it and doesn't have facilities for producing fast code. Common Lisp not only provides type hinting facilities in the language it also provides hinting to the compiler to turn off all run-time type checks in critical code sections.
- WaxProlix 12y ago
  What work specifically are you referencing?
codegeek 12y ago
Good read. Whenever I read stuff like this, I always wonder if it is always true that dynamically typed languages are slower than statically typed languages ? Also, do we have to take this for granted that more higher level the language is, the slower it will be ? Or are there exceptions ? Also, it is worth asking if for majority of use cases of python for data/analysis, the ease and flexibility outweights the slowness.
- YZF 12y ago
  Always is a bit of a strong word but yes, it's always true. Virtual functions in C++ which allow some form of dynamic behaviour are slower than static function calls because they inherently involve another level of indirection. Static calls are known at compile time, they can be inlined by the compiler, they can be optimized in the context in which they're called. Now, nothing prevents the C++ run-time from trying to do the same thing at run-time but you can relatively easily see that it'll have to make some other compromised to do so. Nothing prevents a C++ program from generating C++ code at run-time, compiling it, and loading it into the current process as a .so. Now that's a pretty dynamic behaviour but there's again an obvious price. You can also write self modifying code. At any rate, static languages are capable of the same dynamic behaviour that dynamic languages are capable of but you often have to implement that behaviour yourself (or embed an interpreter...). Fundamentally, a dynamic language can't make the kinds of assumptions a more static language can make, it can try and determine things at run-time (ala JIT) but those take time and still have to adapt to the dynamics of the language. The same line of code "a = b + c" in Python can mean something completely different every time it's executed so the run-time has to figure out what the types are an invoke the right code. Now the real problem is that if you take advantage of that then no one can actually tell what this code is doing and it is utterly unmaintainable. To compound the problems facing dynamic languages is the fact that CPUs are optimized for executing "predictable" code. When your language is dynamic there are more dependencies in the instruction sequence and things like branch prediction may become more difficult. It also doesn't help that some of the dynamic languages we're discussing have poor locality in memory (that's an orthogonal issue though, you could give a dynamic language much better control over memory). EDIT: One would think it should be possible to design a language which has both dynamic and static features where if you restrict yourself to the static portion runs just as fast as any other statically compiled language and still allows to switch to more dynamic concepts and pay the price when you do that.
gregwebs 12y ago
Dynamic typing makes Python easier to use than C I disagree with the implications. The main reasons Python is easier to use is independent of type system. Not having to manually manage memory, for example! Overall it is a higher-level language, whereas C is designed for maximum performance. Dynamic typing is probably preferable to a 40 year old type system. But Python could be easier to use (catch bugs ahead of time!) and execute faster by adding a modern type system. Optional typing (like TypeScript or Facebook's new PHP implemntation) would probably be appropriate.
- Spittie 12y ago
  I'd love to see optional typing in Python, I wonder if there is any official reasons about why it never got introduced. I very rarely change the type of a variable, so it would be essentially free speed and free speeds for me. I actually wonder, do people use dynamic typing so often? I mean, it's nice to do "variable = int(variable)" when I know that I'm getting an integer in a string, but that's probably the only use case can I can think off that doesn't just reuse variables for something else.
  - collyw 12y ago
    I started programming in Java, and moved to Perl and now Python. At first it seemed a bit weird, dynamic typing, but now I embrace it and enjoy the flexibility it offers. I see that many people prefer static typing and I am not sure why. Can someone give me a concrete example of when static typing would be beneficial? (The majority of my work revolves around databases, so I guess I rely on that as a kind of type checking to a degree).
    - antocv 12y ago
      Static typing provides more information for the programmer and for the runtime. More information makes the code somewhat readable and understandable 10 years after when it was written. Dynamic typing often becomes a much more messy affair. Static typing is beneficial when you want to catch a certain type of errors before your code enters production, it makes you write less error prone code.
mrfusion 12y ago
While everyone is on the subject, I thought I'd mention some weird behavior I saw today. I just switched my program from using a defaultdict to a regular dict. I.e., from defaultdict(lambda:'NA') to regular dict using get(val, 'NA') for access. And I got a something like a 100x speedup. It runs in two minutes instead of two hours. I had no idea a defaultdict would be so much slower. Unless there's something funny going on in my program and it's unusual behavior.
- zo1 12y ago
  The docs for the defaultdict are here: https://docs.python.org/2/library/collections.html#collections.defaultdict https://docs.python.org/2/library/collections.html#collectio... And they pretty much say that there is only one method that's been overridden from the normal dict class. And all it does is it executes the anonymous method (lamda) you supplied instead of returning the default value you pass in. Also, this depends on your usage, and what you have in the lambda expression. I'm curious about your code, care to paste/link some of it for us?
  - cefstat 12y ago
    I would guess that the code tries to get the value for keys that do not exist in the defaultdict. Well, if the key does not exist then it is being added with the default value 'NA'. So in this way you can end up with a huge dictionary where most of the values are just 'NA'.
    - mrfusion 12y ago
      Oh, wait, so defaultdicts actually insert missed keys?? I thought they simply gave you a default value instead of a keyerror. If you're right, then that would certainly explain it.
      - bryanh 12y ago
        Yep, take a look! >>> import collections >>> dd = collections.defaultdict(int) >>> print dd['doesnt exist'] 0 >>> print dd defaultdict(<type 'int'>, {'doesnt exist': 0})
rch 12y ago
> Python ends up being an extremely efficient language for the overall task of doing science with code. This is pretty close to how I characterize my time, 'doing X with code', and Python yields great returns in these terms.
VeejayRampay 12y ago
Python (and Ruby for that matter) are slow because there are no billion-dollar companies stuck with an initial choice of language that impedes their ability to grow. PHP and Javascript used to be extremely slow and now after several dozens of millions thrown at JIT, rewrites, forks and redesigns they're starting to get much much faster.
- hyperbovine 12y ago
  Not so. Google uses Python extensively in-house (it's one of the four "blessed" languages) and, in fact, employed GVR until he was recently hired away by Dropbox--another billion-dollar company which relies heavily on Python. At one point Google even spearheaded an ambitious effort (https://code.google.com/p/unladen-swallow/ https://code.google.com/p/unladen-swallow/) to make CPython much faster. It failed. At the point where even Google can't make it happen, it really starts to look like Python performance is limited at some very fundamental level to what we see today. Personally I think this is fine. I use Python for everything day-to-day and offloading the intensive stuff to a C extension (a la Numpy) works just great. There are very few instances where I find myself wishing Python was faster.
  - ihnorton 12y ago
    Unladen Swallow started as an intership project and didn't get very much support at Google [1]. When US started (~2009), Google had been writing their core, high-performance stuff in Java and C++ for many years already so there was no internal or existential need to make Python faster. [1] http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospective.html http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospect...
    - ZenoArrow 12y ago
      On the other hand, there may have been applications higher up the stack that would've benefited from a more efficient Python. I don't know if it still is but I heard GMail was written in Python, you'd think an application of that scale would benefit from better performance.
amit_m 12y ago
The bigger picture is that CPython's core team simply does not care too much about performance. Performance has never been a fundamental requirement, but merely an afterthought. The biggest reminder of this was Python 3, which for me was a complete disappointment. They could have limited python's super-dynamic behavior (e.g. changing builtin functions, patching classes on the fly, etc.) or made them optional. They could've added optional typing annotations a la Cython. Or even changing the builtins and language syntax to allow more inplace operations and preallocations, so that temporary results wouldn't have to be allocated on the heap over and over again. All of these changes would have made python faster and more JIT-able. None of these things happened. Performance-wise, python 3 is no step forward. Python+Cython is still a powerful combination, but eventually Julia or similar languages will eat python's lunch with respect to scientific computing.
- GFK_of_xmaspast 12y ago
  I think if python3 had made major performance improvements (not to mention the GIL) there wouldn't be nearly as many 2.7 holdouts.
  - ris 12y ago
    And it still wouldn't be done.
- est 12y ago
  Yup. the HUGE "CPython is only reference implementation" ego
- walshemj 12y ago
  Not for pure performance Fortran still seems to be king for that type of scientific computing.
dmoney 12y ago
Related: "Python is only slow if you use it wrong" http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf [pdf]

208 points by underyx 12y ago

Yes, Python is slower in execution than some other languages, but: ... efficient use of development time ... That is the reason, why Python counts (not only references). Python has many very good libraries, is a good OOP language, easy to learn, but still very, very powerful. You can express in Python some things in a single line, where you need hundreds of lines in C++ or other languages. The few percent of running speed that you might loose, are neglect-able in most cases against the win in development speed. In many applications you don't need the full CPU power, but often times you are hindered by e.g. the disk speed or other factors ... and than you don't lose anything when you are a little bit slower in some minor tasks.

zo1 12y ago
And if you really need to have a piece of your code execute really quickly, you can just make C extension that python can easily invoke.
- zmmmmm 12y ago
  Python's support for this is awesome, but there is a gap: you lost cross platform. You now have to figure out how to make it compile on every platform that anybody could every possibly want to run it on, including ones not yet invented in the future ... Jython / Java (or other JVM languages) are quite a nice option in that space.
  - PythonicAlpha 12y ago
    Of course you loose a little platform independence, but that is something you don't have anyway when you program in C++ or that kind. Also you get rather rich support for makefiles that are cross platform. I had no problems till now with C code I wrote that integrates into the Python environment. When you don't need OS specifics and have knowledge how to implement things CPU independent (given a common >=32bit architecture) in C, you will have little to no trouble at all. Also one little hint: Use Cython for enhancements. Cython is great for implementing small enhancements when you don't want to hassle with refcounts and the normal Python API. Also your enhancements are automatically compatible to Python 2 + Python 3. You can also call C functions from Cython very easily for full speed. Cython is really great, when you don't need to optimize to the last nanosecond.
nly 12y ago
> You can express in Python some things in a single line, where you need hundreds of lines in C++ or other languages. citation needed.
- PythonicAlpha 12y ago
  I have none, it is my personal experience, since I programmed in C++ (and other languages) before. So cite me, if necessary.
- Borogove 12y ago
  Example: x = {'foo':34,'bar':[1,2,3],'baz':'quux'} The headers alone for STL maps are hundreds of LOC.
  - GFK_of_xmaspast 12y ago
    I agree with the upstream comment about expressiveness, but this isn't a fair example.
  - zmmmmm 12y ago
    Counting headers is rather unfair. C++11: map<int, char> x = {{1, 'a'}, {3, 'b'}, {5, 'c'}, {7, 'd'}}; Any not-ancient C++ using boost Assign: map<int, char> x = map_list_of (1, 'a') (3, 'b') (5, 'c') (7, 'd'); Shamelessly stolen from here: http://stackoverflow.com/questions/138600/initializing-a-static-stdmapint-int-in-c http://stackoverflow.com/questions/138600/initializing-a-sta...
  - nly 12y ago
    That's tricky, because C++ is statically typed and you've selected a varying value type, but here's the closest thing in real terms. #include <map> #include <vector> #include <string> #include <boost/any.hpp> using boost::any; using std::map; using std::string; using std::vector; int main() { map<string,any> x = {{"foo", 34}, {"bar", vector<int>{1,2,3}}, {"bar", "quux"}}; } Syntax overhead amortizes to constant factor ;)
deleted 12y ago
[deleted]

ape4 12y ago

This is hardly breakthru stuff.

phorese 12y ago
True. It also never claims to be. The very first part says that this is only a writeup for people who are not intimately familiar with why "dynamically typed" might slow down Python.
- falcolas 12y ago
  Based off my own experience, dynamic typing has much less to do with slowness than the plethora of string copying which occurs in your typical python script.
  - blt 12y ago
    but for scientific computing with numbers - the author's target audience - dynamic typing is the main bottleneck.
    - freyrs3 12y ago
      Boxing and memory locality are the the main bottlenecks in scientific computations with Python, dynamic typing is orthogonal.
      - andreasvc 12y ago
        If a language is dynamically typed, it is very likely that it will use boxing, so they do not seem orthogonal to me.
  - dragonwriter 12y ago
    There's ways to avoid that, where it matters, though its a pretty common thing in naive Python/Ruby, true.

rpearl 12y ago

> binary_add<int, int>(a, b) no... there is simply not function call overhead for adding two integers in C.

zhemao 12y ago
It's just notation, to make it clear that two integers (and not some other type of object) are being added.
- rpearl 12y ago
  It's just not good notation at all, though. It strongly implies "call". And "object". This is misleading.
Rusky 12y ago
It's inlined ;)

kghose 12y ago

The 'slow' part wasn't so new to me, but the `id` command and the attendant understanding that smply typing 110 in the interpreter creates an OBJECT and when you assign a=110 `a` points at that object (and when you reassign a=30, a new object is created and a points at that), blew my mind. Thanks for this! Previously I had thought that doing a=110 creates an object 'a' that stores the value 110 (and when we do b=a, b simply points to a). I had no idea there is a third object in play.

taejo 12y ago
`a=110; b=a` involves only one object: 110. a and b are just names.
- dekhn 12y ago
  And 110 is probably an interned int, so a and b are likely referencing an object that was compiled into the interpreter.
  - icebraining 12y ago
    Yeap: >>> a = 110 >>> b = 110 >>> id(a) == id(b) True
    - deleted 12y ago
      [deleted]
      - icebraining 12y ago
        Uh, I didn't use assignment between variables; my code is the same as your second example (both variables independently assigned to 110).
        taejo 12y ago
        Indeed; I was confused by the dekhn's aside.

Fede_V 12y ago

Jake's blog is pretty great. I really suggest reading more stuff in there if you care about scientific computing.

williamstein 12y ago

This is convenient, since it's almost exactly what I plan to talk about in the undergrad class I'm teaching today (https://github.com/williamstein/sage2014 https://github.com/williamstein/sage2014). I view this sort of article as good motivation for Cython, and the value of this article is merely that it aims at typical undergraduates not in computer science.

pnathan 12y ago

One thing that bothers me - and has for a long time - is why Python (Perl, Ruby, etc), never have really leveraged the work Common Lisp systems have done (CMUCL, SBCL, etc), which provide very good performance without sacrificing dynamic typing or the REPL.

agentultra 12y ago
It bothers me too. Dynamically typed does not imply no types. A good CL implementation takes that to heart and produces really nice code out of the box. And the language spec provides for an API to allow program developers to specify types and hints to the compiler to optimize certain functions and code paths where it's needed. The benefit from this is that you can rapidly prototype your application and harden it to the platform you deploy on when performance becomes an issue... and with built-in conditional compilation you can do that in a cross-platform way.
- mikeklaas 12y ago
  Python has types; I would even describe it as strongly-typed. It is thus relatively JIT-able (as pypy and psyco have shown). The chief problem is the high level of dynamic typing going, plus the huge amount of globally-changable assumptions (you can add methods to any instance of a class, you can globally change the definition of language built-ins, etc.)
  - agentultra 12y ago
    Some charity, please; I know Python is strongly-typed. It just doesn't take advantage of it and doesn't have facilities for producing fast code. Common Lisp not only provides type hinting facilities in the language it also provides hinting to the compiler to turn off all run-time type checks in critical code sections.
WaxProlix 12y ago
What work specifically are you referencing?

codegeek 12y ago

Good read. Whenever I read stuff like this, I always wonder if it is always true that dynamically typed languages are slower than statically typed languages ? Also, do we have to take this for granted that more higher level the language is, the slower it will be ? Or are there exceptions ? Also, it is worth asking if for majority of use cases of python for data/analysis, the ease and flexibility outweights the slowness.

YZF 12y ago
Always is a bit of a strong word but yes, it's always true. Virtual functions in C++ which allow some form of dynamic behaviour are slower than static function calls because they inherently involve another level of indirection. Static calls are known at compile time, they can be inlined by the compiler, they can be optimized in the context in which they're called. Now, nothing prevents the C++ run-time from trying to do the same thing at run-time but you can relatively easily see that it'll have to make some other compromised to do so. Nothing prevents a C++ program from generating C++ code at run-time, compiling it, and loading it into the current process as a .so. Now that's a pretty dynamic behaviour but there's again an obvious price. You can also write self modifying code. At any rate, static languages are capable of the same dynamic behaviour that dynamic languages are capable of but you often have to implement that behaviour yourself (or embed an interpreter...). Fundamentally, a dynamic language can't make the kinds of assumptions a more static language can make, it can try and determine things at run-time (ala JIT) but those take time and still have to adapt to the dynamics of the language. The same line of code "a = b + c" in Python can mean something completely different every time it's executed so the run-time has to figure out what the types are an invoke the right code. Now the real problem is that if you take advantage of that then no one can actually tell what this code is doing and it is utterly unmaintainable. To compound the problems facing dynamic languages is the fact that CPUs are optimized for executing "predictable" code. When your language is dynamic there are more dependencies in the instruction sequence and things like branch prediction may become more difficult. It also doesn't help that some of the dynamic languages we're discussing have poor locality in memory (that's an orthogonal issue though, you could give a dynamic language much better control over memory). EDIT: One would think it should be possible to design a language which has both dynamic and static features where if you restrict yourself to the static portion runs just as fast as any other statically compiled language and still allows to switch to more dynamic concepts and pay the price when you do that.

gregwebs 12y ago

Dynamic typing makes Python easier to use than C I disagree with the implications. The main reasons Python is easier to use is independent of type system. Not having to manually manage memory, for example! Overall it is a higher-level language, whereas C is designed for maximum performance. Dynamic typing is probably preferable to a 40 year old type system. But Python could be easier to use (catch bugs ahead of time!) and execute faster by adding a modern type system. Optional typing (like TypeScript or Facebook's new PHP implemntation) would probably be appropriate.

Spittie 12y ago
I'd love to see optional typing in Python, I wonder if there is any official reasons about why it never got introduced. I very rarely change the type of a variable, so it would be essentially free speed and free speeds for me. I actually wonder, do people use dynamic typing so often? I mean, it's nice to do "variable = int(variable)" when I know that I'm getting an integer in a string, but that's probably the only use case can I can think off that doesn't just reuse variables for something else.
- collyw 12y ago
  I started programming in Java, and moved to Perl and now Python. At first it seemed a bit weird, dynamic typing, but now I embrace it and enjoy the flexibility it offers. I see that many people prefer static typing and I am not sure why. Can someone give me a concrete example of when static typing would be beneficial? (The majority of my work revolves around databases, so I guess I rely on that as a kind of type checking to a degree).
  - antocv 12y ago
    Static typing provides more information for the programmer and for the runtime. More information makes the code somewhat readable and understandable 10 years after when it was written. Dynamic typing often becomes a much more messy affair. Static typing is beneficial when you want to catch a certain type of errors before your code enters production, it makes you write less error prone code.

mrfusion 12y ago

While everyone is on the subject, I thought I'd mention some weird behavior I saw today. I just switched my program from using a defaultdict to a regular dict. I.e., from defaultdict(lambda:'NA') to regular dict using get(val, 'NA') for access. And I got a something like a 100x speedup. It runs in two minutes instead of two hours. I had no idea a defaultdict would be so much slower. Unless there's something funny going on in my program and it's unusual behavior.

zo1 12y ago
The docs for the defaultdict are here: https://docs.python.org/2/library/collections.html#collections.defaultdict https://docs.python.org/2/library/collections.html#collectio... And they pretty much say that there is only one method that's been overridden from the normal dict class. And all it does is it executes the anonymous method (lamda) you supplied instead of returning the default value you pass in. Also, this depends on your usage, and what you have in the lambda expression. I'm curious about your code, care to paste/link some of it for us?
- cefstat 12y ago
  I would guess that the code tries to get the value for keys that do not exist in the defaultdict. Well, if the key does not exist then it is being added with the default value 'NA'. So in this way you can end up with a huge dictionary where most of the values are just 'NA'.
  - mrfusion 12y ago
    Oh, wait, so defaultdicts actually insert missed keys?? I thought they simply gave you a default value instead of a keyerror. If you're right, then that would certainly explain it.
    - bryanh 12y ago
      Yep, take a look! >>> import collections >>> dd = collections.defaultdict(int) >>> print dd['doesnt exist'] 0 >>> print dd defaultdict(<type 'int'>, {'doesnt exist': 0})

rch 12y ago

> Python ends up being an extremely efficient language for the overall task of doing science with code. This is pretty close to how I characterize my time, 'doing X with code', and Python yields great returns in these terms.

VeejayRampay 12y ago

Python (and Ruby for that matter) are slow because there are no billion-dollar companies stuck with an initial choice of language that impedes their ability to grow. PHP and Javascript used to be extremely slow and now after several dozens of millions thrown at JIT, rewrites, forks and redesigns they're starting to get much much faster.

hyperbovine 12y ago
Not so. Google uses Python extensively in-house (it's one of the four "blessed" languages) and, in fact, employed GVR until he was recently hired away by Dropbox--another billion-dollar company which relies heavily on Python. At one point Google even spearheaded an ambitious effort (https://code.google.com/p/unladen-swallow/ https://code.google.com/p/unladen-swallow/) to make CPython much faster. It failed. At the point where even Google can't make it happen, it really starts to look like Python performance is limited at some very fundamental level to what we see today. Personally I think this is fine. I use Python for everything day-to-day and offloading the intensive stuff to a C extension (a la Numpy) works just great. There are very few instances where I find myself wishing Python was faster.
- ihnorton 12y ago
  Unladen Swallow started as an intership project and didn't get very much support at Google [1]. When US started (~2009), Google had been writing their core, high-performance stuff in Java and C++ for many years already so there was no internal or existential need to make Python faster. [1] http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospective.html http://qinsb.blogspot.com/2011/03/unladen-swallow-retrospect...
  - ZenoArrow 12y ago
    On the other hand, there may have been applications higher up the stack that would've benefited from a more efficient Python. I don't know if it still is but I heard GMail was written in Python, you'd think an application of that scale would benefit from better performance.

amit_m 12y ago

The bigger picture is that CPython's core team simply does not care too much about performance. Performance has never been a fundamental requirement, but merely an afterthought. The biggest reminder of this was Python 3, which for me was a complete disappointment. They could have limited python's super-dynamic behavior (e.g. changing builtin functions, patching classes on the fly, etc.) or made them optional. They could've added optional typing annotations a la Cython. Or even changing the builtins and language syntax to allow more inplace operations and preallocations, so that temporary results wouldn't have to be allocated on the heap over and over again. All of these changes would have made python faster and more JIT-able. None of these things happened. Performance-wise, python 3 is no step forward. Python+Cython is still a powerful combination, but eventually Julia or similar languages will eat python's lunch with respect to scientific computing.

GFK_of_xmaspast 12y ago
I think if python3 had made major performance improvements (not to mention the GIL) there wouldn't be nearly as many 2.7 holdouts.
- ris 12y ago
  And it still wouldn't be done.
est 12y ago
Yup. the HUGE "CPython is only reference implementation" ego
walshemj 12y ago
Not for pure performance Fortran still seems to be king for that type of scientific computing.

dmoney 12y ago

Related: "Python is only slow if you use it wrong" http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf [pdf]