14 ms·

Yes, Python is Slow, and I Don’t Care

136 points by mithunmanohar1 9y ago

VHRanger 9y ago
The problem is not so much that python is slow. It's that in some scenarios python can't be made fast. Fast prototyping is great but being stuck with a prototype for deployment isn't.
- ColinWright 9y ago
  Did you read the section on "Optimizing Python" ?? If so, I'm assuming you read about using CPython to migrate your package/program/module painlessly to C. So given that, can you elaborate on your objection? I'd like to know why you think the article is wrong about optimising your Python to make it fast enough.
  - VHRanger 9y ago
    "migrate your package/program/module painlessly to C" Do you or anyone on your python using team can use C professionally for things you'll use in production? I work with python everyday on a team that uses python everyday, and I worked professionally in C++ for a >2 years time, and I still wouldn't trust myself to write a reliable/safe C backend to something. And using C++ behind python with something like pybind11 is anything but painless on the other end; it requires careful consideration. Python is "good enough" in nearly all cases, but there are some instances where you sort of hate yourself for using it for the prototype since you're forced to effectively rewrite afterwards.
    - traverseda 9y ago
      You should try python-cffi. Using python code as a shared library under C is pretty easy, and calling C from python is pretty easy. The documentation is a bit crappy, but it's alright once you get the hang of it. I recently used it to prototype a shared library that overrides the `getenv` syscall, with the intent of providing decrypt-on-use environment variables. It's pretty simple, stolen from lua I believe. https://cffi.readthedocs.io/en/latest/overview.html#embedding https://cffi.readthedocs.io/en/latest/overview.html#embeddin...
- auserperson 9y ago
  In pure Python that is true, but the article discuss this issue.
- traverseda 9y ago
  >It's that in some scenarios python can't be made fast. Can you give some examples of this? I mean, obviously with enough effort you can "make python fast" since it has good C bindings, and can just be a thin wrapper around fast stuff. Similar to how command line tools can be ridiculously fast[^1] despite, ostensibly, running in bash. So I'm a bit confused about what you're claiming. Organizational issues, it's difficult to get management on board with an optimization pass? [^1]: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
  - deleted 9y ago
    [deleted]
  - VHRanger 9y ago
    My point is do you have competent C programmers on your team? That said, there is a point in some python prototypes where you "hit the performance wall". For whatever reason, you'll need to look at one of the options to make python faster and none of them are painless unless you're already a serious C programmer.
- kerkeslager 9y ago
  In ten years of Python development I have yet to come across an instance where Python couldn't be made fast. In some cases critical sections had to be delegated to C but even that is very rare.
- fnord123 9y ago
  One of the best features of Python is the almost free FFI into C. This means you can prototype and then dial up the performance to 11 if required.
wyldfire 9y ago
> Your bottleneck is most likely not CPU or Python itself. I've found that this is often the case. Nearly always disk or network. But it's sometimes surprising how little work you need to do to become CPU-bound. This is the price we pay for such a tremendously dynamic language. Indeed, the article's suggestions of C/Cython/PyPy are good ones to remedy the problem when it occurs.
booshi 9y ago
This keeps getting posted, and while it makes some valid points, it's a lot of handwaving. Arguably, other languages can get code out faster depending on the dev, language, etc.
- blumomo 9y ago
  Definitely depending on the developer. That's why his disclaimer on the article's beginning: "I'm a Python fan boy". Nevertheless a very good read to me.
- 0xcde4c3db 9y ago
  Agreed. Things that are handwaved include: 1) Performance can be a genuine requirement of the product, i.e. if it's not fast enough, it doesn't ship. You can't ship faster and cheaper by sacrificing the thing you need to ship (well, you can, but then you're shipping a different product, not meeting the same requirements sooner; it's no different than cutting a feature). 2) Many processes can't be horizontally scaled in an efficient way, period. Not because the programmer is ignorant of some cool algorithm, but because the problem is fundamentally expensive to parallelize. Maybe you end up getting something like a 20% boost by having twice as many nodes, even after applying all the cool algorithms. And you don't necessarily get that scalability in your code base for free, either. 3) "Speed" in the mobile and embedded spaces is often as much about energy efficiency and thermal management as getting done sooner. 4) The metrics for deciding that Python is faster to develop in only measure small problems. People tend to shy away from Python for bigger projects, and the reasons for this are pretty hotly debated.
mangecoeur 9y ago
Python is also heavily used in science, where performance really does matter. It's successful because of how highly ergonomic python apis can be built on top of optimised C/C++/Fortran libraries. That said, there is clearly a desire to write 'fast' code in python itself without swapping to C. Cython helps, but to get really fast Cython code you actually have to write with C-semantics (so you are basically writing C with Python syntax). Projects like numba JIT are interesting in that they can optimise domain-specific code (i.e. numerical/array code) that's written in normal python style. It also means jumping through a few hoops (although with the latest version in many cases all you need is a single decorator on your hot function). You can even do GIL-less multithreading in some cases. Overall things are looking promising, with the addition of the frame evaluation API and possible improvements to the python C-api that could make JIT and similar extentions easier.
dom96 9y ago
The fact that Python is slow isn't its only problem. What I care more about nowadays is wasting my time hunting bugs that could have been avoided by a static type system.
- blumomo 9y ago
  Please tell me, do you write tests? I've learned that it's necessary. I do 100% code coverage and I'm enjoying TDD a lot.
  - virmundi 9y ago
    The whole point of static type systems is that they give you the "type tests". 100% code coverage just to get what the compiler would give you is a waste of time. If python is supposed make developers more productive, this is just dragging them down.
  - adwn 9y ago
    Time/cost to locate and fix a bug: at compile time < in unit test < in integration test < in production. Besides, tests cannot find all the bugs a good static type system can find, and vice versa. Tests, even at 100% coverage, are not a complete substitute.
  - vog 9y ago
    If you haven't already tried, I recommend to port [1] a Python program to OCaml, F# or Haskell. There are so many tests you can throw away, so many corner cases you don't need to test anymore. (I'm talking about tests that you can't even write down without getting a compiler error.) Not to mention the assertions within your functions that aren't needed anymore. And those little type annotations are so much simpler and easier to write down than corresponding tests. If you really head for 100% testing, not just code coverage, but also all corner cases, you will love the modern static type systems. (However, you should really use an ML type system, because doing the same with Java or C++ is cumbersome and not much fun.) [1] I did so for my mathematics diploma thesis, starting implementing my formalism in Python, having lots of trouble when refactoring, then moving all code to OCaml and then being able to refactor the code alongside the developing formalism in the thesis.
icebraining 9y ago
"It doesn't matter than Python is slow, besides we can use compiled libraries to speed it up" "People saying it doesn't matter that Python is slow are deluding themselves and preventing Python from getting faster like JS did" "Python is inherently harder to optimize than JS since it has <very dynamic features>" "Smalltalk/Lisp/etc are also very dynamic yet are much faster" "The slowness of Python is harming the planet by being inefficient and therefore wasting more energy/producing more pollution" Did I miss any arguments? I know certain topics are bound to attract some repetitive discussion, but "Python is slow" has been one of the worst.
- jrs95 9y ago
  "Because Guido said so" isn't on your list, but otherwise I think you got them all
- yen223 9y ago
  "The slowness isn't worth the metaclass abuse" would be my take
- dom0 9y ago
  > "Python is inherently harder to optimize than JS since it has <very dynamic features>" Python is not a very dynamic language in the sense that you actually can't change a lot of stuff (and a number of the things you can change just segfault CPython). I think JS is more dynamic, for example. Or Ruby.
  - icebraining 9y ago
    These are not my arguments, mind you; I don't know enough to make them. You've piqued my interest, though: can you give me an example of those things that you can't change or that break CPython?
    - dom0 9y ago
      Things like the Carlo Verre hack (also a thing you can't change —any more— in Python: builtins), editing objects during their construction (via e.g. gc)... generally, the gc module allows other ways as well to crash your interpreter. >>> import gc >>> 'foo'.lower() >>> gc.get_referents(str.__dict__)[0]['lower'] = str.upper >>> 'foo'.lower() segmentation fault (core dumped) python (That's the method lookup cache) A talk in this direction is https://www.youtube.com/watch?v=qCGofLIzX6g https://www.youtube.com/watch?v=qCGofLIzX6g
jayflux 9y ago
I get the point this guy is making, but if you need something parallel for a cpu bound task, throwing more hardware at the problem isn't the most efficient solution if you can just use more cores. For example adding another quad core when the first cpu is only using one core anyway is inefficient and expensive. Right tool for the right job I suppose.
- nhumrich 9y ago
  Python does multiprocess very well. You can easily use all cores on your machine. Pythons main "disadvantage" is threading because of the GIL. But each process gets its own GIL. So when you multi process, your not limited to one core.
  - classybull 9y ago
    This. I had a problem where I needed to scrape roughly 20,000 html documents daily, which is normally a pretty slow task. You have to open the file, load it into memory, parse the DOM, and then run all of your selection methods. Sequentially, it took about 60 minutes daily. Multithreading slowed it down because it was CPU bound. Multiprocessing allowed me to run 12 processes across 8 cores. That took the total processing time down to about 4 minutes or so. And I was able to write the code in a day. Writing something similar in Java or C++ would have taken me a week.
- aeorgnoieang 9y ago
  The point is that (modifying existing code or writing new code to) "just use more cores" may be less efficient, for the business or organization that is employing the programmer, given that programmer salaries, over even a fairly short amount of time, can be more expensive than hardware.
Waterluvian 9y ago
Python is my Swiss army knife. I love it because it is a single tool that can aid in almost every project I do. But if I'm doing one specific thing a lot, I want that thing to be done well and done efficiently, so I'll reach for the specific screwdriver I need. Also most of my problems are IO bound so single threaded concurrency is fine. But I represent a very small portion of the global problem space.
_pmf_ 9y ago
Somewhat ironically, Python is used a lot for things that would benefit from raw speed (data processing pipelines) and do not benefit at all from dynamic typing (since the kind of property bags / data frame views over data are easily replicated in statically typed languages). But Python's C extension API is quite a bit easier than p.e. Matlab's MEX API (to me at least); can typical Python IDEs compile and relink extension modules without an external build step? > Your bottleneck is most likely not CPU or Python itself. With applications that are dominated by raw data processing, it's very, very easy to be CPU dominated. Hell, I had one quite trivial data converter for logfiles where the "parsing the printf string" part of Java's printf dominated processing and writing a custom formatter halved processing time (while regexes can be compiled, the format string cannot be precompiled and will be interpreted each time); it's one of those things where I would intuitively say "why did this moron write his custom formatter" if I stumbled upon it in a code review. Intuitively, you'd expect this to be a simple case of an IO dominated task (which it is now once the bottleneck has been removed). If it's fire-and-forget batch jobs, you can get away with it, but if the converter is part of a user facing fat client application that runs on a old office laptop, you don't have that luxury.
freetime2 9y ago
I pretty much agree with everything in the article - except for the bit where he tries to quantify why python is better from a developer efficiency perspective than other languages. The main example he cites is a study that compares the amount of time writing string processing routines in different languages - which is quite a bit different from the work I do every day. I develop web apps which means I generally work in very large code bases, and spend most of my time modifying existing code rather than writing fresh code from scratch. I have found that statically typed languages (java + typescript) and the fantastic IDE support that comes along with them make it really easy to navigate around the code and refactor things. Also - the compiler tends to catch and prevent a whole class of bugs that you might otherwise only catch at runtime in a dynamically typed language. Of course there are other situations where I prefer to use Ruby as my scripting language of choice - it all comes down to using the right tool for the job at hand. Unfortunately I don't think the author gives enough consideration to the trade-offs between static vs. dynamically typed languages, and I think he would have been better just leaving that section out as it isn't really necessary to prove his point that CPU efficiency isn't important in a lot of applications. Ultimately though I completely agree with his main point: "Optimize for your most expensive resource. That’s YOU, not the computer."
scarface74 9y ago
I was all ready to savage his opinion after reading the headline but I agree looking at my architecture that I designed for the company I work for, CPU isn't the bottleneck. Every time I try to increase performance by multi threading as much as possible, the databases start screaming. On the other hand, the idea that dynamic languages are more productive than static languages are laughable. Statically type languages prevent a lot of bugs and allow for a lot of automated provably correct refactorings that simple cannot be done with a statically typed languages. You can't even reliably do a "find usages" of classes using a dynamically typed language.
- simonh 9y ago
  'Dynamic languages' is too often used as a shorthand, or interchangeable with 'scripting language', as here. There are a ton of more relevant language features when it comes to productivity, automatic memory management being the biggest IMHO. Interpreted vs compiled makes a difference too because your code->launch->test cycle can be so fast. But I agree, I'm a huge Python fan and even though I appreciate not having to gum up my code with type declarations, it's not really that big of a win in terms of productivity.
  - rockostrich 9y ago
    But with a statically typed language, you don't need to go through that code -> launch -> test cycle as often because everything is type-safe.
    - deleted 9y ago
      [deleted]
    - hpcjoe 9y ago
      Uh ... no. Type safety is rarely a problem from my personal experience (YMMV) in code dev. The issues I run across have to do with whether or not the code is an accurate reflection of the algorithm in question, or even if the algorithm itself is correct, if core assumptions are correct, dealing with corner cases, etc. Types rarely have anything constructive to add to this mix, regardless of whether or not I am working in a weakly/non-typed language, or strongly typed language. I personally tend to do something much closer to TDD, and break my code out so I test each part with its own test rig. So I see when I do dumb things that need re-work. Again, typing is pretty much never an issue there. While this is all anecdotal, I personally have found that enforcing boilerplate type defs actually increases my coding error rate. More LOC gives you more opportunities for errors. FWIW, Julia (http://julialang.org http://julialang.org) is IMO Python done right, and it is strongly typed. But done in such a way as to not actually get in your way most of the time. Like Python it has a great REPL, fantastic FFI, rapidly growing module list, very active development. Unlike Python, it has no issues with whitespace[1], is compiled (JIT for now, though static is planned from what I understand), is very fast, has multiple dispatch and a strong typing system. The real point the article was making is, programmer time is the most valuable thing to optimize for. I completely agree with this, even though I disagree that Python is the right tool for (most) every job. There are better tools IMO, which allow me to be far more productive, and spend as little time as possible on worrying about types ... which are rarely ever a concern for me. I thought his comment about string processing was interesting, though he completely ignored Perl in the mix. I've found string processing in Perl to be almost trivial. Its harder (far more verbose boilerplate) in Python, and gets worse in other languages. Perl6 lets you embed grammars and write parsers without resorting to external modules (Marpa::* in Perl5). Takes building string processing, parsers, etc. to a whole new level, while making you even more productive. But, as the author had noted, he is a pythonista. Everyone has their biases (including me). I want to code correctly quickly, and not have things that shouldn't get in my way, get in my way. Including type systems, massive boilerplate/odd syntax rules, etc. [1] It is 2017 ... structure by indentation is somehow, still a thing. It shouldn't be IMO.
traverseda 9y ago
There's are still some big gains python could make, if python implementations were better. Micropython is equivalent to a real-time cooperative-multitasking OS. If it had ~~better~~ support for things like cffi, you could implement posix on top of it. I can imagine a laptop that runs gnu+python in the next few years. That's a whole new usecase, simply because that implementation uses a lot less ram. What usecases would we discover for a faster python? Shared objects and proper sandboxing would also be huge.
- jerf 9y ago
  "There's are still some big gains python could make, if python implementations were better." At this point, I would find it far easier to believe that you are underestimating the difficulty involved in what it takes to speed up Python than that there are enormous gains yet to be had in speeding up Python. I suspect JS has had more optimization effort expended overall, but Python has still had a ton of work by lots of smart people, and generally got an earlier head start on optimization. (They didn't start trying to make JS "fast" right away; they spent rather a lot of time getting JS's hookup to the DOM in the face of things like .innerHTML working first, before anyone even cared to do what we today do routinely without thought in plain ol' Javascript, let alone with our glorious frameworks.) There are enormous gains to be had in speeding up "a language that is like Python except certain things are banned", but people have already done that analysis too and discovered that broadly speaking, if you do that, too much existing Python breaks. If you want to see something like that, check out the RPython aspect of the PyPy project, which successfully implements a fast subset of Python. But it is a noticeably restricted subset of Python; AIUI it's not even close to something you can just drop in to your code and get faster speeds. One of the things that I've learned from Python and the other attempts to speed up the scripting languages is that despite the mantra, yes, there is in practice such a thing as an intrinsically slow language. (The theoretical existence of a Python interpreter that can run all existing Python code at C speeds doesn't do us much good if we have no idea after decades of very smart people banging on the problem how to manifest it. Personally I'd suspect that while such a beast theoretically exists it has an exponential complexity startup cost or, if you prefer, exponential compilation costs. And probably a pretty decent code and/or RAM bloat cost, too.) And Python is one of them. Some of the reasons why it is so much fun to use are part of that intrinsic slowness. Some of them really aren't. I personally think there's a lot of up-and-coming languages that are exploring the space of how to get those nicer programming abstractions and programmer-convenient code without paying anywhere near the runtime cost that the dynamic scripting languages of today do; it's one of the more exciting developments I see coming up. People complain a lot about code bloat and poor performance of our code since right now we have to choose between "fairly inconvenient but fast" and "convenient but slow and bloated". Patience! Better choices are developing, but they're still young.
hellofunk 9y ago
Yeah? Well, Jimmy Crack Corn and I don't care.
11235813213455 9y ago
Nice article, I'm more in JS nowadays, it's probably now as good as a scripting language than Python
agentgt 9y ago
Unless we are talking like circa 1999 I don't think I have heard a complaint yet that Python is slow. I'm curious who or where the author heard that from (not specifically the people themselves but the domain they are in). What I have heard complaints about Python are (and I don't agree with all these points): * Its not statically typed * The python 2/3 compatibility * It has some design flaws: GIL, variable assigning, mutable variables, lambdas, indentation (I don't agree with all these but this is complaints I have heard). * The plethora of packaging (ie its not unified) I guess one could argue its slow because it can't do concurrency well but that really isn't raw speed. Then the author started comparing string processing of programmer time from a study which... doesn't help the authors point at all. * Python has and will always be fast at string processing and most people know this * The people that complain about python speed are almost certainly not doing string processing * I have serious questions about the study in general (many languages have changed quite a bit since then)
- wheaties 9y ago
  Oh, I've heard "Python is slow" from Node folks, Java folks, Scala folks (in terms of productivity) and of course, die hard C++ fans. I'm a fan of Python but i also like languages with a stronger FP bent (and statically typed.)
  - agentgt 9y ago
    > Oh, I've heard "Python is slow" from Node folks, Java folks, Scala folks (in terms of productivity) I have hard time believing Java folks made an argument on productivity unless you meant for that parenthetical statement to be only applied to Scala. Regardless productivity is a far more complicated than raw speed (particularly if you get into maintenance as some languages are easy to pump out code but harder to maintain... oh perl.. the pain...). And yeah I have seen the C++ folks complain about Python but this in my experience has been video game programming which is clearly not the domain the author is in or talking about (I think given the microservice discussions). Hence why I would like to know more about these folks complaining.
- rockostrich 9y ago
nadam 9y ago
"It used to be the case that programs took a really long time to run. CPU’s were expensive, memory was expensive. Running time of a program used to be an important metric." As hardware gets faster we give it new tasks that could not be achieved before. Like rendering high resolution stereoscopic images using physically based shading at 90 FPS on relatively cheap consumer hadware (VR). There are still quite a lot of code that we call 'performance critical'. Most of that code is written in C/C++ (and CUDA and glsl, and hlsl, etc...) today.
- carlmr 9y ago
  Yeah, also we have a lot of code that a lot of engineers in our company run. Some of it takes hours instead of a few minutes to run, because it's a bunch of slowness combined. Sure you can say the engineer doesn't have to watch. But sometimes you need this result, and can't do much else until it's there. Doing a few loops at a few hours at a time is very time consuming. Being lazy about slowness is bad in the long run, if you or your customers waste inordinate amounts of time, because one of your devs didn't want to think. Sure you should optimize from the bottleneck, look at hot path, measure where optimization makes sense. But using tools that make some stuff 10x faster without much effort is not premature optimization. It's just not being stupid.
- ivm 9y ago
  It's still expensive on client machines because most of the persons in the world are NOT software engineers with 6-digit salaries. They run cheap computers with HDDs and Windows polluted by a ton of 3rd party crap. They don't know how to fix it and silently suffer. I was cleaning a local vet clinic's devices recently – they were literally switching between two computers to not wait 5 minutes of non-responsiveness because some bloated software was occasionally consuming 100% of CPU.
  - mark-r 9y ago
    A lot of businesses these days prefer web apps. It's not hard to understand why - all the hassle of system maintenance falls to the people who host the app and can afford to know their stuff. If your Windows PC is suffering from rot just replace it with a Chromebook.
fiatjaf 9y ago
> It’s more important to get stuff done than to make it go fast. This is not a real absolute. It is only valid when what you have to run will not benefit a lot from performance or suffer a lot from lack of it. The real guidance you can have in these matters is: how many times is my code going to run per second? Some programs are written to be run once a day, others 10000000 times in a second. The first ones should be written in the language you're most productive in, the second ones in the fastest possible language.
donatj 9y ago
I know it's not trendy, but I would argue PHP is as productive for developers as Python and has a MUCH faster runtime, particularly after 7.
__s 9y ago
> without getting stuck in the weeds of the small things such as whether you should use a vector or an array Yes, instead get into the weeds of tuple vs list Not included in the graph of time-to-solve-problem static languages: statically typed languages with type inference
- scbrg 9y ago
  Given that they have exactly the same interface, that choice is really easy. You go with one until it turns out to be insufficient, and then you switch to the other and not a single line of code has to change, except at the point where you create the thing. Incidentally, the same is true in many situations in Python, and that is (IMO) one of its strengths.
snarfy 9y ago
Slow doesn't matter when you scale horizontally.
nhumrich 9y ago
Author here. Surprised to see this toping HN. Appreciate all the feedback. Let me know if you have any questions.
kodablah 9y ago
The article could be titled: "Yes, Python is Slow To Refactor and Maintain, and I Still Don't Care". I never understand why dynamic language enthusiasts primarily focus on new code only. You have to discuss all sides of increased or decreased productivity to make a rational argument.
- hasenj 9y ago
  Python is optimized for getting interesting things done in a few lines of code. Small scripts you write once and then forget. For serious projects? IMO python is a disaster.
Thaxll 9y ago
Giving EvE Online as an example is bad because that game artificially slow the game loop to keep up with the number of players, would this happen with C++ on a recent architecture? Probably not.
nervous123 9y ago
This sentiment is the reason for almost all software (especially on the web) beeing a load of crap. It's slow, it's buggy and developers always give the same excuse: CPUs and memory are cheap, therefore we can waste our customers time. Imagine what we could do with the amazing hardware we have, if people started to do the sane thing and actually use the hardware to do things efficiently.
deleted 9y ago
[deleted]
dahart 9y ago
Python's value to me has always been that it's easier to get things done, not it's speed. One time when I was interviewing a candidate for a coding job, the candidate said she loved Python the most "because you can just yell at it and it'll work." It's both the breadth of the standard library and ecosystem, and the simple language design, that make developing things in Python faster for me. Doing problems on Project Euler has been an education for me in how algorithm matters more than speed. Lots and lots of people spend hours writing long C++ codes that are easily beaten by a few lines of Python. It certainly goes the other way too, and the wrong algorithm in Python is even that much slower and more painful than the right algorithm in C++. But when the right algorithm is used and the problem is solved in a few milliseconds, it really doesn't matter which language uses more CPU cycles, all that matters is whether you saw the insight that let you skip 99% of the search space, and how much time you spend writing code.
progman 9y ago
Yes, time to market is important. However, you don't need to compromise convenience of development for the sake of performance. If you twist your Python code to get performance it takes time. If you need performance, and like the syntax of Python then you should take a look at Nim [1]. With Nim I develop as quickly as in Python while I get the performance of C. [1] https://nim-lang.org https://nim-lang.org I believe application performance is important on servers. It makes a difference if your Shop software written in Python is able to handle 50 requests per second, or if the same software written in Nim can handle 500 rps. And by the way, Nim provides static typing which helps a lot to catch errors at compile time.
- cup-of-tea 9y ago
  Nim seems really good, but does it have a decent REPL these days? I'm not sure if it would be as convenient with a statically typed language, but I like the incremental development approach so much that I only use C if I absolutely have to.
  - dom96 9y ago
    It doesn't. But you can grab Aporia (or some other tool) to quickly compile and run some code, it replaces a REPL very well in my experience.
bluedino 9y ago
Many times when Python is blamed for being slow, it's the programmers fault. Python is great that you can 'regular' people writing code in it quickly. The problem is, these regular people don't always understand algorithms or things like caches, threads, databases... A lot of these users can just say "My department needs a $40,000 24 CPU server with maximum RAM from MicroWay/SuperMicro, we need to run our codes faster", when they are just trying to brute force things. They understand the problem domain but don't have the programming skills to use a computer to efficiently solve it. But, these guys are all a step ahead of the ones who are stuck in the mindset of "C is the only language fast enough for my work", while not even understanding pointers and basic syntax and getting stuck on silly things like text processing, which could be done in minutes in Python.
sedlich 9y ago
In "What if CPU time is an issue?" we could also mention the nim language (and not only cython) because it compiles (not only) to C and feels like python.
deadsy 9y ago
Premise: It's more important to be productive than to have fast code. Conclusion: Use Python. Is the premise true? For many cases- yes, but it depends. If you are running an application on the cloud and your metric is $/user/year and you have many users then saving some compute resources for each user gets attractive and you don't want to just throw another VM at it. Is the conclusion true? Garbage collection gives big productivity gains. Other languages have GC. It's not nice to see your Python code die after a few days because you messed up the type passed to a function. Other languages fix that at compile time. Multicore is now. Other languages are built with better multicore awareness.
boomlinde 9y ago
The author argues from his professional experience as a Python developer that it's fast enough, that you'll spend most time waiting for I/O anyway, that you can just throw more servers at the problem etc. The problem is that his experience as a Python developer doesn't accurately reflect the prevalence of problems where runtime CPU performance actually is an issue. Of course not, because who in their right mind would make an informed decision to solve such a problem in Python? Python has worked for him because it is only useless for a category of problems that he hasn't had the opportunity to solve because he's a Python developer. Outside this professional experience, not everything is a trivially parallel web service that you can just throw more servers at if CPU time exceeds I/O waiting. It all really boils down to what your requirements are, whether you have all the time and memory of a whole server park at your hands, or a fraction of the time available in a smaller embedded system, how timely the delivery of the software has to be and how timely it needs to deliver runtime results once it's up and running. There are times where Python just isn't fast enough, or where getting it fast enough is possible, but more convoluted and tricky than implementing the solution in a more performant language. Developer time may be more expensive than the platform that my solution is for, but that doesn't get around the fact that it eventually will need to run with the available resources.
agnivade 9y ago
> However, this is no longer true, as silicon is now cheap. Like really cheap. Run time is no longer your most expensive resource. Our client won't spend more money than a t2.medium instance on aws. Nothing we can do about it. In that case, run time does become an expensive resource. But I get the point that OP is trying to make. Just wanted to mention that not all of us have the comfort of having enough resources on which our app runs.
karmakaze 9y ago
Putting aside the discussion regarding productivity, there is a case where I have found execution time to matter. Scaling an application which uses an unsharded database. The long transaction durations and number of connctions were bottlenecking db throughput. The particular app was a Ruby/Rails monolith.

The problem is not so much that python is slow. It's that in some scenarios python can't be made fast. Fast prototyping is great but being stuck with a prototype for deployment isn't.

ColinWright 9y ago
Did you read the section on "Optimizing Python" ?? If so, I'm assuming you read about using CPython to migrate your package/program/module painlessly to C. So given that, can you elaborate on your objection? I'd like to know why you think the article is wrong about optimising your Python to make it fast enough.
- VHRanger 9y ago
  "migrate your package/program/module painlessly to C" Do you or anyone on your python using team can use C professionally for things you'll use in production? I work with python everyday on a team that uses python everyday, and I worked professionally in C++ for a >2 years time, and I still wouldn't trust myself to write a reliable/safe C backend to something. And using C++ behind python with something like pybind11 is anything but painless on the other end; it requires careful consideration. Python is "good enough" in nearly all cases, but there are some instances where you sort of hate yourself for using it for the prototype since you're forced to effectively rewrite afterwards.
  - traverseda 9y ago
    You should try python-cffi. Using python code as a shared library under C is pretty easy, and calling C from python is pretty easy. The documentation is a bit crappy, but it's alright once you get the hang of it. I recently used it to prototype a shared library that overrides the `getenv` syscall, with the intent of providing decrypt-on-use environment variables. It's pretty simple, stolen from lua I believe. https://cffi.readthedocs.io/en/latest/overview.html#embedding https://cffi.readthedocs.io/en/latest/overview.html#embeddin...
auserperson 9y ago
In pure Python that is true, but the article discuss this issue.
traverseda 9y ago
>It's that in some scenarios python can't be made fast. Can you give some examples of this? I mean, obviously with enough effort you can "make python fast" since it has good C bindings, and can just be a thin wrapper around fast stuff. Similar to how command line tools can be ridiculously fast[^1] despite, ostensibly, running in bash. So I'm a bit confused about what you're claiming. Organizational issues, it's difficult to get management on board with an optimization pass? [^1]: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
- deleted 9y ago
  [deleted]
- VHRanger 9y ago
  My point is do you have competent C programmers on your team? That said, there is a point in some python prototypes where you "hit the performance wall". For whatever reason, you'll need to look at one of the options to make python faster and none of them are painless unless you're already a serious C programmer.
kerkeslager 9y ago
In ten years of Python development I have yet to come across an instance where Python couldn't be made fast. In some cases critical sections had to be delegated to C but even that is very rare.
fnord123 9y ago
One of the best features of Python is the almost free FFI into C. This means you can prototype and then dial up the performance to 11 if required.

wyldfire 9y ago

> Your bottleneck is most likely not CPU or Python itself. I've found that this is often the case. Nearly always disk or network. But it's sometimes surprising how little work you need to do to become CPU-bound. This is the price we pay for such a tremendously dynamic language. Indeed, the article's suggestions of C/Cython/PyPy are good ones to remedy the problem when it occurs.

booshi 9y ago

This keeps getting posted, and while it makes some valid points, it's a lot of handwaving. Arguably, other languages can get code out faster depending on the dev, language, etc.

blumomo 9y ago
Definitely depending on the developer. That's why his disclaimer on the article's beginning: "I'm a Python fan boy". Nevertheless a very good read to me.
0xcde4c3db 9y ago
Agreed. Things that are handwaved include: 1) Performance can be a genuine requirement of the product, i.e. if it's not fast enough, it doesn't ship. You can't ship faster and cheaper by sacrificing the thing you need to ship (well, you can, but then you're shipping a different product, not meeting the same requirements sooner; it's no different than cutting a feature). 2) Many processes can't be horizontally scaled in an efficient way, period. Not because the programmer is ignorant of some cool algorithm, but because the problem is fundamentally expensive to parallelize. Maybe you end up getting something like a 20% boost by having twice as many nodes, even after applying all the cool algorithms. And you don't necessarily get that scalability in your code base for free, either. 3) "Speed" in the mobile and embedded spaces is often as much about energy efficiency and thermal management as getting done sooner. 4) The metrics for deciding that Python is faster to develop in only measure small problems. People tend to shy away from Python for bigger projects, and the reasons for this are pretty hotly debated.

mangecoeur 9y ago

Python is also heavily used in science, where performance really does matter. It's successful because of how highly ergonomic python apis can be built on top of optimised C/C++/Fortran libraries. That said, there is clearly a desire to write 'fast' code in python itself without swapping to C. Cython helps, but to get really fast Cython code you actually have to write with C-semantics (so you are basically writing C with Python syntax). Projects like numba JIT are interesting in that they can optimise domain-specific code (i.e. numerical/array code) that's written in normal python style. It also means jumping through a few hoops (although with the latest version in many cases all you need is a single decorator on your hot function). You can even do GIL-less multithreading in some cases. Overall things are looking promising, with the addition of the frame evaluation API and possible improvements to the python C-api that could make JIT and similar extentions easier.

dom96 9y ago

The fact that Python is slow isn't its only problem. What I care more about nowadays is wasting my time hunting bugs that could have been avoided by a static type system.

blumomo 9y ago
Please tell me, do you write tests? I've learned that it's necessary. I do 100% code coverage and I'm enjoying TDD a lot.
- virmundi 9y ago
  The whole point of static type systems is that they give you the "type tests". 100% code coverage just to get what the compiler would give you is a waste of time. If python is supposed make developers more productive, this is just dragging them down.
- adwn 9y ago
  Time/cost to locate and fix a bug: at compile time < in unit test < in integration test < in production. Besides, tests cannot find all the bugs a good static type system can find, and vice versa. Tests, even at 100% coverage, are not a complete substitute.
- vog 9y ago
  If you haven't already tried, I recommend to port [1] a Python program to OCaml, F# or Haskell. There are so many tests you can throw away, so many corner cases you don't need to test anymore. (I'm talking about tests that you can't even write down without getting a compiler error.) Not to mention the assertions within your functions that aren't needed anymore. And those little type annotations are so much simpler and easier to write down than corresponding tests. If you really head for 100% testing, not just code coverage, but also all corner cases, you will love the modern static type systems. (However, you should really use an ML type system, because doing the same with Java or C++ is cumbersome and not much fun.) [1] I did so for my mathematics diploma thesis, starting implementing my formalism in Python, having lots of trouble when refactoring, then moving all code to OCaml and then being able to refactor the code alongside the developing formalism in the thesis.

icebraining 9y ago

"It doesn't matter than Python is slow, besides we can use compiled libraries to speed it up" "People saying it doesn't matter that Python is slow are deluding themselves and preventing Python from getting faster like JS did" "Python is inherently harder to optimize than JS since it has <very dynamic features>" "Smalltalk/Lisp/etc are also very dynamic yet are much faster" "The slowness of Python is harming the planet by being inefficient and therefore wasting more energy/producing more pollution" Did I miss any arguments? I know certain topics are bound to attract some repetitive discussion, but "Python is slow" has been one of the worst.

jrs95 9y ago
"Because Guido said so" isn't on your list, but otherwise I think you got them all
yen223 9y ago
"The slowness isn't worth the metaclass abuse" would be my take
dom0 9y ago
> "Python is inherently harder to optimize than JS since it has <very dynamic features>" Python is not a very dynamic language in the sense that you actually can't change a lot of stuff (and a number of the things you can change just segfault CPython). I think JS is more dynamic, for example. Or Ruby.
- icebraining 9y ago
  These are not my arguments, mind you; I don't know enough to make them. You've piqued my interest, though: can you give me an example of those things that you can't change or that break CPython?
  - dom0 9y ago
    Things like the Carlo Verre hack (also a thing you can't change —any more— in Python: builtins), editing objects during their construction (via e.g. gc)... generally, the gc module allows other ways as well to crash your interpreter. >>> import gc >>> 'foo'.lower() >>> gc.get_referents(str.__dict__)[0]['lower'] = str.upper >>> 'foo'.lower() segmentation fault (core dumped) python (That's the method lookup cache) A talk in this direction is https://www.youtube.com/watch?v=qCGofLIzX6g https://www.youtube.com/watch?v=qCGofLIzX6g

jayflux 9y ago

I get the point this guy is making, but if you need something parallel for a cpu bound task, throwing more hardware at the problem isn't the most efficient solution if you can just use more cores. For example adding another quad core when the first cpu is only using one core anyway is inefficient and expensive. Right tool for the right job I suppose.

nhumrich 9y ago
Python does multiprocess very well. You can easily use all cores on your machine. Pythons main "disadvantage" is threading because of the GIL. But each process gets its own GIL. So when you multi process, your not limited to one core.
- classybull 9y ago
  This. I had a problem where I needed to scrape roughly 20,000 html documents daily, which is normally a pretty slow task. You have to open the file, load it into memory, parse the DOM, and then run all of your selection methods. Sequentially, it took about 60 minutes daily. Multithreading slowed it down because it was CPU bound. Multiprocessing allowed me to run 12 processes across 8 cores. That took the total processing time down to about 4 minutes or so. And I was able to write the code in a day. Writing something similar in Java or C++ would have taken me a week.
aeorgnoieang 9y ago
The point is that (modifying existing code or writing new code to) "just use more cores" may be less efficient, for the business or organization that is employing the programmer, given that programmer salaries, over even a fairly short amount of time, can be more expensive than hardware.

Waterluvian 9y ago

Python is my Swiss army knife. I love it because it is a single tool that can aid in almost every project I do. But if I'm doing one specific thing a lot, I want that thing to be done well and done efficiently, so I'll reach for the specific screwdriver I need. Also most of my problems are IO bound so single threaded concurrency is fine. But I represent a very small portion of the global problem space.

_pmf_ 9y ago

Somewhat ironically, Python is used a lot for things that would benefit from raw speed (data processing pipelines) and do not benefit at all from dynamic typing (since the kind of property bags / data frame views over data are easily replicated in statically typed languages). But Python's C extension API is quite a bit easier than p.e. Matlab's MEX API (to me at least); can typical Python IDEs compile and relink extension modules without an external build step? > Your bottleneck is most likely not CPU or Python itself. With applications that are dominated by raw data processing, it's very, very easy to be CPU dominated. Hell, I had one quite trivial data converter for logfiles where the "parsing the printf string" part of Java's printf dominated processing and writing a custom formatter halved processing time (while regexes can be compiled, the format string cannot be precompiled and will be interpreted each time); it's one of those things where I would intuitively say "why did this moron write his custom formatter" if I stumbled upon it in a code review. Intuitively, you'd expect this to be a simple case of an IO dominated task (which it is now once the bottleneck has been removed). If it's fire-and-forget batch jobs, you can get away with it, but if the converter is part of a user facing fat client application that runs on a old office laptop, you don't have that luxury.

freetime2 9y ago

I pretty much agree with everything in the article - except for the bit where he tries to quantify why python is better from a developer efficiency perspective than other languages. The main example he cites is a study that compares the amount of time writing string processing routines in different languages - which is quite a bit different from the work I do every day. I develop web apps which means I generally work in very large code bases, and spend most of my time modifying existing code rather than writing fresh code from scratch. I have found that statically typed languages (java + typescript) and the fantastic IDE support that comes along with them make it really easy to navigate around the code and refactor things. Also - the compiler tends to catch and prevent a whole class of bugs that you might otherwise only catch at runtime in a dynamically typed language. Of course there are other situations where I prefer to use Ruby as my scripting language of choice - it all comes down to using the right tool for the job at hand. Unfortunately I don't think the author gives enough consideration to the trade-offs between static vs. dynamically typed languages, and I think he would have been better just leaving that section out as it isn't really necessary to prove his point that CPU efficiency isn't important in a lot of applications. Ultimately though I completely agree with his main point: "Optimize for your most expensive resource. That’s YOU, not the computer."

scarface74 9y ago

I was all ready to savage his opinion after reading the headline but I agree looking at my architecture that I designed for the company I work for, CPU isn't the bottleneck. Every time I try to increase performance by multi threading as much as possible, the databases start screaming. On the other hand, the idea that dynamic languages are more productive than static languages are laughable. Statically type languages prevent a lot of bugs and allow for a lot of automated provably correct refactorings that simple cannot be done with a statically typed languages. You can't even reliably do a "find usages" of classes using a dynamically typed language.

simonh 9y ago
'Dynamic languages' is too often used as a shorthand, or interchangeable with 'scripting language', as here. There are a ton of more relevant language features when it comes to productivity, automatic memory management being the biggest IMHO. Interpreted vs compiled makes a difference too because your code->launch->test cycle can be so fast. But I agree, I'm a huge Python fan and even though I appreciate not having to gum up my code with type declarations, it's not really that big of a win in terms of productivity.
- rockostrich 9y ago
  But with a statically typed language, you don't need to go through that code -> launch -> test cycle as often because everything is type-safe.
  - deleted 9y ago
    [deleted]
  - hpcjoe 9y ago
    Uh ... no. Type safety is rarely a problem from my personal experience (YMMV) in code dev. The issues I run across have to do with whether or not the code is an accurate reflection of the algorithm in question, or even if the algorithm itself is correct, if core assumptions are correct, dealing with corner cases, etc. Types rarely have anything constructive to add to this mix, regardless of whether or not I am working in a weakly/non-typed language, or strongly typed language. I personally tend to do something much closer to TDD, and break my code out so I test each part with its own test rig. So I see when I do dumb things that need re-work. Again, typing is pretty much never an issue there. While this is all anecdotal, I personally have found that enforcing boilerplate type defs actually increases my coding error rate. More LOC gives you more opportunities for errors. FWIW, Julia (http://julialang.org http://julialang.org) is IMO Python done right, and it is strongly typed. But done in such a way as to not actually get in your way most of the time. Like Python it has a great REPL, fantastic FFI, rapidly growing module list, very active development. Unlike Python, it has no issues with whitespace[1], is compiled (JIT for now, though static is planned from what I understand), is very fast, has multiple dispatch and a strong typing system. The real point the article was making is, programmer time is the most valuable thing to optimize for. I completely agree with this, even though I disagree that Python is the right tool for (most) every job. There are better tools IMO, which allow me to be far more productive, and spend as little time as possible on worrying about types ... which are rarely ever a concern for me. I thought his comment about string processing was interesting, though he completely ignored Perl in the mix. I've found string processing in Perl to be almost trivial. Its harder (far more verbose boilerplate) in Python, and gets worse in other languages. Perl6 lets you embed grammars and write parsers without resorting to external modules (Marpa::* in Perl5). Takes building string processing, parsers, etc. to a whole new level, while making you even more productive. But, as the author had noted, he is a pythonista. Everyone has their biases (including me). I want to code correctly quickly, and not have things that shouldn't get in my way, get in my way. Including type systems, massive boilerplate/odd syntax rules, etc. [1] It is 2017 ... structure by indentation is somehow, still a thing. It shouldn't be IMO.

traverseda 9y ago

There's are still some big gains python could make, if python implementations were better. Micropython is equivalent to a real-time cooperative-multitasking OS. If it had ~~better~~ support for things like cffi, you could implement posix on top of it. I can imagine a laptop that runs gnu+python in the next few years. That's a whole new usecase, simply because that implementation uses a lot less ram. What usecases would we discover for a faster python? Shared objects and proper sandboxing would also be huge.

jerf 9y ago
"There's are still some big gains python could make, if python implementations were better." At this point, I would find it far easier to believe that you are underestimating the difficulty involved in what it takes to speed up Python than that there are enormous gains yet to be had in speeding up Python. I suspect JS has had more optimization effort expended overall, but Python has still had a ton of work by lots of smart people, and generally got an earlier head start on optimization. (They didn't start trying to make JS "fast" right away; they spent rather a lot of time getting JS's hookup to the DOM in the face of things like .innerHTML working first, before anyone even cared to do what we today do routinely without thought in plain ol' Javascript, let alone with our glorious frameworks.) There are enormous gains to be had in speeding up "a language that is like Python except certain things are banned", but people have already done that analysis too and discovered that broadly speaking, if you do that, too much existing Python breaks. If you want to see something like that, check out the RPython aspect of the PyPy project, which successfully implements a fast subset of Python. But it is a noticeably restricted subset of Python; AIUI it's not even close to something you can just drop in to your code and get faster speeds. One of the things that I've learned from Python and the other attempts to speed up the scripting languages is that despite the mantra, yes, there is in practice such a thing as an intrinsically slow language. (The theoretical existence of a Python interpreter that can run all existing Python code at C speeds doesn't do us much good if we have no idea after decades of very smart people banging on the problem how to manifest it. Personally I'd suspect that while such a beast theoretically exists it has an exponential complexity startup cost or, if you prefer, exponential compilation costs. And probably a pretty decent code and/or RAM bloat cost, too.) And Python is one of them. Some of the reasons why it is so much fun to use are part of that intrinsic slowness. Some of them really aren't. I personally think there's a lot of up-and-coming languages that are exploring the space of how to get those nicer programming abstractions and programmer-convenient code without paying anywhere near the runtime cost that the dynamic scripting languages of today do; it's one of the more exciting developments I see coming up. People complain a lot about code bloat and poor performance of our code since right now we have to choose between "fairly inconvenient but fast" and "convenient but slow and bloated". Patience! Better choices are developing, but they're still young.

hellofunk 9y ago

Yeah? Well, Jimmy Crack Corn and I don't care.

11235813213455 9y ago

Nice article, I'm more in JS nowadays, it's probably now as good as a scripting language than Python

agentgt 9y ago

Unless we are talking like circa 1999 I don't think I have heard a complaint yet that Python is slow. I'm curious who or where the author heard that from (not specifically the people themselves but the domain they are in). What I have heard complaints about Python are (and I don't agree with all these points): * Its not statically typed * The python 2/3 compatibility * It has some design flaws: GIL, variable assigning, mutable variables, lambdas, indentation (I don't agree with all these but this is complaints I have heard). * The plethora of packaging (ie its not unified) I guess one could argue its slow because it can't do concurrency well but that really isn't raw speed. Then the author started comparing string processing of programmer time from a study which... doesn't help the authors point at all. * Python has and will always be fast at string processing and most people know this * The people that complain about python speed are almost certainly not doing string processing * I have serious questions about the study in general (many languages have changed quite a bit since then)

wheaties 9y ago
Oh, I've heard "Python is slow" from Node folks, Java folks, Scala folks (in terms of productivity) and of course, die hard C++ fans. I'm a fan of Python but i also like languages with a stronger FP bent (and statically typed.)
- agentgt 9y ago
  > Oh, I've heard "Python is slow" from Node folks, Java folks, Scala folks (in terms of productivity) I have hard time believing Java folks made an argument on productivity unless you meant for that parenthetical statement to be only applied to Scala. Regardless productivity is a far more complicated than raw speed (particularly if you get into maintenance as some languages are easy to pump out code but harder to maintain... oh perl.. the pain...). And yeah I have seen the C++ folks complain about Python but this in my experience has been video game programming which is clearly not the domain the author is in or talking about (I think given the microservice discussions). Hence why I would like to know more about these folks complaining.
rockostrich 9y ago

nadam 9y ago

"It used to be the case that programs took a really long time to run. CPU’s were expensive, memory was expensive. Running time of a program used to be an important metric." As hardware gets faster we give it new tasks that could not be achieved before. Like rendering high resolution stereoscopic images using physically based shading at 90 FPS on relatively cheap consumer hadware (VR). There are still quite a lot of code that we call 'performance critical'. Most of that code is written in C/C++ (and CUDA and glsl, and hlsl, etc...) today.

carlmr 9y ago
Yeah, also we have a lot of code that a lot of engineers in our company run. Some of it takes hours instead of a few minutes to run, because it's a bunch of slowness combined. Sure you can say the engineer doesn't have to watch. But sometimes you need this result, and can't do much else until it's there. Doing a few loops at a few hours at a time is very time consuming. Being lazy about slowness is bad in the long run, if you or your customers waste inordinate amounts of time, because one of your devs didn't want to think. Sure you should optimize from the bottleneck, look at hot path, measure where optimization makes sense. But using tools that make some stuff 10x faster without much effort is not premature optimization. It's just not being stupid.
ivm 9y ago
It's still expensive on client machines because most of the persons in the world are NOT software engineers with 6-digit salaries. They run cheap computers with HDDs and Windows polluted by a ton of 3rd party crap. They don't know how to fix it and silently suffer. I was cleaning a local vet clinic's devices recently – they were literally switching between two computers to not wait 5 minutes of non-responsiveness because some bloated software was occasionally consuming 100% of CPU.
- mark-r 9y ago
  A lot of businesses these days prefer web apps. It's not hard to understand why - all the hassle of system maintenance falls to the people who host the app and can afford to know their stuff. If your Windows PC is suffering from rot just replace it with a Chromebook.

fiatjaf 9y ago

> It’s more important to get stuff done than to make it go fast. This is not a real absolute. It is only valid when what you have to run will not benefit a lot from performance or suffer a lot from lack of it. The real guidance you can have in these matters is: how many times is my code going to run per second? Some programs are written to be run once a day, others 10000000 times in a second. The first ones should be written in the language you're most productive in, the second ones in the fastest possible language.

donatj 9y ago

I know it's not trendy, but I would argue PHP is as productive for developers as Python and has a MUCH faster runtime, particularly after 7.

__s 9y ago

> without getting stuck in the weeds of the small things such as whether you should use a vector or an array Yes, instead get into the weeds of tuple vs list Not included in the graph of time-to-solve-problem static languages: statically typed languages with type inference

scbrg 9y ago
Given that they have exactly the same interface, that choice is really easy. You go with one until it turns out to be insufficient, and then you switch to the other and not a single line of code has to change, except at the point where you create the thing. Incidentally, the same is true in many situations in Python, and that is (IMO) one of its strengths.

snarfy 9y ago

Slow doesn't matter when you scale horizontally.

nhumrich 9y ago

Author here. Surprised to see this toping HN. Appreciate all the feedback. Let me know if you have any questions.

kodablah 9y ago

The article could be titled: "Yes, Python is Slow To Refactor and Maintain, and I Still Don't Care". I never understand why dynamic language enthusiasts primarily focus on new code only. You have to discuss all sides of increased or decreased productivity to make a rational argument.

hasenj 9y ago
Python is optimized for getting interesting things done in a few lines of code. Small scripts you write once and then forget. For serious projects? IMO python is a disaster.

Thaxll 9y ago

Giving EvE Online as an example is bad because that game artificially slow the game loop to keep up with the number of players, would this happen with C++ on a recent architecture? Probably not.

nervous123 9y ago

This sentiment is the reason for almost all software (especially on the web) beeing a load of crap. It's slow, it's buggy and developers always give the same excuse: CPUs and memory are cheap, therefore we can waste our customers time. Imagine what we could do with the amazing hardware we have, if people started to do the sane thing and actually use the hardware to do things efficiently.

deleted 9y ago

[deleted]

dahart 9y ago

Python's value to me has always been that it's easier to get things done, not it's speed. One time when I was interviewing a candidate for a coding job, the candidate said she loved Python the most "because you can just yell at it and it'll work." It's both the breadth of the standard library and ecosystem, and the simple language design, that make developing things in Python faster for me. Doing problems on Project Euler has been an education for me in how algorithm matters more than speed. Lots and lots of people spend hours writing long C++ codes that are easily beaten by a few lines of Python. It certainly goes the other way too, and the wrong algorithm in Python is even that much slower and more painful than the right algorithm in C++. But when the right algorithm is used and the problem is solved in a few milliseconds, it really doesn't matter which language uses more CPU cycles, all that matters is whether you saw the insight that let you skip 99% of the search space, and how much time you spend writing code.

progman 9y ago

Yes, time to market is important. However, you don't need to compromise convenience of development for the sake of performance. If you twist your Python code to get performance it takes time. If you need performance, and like the syntax of Python then you should take a look at Nim [1]. With Nim I develop as quickly as in Python while I get the performance of C. [1] https://nim-lang.org https://nim-lang.org I believe application performance is important on servers. It makes a difference if your Shop software written in Python is able to handle 50 requests per second, or if the same software written in Nim can handle 500 rps. And by the way, Nim provides static typing which helps a lot to catch errors at compile time.

cup-of-tea 9y ago
Nim seems really good, but does it have a decent REPL these days? I'm not sure if it would be as convenient with a statically typed language, but I like the incremental development approach so much that I only use C if I absolutely have to.
- dom96 9y ago
  It doesn't. But you can grab Aporia (or some other tool) to quickly compile and run some code, it replaces a REPL very well in my experience.

bluedino 9y ago

Many times when Python is blamed for being slow, it's the programmers fault. Python is great that you can 'regular' people writing code in it quickly. The problem is, these regular people don't always understand algorithms or things like caches, threads, databases... A lot of these users can just say "My department needs a $40,000 24 CPU server with maximum RAM from MicroWay/SuperMicro, we need to run our codes faster", when they are just trying to brute force things. They understand the problem domain but don't have the programming skills to use a computer to efficiently solve it. But, these guys are all a step ahead of the ones who are stuck in the mindset of "C is the only language fast enough for my work", while not even understanding pointers and basic syntax and getting stuck on silly things like text processing, which could be done in minutes in Python.

sedlich 9y ago

In "What if CPU time is an issue?" we could also mention the nim language (and not only cython) because it compiles (not only) to C and feels like python.

deadsy 9y ago

Premise: It's more important to be productive than to have fast code. Conclusion: Use Python. Is the premise true? For many cases- yes, but it depends. If you are running an application on the cloud and your metric is $/user/year and you have many users then saving some compute resources for each user gets attractive and you don't want to just throw another VM at it. Is the conclusion true? Garbage collection gives big productivity gains. Other languages have GC. It's not nice to see your Python code die after a few days because you messed up the type passed to a function. Other languages fix that at compile time. Multicore is now. Other languages are built with better multicore awareness.

boomlinde 9y ago

The author argues from his professional experience as a Python developer that it's fast enough, that you'll spend most time waiting for I/O anyway, that you can just throw more servers at the problem etc. The problem is that his experience as a Python developer doesn't accurately reflect the prevalence of problems where runtime CPU performance actually is an issue. Of course not, because who in their right mind would make an informed decision to solve such a problem in Python? Python has worked for him because it is only useless for a category of problems that he hasn't had the opportunity to solve because he's a Python developer. Outside this professional experience, not everything is a trivially parallel web service that you can just throw more servers at if CPU time exceeds I/O waiting. It all really boils down to what your requirements are, whether you have all the time and memory of a whole server park at your hands, or a fraction of the time available in a smaller embedded system, how timely the delivery of the software has to be and how timely it needs to deliver runtime results once it's up and running. There are times where Python just isn't fast enough, or where getting it fast enough is possible, but more convoluted and tricky than implementing the solution in a more performant language. Developer time may be more expensive than the platform that my solution is for, but that doesn't get around the fact that it eventually will need to run with the available resources.

agnivade 9y ago

> However, this is no longer true, as silicon is now cheap. Like really cheap. Run time is no longer your most expensive resource. Our client won't spend more money than a t2.medium instance on aws. Nothing we can do about it. In that case, run time does become an expensive resource. But I get the point that OP is trying to make. Just wanted to mention that not all of us have the comfort of having enough resources on which our app runs.

karmakaze 9y ago

Putting aside the discussion regarding productivity, there is a case where I have found execution time to matter. Scaling an application which uses an unsharded database. The long transaction durations and number of connctions were bottlenecking db throughput. The particular app was a Ruby/Rails monolith.