25 ms·

Python’s Weak Performance Matters

• matt_wulfeck 9y ago
  > The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.
  • thinkpad20 9y ago
    How so? Haskell has a very high performance ceiling.
    • gameswithgo 9y ago
      it does have a high perf ceiling, but the code isn't easy to write when you approach it. Things like C#, F#, Java, Kotlin, Nim, Lua would be more natural things to turn to when you want something "Easy" like python but faster, I think.
      • nightski 9y ago
        I think you could replace "isn't easy" with "isn't familiar". Haskell is very likely to be the first language a developer encounters which is lazy by default instead of strict by default (for many good reasons). So no, it's not easy in a similar fashion that pointers or double pointers in C/C++ are not easy. Or understanding call by value vs. call by reference semantics are not easy. The list goes on. It's probably the largest barrier to learning the language. But once you get a handle on the evaluation model it becomes a lot more natural. At least that was my experience, maybe it is not typical.
      • gaius 9y ago
        I use OCaml when I need something like a fast, type-safe Python
    • banachtarski 9y ago
      It really doesn't... I think it's "better" but I definitely wouldn't classify it as high.
  • bringtheaction 9y ago
    Me too. I have heard from someone that used Haskell in production that lazy evaluation caused trouble in terms of achieving predictable performance.
    • nightski 9y ago
      Yes, naive use of lazy evaluation can cause performance problems. So can naive use of strict evaluation. It's important to have a solid understanding of your languages evaluation model. This is probably the largest barrier to writing highly performant Haskell. Not because it is vastly more difficult or anything, but it is very different than pretty much any other widely used language.
  • zwerdlds 9y ago
    Not saying you're wrong - you definitely don't get great performance easily in Haskell, but it seems to have better benchmarks than python, by a lot.
    • ohyes 9y ago
      “Lies, damn lies and benchmarks” I think is the saying. Python itself is a slow language but it has a lot of fast packages, so it shows poorly when you actually write your benchmark in python. Haskell is a faster language but because it is high level there are more pitfalls you’ll get into if you don’t know the ins and outs of getting fast code out of the compiler. The guys who write fast benchmark code aren’t ‘average’ developers. So in Haskell it’s “the code is slow and I don’t know why” vs python “the code is slow because python is slow, import fast package someone wrote to speed it up.” All that said, I think Haskell is the better language but you have to put in more effort to get experienced in it before you see returns on the investment. Python has a shallower learning curve and an easy way to get “good enough” performance (a bit slower than C). The best criticism in the article is of the multi-core deficiency of python’s interpreter. But that’s only briefly touched on. It isn’t a friendly environment to write complicated multi core code.
      • acqq 9y ago
        > Python has a shallower learning curve and an easy way to get “good enough” performance (a bit slower than C). The article we all reply to exactly claims that as soon as you don't use e.g. NumPy, it's not "good enough" anymore, and I agree with that. The article also argues that e.g. JavaScript isn't more in the same category with Python, but much faster, even if it's not less dynamic. I think the reason for JavaScript's speed vs. Python's slowness is obvious: there were wealthy companies involved, which were, due to competition pressures, motivated to speed up their own JavaScript engines. To get to the point where Python has similar speeds somebody would have to be motivated enough to invest heavily, and then it could happen. As far as I know, there aren't technical limitations against that.
        • pjmlp 9y ago
  • eksemplar 9y ago
    I read it as a compromise. I don’t think the author is primarily a programmer, but mainly uses high-level “easy” languages to do data science. C++ would probably increases his development time significantly compared to Haskell.
  • willtim 9y ago
    GHC Haskell is an advanced optimising compiler, which can get very near the speed of C and C++. However, to write fast programs, one must use the right data structures and algorithms. Often this means array-based strings and streaming IO, unfortunately many Haskell textbooks don't tend to cover this.
    • luispedrocoelho 9y ago
      Yep, my Haskell usage is "conduit all the way down". I even wrote up a few utilities to make use of multiple threads while working at a high level: https://hackage.haskell.org/package/conduit-algorithms-0.0.7.1/docs/Data-Conduit-Algorithms-Async.html https://hackage.haskell.org/package/conduit-algorithms-0.0.7...
    • bjoli 9y ago
      All comparisons of Haskell and C I have seen where Haskell has been near the performance of C(++) has always compared a highly optimised Haskell program to a moderately fast c program. I have found that most managed languages generally come within 2-5 times slower than C and C++. Which is good enough for me.
      • imtringued 9y ago
        The JVM is fast enough for me but for some reason any JVM project requires hundreds of megabytes of RAM even for the simplest things. cpp webserver + sqlite database + cygwin overhead? 9MB RAM lua worker + websocket client? 4MB RAM JVM server + static html page + websocket relay + kurento api? 500MB RAM With the exception of the cpp webserver none of these tasks are CPU or memory intensive yet the JVM is still off by orders of magnitude. It's ok if it's the only application running on a server with multiple users but there is just a single user and that's me.
        • willtim 9y ago
          Have you tried specifying a smaller maximum heap size? The biggest memory issue IMHO, is that Java and the JVM do not specialise any "generic" code and so there can be a lot of unnecessary boxing going on.
  • luispedrocoelho 9y ago
    OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.
    • pletnes 9y ago
      That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.
      • luispedrocoelho 9y ago
        If I can fit the code into numpy-like structure, then Python is typically fine. The issue is when I cannot.
        • fxmc 9y ago
          Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.
          • VHRanger 9y ago
            Dask doesn't solve that problem since it's a wrapper around pandas functions. If you can't make the core pandas code decently fast, dask won't save you.
            • kiriliponi 9y ago
              dask.dataframe might not help but dask.distributed could in that case. I've had success using it on non vanilla stuff (i.e. code that could not get converted to play natively with numpy/pandas structures) As a bonus, the nice profiling tools (built within dask) have also helped me improve the performance of the code. See https://distributed.readthedocs.io/en/latest/ https://distributed.readthedocs.io/en/latest/
• Insanity 9y ago
  does this not come down to using the right tool for the job? I love writing in Python but would not use it for something where performance matters.
  • dfinninger 9y ago
    I'm with you here. I tend to use Python, but if I need raw speed/concurrency I'll reach for something else. Currently Go.
  • anameaname 9y ago
    It's hard to tell ahead of time if performance matters. When you hack something out, and are able to get it to work, it inevitably starts to grow features. At some point (around 1000 lines in my experience), Python no longer is fast enough, but the code is difficult to port to something faster. It would be much better if you didn't have to back track on all the code you wrote. I've been burned by this enough times, that now I am wary of starting anything in Python, because I know it will grow to be bigger, and I will regret having picked Python!
  • luispedrocoelho 9y ago
    I think this is part of my argument: as datasets grow faster than single-core CPU speed, performance matters more and more.
  • collyw 9y ago
    Exactly. For web dev stuff, the database is usually the bottleneck. Caching is the next lace to put your efforts. I think I have had one problem where Python was too slow.
• indubitable 9y ago
  Something I often wonder in these sort of discussions is why C# is generally omitted. Its performance is comparable to C++, with none of the trappings. It also does an excellent job of integrating some of the most useful features of functional programming into an imperative language. And multi-processor programming with the language is also incredibly simple. But I think the best part is in programmer time. An anecdote I find endlessly entertaining is on another forum I shared some code to solve a problem people were having an issue with, and it was assumed my code was pseudo-code. It was correct, compilable C#. And they're constantly adding incredibly useful features. For instance a recent addition is more expressive tuples: (int number, string s, char c) triple = (2, "two", '2'); triple.number = 13; And lambda functionality is similarly clean. A lambda value might be: x => 2*x + 5; Equivalently, as an anonymous method: delegate(double x){return 2*x + 5;} And an example of how simple arbitrary processor count parallel programming can be (using lambda syntax as above): Parallel.ForEach(listOfThings, thing => DoSomething(thing)); Yet as typical in scenarios like this one, the author sees the decision as being between opposite extremes of C++ and Python. The only downsides of the language I've run into are a lack of some shoot yourself in the foot features of C++, like multiple inheritance, and the fact that template specialization is awkward. Garbage collection is vastly overblown. My main work is with projects that have in memory collections in the gigs of size and you'd think the collector would be a huge issue, yet it's mostly transparent and can be controlled if necessary - which in the vast majority of cases, is not.
  • blub 9y ago
    C# started out as a boring Java clone enterprise language from the company famous for its "embrace, extend, extinguish" strategy. In fact they invented C# because they got sued for trying to "extend" Java. And until now no amount of cool features and Microsoft PR were able to remove that stench.
  • pletnes 9y ago
    The author is a scientist analyzing his data. I never met anyone in that crowd using C#. Are there even any good data science/numerics libs out there? C++ has a lot of number crunching libs, python even more.
• hyperion2010 9y ago
  I recently discovered that pypy3 can run all my day to day Python code. It has some issues with slightly different behavior from cpython when using threads but other than that I see a 4x speedup on most of my slowest pure python workloads (parsing large rdf files and reserializing them after computing a total order on all their nodes). Huge win for productivity.
  • civility 9y ago
    > I see a 4x speedup on most of my slowest pure python workloads Heh, only 25..250 X to go. We did a direct line for line translation of some numerically intensive code from Python to C++ and saw a literal 1000X speedup. On other projects, it's been more like 100X slower. That says two things: first Python can be really slow, second, for some programs, Python doesn't really save on lines of code over modern C++. I've been very impressed with PyPy however. In testing, it can sometimes sneak up to less than a factor of 2 slower than C. However, the bummer comes when it doesn't hit that mark and you have no idea how to trick the JIT to do better. If it works, great. If it doesn't, you don't have much insight into why. Finally, I've always been able to get Cython to parity with C++. However, when I'm done, I wonder what I gained. The C++ isn't that much more complicated than adequately type annotated Cython.
    • joshuamorton 9y ago
      While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.
      • pjmlp 9y ago
        Depends, I bet they can be easily translated to some LINQ like implementation in C++17. List comprehensions are just syntax sugar for map/filter/fold.
• pletnes 9y ago
  There’s a project to plug different JIT compilers into CPython, so there’s hope. https://github.com/Microsoft/Pyjion/blob/master/README.md https://github.com/Microsoft/Pyjion/blob/master/README.md Also, I’ve more than once seen cpython beat C++/Fortran since it’s easier to do the right algo/datastructure things, plus numpy is more optimized than most «amateur» C loop-over-arrays. That being said, faster python is always welcome.
  • gh02t 9y ago
    Honestly, NumPy is gonna be hard to beat even for someone knowledgeable in certain use cases, especially ones where the overhead in Python is trumped by time spent in library calls. It's the same reason that it's hard to beat MATLAB or Mathematica in cases they are optimized for despite being relatively slow languages. They are calling some of the most heavily optimized libraries in existence (e.g., BLAS) and using heuristics to help choose the smartest evaluation strategy. Edit: More speed on the Python side is good though, because it gives you flexibility. Sometimes it's hard to figure out how to do stuff optimally in NumPy, versus just banging things out in a for loop. I've definitely done that when I wanted something to just work, versus spending an hour figuring out what arcane incantation I need to pass to np.einsum to get the operation I want.
    • luispedrocoelho 9y ago
      "spending an hour figuring out what arcane incantation I need to pass to np.einsum to get the operation I want" Yes, I have also had this experience and I hate how in the end, the code is very hard to read, while the for loop was probably trivial.
      • cjalmeida 9y ago
        Every time I thought I needed einsum or similar arcane ops, I found that a Numba optimization for loop did the job.
        • gh02t 9y ago
• ChrisSD 9y ago
  I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's frustrated with Python's performance but it would be really interesting to hear from someone who knows the technology involved rather than simple speculation.
  • civility 9y ago
    If you neglect some of the metaclass stuff, which I believe most people do, then Python is nearly isomorphic to JavaScript. I think the comparison is very fair. I also believe the reason Python is unlikely to ever catch up to JavaScript is the same reason that CPython will always be the dominant implementation - They've exposed so much of the C internals, that everyone is bound to the actual slow and single threaded implementation. JavaScript implementations in web browsers can do lots of magic behind the curtains because the majority of users don't rely on the actual innards being consistent from release to release.
  • shakna 9y ago
    Viper is an interesting approach on speeding up Python. It's developed for MicroPython, which does give them room for breaking changes, but has trade-offs. Arithmetic is much faster, but dictionary lookups take much longer compared to CPython. Viper is a code-emitter from a large subset of Python, and even allows for inline assembly. But it's only for a few architectures at the moment, like ARM and x86.
    • cromat3 9y ago
      Viper is called Zerynth today: https://wiki.python.org/moin/Zerynth https://wiki.python.org/moin/Zerynth
      • shakna 9y ago
        They're similar, but not the same. Zerynth is a development suite. Notably, it makes use of a VM. Viper is just one of the code emitters buried inside the MicroPython source code, like here [0]. Notably, it produces native code, not bytecode for a VM. [0] https://github.com/micropython/micropython/blob/master/py/objfun.c#L417 https://github.com/micropython/micropython/blob/master/py/ob...
• knicholes 9y ago
  When I was taking a python class in school, the professor did something to generate C code from the Python code, and it gave something like a 60% speedup.
  • repsilat 9y ago
    Probably Cython. Sometimes it helps, sometimes it doesn't work (doesn't like generator comprehensions iirc) but mostly it provides a "sliding scale" into C or C++ land -- after the first compile, you can start littering type declarations around the code and you can stop whenever you hit the speed you want. It has been a good solution for me in the past, but probably only because I started with a Python codebase. If I were starting afresh I wouldn't bother.
• cturner 9y ago
  Where are the main blowouts in python performance? For example, is it compilation, evaluation overhead, or memory management?
  • shalabhc 9y ago
    A good writeup is here: http://faster-cpython.readthedocs.io/notes_2017.html http://faster-cpython.readthedocs.io/notes_2017.html
  • PyComfy 9y ago
    Naive interpretation of the bytecode (not even pre-decoded, just a switch statement). And almost everything is resolved in the dynamic environment. for example, a = foo.bar(b) is actually ldict = locals() ldict['a'] = ldict['foo'].__getattribute__('bar')(ldict['b'])
    • tom_mellior 9y ago
      This is a bit misleading. You suggest that local variables are looked up by name in a dictionary, which is not the case. They are looked up by indexing into a C array, with the index being a constant in the bytecode. That's quite a lot simpler. Here is the corresponding code (look above for the definition of the GETLOCAL macro): https://github.com/python/cpython/blob/fc1ce810f1da593648b4d19e7d582a235ec1dd37/Python/ceval.c#L1066 https://github.com/python/cpython/blob/fc1ce810f1da593648b4d... So your code should be more like: locals[a_idx] = locals[foo_idx]->__getattribute__('bar')(locals[b_idx]) But this isn't a very good rendering of the thing because it doesn't show the many redundant reference count increment/decrement pairs every time you touch a variable. (Also, interpreter dispatch uses computed GOTOs instead of the plain switch on C compilers that support it.)
      • PyComfy 9y ago
        ah sorry. i did import dis dis.dis("a = foo.bar(b)") which gave 1 0 LOAD_NAME 0 (foo) 2 LOAD_ATTR 1 (bar) 4 LOAD_NAME 2 (b) 6 CALL_FUNCTION 1 8 STORE_NAME 3 (a) 10 LOAD_CONST 0 (None) 12 RETURN_VALUE
• donarb 9y ago
  At PyCon US last year, Intel had a booth promoting their version of Python and data libraries optimized for Intel processors. Wonder if any of this would have helped. https://software.intel.com/en-us/distribution-for-python https://software.intel.com/en-us/distribution-for-python
  • ddorian43 9y ago
    I don't think they made raw-python-code faster (like pypy) but bundle some libraries and make them faster (like they have some if cpu=intel: enable optimizaion they've done elsewhere).
• hidenotslide 9y ago
  I don't find this a very compelling argument. The author doesn't mention any attempts to profile or speed up the code. Specifically with pandas I've found if you aren't careful you can do a lot of unnecessary copying. Not sure if that's what is going on here, but cProfile can help find the bottlenecks.
  • joshuamorton 9y ago
    Seconding this, there are a couple of things that jump out at me as immediately non-optimal, and which together would probably give an order of magnitude speedup. - Defining compute_diversity inside a double for loop - `sample1.ix[sample1.index[sample1.index.duplicated()]]` appears overengineered (I think you can just remove the `sample1.index` here (edit: you can't , but I think you could refactor to remove the indexing and reindexing and index resetting, and then you could)) - Depending on the data size, swapping from `[` to `(` everywhere would give a nice speedup just because you no longer need to store everything in memory/swap to disk, whereas in haskell the list comprehensions would be lazy by default. (edit: seeing as the databases downloaded are 12 and 33 GB, and Pandas requires generally 2-3X ram, its likely that there's swapping happening somewhere. I'd bet that using generators would be a big speed boost) - Overall I think genetic_distance can be significantly simplified, a lot of the index-massaging doesn't look necessary. I could be wrong, but this looks sloppy, and sloppy often implies slower than necessary. Unfortunately, the provided data files are big enough that I can't easily benchmark on my computer. I can't even fit the dataset in memory!
    • luispedrocoelho 9y ago
      You are commenting on the variant of the code that is fast enough that it doesn't matter.
      • joshuamorton 9y ago
        While that may be true, my point is that it is almost certainly possible to make your code go faster than it is already, and also become more readable in the process. And so saying that python is either slow or ugly and unreadable is perhaps an unfair characterization. I may be wrong here. I haven't benchmarked the code in question, but I think that even for the algorithm you're trying to do, with the special casing, that function could be significantly simplified. Edit: I'd be curious to see example data that is passed into this function.
• fermigier 9y ago
  The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.
  • hoschicz 9y ago
    Do type annotations really make any difference to the interpreter? I thought that the interpreter doesn't care about what type a variable is annotated to...
    • klibertp 9y ago
      Cython compiles to native library exposing Python extensions API.
    • deleted 9y ago
      [deleted]
    • dragonwriter 9y ago
      > Do type annotations really make any difference to the interpreter? Note the advice was first use Cython (a compiler for a superset of Python), and then tweak as needed with type annotations. Cython’s compiler definitely uses type annotations.
  • kankroc 9y ago
    I don't understand why your comment is so low. Cython is the easy-to-implement solution here.
• banku_brougham 9y ago
  I’d like to thank the author for sharing a very practical view of problem solving in the data science space. Can I suggest julia? Its very easy to understand coming from python, and performant code can be had usually in easy to read implementation of the expressions in whatever paper you are basing your work upon.
  • loeg 9y ago
    Anytime someone brings up Julia, I think of Dan Luu's review of the language: https://danluu.com/julialang/ https://danluu.com/julialang/
    • myopicgoat 9y ago
      From 4 years ago... a lot has changed since. Consider this recent post for example: https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-for-the-giving-up-on-julia-blog-post/ https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...
      • loeg 9y ago
        The linked article doesn't attempt to refute the significant claims from Dan's article (which has an update from a year later, so 2015, at the bottom): 1. That the language is (was) undertested, and as a result, full of easy to run into bugs 2. The language makes it easy to ignore errors 3. APIs are inconsistent 4. The head branch isn't kept build-clean (i.e., often it fails to build) 5. Code is often undocumented, with non-descriptive naming. The combination makes it difficult to understand what code is doing. Dan also writes that he expects the performance issues would be fixed and those aren't what concerns him: > The purely technical problems, like slow load times or the package manager, are being fixed or will be fixed, so there’s not much to say there. And bonus community problem: > Update: this post was edited a bit to remove a sentence about how friendly the Julia community is since that no longer seemed appropriate in light of recent private and semi-private communications from one of the co-creators of Julia. They were, by far, the nastiest and most dishonest responses I’ve ever gotten to any blog post. Some of those responses were on a private discussion channel; multiple people later talked to me about how shocked they were at the sheer meanness and dishonesty of the responses. Oh, and there’s also the public mailing list. The responses there weren’t in the same league, but even so, I didn’t stick around long since I unsubscribed when one the Julia co-creators responded with something bad enough that it prompted someone else to to suggest sticking to the facts and avoiding attacks. Now, I don't know if any or how many of those issues have been fixed since 2014-2015. But the 2018 blog post about microbenchmark improvements doesn't really address these concerns.
• ivanb 9y ago
  I have an impression that there are features in Python that give very little programmer productivity but make the language slow. It should be possible to implement a hypothetical FastPython without such features but with great performance gains. Of course it wouldn't be compatible with most of the libraries. I can imagine though that porting most of the libraries to FastPython still would be a manageable task. I wonder if such projects had been attempted.
  • benou 9y ago
    You mean like Cython [1] or RPython [2] from PyPy [3] ? [1] http://cython.org/ http://cython.org/ [2] https://rpython.readthedocs.io/en/latest/rpython.html https://rpython.readthedocs.io/en/latest/rpython.html [3] http://pypy.org/ http://pypy.org/
    • ivanb 9y ago
      Yes, RPython looks close to what I'm talking about.
  • Certhas 9y ago
    Julia goes in that direction.
    • ChrisRackauckas 9y ago
      I'd go even further and say Julia is the instantiation of that idea.
• marmaduke 9y ago
  His example of a function which is unreadable, is pretty typical. It still might be slower than a tight loop in C, but it’s only unreadable the first time you write something like that. That said, Numba would be a natural tool here.
  • klibertp 9y ago
    I'll buy you a beer if you can derive the original formula from this code.
    • marmaduke 9y ago
      I admit I recognized only some general NumPy things like masks, unique, reductions, outer etc: I don’t use Pandas and not sure what the non-Numpy stuff does. I still don’t think it’s more obscure than equivalent for loops or FP folds or similar.
• WillReplyfFood 9y ago
  The problem here is that most language are tailored to easy learning by humans and not to strong performance. If you had a language tailored to strong performance, it would force you to bundle together seemingly unrelated data into structures used in the main hot codepath of the core algorithm. It would then force you to specify data delivery routes and processing- and would craft a cache optimal process loop dependent on the used architecture. The result would seem like a enormous while loop, that takes seemingly arbitrary number of for loops to preprocess data, glued together in strange overleafing unions to shove the endresult to the main process algorithm. This would be the most optimal result for a processor- but to write a language to describe this- and compilers to implement this) - the horror.
  • walterbell 9y ago
    Have you looked at http://terralang.org http://terralang.org?
• dsign 9y ago
  Interesting article, and the comments.... going to research that one about Numpy with MKL....
• tworc 9y ago
  Numba makes python highly performant, and is simple to use. Definitely worth considering.
• munro 9y ago
  > At the same time, data keeps getting bigger and computers come with more and more cores (which Python cannot easily take advantage of), while single-core performance is only slowly getting better. Thus, Python is a worse and worse solution, performance-wise. PySpark is makes it really easy to take advantage of multiple cores & machines. Most operations I want to do to my data I can find in PySpark's pyspark.sql.functions, so I get all the benefits of the JVM. In the cases I need something from Python, I can just UDF, it's a little slower than JVM but still extremely fast when distributed--I find all problems come down to time or memory complexity, which is independent to whatever your programming in. Also, it's very easy to take advantage of spot instances with Spark... I'm usually working with 2-20 spot instances, and sometimes go up to 60 depending on what I'm doing.
  • pcx 9y ago
    This is true, but using a Distributed system like Spark itself adds a ton of complexity in having to understand and manage it. If one can do something with a set of stateless processes, even if it's more performant, I feel it's a bad idea to use a distributed system instead. Not always, but a good majority of cases that I've seen. I've seen projects where Celery would be enough but instead they chose to use Spark/Storm and never delivered.
    • deleted 9y ago
      [deleted]
  • quietbritishjim 9y ago
    The original article said that one reason it doesn't matter that pure Python's performance is poor is that you can use numpy (and pandas) to vectorise things, which then has native code performance. It goes on to say that his current problem is that the things he's doing today can't be vectorised with numpy – so that poor performance does matter after all. If he can't even express his code in terms of numpy operations (and other C-based libraries like scipy), I doubt they're going to be expressible in terms of Spark's primitives, which are a considerably smaller subset.
  • lmm 9y ago
• ggm 9y ago
  To me, the critical quality exposed is the abstraction/synthesis moment of Haskell/types/FP thinking. Python is what I use, but I rely on insights from a Haskell person to get solutions of merit. Left to my own devices I frequently derive python solutions with bad scaling, few and weak opportunitistic parallelism moments and heaps of errors. When driven to think in types and simple function composition the solutions seem to run better
• stesch 9y ago
  Don't forget marketing. "We need to switch from Python 2 to Python 3!" – "Yeah! We do! Faster running programs!!" – "Well, actually …"
• Annatar 9y ago
  Update: Here is a “fun” Python performance bug that I ran into the other day: deleting a set of 1 billion strings takes >12 hours. Buddy, it’s time for AWK.
• narimiran 9y ago
  If you would like 10-100x faster performance than Python, but would like to keep the easy-to-read code, give Nim [0] a try. I do all my work in Python, and I've been using Nim in last couple of months - it took me a week or two until I was able to be productive in Nim. Don't expect Python's large ecosystem, nor some Python goodies, but if you're looking for a readable, writable, high-performance post-Python language - Nim is the way to go! [0] https://nim-lang.org/ https://nim-lang.org/
  • platz 9y ago
    I mean, if you're willing to give up the ecosystem, there are tons of options out there.
    • carlmr 9y ago
      Exactly, the ecosystem is the only reason IMHO why Python is "easy". I find I'm much quicker at developing C#/F# if the library is available on NuGet (which is often, but not always the case). They're both reasonably fast, too. Python only has a very large community and thus ecosystem to offer.
      • dTal 9y ago
        It can't be the only reason, or how would it ever have attracted such an ecosystem in the first place? Especially with its performance characteristics, and lack of corporate backing. No, Python was invented at a time when its closest competitor was Perl - and you need only compare typical Perl with typical Python to appreciate that Python really was a usability revelation. But that was nearly 30 years ago. I do think we can do better now.
        • vgy7ujm 9y ago
          IME Perl can be more readable if you are experienced in Perl.
        • rthomas6 9y ago
          Another reason is the documentation. It's fantastic.
• ptype 9y ago
  I love Python. The language is a joy, the eco system is fantastic. But yes, let’s be honest, if you can not vectorise your code it is slow, and I think that will be its downfall eventually. I’m excited about Julia, I hope it gains popularity and the eco system grows. Until then, and in particular until the data frames story can compete with pandas, it Python with Cython for me, but I’d rather skip the Cython if it was not necessary for performance. Any early adopters running Julia in production with stories to share?
  • wirrbel 9y ago
    This is maybe more hearsay, I only briefly tried to use Julia. It is true, Julia has great features for performant code. It is, however, focussing too much on being a matlab competitor in my opinion. It will not be a language that you use to write a "normal" (i.e. non-numeric or CRUD) dynamic website in. My general observation however is, that you need to attract this crow, if you want to have an ecosystem with a variety of tooling. And it is the neat thing about Python (and Haskell).
  • tekkk 9y ago
    I feel bit ambivalent about Python as it's a nice language for prototyping and quickly hacking things done. Yet I'm always baffled when I read Numpy's or Matplotlib's documentation and try to make sense of it as it can be (or at least feel) so complex and highly ambiguous. Eg. sometimes there is no/very brief examples at Numpy's documentation pages how the method works and most results from Google are only about advanced implementations, not about the basics of the method itself. In Matplotlib I still don't understand what is the right way of initializing a pyplot, there seems to be a million ways to do it and a million parameters you can give. API changes and inconsistencies too pain me at times (Pandas comes to my mind). While not a fault of Python as a language I think they greatly contribute to the experience of using Python. Also I don't feel like the culture of Python programming focuses too much on documenting things which makes reading code at times like transcribing ancient Latin manuscripts. Maybe a good analogy would be JS back in the days with global jQuery scripts. Too unrestricted and free-form maybe. I'd wish Python became more like Kotlin with very clear patterns and great IDE support (in addition to PyCharm). Well those are at least my experiences and feel free to disagree with me.
• AllegedAlec 9y ago
  Interesting article. I mostly agree. OP, could I ask a question? You mention 1TB files. Why do you guys at embl not use a database for this sort of stuff? I'd figure that with some proper indexing, I figure you could see pretty decent speedups just from that already.
  • jerven 9y ago
    Not OP: but working on a downstream project and my current boss used to work on EMBL-bank in the day. A lot of this stuff is in databases. e.g. Oracle and I think for advanced search it was in Teradata. However, databases are hard to share so many steps require dumping the database into some interchange formats (custom and often from before the age of XML or JSON, yeah for ASN.1 parsing!) Sharing database dumps is done but commercial licenses and version mismatches do add issues here as well. Remember EMBL/ENA is older than MySQL. The databases tend to have the wrong shape for the next downstream step i.e. table design is related to work flow and if your next step in working is completely different we end up with issues. Also some data can't be published until a certain date so that needs to be filtered from the dumps in some way. Consider as well that this project is 3 decades old and used to be printed in books at some point, and shipped on DVD as recently as 2004. File based operations can be extremely efficient.
  • xyhopguy 9y ago
    Bioinformatics is perpetually ten years behind. The de facto standard for sequencing data is effectively a stripped down bzipped plain text file. It's madness.
  • luispedrocoelho 9y ago
    For some things, we do. But databases are not magical and setting up a good table/index system &c is also work and there is overhead. Thus, if we are talking about having (for example) a webservice where queries have a form that is known apriori, then it's a good solution. If you have output data from your processing that you will be slicing and dicing in different ways which you cannot predict ahead of time, then, they are not appropriate. (Loading Terabytes of data into a database takes a while too).
• Rotareti 9y ago
  I use a lot of Python for web stuff and I haven't been in a situation where Python itself was the performance bottleneck. I always thought that when you run into a situation where Python is the bottleneck, you replace the critical bits with something like C/C++/Rust. Following this approach, you would get the best of both worlds: rapid proof-of-concept/time-to-marked with the option to improve performance critical parts later (which often isn't necessary). Could anybody share some experience with this?
  • Erwin 9y ago
    It requires that your code is architected so performance critical sections of Python can move into C etc. Let's say that your code creates a complex object tree from some configuration settings, and executes Python methods and code from all over it, using heavy OO. That is difficult to move to C++ as your performance is spent on Python bookkeeping -- you are calling methods and thus looking things up in dictionary, you are modifying fields and also looking up more thing in dictionaries, increasing decreasing refcounts etc. If you have a million 32-bit numbers that you current run Python code on, great, you don't have to convert Python objects to C at all.
    • Rotareti 9y ago
      Thanks for the insights! > Let's say that your code creates a complex object tree from some configuration settings, and executes Python methods and code from all over it, using heavy OO. Luckily, this does not apply to the codebases I'm working on, which are all quite functional (no classes, no inheritance, pure functions exclusively, immutable data types, etc.) I have the feeling that this will not strike me that hard. If you rely on pure functions you have all the application state that you need for the function on the parameters and you pass all new state back through `return`. I guess all I had to do is convert the types once for the C function call (from Python to C) and once for the `return` (from C to Python)?
• otorrillas 9y ago
  OP, I would encourage you take some courses on High-performance computing and, specially, on Architecture Awareness in Programming. These types of courses will help you increasing the performance of your programs by being aware of what's running "under the hood" and ways to "help" the compiler/interpreter better optimisations. Although it's accurate to say that Haskell or C++ are faster than Python, by having had a quick look on the examples you posted around here, I believe there's still a lot of room to improve (performance-wise) on your Python code that could bring a significant speedup boost. However, bear in mind that you shouldn't expect Python to be close to a C++ performance unless you start using libraries such as NumPy that are, essentially, written in C/C++.
• __s 9y ago
  They should use the Y shortcut, available on gitlab & github, to convert their links to mutable master instead to a link to that immutable commit
• passive 9y ago
  This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's the right tool for every part of every job. What we will say is it's usually a good "first" tool for every job. Building a system out in Python allows you to get something representative fairly quickly, which helps identify if there are areas where Python alone is not enough.
  • deleted 9y ago
    [deleted]
  • scribu 9y ago
    I think the article makes a lot more sense if you consider it in the context of "Python for data science". In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).
    • autokad 9y ago
      i dont seem to follow. if you are doing data science, all the bottle necked stuff will be running in numpy or pyspark. Choosing python over R, SPSS, Matlab usually doesnt come down to which one is faster, and R as far as i know is at least not vastly superior in speed.
      • make3 9y ago
        or a cuda wrapper
      • michaelsbradley 9y ago
        As with Python, the fast libraries written for R are usually implemented with something else under the hood. Take the data.table library, for example: https://github.com/Rdatatable/data.table https://github.com/Rdatatable/data.table It's wicked fast for many kinds of tasks, but its R API is just a thin layer on top of C.
• carlmr 9y ago
  >I used to make this argument. Some of it is just a form of utilitarian programming: having a program that runs 1 minute faster but takes 50 extra hours to write is not worth it unless you run it >3000 times. For code that is written as part of data analysis, this is rarely the case. I find this argument breaks down if you consider human psychology. Especially with a program taking 15 seconds or 30 minutes (which is a reasonable time span between say a C++/Rust implementation and a Python implementation in some cases I've experienced). With 15 seconds exec time you might stay in flow. With 30 minutes you're almost guaranteed to have started something else. Maybe you even forget and only get back to it the next day. All of a sudden your 30minutes delay become a day. Then you notice you made some wrong inputs, and you lose another day. In the other case you're still under a minute. I find small increases in program delay often lead to big increases in time inefficiency. It's hard to constantly context switch in and out of tasks. I think the utilitarian argument should take human psychology into effect and weight more towards faster programs.
• lpmay 9y ago
  I don't see it mentioned here, but the dask library looks like a promising solution,. It has ways to handle to these kinds of large datasets, and efficiently schedule computations that don't fit a numpy model. Worth a look.
• b0rsuk 9y ago
  Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow parts. Later, you could replace more of it if needed. Is writing extensions so very problematic in practice? I know Go has runtime issues making it not very good for mixing with other languages, so it often encourages rewriting the whole application in it.
  • rgperkins 9y ago
    It is not a lost art, but it is a thankless job. If an extension just works, people take it, the author gets two or three positive remarks and is ignored from then on because the extension now has utility status. Much better to write an application in Python using 30 slow Python module dependencies, so there are always some fires to fight and the application always gets publicity.
  • jerf 9y ago
    Extensions aren't a total solution, though, which people often sell them as. You have an impedance mismatch between Python and C code, because Python has all of its objects packed in a way that is very strange to C, so you end up essentially deserializing all objects into C, then back out into Python, in a very expensive and allocation-heavy (on both sides) conversion. If you can set up your computation in Python and run it in C, as with a lot of NumPy code, you can have your entire program basically run at C speeds. But if you have a complicated algorithm in Python, perhaps implementing business logic, you can very easily see a slowdown if you try to move bits of that logic into C piecemeal, as you end up paying more in cross-language serialization and overhead than you can win back. In addition, writing extensions can be hard. First you've got a maze of choices nowadays, and while many of them are quite good at what they do, it can be difficult to figure out whether you're going to do something they aren't good at and have to switch options later, and it's really hard to figure out how to even analyze what they are and are not good at when you're not already familiar with the space. Then, if you do end up having to delve into the raw C, it's very tedious code, very tricky code to deal with the PyObjs, and code that can segfault the interpreter instantly if you don't get it right, which is not what you want to read about your multi-hour processing code. And for a greenfield or very young product, this maze of options is competing against other ecosystems where you can simply implement your code and get it to run 20-50x faster while writing code that is easier to write than a Python extension. They are a solution to some problems. I don't deny this. NumPy is an existence proof of that statement. But I'd write this post because I'd say "if it's slow, just write the slow bit in C!" has been oversold in the dynamic language community now for at least the 15 years I've been paying attention, and it still seems to be going strong. In 2003, it may still have been a good choice; in 2018, my recommendation to anybody writing the sort of code where this matters is to pick up one of the several languages that are simply faster to start with, and are much more convenient (even when statically typed) to work with than the competition was in 2003. And I also want to say that Python is still good for many things; I still whip it out every couple of months for something; it's still on my very short list of best languages. But the ground on which it is the best choice is definitely getting squeezed by a lot of very good competition and the changing nature of computer hardware, and a wise engineer pays attention to that and adjusts as needed.
• pcarolan 9y ago
  This is why God invented Spark.
• weberc2 9y ago
  Let me disclaim by saying I like Python, and I've used it for a decade and it pays my bills. The author claims that Python has the lowest developer cost. I used to think that was true, and maybe it is in data science applications, but I regularly find that I'm quite a bit more productive in Go than in Python (largely thanks to the type checker and other static analysis tooling). As an added bonus, Go programs are regularly 100 times faster than Python programs, and usually Python programs are much more difficult to optimize than Go programs. Library availability notwithstanding, starting new projects (of any significance at all) in Python is looking like a worse and worse choice all the time.
• a3n 9y ago
  CPython is obviously the way it is on purpose, by design, and quite successful. Yet I still find it ironic that this "slow" interpreter is written in C, the go to, general purpose, "low level and a half" fast language, and that "C" is right there in the name. Not knowing anything else, I might expect a project with "C" at the front of the name would be at least fast-ish, and that the naming was intended to signal that. Not a complaint, just an observation.
  • wolf550e 9y ago
    What? No. The c in cpython was not meant to signal performance. The c was added after alternative implementations were created to mean "the original/reference implementation". Python's competitors perl ruby and php are likewise interpreters implemented in c and are equally slow. Anyway, what alternative did they have to implement a bytecode interpreter? C is portable and fast enough, especially with computed goto extension.
    • a3n 9y ago
      > The c in cpython was not meant to signal performance. I thought it was clear that I understood that: "Not knowing anything else, I might expect ..." I guess not.
• skywhopper 9y ago
  The quoted argument that "easy to write but slow languages are better because programmer time is far more costly than CPU speed" was pretty common, and I honestly think correct, 10-15 years ago. But things have changed. CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed really is more expensive than programmer time, if you are running that code at a big enough scale. Add in containerization and cloud VM platforms where the tradeoffs of space and performance versus money start to become very clear. Add in better, safer languages like Rust and Go for writing high-performance code. And today if you can spend three times the programming time writing in a faster and more efficient language, and it runs 100x as fast in a tenth the memory footprint, you are talking about massive overall cost savings.
  • commandlinefan 9y ago
    I've been "premature optimization is the root of all evil"ed in more code reviews than I can count - yet when people complain about software, they first complain that it's ugly and then they complain that it's slow. Users don't care what it's written in.
    • sitkack 9y ago
      There is fast (optimization) and slow (the most straightforward thing with no consideration to perf) and then not-slow. I think not-slow is a happy medium. Fast code should and probably is, totally unreadable, looking like AlphaGo optimized your program.
      • smitherfield 9y ago
        >Fast code should and probably is, totally unreadable, looking like AlphaGo optimized your program. Yes and no; multithreading and optimizing to remove branches can do a number on readability, but I find optimizing for cache locality and compile-time evaluation often makes code more readable. It depends on the language obviously.
      • 9y ago
• dahart 9y ago
  > It is true that programmer time is more valuable than computer time, but waiting for results to finish computing is also a waste of my time (I suppose I could do something else in the meanwhile, but context switches are such a killer of my performance that I often just wait). In film CG production, we had a rule of thumb. If running an interactive program takes longer than about ten seconds, the artist (user) becomes more likely than not to get up and go get some coffee or talk to someone else. We consciously made an effort to keep anything someone needed to wait for to under ten seconds, and save anything longer than that for nightly farm renders. We were writing the code in C, btw.
  • cbcoutinho 9y ago
    Somewhat tangential to this, but I remember reading an article on HN about a group creating some web app that was constrained by some size limit (50 KB or so). Groups ended up putting 'dead code' in their projects to guard against other groups taking their space while they were working on their feature. I think this made it so the app never got less than 50... Did you ever see something similar where devs put in some code as filler to make room for future features they were working on?
    • pfranz 9y ago
      In my experience of vfx usually the same author has ownership over the whole tool, so there's no incentive to pad time. We have joked about adding in sleeps and removing them look impressive to users, though. Games probably have a situation more similar to what you're describing. Many games target 30fps, which is 33.3 milliseconds per frame. Each department usually gets a "budget" of how long they can spend. I've heard similar stories about padding memory usage and time, but it's hard to tell if they were serious and it's not common.
    • dahart 9y ago
      Oh yeah, this is very common in game development, and I'm pretty sure in other embedded dev too. There's a famous story about a game programmer who saved an entire production when it started crashing something like two weeks before shipping by commenting one line of code. Turned out the line of code was a malloc (of like a megabyte) that he'd added a year earlier, in anticipation of the game running out of memory. My details are probably wrong, but I'm pretty sure I've seen this story linked on HN. I was in game dev for a decade, and I saw this happen where I worked, the studio technical director adopted the practice of saving some space, because we always started running out of memory near the deadline as artists threw in all their content. *edit: http://www.dodgycoder.net/2012/02/coding-tricks-of-game-developers.html http://www.dodgycoder.net/2012/02/coding-tricks-of-game-deve...
• arithma 9y ago
  Maybe there's merit in a Python version that compiles to JS where we allow engines like V8 to do the optimisation. The dynamic nature of JS maybe enough impedance-MATCH to allow this to happen? Writing this up, I'll probably get someone suggesting that this is already a reality with some tool. Glad to be taught.
• mschaef 9y ago
  Great article. The way I look at it is something like the swordsman scene in Raiders of the Lost Ark. The swordsman's doing fancy optimizations and C just blows away the need. If you can find a language that's x100 the performance of an interpreted language, that speed delta will cover up lots of naivety in the code you write.
• coleifer 9y ago
  Sad not to see Cython getting a mention in this post. It is a superset of Python, so your vanilla code will run just fine, and you can optimize slow parts to native-speed. It's a fantastic tool.
• deathanatos 9y ago
  I agree with some of this, such as, > Python, it is slow as molasses. I don’t mean slower in the sense of “wait a couple of seconds”, I mean “wait several hours instead of 2 minutes.” Python can be multiple orders of magnitude slower than the equivalent in C/C++/Rust, but, > more cores (which Python cannot easily take advantage of) Python's multiprocessing makes launching new processes (which can take advantage of more cores) pretty much as easy as launching threads. But as a developer, I am frustrated by a lot of the things people believe are options. They are options, but … they're hard to use, and hard to take advantage of. * Writing a Python extension requires dropping down to C, which has so many foot-guns, I'd like to delay doing so as long as absolutely possible. Even then, you might not be able to win back that much performance, if most of your time is spent manipulating Python objects. Programmers, in my experience, also vastly overestimate their ability to write correct C. * Cython can compile Python to "C", but in Python 2 (which I am alas stuck with, despite my will; someday…), has a bug that miscompiles code dealing with metaclasses. Worse still, the latest version of six will trigger this bug. (The Cython developers do not consider this — Cython's compiled version of code behaving differently from Python and CPython — a bug.) * rust-cpython is theoretically great, but has a bug on OS X that causes aborts (it erroneously links against the Python binary, I think, and this causes issues w/ virtual environments, where a different binary ends up getting used. I don't think this effects Linux, but I need to support OS X.) (Throw in the enormous amount of time that I spend debugging object of type NoneType has no attribute "static_typing", and the amount of time that I spend wonder "what type is this variable supposed to be? and working it out by reverse engineering the code, and I honestly wonder if Python is actually "faster".) [1]: https://github.com/dgrunwald/rust-cpython/issues/59 https://github.com/dgrunwald/rust-cpython/issues/59
• deleted 9y ago
  [deleted]