6 ms·

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

pletnes 9y ago
That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.
- luispedrocoelho 9y ago
  If I can fit the code into numpy-like structure, then Python is typically fine. The issue is when I cannot.
  - fxmc 9y ago
    Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.
    - VHRanger 9y ago
      Dask doesn't solve that problem since it's a wrapper around pandas functions. If you can't make the core pandas code decently fast, dask won't save you.
      - kiriliponi 9y ago
        dask.dataframe might not help but dask.distributed could in that case. I've had success using it on non vanilla stuff (i.e. code that could not get converted to play natively with numpy/pandas structures) As a bonus, the nice profiling tools (built within dask) have also helped me improve the performance of the code. See https://distributed.readthedocs.io/en/latest/ https://distributed.readthedocs.io/en/latest/
      - pletnes 9y ago
        There’s dask.array which works on numpy arrays instead of dataframes. Otherwise, your argument holds.
  - kalefranz 9y ago
    Also look into numba as a jit decorator for python functions.
  - fnord123 9y ago
    Then move the function to a pyx file and build it with Cython. Problem solved.
- kamaal 9y ago
  Or use languages where you don't have to these extreme workarounds for what should happen by default.
inciampati 9y ago
If you write more C++ than python, it will have a lower TimeToWriteCode. Despite having spent years writing python I don't find it any more productive than C++. C++11 has all the nice features you might expect from python with the only drawback being the lack of a REPL.
- banachtarski 9y ago
  +1 people shouldn't overlook things that are bundled in the C++ stdlib now (chrono, random, thread, algorithm, mutex, containers, etc)
  - cjalmeida 9y ago
    They're great and incredibly useful. And one should not forget that you can easily use them in a Python extension written in C++14 and exported using Cython or SWIG.
- klibertp 9y ago
  The lack of REPL compounds with long compilation times, which is practically a feature of C++ and not going to go away anytime soon. The effect is that, when you explore a new API or need to tune parameters to some function call deep in the call stack, you're an order of magnitude slower than with Python (or Lisp, Scala, F#, Haskell, or even Nim or plain C (b/c compilation times)). If you know exactly what you need to write, you're just as quick in C++ as in Python, that's true. Programming is mostly about learning what to write, though, and here C++ loses.
  - kamaal 9y ago
    If you are developing your code as a small tiny functions getting stitched later. Then writing unit test cases will solve this problem too.
    - klibertp 9y ago
      No, it will help with lack of REPL but not with long compilation times. Long compilation times are bad across the board. Go advertises "fast compilation" as one of its key features for a reason. EDIT: Not to mention, if you write your code as a lot of tiny functions you could just as well write it in C. Once you go for classes and templates, that's where C++ power is visible, but that's also where its compile times suck.
inciampati 9y ago
As for high level C++ threading you have OMP. It's incredibly easy to use. In the simplest case you just use a preprocessor directive before a loop to say it should run in parallel. It's probably not as nice as what you get in Haskell because it needs to be done explicitly but it is really easy to use.
aldanor 9y ago
@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-contained code example (with data required to run it) that is “too slow”, I’d be willing to try and optimise it to support my point above. Also, have you tried Numba? It maybe a matter of just applying a “@jit” decorator and restructuring your code a bit in which case it may get magically boosted a few hundred times in speed. [1] https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post.py#L331 https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post....
- pbowyer 9y ago
  Brian Moore's quip [0] about mod_rewrite comes to mind every time I use Numba: "Despite the examples and docs, Numba is voodoo. Damned cool voodoo, but still voodoo" 0. https://httpd.apache.org/docs/2.0/rewrite/ https://httpd.apache.org/docs/2.0/rewrite/
- luispedrocoelho 9y ago
  That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e04aaf72bcd4feb1a645afb#ca7d49b27cf92be478d916df4f3b59edf91ff0b5_328_328 https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in open('interesting.txt')) total = 0 for line in open('data.txt'): id,val = line.split('\t') if id in interesting: total += int(val) This is not unlike a lot of code I write, actually.
  - BerislavLopac 9y ago
    The speed could possibly be improved by using map. Also, not related to speed if this is all of the code, but might affect it in a larger programs: you should make sure your file pointers are closed. Something like: with open('interesting.txt') as interesting_file: interesting = {line.strip() for line in interesting_file} with open('data.txt') in data_file: total = sum(int(val) for id, val in map(lambda line: line.split('\t'), data_file) if id in interesting)
biztos 9y ago
Interesting assertion re: TimeToWriteCode, but I think there's TimeToWriteCode vs. TimeToWriteGoodCode. I'm working on my first serious Python project right now, and I find it's super easy to throw together some code that more or less works; but for solid, readable, documented, properly unit-tested code I hope is production-ready, it's not any faster than Perl or Golang. (Sure, if you're a Python expert it's faster for you than for me, but if it's about TimeForExpertsToWriteGoodCode I'm not any more convinced.)
- guitarbill 9y ago
  Production-ready is so complex, it's hard to make any comparison. E.g. for a library, writing good documentation (with diagrams and decent technical writing) takes me way longer coding anyway - probably by an order of magnitude. Proper unit-testing is also going to take roughly the same time in any language, just because you have to think hard about sensible tests (although I still love mocking/patching in Python, so I'd give it an edge, plus pdb/ipdb for debugging tests is cool). Production-ready also includes deployment, which for anything non-trivial I'd say Golang > Python > Perl. Finally, if we're talking "serious project", IMO tooling and how that tooling integrates into a CI pipeline are more important than development speed, because as a team or project goes, terrible CI will slow developers more than any language. Although again here I think Python does quite well with decent linting, unit test frameworks, and code coverage options, Golang's opinionated tools are simpler in this respect. (I enjoyed C# for similar reasons, although I don't think it's kept up w.r.t. tooling - been ages since I used it though.)
  - biztos 9y ago
    Good points. So far I find I really like Python's mocking, "with self.some_useful_patch()" is really nice, and I like the idea of side effects especially with boto. Of course in some cases it's really difficult, but every language has its tricky unit-testing problems. One big point I would give to Golang, about which lots of people disagree with me, is the "opinionatedness" of it. It seems to me that Python, like Perl, has a "There's More Than One Way To Do It" mentality, and after many years of that I really appreciated Golang's emphasis on the "idiomatic." That goes for the tooling too. I have also noticed that the Python ecosystem doesn't have a strong documentation culture, which I find annoying as a relative newbie. But that presumably matters less over time, and it seems to be part of the Python Way to use libraries that "just work" and not worry about the details.
boomlinde 9y ago
Is total time really that interesting as a metric? Factor in cost, both in terms of, say, what the employer pays you and what they pay for CPU time, sprinkle it with costs in terms of externalities (e.g. the cost of millions of clients executing poorly performing code vs the cost of millions of clients paying for the additional development overhead of well performing code) and the equation is a lot more complex and application-dependent. Then weigh in the hard realities of some engineering problems. It won't matter that it takes 1% of the time to implement a video decoder in python if it can't deliver decoded frames in a timely manner. It won't matter that the C solution will run 1000x faster if you need a month to develop what should be delivered on Friday. I'm sorry if this is already covered in the article. I had a brief look before but it won't currently load.

pletnes 9y ago

That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.

luispedrocoelho 9y ago
If I can fit the code into numpy-like structure, then Python is typically fine. The issue is when I cannot.
- fxmc 9y ago
  Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.
  - VHRanger 9y ago
    Dask doesn't solve that problem since it's a wrapper around pandas functions. If you can't make the core pandas code decently fast, dask won't save you.
    - kiriliponi 9y ago
      dask.dataframe might not help but dask.distributed could in that case. I've had success using it on non vanilla stuff (i.e. code that could not get converted to play natively with numpy/pandas structures) As a bonus, the nice profiling tools (built within dask) have also helped me improve the performance of the code. See https://distributed.readthedocs.io/en/latest/ https://distributed.readthedocs.io/en/latest/
    - pletnes 9y ago
      There’s dask.array which works on numpy arrays instead of dataframes. Otherwise, your argument holds.
- kalefranz 9y ago
  Also look into numba as a jit decorator for python functions.
- fnord123 9y ago
  Then move the function to a pyx file and build it with Cython. Problem solved.
kamaal 9y ago
Or use languages where you don't have to these extreme workarounds for what should happen by default.

inciampati 9y ago

If you write more C++ than python, it will have a lower TimeToWriteCode. Despite having spent years writing python I don't find it any more productive than C++. C++11 has all the nice features you might expect from python with the only drawback being the lack of a REPL.

banachtarski 9y ago
+1 people shouldn't overlook things that are bundled in the C++ stdlib now (chrono, random, thread, algorithm, mutex, containers, etc)
- cjalmeida 9y ago
  They're great and incredibly useful. And one should not forget that you can easily use them in a Python extension written in C++14 and exported using Cython or SWIG.
klibertp 9y ago
The lack of REPL compounds with long compilation times, which is practically a feature of C++ and not going to go away anytime soon. The effect is that, when you explore a new API or need to tune parameters to some function call deep in the call stack, you're an order of magnitude slower than with Python (or Lisp, Scala, F#, Haskell, or even Nim or plain C (b/c compilation times)). If you know exactly what you need to write, you're just as quick in C++ as in Python, that's true. Programming is mostly about learning what to write, though, and here C++ loses.
- kamaal 9y ago
  If you are developing your code as a small tiny functions getting stitched later. Then writing unit test cases will solve this problem too.
  - klibertp 9y ago
    No, it will help with lack of REPL but not with long compilation times. Long compilation times are bad across the board. Go advertises "fast compilation" as one of its key features for a reason. EDIT: Not to mention, if you write your code as a lot of tiny functions you could just as well write it in C. Once you go for classes and templates, that's where C++ power is visible, but that's also where its compile times suck.

inciampati 9y ago

As for high level C++ threading you have OMP. It's incredibly easy to use. In the simplest case you just use a preprocessor directive before a loop to say it should run in parallel. It's probably not as nice as what you get in Haskell because it needs to be done explicitly but it is really easy to use.

aldanor 9y ago

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-contained code example (with data required to run it) that is “too slow”, I’d be willing to try and optimise it to support my point above. Also, have you tried Numba? It maybe a matter of just applying a “@jit” decorator and restructuring your code a bit in which case it may get magically boosted a few hundred times in speed. [1] https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post.py#L331 https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post....

pbowyer 9y ago
Brian Moore's quip [0] about mod_rewrite comes to mind every time I use Numba: "Despite the examples and docs, Numba is voodoo. Damned cool voodoo, but still voodoo" 0. https://httpd.apache.org/docs/2.0/rewrite/ https://httpd.apache.org/docs/2.0/rewrite/
luispedrocoelho 9y ago
That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e04aaf72bcd4feb1a645afb#ca7d49b27cf92be478d916df4f3b59edf91ff0b5_328_328 https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in open('interesting.txt')) total = 0 for line in open('data.txt'): id,val = line.split('\t') if id in interesting: total += int(val) This is not unlike a lot of code I write, actually.
- BerislavLopac 9y ago
  The speed could possibly be improved by using map. Also, not related to speed if this is all of the code, but might affect it in a larger programs: you should make sure your file pointers are closed. Something like: with open('interesting.txt') as interesting_file: interesting = {line.strip() for line in interesting_file} with open('data.txt') in data_file: total = sum(int(val) for id, val in map(lambda line: line.split('\t'), data_file) if id in interesting)

biztos 9y ago

Interesting assertion re: TimeToWriteCode, but I think there's TimeToWriteCode vs. TimeToWriteGoodCode. I'm working on my first serious Python project right now, and I find it's super easy to throw together some code that more or less works; but for solid, readable, documented, properly unit-tested code I hope is production-ready, it's not any faster than Perl or Golang. (Sure, if you're a Python expert it's faster for you than for me, but if it's about TimeForExpertsToWriteGoodCode I'm not any more convinced.)

guitarbill 9y ago
Production-ready is so complex, it's hard to make any comparison. E.g. for a library, writing good documentation (with diagrams and decent technical writing) takes me way longer coding anyway - probably by an order of magnitude. Proper unit-testing is also going to take roughly the same time in any language, just because you have to think hard about sensible tests (although I still love mocking/patching in Python, so I'd give it an edge, plus pdb/ipdb for debugging tests is cool). Production-ready also includes deployment, which for anything non-trivial I'd say Golang > Python > Perl. Finally, if we're talking "serious project", IMO tooling and how that tooling integrates into a CI pipeline are more important than development speed, because as a team or project goes, terrible CI will slow developers more than any language. Although again here I think Python does quite well with decent linting, unit test frameworks, and code coverage options, Golang's opinionated tools are simpler in this respect. (I enjoyed C# for similar reasons, although I don't think it's kept up w.r.t. tooling - been ages since I used it though.)
- biztos 9y ago
  Good points. So far I find I really like Python's mocking, "with self.some_useful_patch()" is really nice, and I like the idea of side effects especially with boto. Of course in some cases it's really difficult, but every language has its tricky unit-testing problems. One big point I would give to Golang, about which lots of people disagree with me, is the "opinionatedness" of it. It seems to me that Python, like Perl, has a "There's More Than One Way To Do It" mentality, and after many years of that I really appreciated Golang's emphasis on the "idiomatic." That goes for the tooling too. I have also noticed that the Python ecosystem doesn't have a strong documentation culture, which I find annoying as a relative newbie. But that presumably matters less over time, and it seems to be part of the Python Way to use libraries that "just work" and not worry about the details.

boomlinde 9y ago

Is total time really that interesting as a metric? Factor in cost, both in terms of, say, what the employer pays you and what they pay for CPU time, sprinkle it with costs in terms of externalities (e.g. the cost of millions of clients executing poorly performing code vs the cost of millions of clients paying for the additional development overhead of well performing code) and the equation is a lot more complex and application-dependent. Then weigh in the hard realities of some engineering problems. It won't matter that it takes 1% of the time to implement a video decoder in python if it can't deliver decoded frames in a timely manner. It won't matter that the C solution will run 1000x faster if you need a month to develop what should be delivered on Friday. I'm sorry if this is already covered in the article. I had a brief look before but it won't currently load.