10 ms·

A comparison of programming languages in economics

104 points by scarmig 12y ago

cschmidt 12y ago
Direct download link of the original paper: http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf http://economics.sas.upenn.edu/~jesusfv/comparison_languages...
minimax 12y ago
Their code is on github [1]. I am a C++ programmer so I'm happy that C++ came out on top, but the comparison is obviously flawed (as these sorts of comparisons always are). The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version. See Russ Cox's response [2] to a Google paper [3] of a similar nature comparing C++, Go, Java, Scala for an example of a similar boondoggle. The R code, for instance, is obviously unidiomatic. If you have a four levels deep nested loop in R, you're doing it wrong, and it's taking every ounce of willpower to keep from spending the rest of the afternoon trying to fix up their code and make it faster. 1. https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-... 2. http://blog.golang.org/profiling-go-programs http://blog.golang.org/profiling-go-programs 3. http://research.google.com/pubs/pub37122.html http://research.google.com/pubs/pub37122.html
- danielweber 12y ago
  I'm hoping that -- and will be pleased by the irony if -- the HN discussion has less language war than the MR discussion. It's worth knowing, for economists, how well economists can write code in different languages. But I'm sure I, too, would write better C++ than R, since I've not written any R at all. They would need to ask economists skilled in each language to write the code to get a better answer, and there would still be a lot of noise in the signal. EDIT last sentence was gibbgersh
- bradfordarner 12y ago
  It is unfortunate that the authors of this paper didn't reach out to experts in each programming language in order to write high-quality code against which they could have run the analysis. This seems like the perfect use-case for open-source: academic study, subject-matter experts in one area who are stepping outside their areas of expertise and an interesting subject matter. They should have crowd-sourced the code for this study with a Top Coder-like contest.
  - ViralBShah 12y ago
    I don't know about other languages, but the authors did reach out to us to sort out performance issues with the Julia code.
    - bradfordarner 12y ago
      That is very good to know. That reassures me a little. I was really impressed with the speed difference they noted between Python and R and the fact that Numba wasn't too much slower than C++ implementations. However, given the state of the R code I became wary of accepting the benchmarks at face value. I haven't made the jump from R to Python for statistical applications yet. So, this article gave me something to think about. In terms of programmer happiness, I'm willing to live with a little slower performance in Python and not have to spend hours in C++.
      - Fede_V 12y ago
        I've used Numba to speed up some tight numerical computations written on NumPy arrays and it's incredibly good - about as fast as rewriting it from scratch in Cython, but much, much faster. The flipside is that it only really results in a speed up for pure numerical code - if you have to use any Python objects, then you lose everything.
  - k2enemy 12y ago
    From the paper... Second, to make the comparison as unbiased as possible, we coded the same algorithm in each language without adapting it to the peculiarities of each language (which could reflect more about our knowledge of each language than of its objective virtues). They have invited others to fork the code on github and write the fastest possible for each language.
- ViralBShah 12y ago
  The stochastic growth model, which is the workhorse of modern economics, cannot be easily vectorized, and usually requires tight inner loops to get good performance, or rewrite the computational kernels in C. Codes such as Dynare (http://www.dynare.org http://www.dynare.org) do exactly that. In my opinion, this is where Julia shines - it provides high performance in a dynamic language, much at the level of R or MATLAB. Then again, being a co-creator of Julia, I am probably biased, but this is exactly the kind of thing that motivated work on Julia in the first place.
  - minimax 12y ago
    The two things I like about Julia are actually using it and all the smart people I have met who are also using it. It's awesome. After playing with this particular problem a bit, I do see how it is a pathological case for R due to the recursive nature of the value function. I still think I could do better than 500x slower than C++.
  - bachmeier 12y ago
    The stochastic growth model is the workhorse of a narrow subfield of economics. It is not in any way representative of the research we do. Further, you will not find any examples that are better suited to compiled languages or less suited to using R than the one in this paper. To my knowledge, value function iteration isn't even used much in this area of research, so the claim that the stochastic growth model is a "workhorse" is a bit misleading - the stochastic growth model gets used, but it is not solved very often using the methodology in this paper.
  - c0g 12y ago
    Hi Viral, do you have an intuition for why Numba would be faster than Julia here? It appears Julia spends a lot of time calculating the log on line 71.
    - ihnorton 12y ago
      (not Viral, but) yes - log was the first thing that came up when this paper was discussed on the Julia mailing list. Julia's openlibm is intended to provide cross-platform accuracy and consistency, but the log function turned out to be slower than log in the system libm used by C++ and Python. The other big reason that Numba was faster was GC performance. There is an outstanding pull request adding incremental GC to Julia, which should help when it is fully integrated, and several other improvements are planned.
- cs702 12y ago
  I almost gasped when I saw that their R code[1] does the bulk of its computations inside a while(...for(...for(...for(...if(...){...}else{...})))) block, instead of using functionals like apply and 'vectorized' operations like ifelse. If this criticism seems harsh, compare the authors' R code to Hadley Wickham's advice at http://adv-r.had.co.nz/Profiling.html http://adv-r.had.co.nz/Profiling.html -- or just search Google for "speed-up loops in R". -- [1] https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_R.R https://github.com/jesusfv/Comparison-Programming-Languages-...
  - collyw 12y ago
    Many of the bioinformaticains I know use R. Not knowing how to use it properly wouldn't surprise me in the slightest. I struggle to get some of them to indent their Perl code.
- briandh 12y ago
  >The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version One of they authors addresses this in the comments on the OP: > As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. > Yes, we know how to code better and prettier R code with the plyr package and related tools. Again, that was not what we wanted to do in this exercise. But we will be thrilled to see alternative versions on github. That is why we posted all our code there. > My personal guess is that, given how large our capital grid is, plyr will go less far than some people would have forecast, but I’d be happy to be proven wrong. And if zissou is correct below in his assertion that the "vast majority of economists are horrible programmers", a comparison of what a naive user would write may be the most useful.
  - Alupis 12y ago
    The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their tests on consumer desktop general purpose operating systems instead of a more specialized system designed for the task... or at the very least a more specialized OS such as a flavor of BSD or Linux without the desktop and other generalized packages. I don't really consider this a well done test, and the results are only surface-level... certainly nothing you should base a serious decision on.
    - taeric 12y ago
      JIT'ed Java code is lacking a major facility that truly performant native code has, and that is cache locality. That is, it isn't enough that the code to perform the algorithm is "well tuned" if the data structures are not. (Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)
- rlvesco7 12y ago
  It would have been nice to see R with c++. Using the Rcpp package you can write inline c++ for the heavy loops like a boss.
- sveme 12y ago
  Given that their code is on github and many people criticized the unidiomatic nature of their code, wouldn't it make sense to set up a competition to improve the code? Everyone can submit pull requests with their improved codes and the authors should simply rerun the codes on a standard machine to get a benchmark. Obviously, a couple of rules should apply: 1. when you improve the code, do not move any elements outside the benchmarked language, i.e., don't use mex files in Matlab, c++ in R etc. 2. maybe create two or three different versions of your code - single processor, multiple cpus, gpu and run it on a single machine with a predefined set up. This will remove at least some of the criticism thrown at the economists and has an additional benefit: researchers trying to find the best-suited language for their problem can not only use speed but also expressiveness, aesthetics, code length, boilerplate and ease of use to guide their decision.
  - igouy 12y ago
    http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/
zissou 12y ago
I used to be gung-ho about convincing economists to use Python (started econpy.org in the 1st year of my PhD in economics, stopped updating it after 2 years). But now I just don't care. The vast majority of economists are horrible programmers, however most know how to script in at least 1 language, and a small number of them are actually really good at scripting. An extremely small number of economists know they're way around at least 1 entire general purpose programming language. It seems that every time this gets brought up online a sea of infamous Fortran-programming-economists always tell you how fast Fortran is compared to everything else, signing it with "Fortran or GTFO". I've met hundreds/thousands of economists and econ grad students at schools ranked from 1 to 200, and I don't recall a single one that actually used Fortran (I'm sure there is a non-zero quantity -- I probably just haven't cared to talk with them b/c they are most likely macro theorists). The fact of the matter is that in economics graduate school, your professors could care less what language you use. It isn't an algorithm competition, it's a story telling competition. But the type of story telling economists do is no less noble than writing elegant algos. Writing a story based on economic mechanisms and behaviors and supporting it with quality data, sound statistics, and logical/exact theories is no simple task.
- cwal37 12y ago
  Hey, that's really neat. I'm doing a post-masters right now in energy economics and have been transitioning into using python for everything (from previously using Matlab or whatever was available), but I'm eyeing an economics PhD down the line. You have any advice/opinions on economics PhDs?
  - bachmeier 12y ago
    Do you have any specific questions? I teach in a PhD-granting department and do research in energy economics. (I'm an associate editor of one of the energy field journals.)
    - cwal37 12y ago
      Huh, some of your papers sound like ideas I sketched out in my commonplace book (particularly the ones on oil prices in the context of the larger economy), looks like I'm a few years behind. I guess my biggest concern is the viability of pursuing a PhD in economics without having a previous degree in econ. I have a BS and MS in environmental science and an MPA in energy policy/economics. I think I have stronger scientific chops than many economists, but I don't know how valuable that actually is. Also, it seems to me that the general market for energy economists is very strong, and will remain that way for quite some time. Do you think that's accurate? I love plowing through data and developing methods and analysis to try and draw out relationships and show why things are what they are. I know that sounds incredibly vague, but as far as I can tell economics is the field that most closely aligns with this in the context of energy and the environment.
wcember 12y ago
I think the paper should be called "A comparison of programming language run times in economics." It's not obvious to me that run time is an important quality of a programming language for economic research.
oniTony 12y ago
This seems more a survey of compilers than programming languages. Python is represented by 3 different compilers, spanning 2 orders of magnitude in performance: CPython (155x), PyPy (44x), Numba (1.57x). It's hinted that C++ similarly depends on compiler choice: > although one needs to be careful with the choice of compiler [for C++ and Fortran]
MrBuddyCasino 12y ago
In case anyone missed the Java results, I do think it is worth mentioning that it is very competitive: "Our fourth result is that Java imposes a speed penalty of 110 to 169 percent. Given the similarity between Java and C++ syntax, there does not seem to be an obvious advantage for choosing Java unless portability across platforms or the wide availability of Java programmers is an important factor."
- dodders 12y ago
  In my professional life I have rarely (if ever?) seen a language chosen on the basis of it's syntax.
  - MrBuddyCasino 12y ago
    I get what you mean, but lots of languages have seen limited popularity due to their unfamiliar or odd syntax (e.g. lisp).
  - collyw 12y ago
    Plenty of people claim they won't use Python for its whitespace. Not quite the same, but it seem that elements of syntax are influencing peoples decisions. I would also recommend Python to new programmers, partly for the syntax.
  - adrusi 12y ago
    It's true, most good programmers won't choose a language based on just syntax. And even though most economists aren't good programmers (in my experience), they're still smart people who can see past the syntax. The paper is probably misusing the term syntax though. C++ and java really are quite similar in many ways, far beyond just syntax, java just letting the programmer forget about some annoying things like memory management and portability. If most of what they're doing in the language is numeric analysis, then there probably isn't much of a difference at all, and they will probably get slightly better performance from C++.
- thinkpad20 12y ago
  There are legions of reasons to choose Java over C++, or vice-versa, none of which have anything to do with syntax. Is this a strawman on the part of the authors, or are they unaware of the legion of things that make C++ a far more complex and dangerous language than Java? Or are those things irrelevant to the needs of economists?
psychometry 12y ago
Economic is a large, diverse field and it would be reasonable to assume, for instance, that R is used in very different ways than C++, so this study is already of limited use. Furthermore, they claim to be comparing the "strength and weakness of each language" yet only analyze the performance of naively-written, unoptimized code.
- zissou 12y ago
  You bring up a good point in that the use cases for each language varies. R is a good example because one of the main reasons it is chosen is due to the fact that it already has the largest set of drop in econometrics operations for tons of extremely specific situations. In any other language, you'd often have to code a bunch of complex linear algebra by hand, which would totally negate any marginal benefits due to computation time. In terms of econometrics, Python is going to surpass closed source software packages like SPSS, SAS, eViews, etc soon simply because the rate at which econometric procedures are being implemented in Python is growing steadily (e.g. via a statsmodels/pandas/numpy/scipy based stack). I don't think Python will ever pass R in this respect though as R has a large share of the statisticians helping add the implementations relative to Python.
larrydag 12y ago
It seems like this is a comparison of speed. Not a comparison of languages. Each language has their own pro's and con's. For instance, I use R on a daily basis. I don't require speed to do my job because I'm analyzing data and drawing inferences to help make business decisions. I like the fact that there is a huge library and an active community that can help me implement solutions. Perhaps if a solution/product I come up with needs speed then I'll implement a tool that will do that job well. But my "toolbox" requires fast prototyping and development.
nemoniac 12y ago
Two economists judge the relative suitability of a bunch of programming languages. I have to wonder how it would be received if two computer scientists wrote an article judging the relative merits of a bunch of economic models.
civilian 12y ago
What about Excel? I know, I'm also horrified of how economists use excel. But it does tend to be their go-to tool, and they are all comfortable in it. It'd be nice to show them just how poor the performance is in excel, to tempt them into learning scripting languages if they have an excel workbook that will barely refresh.
plumeria 12y ago
It would be cool that people from different areas of expertise (whether C++, Rust, Go, etc...) committed their most idiomatic implementation for the benchmark.
adrusi 12y ago
It's important to note that the available libraries for each language varies a lot, and the best performing ones are proprietary for the most part. Fortran, for example, has a set of proprietary libraries that many employers buy that have been continually optimized for nearly 50 years. Both C and Python can readily bind to Fortran libraries, but for other languages like R, Matlab, Mathematica, etc. this is a major disadvantage.
- bachmeier 12y ago
  Not only is it easy to call Fortran functions from R, the inline package actually allows you to embed Fortran functions inside your R program.
mikecb 12y ago
Another interesting comparison from the github repo[1] is the proportions of lines that it took in each language. [1] https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-...
- igouy 12y ago
  Programs for some languages are tall and narrow, while comparable programs for other languages are short and wide.
georgewfraser 12y ago
There are two glaring problems with this paper: 1. The algorithm is just a bunch of arrays and loops in one giant block. This is the easiest possible case for compilers that optimize through specialization. Even a bad specializing compiler will be good at this but will blow up when faced with a big piece of code with lots of functions and data structures. 2. The runtimes are only ~2s, which is not long enough for the JIT-compiled languages to "warm up". I suspect Java in particular would have had the exact same runtime after warming up, because in an easy example like this Hotspot will generate the exact same instructions as the C program. EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s.
- Alupis 12y ago
  > EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s. Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)
  - georgewfraser 12y ago
    True though oddly I compiled and ran them exactly the same way he described in the paper -- one time through, no warm-up time.
    - Terr_ 12y ago
      I think the command-line option -XX:+PrintCompilation should show you if/when the JIT kicks in for a method.
    - Alupis 12y ago
      The sample code is probably too predictable and the JVM was able to optimize it well. In any event... the "myth" that Java is slow is bogus :)
JAlexoid 12y ago
Wasted time on a wasted subject. The person that compiled it had nothing better to do... Economists are not developers and should not imagine to be ones. It takes a lot of time and effort to be one.

104 points by scarmig 12y ago

Direct download link of the original paper: http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf http://economics.sas.upenn.edu/~jesusfv/comparison_languages...

minimax 12y ago

Their code is on github [1]. I am a C++ programmer so I'm happy that C++ came out on top, but the comparison is obviously flawed (as these sorts of comparisons always are). The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version. See Russ Cox's response [2] to a Google paper [3] of a similar nature comparing C++, Go, Java, Scala for an example of a similar boondoggle. The R code, for instance, is obviously unidiomatic. If you have a four levels deep nested loop in R, you're doing it wrong, and it's taking every ounce of willpower to keep from spending the rest of the afternoon trying to fix up their code and make it faster. 1. https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-... 2. http://blog.golang.org/profiling-go-programs http://blog.golang.org/profiling-go-programs 3. http://research.google.com/pubs/pub37122.html http://research.google.com/pubs/pub37122.html

danielweber 12y ago
I'm hoping that -- and will be pleased by the irony if -- the HN discussion has less language war than the MR discussion. It's worth knowing, for economists, how well economists can write code in different languages. But I'm sure I, too, would write better C++ than R, since I've not written any R at all. They would need to ask economists skilled in each language to write the code to get a better answer, and there would still be a lot of noise in the signal. EDIT last sentence was gibbgersh
bradfordarner 12y ago
It is unfortunate that the authors of this paper didn't reach out to experts in each programming language in order to write high-quality code against which they could have run the analysis. This seems like the perfect use-case for open-source: academic study, subject-matter experts in one area who are stepping outside their areas of expertise and an interesting subject matter. They should have crowd-sourced the code for this study with a Top Coder-like contest.
- ViralBShah 12y ago
  I don't know about other languages, but the authors did reach out to us to sort out performance issues with the Julia code.
  - bradfordarner 12y ago
    That is very good to know. That reassures me a little. I was really impressed with the speed difference they noted between Python and R and the fact that Numba wasn't too much slower than C++ implementations. However, given the state of the R code I became wary of accepting the benchmarks at face value. I haven't made the jump from R to Python for statistical applications yet. So, this article gave me something to think about. In terms of programmer happiness, I'm willing to live with a little slower performance in Python and not have to spend hours in C++.
    - Fede_V 12y ago
      I've used Numba to speed up some tight numerical computations written on NumPy arrays and it's incredibly good - about as fast as rewriting it from scratch in Cython, but much, much faster. The flipside is that it only really results in a speed up for pure numerical code - if you have to use any Python objects, then you lose everything.
- k2enemy 12y ago
  From the paper... Second, to make the comparison as unbiased as possible, we coded the same algorithm in each language without adapting it to the peculiarities of each language (which could reflect more about our knowledge of each language than of its objective virtues). They have invited others to fork the code on github and write the fastest possible for each language.
ViralBShah 12y ago
The stochastic growth model, which is the workhorse of modern economics, cannot be easily vectorized, and usually requires tight inner loops to get good performance, or rewrite the computational kernels in C. Codes such as Dynare (http://www.dynare.org http://www.dynare.org) do exactly that. In my opinion, this is where Julia shines - it provides high performance in a dynamic language, much at the level of R or MATLAB. Then again, being a co-creator of Julia, I am probably biased, but this is exactly the kind of thing that motivated work on Julia in the first place.
- minimax 12y ago
  The two things I like about Julia are actually using it and all the smart people I have met who are also using it. It's awesome. After playing with this particular problem a bit, I do see how it is a pathological case for R due to the recursive nature of the value function. I still think I could do better than 500x slower than C++.
- bachmeier 12y ago
  The stochastic growth model is the workhorse of a narrow subfield of economics. It is not in any way representative of the research we do. Further, you will not find any examples that are better suited to compiled languages or less suited to using R than the one in this paper. To my knowledge, value function iteration isn't even used much in this area of research, so the claim that the stochastic growth model is a "workhorse" is a bit misleading - the stochastic growth model gets used, but it is not solved very often using the methodology in this paper.
- c0g 12y ago
  Hi Viral, do you have an intuition for why Numba would be faster than Julia here? It appears Julia spends a lot of time calculating the log on line 71.
  - ihnorton 12y ago
    (not Viral, but) yes - log was the first thing that came up when this paper was discussed on the Julia mailing list. Julia's openlibm is intended to provide cross-platform accuracy and consistency, but the log function turned out to be slower than log in the system libm used by C++ and Python. The other big reason that Numba was faster was GC performance. There is an outstanding pull request adding incremental GC to Julia, which should help when it is fully integrated, and several other improvements are planned.
cs702 12y ago
I almost gasped when I saw that their R code[1] does the bulk of its computations inside a while(...for(...for(...for(...if(...){...}else{...})))) block, instead of using functionals like apply and 'vectorized' operations like ifelse. If this criticism seems harsh, compare the authors' R code to Hadley Wickham's advice at http://adv-r.had.co.nz/Profiling.html http://adv-r.had.co.nz/Profiling.html -- or just search Google for "speed-up loops in R". -- [1] https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_R.R https://github.com/jesusfv/Comparison-Programming-Languages-...
- collyw 12y ago
  Many of the bioinformaticains I know use R. Not knowing how to use it properly wouldn't surprise me in the slightest. I struggle to get some of them to indent their Perl code.
briandh 12y ago
>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version One of they authors addresses this in the comments on the OP: > As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. > Yes, we know how to code better and prettier R code with the plyr package and related tools. Again, that was not what we wanted to do in this exercise. But we will be thrilled to see alternative versions on github. That is why we posted all our code there. > My personal guess is that, given how large our capital grid is, plyr will go less far than some people would have forecast, but I’d be happy to be proven wrong. And if zissou is correct below in his assertion that the "vast majority of economists are horrible programmers", a comparison of what a naive user would write may be the most useful.
- Alupis 12y ago
  The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their tests on consumer desktop general purpose operating systems instead of a more specialized system designed for the task... or at the very least a more specialized OS such as a flavor of BSD or Linux without the desktop and other generalized packages. I don't really consider this a well done test, and the results are only surface-level... certainly nothing you should base a serious decision on.
  - taeric 12y ago
    JIT'ed Java code is lacking a major facility that truly performant native code has, and that is cache locality. That is, it isn't enough that the code to perform the algorithm is "well tuned" if the data structures are not. (Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)
rlvesco7 12y ago
It would have been nice to see R with c++. Using the Rcpp package you can write inline c++ for the heavy loops like a boss.
sveme 12y ago
Given that their code is on github and many people criticized the unidiomatic nature of their code, wouldn't it make sense to set up a competition to improve the code? Everyone can submit pull requests with their improved codes and the authors should simply rerun the codes on a standard machine to get a benchmark. Obviously, a couple of rules should apply: 1. when you improve the code, do not move any elements outside the benchmarked language, i.e., don't use mex files in Matlab, c++ in R etc. 2. maybe create two or three different versions of your code - single processor, multiple cpus, gpu and run it on a single machine with a predefined set up. This will remove at least some of the criticism thrown at the economists and has an additional benefit: researchers trying to find the best-suited language for their problem can not only use speed but also expressiveness, aesthetics, code length, boilerplate and ease of use to guide their decision.
- igouy 12y ago
  http://benchmarksgame.alioth.debian.org/ http://benchmarksgame.alioth.debian.org/

zissou 12y ago

I used to be gung-ho about convincing economists to use Python (started econpy.org in the 1st year of my PhD in economics, stopped updating it after 2 years). But now I just don't care. The vast majority of economists are horrible programmers, however most know how to script in at least 1 language, and a small number of them are actually really good at scripting. An extremely small number of economists know they're way around at least 1 entire general purpose programming language. It seems that every time this gets brought up online a sea of infamous Fortran-programming-economists always tell you how fast Fortran is compared to everything else, signing it with "Fortran or GTFO". I've met hundreds/thousands of economists and econ grad students at schools ranked from 1 to 200, and I don't recall a single one that actually used Fortran (I'm sure there is a non-zero quantity -- I probably just haven't cared to talk with them b/c they are most likely macro theorists). The fact of the matter is that in economics graduate school, your professors could care less what language you use. It isn't an algorithm competition, it's a story telling competition. But the type of story telling economists do is no less noble than writing elegant algos. Writing a story based on economic mechanisms and behaviors and supporting it with quality data, sound statistics, and logical/exact theories is no simple task.

cwal37 12y ago
Hey, that's really neat. I'm doing a post-masters right now in energy economics and have been transitioning into using python for everything (from previously using Matlab or whatever was available), but I'm eyeing an economics PhD down the line. You have any advice/opinions on economics PhDs?
- bachmeier 12y ago
  Do you have any specific questions? I teach in a PhD-granting department and do research in energy economics. (I'm an associate editor of one of the energy field journals.)
  - cwal37 12y ago
    Huh, some of your papers sound like ideas I sketched out in my commonplace book (particularly the ones on oil prices in the context of the larger economy), looks like I'm a few years behind. I guess my biggest concern is the viability of pursuing a PhD in economics without having a previous degree in econ. I have a BS and MS in environmental science and an MPA in energy policy/economics. I think I have stronger scientific chops than many economists, but I don't know how valuable that actually is. Also, it seems to me that the general market for energy economists is very strong, and will remain that way for quite some time. Do you think that's accurate? I love plowing through data and developing methods and analysis to try and draw out relationships and show why things are what they are. I know that sounds incredibly vague, but as far as I can tell economics is the field that most closely aligns with this in the context of energy and the environment.

wcember 12y ago

I think the paper should be called "A comparison of programming language run times in economics." It's not obvious to me that run time is an important quality of a programming language for economic research.

oniTony 12y ago

This seems more a survey of compilers than programming languages. Python is represented by 3 different compilers, spanning 2 orders of magnitude in performance: CPython (155x), PyPy (44x), Numba (1.57x). It's hinted that C++ similarly depends on compiler choice: > although one needs to be careful with the choice of compiler [for C++ and Fortran]

MrBuddyCasino 12y ago

In case anyone missed the Java results, I do think it is worth mentioning that it is very competitive: "Our fourth result is that Java imposes a speed penalty of 110 to 169 percent. Given the similarity between Java and C++ syntax, there does not seem to be an obvious advantage for choosing Java unless portability across platforms or the wide availability of Java programmers is an important factor."

dodders 12y ago
In my professional life I have rarely (if ever?) seen a language chosen on the basis of it's syntax.
- MrBuddyCasino 12y ago
  I get what you mean, but lots of languages have seen limited popularity due to their unfamiliar or odd syntax (e.g. lisp).
- collyw 12y ago
  Plenty of people claim they won't use Python for its whitespace. Not quite the same, but it seem that elements of syntax are influencing peoples decisions. I would also recommend Python to new programmers, partly for the syntax.
- adrusi 12y ago
  It's true, most good programmers won't choose a language based on just syntax. And even though most economists aren't good programmers (in my experience), they're still smart people who can see past the syntax. The paper is probably misusing the term syntax though. C++ and java really are quite similar in many ways, far beyond just syntax, java just letting the programmer forget about some annoying things like memory management and portability. If most of what they're doing in the language is numeric analysis, then there probably isn't much of a difference at all, and they will probably get slightly better performance from C++.
thinkpad20 12y ago
There are legions of reasons to choose Java over C++, or vice-versa, none of which have anything to do with syntax. Is this a strawman on the part of the authors, or are they unaware of the legion of things that make C++ a far more complex and dangerous language than Java? Or are those things irrelevant to the needs of economists?

psychometry 12y ago

Economic is a large, diverse field and it would be reasonable to assume, for instance, that R is used in very different ways than C++, so this study is already of limited use. Furthermore, they claim to be comparing the "strength and weakness of each language" yet only analyze the performance of naively-written, unoptimized code.

zissou 12y ago
You bring up a good point in that the use cases for each language varies. R is a good example because one of the main reasons it is chosen is due to the fact that it already has the largest set of drop in econometrics operations for tons of extremely specific situations. In any other language, you'd often have to code a bunch of complex linear algebra by hand, which would totally negate any marginal benefits due to computation time. In terms of econometrics, Python is going to surpass closed source software packages like SPSS, SAS, eViews, etc soon simply because the rate at which econometric procedures are being implemented in Python is growing steadily (e.g. via a statsmodels/pandas/numpy/scipy based stack). I don't think Python will ever pass R in this respect though as R has a large share of the statisticians helping add the implementations relative to Python.

larrydag 12y ago

It seems like this is a comparison of speed. Not a comparison of languages. Each language has their own pro's and con's. For instance, I use R on a daily basis. I don't require speed to do my job because I'm analyzing data and drawing inferences to help make business decisions. I like the fact that there is a huge library and an active community that can help me implement solutions. Perhaps if a solution/product I come up with needs speed then I'll implement a tool that will do that job well. But my "toolbox" requires fast prototyping and development.

nemoniac 12y ago

Two economists judge the relative suitability of a bunch of programming languages. I have to wonder how it would be received if two computer scientists wrote an article judging the relative merits of a bunch of economic models.

civilian 12y ago

What about Excel? I know, I'm also horrified of how economists use excel. But it does tend to be their go-to tool, and they are all comfortable in it. It'd be nice to show them just how poor the performance is in excel, to tempt them into learning scripting languages if they have an excel workbook that will barely refresh.

plumeria 12y ago

It would be cool that people from different areas of expertise (whether C++, Rust, Go, etc...) committed their most idiomatic implementation for the benchmark.

adrusi 12y ago

It's important to note that the available libraries for each language varies a lot, and the best performing ones are proprietary for the most part. Fortran, for example, has a set of proprietary libraries that many employers buy that have been continually optimized for nearly 50 years. Both C and Python can readily bind to Fortran libraries, but for other languages like R, Matlab, Mathematica, etc. this is a major disadvantage.

bachmeier 12y ago
Not only is it easy to call Fortran functions from R, the inline package actually allows you to embed Fortran functions inside your R program.

mikecb 12y ago

Another interesting comparison from the github repo[1] is the proportions of lines that it took in each language. [1] https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-...

igouy 12y ago
Programs for some languages are tall and narrow, while comparable programs for other languages are short and wide.

georgewfraser 12y ago

There are two glaring problems with this paper: 1. The algorithm is just a bunch of arrays and loops in one giant block. This is the easiest possible case for compilers that optimize through specialization. Even a bad specializing compiler will be good at this but will blow up when faced with a big piece of code with lots of functions and data structures. 2. The runtimes are only ~2s, which is not long enough for the JIT-compiled languages to "warm up". I suspect Java in particular would have had the exact same runtime after warming up, because in an easy example like this Hotspot will generate the exact same instructions as the C program. EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s.

Alupis 12y ago
> EDIT: I stand corrected. I downloaded his examples off Github and ran them the same way as the paper. I get 1.87s for C, and for Java... also exactly 1.87s. Your code probably got JIT'ed... which essentially makes your java code into native C... So... it's no surprise they perform around or exactly the same (given the JVM is allowed time to warm up, discover "hot spots", and then perform the JITing)
- georgewfraser 12y ago
  True though oddly I compiled and ran them exactly the same way he described in the paper -- one time through, no warm-up time.
  - Terr_ 12y ago
    I think the command-line option -XX:+PrintCompilation should show you if/when the JIT kicks in for a method.
  - Alupis 12y ago
    The sample code is probably too predictable and the JVM was able to optimize it well. In any event... the "myth" that Java is slow is bogus :)

JAlexoid 12y ago

Wasted time on a wasted subject. The person that compiled it had nothing better to do... Economists are not developers and should not imagine to be ones. It takes a lot of time and effort to be one.