4 ms·

>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each ver

by briandh 12y ago

>The problem is that the implementers don't have the same level of skill in each language and who knows how much time they spent analyzing and refining each version One of they authors addresses this in the comments on the OP: > As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. > Yes, we know how to code better and prettier R code with the plyr package and related tools. Again, that was not what we wanted to do in this exercise. But we will be thrilled to see alternative versions on github. That is why we posted all our code there. > My personal guess is that, given how large our capital grid is, plyr will go less far than some people would have forecast, but I’d be happy to be proven wrong. And if zissou is correct below in his assertion that the "vast majority of economists are horrible programmers", a comparison of what a naive user would write may be the most useful.

Alupis 12y ago
The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their tests on consumer desktop general purpose operating systems instead of a more specialized system designed for the task... or at the very least a more specialized OS such as a flavor of BSD or Linux without the desktop and other generalized packages. I don't really consider this a well done test, and the results are only surface-level... certainly nothing you should base a serious decision on.
- taeric 12y ago
  JIT'ed Java code is lacking a major facility that truly performant native code has, and that is cache locality. That is, it isn't enough that the code to perform the algorithm is "well tuned" if the data structures are not. (Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)
  - pkolaczk 12y ago
    While I agree Java makes writing cache-friendly code much harder than C, it is no-way impossible. Nor is any C or C++ code implicitly cache-friendly. Generally, in both cases you must know what you're doing. I also hope some of those problems are going to be addressed in Java 9.
    - taeric 12y ago
      Fair enough. My point was supposed to be more limited, saying that a JIT'ed solution is not a cure all for performance considerations. Sure, it is great that there is a JIT compiler helping out. It just is not necessarily enough on its own to even the performance gap between native and not.
      - Alupis 12y ago
        It is enough. You can perf test the same non-trivial program (such as a biz application) in C and Java, and given you allow the JVM to warm up, discover hot spots, and JIT your code... they will run just about the same performance level... every time. The JVM is impressive.
        sixbrx 12y ago
        JVM is impressive until closures are involved, in which case when inlining is not done (either because it cannot be or ever-changing heuristics are not met), you can drop back to virtual calls and performance suffers a great deal. Compare to C++ where closures can be used with virtually no penalty and aggressively optimized. Even C with its trick of passing an extra *void for environment comes out shining performance-wise in comparison. Edit: There's a good discussion of the problems that JVMs have in this area at http://www.azulsystems.com/blog/cliff/2011-04-04-fixing-the-inlining-problem http://www.azulsystems.com/blog/cliff/2011-04-04-fixing-the-.... "The Problem is a hard one to solve; (i.e. it’s not the case that we’ll wave our wands and do a quick fix & rebuild HotSpot and the problem will go away). John Rose, Doug Lea and I all agree that it’s a Major Problem facing JVMs with long and far reaching implications." Maybe progress will be made soon though, with more pressure caused by increasing use of closures in the new streams library and with anonymous function syntax available now.
- jeremyjh 12y ago
  They also didn't account for the possiblity that a Java programmer would use an ArrayList<Double> instead of double[]. You can't JIT that away; JIT does not change the semantics of the Java code so you have to code Java in a sympathetic way. I can imagine this may be more idiomatic in numeric intensive fields though.
- maxlybbert 12y ago
  The authors do have a comment on the Github project that code needs to be allowed to run for a while so the JIT can kick in, although I didn't see any statement of how they did that. The paper also mentions a decision to turn a loop body into a function to allow a JIT to optimize things. But, generally, I'm not interested in whether "for" loops run faster in language X or language Y. I am interested in whether some languages encourage code styles that happen to be easily optimized (I'm also interested in whether some languages encourage code styles that are easily maintained, easily extensible, or easily run into a ditch). This paper is meaningful to somebody, but it's clearly written by people who spend most of their thinking about things other than programming. That's not an insult; it's a simple fact that a lot of programs are written by people who are experts in other fields. I happen to work with a former physicist who has experience writing programs to run on one of the nation's largest supercomputers. The depressing thing (to me) is that while he knows how to write numerical programs that run close to a computer's theoretical maximum speed, that information is only well-known among a small group of people (namely people who have to show that if they're allowed to use a supercomputer, they won't make it run wasteful code). I wish it were shared more broadly.
- gd1 12y ago
  >JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed No way. It may be native C, but it's not native C you've hand-coded yourself.
  - fulafel 12y ago
    C doesn't enter into it, any way you look at it. Unless you are using some obscure Java -> C compiler.
    - Alupis 12y ago
      No, the optimizing compiler inside the JVM does produce a lot of native code for your system... c/assembler. It just has to discover hot spots then JIT them.
grayclhn 12y ago
> One of they authors addresses this in the comments on the OP: >> As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. Sure, but this makes the paper less useful for answering the question, "which programming language should I use?" Or, "which programming language should I learn better?" But, yes, most economists are pretty bad programmers, so "which language should I use poorly?" is probably a useful question to answer.
- igouy 12y ago
  Which language should I learn to use better? is probably a useful question to answer.
Goladus 12y ago
We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. This is rather half-baked reasoning. The only ways I can think of to control for programmer skill or at least account for it is to 1. Use "idiomatic" code for all languages. 2. Use "optimized" code for all languages. 3. Use "real" code for all languages with a large and detailed sample set. In number 3 you aren't "controlling" the variables so much as diligently including them in the observations. The baseline skill of a programmer sounds to me more like an unknown than a variable-- something more worth studying first than controlling.
- igouy 12y ago
  The closest to #3 available seems to be -- "Plat_Forms - The web development platform comparison" https://www.plat-forms.org/ https://www.plat-forms.org/

by briandh 12y ago

Alupis 12y ago

The "study" does not account for nuances in each language. JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed. The tests did not account for JVM warm-up time, nor allow the code to become "hot" before running their tests. They also executed their test code on poorly performing operating systems not designed to execute high performance code, ie. they ran their tests on consumer desktop general purpose operating systems instead of a more specialized system designed for the task... or at the very least a more specialized OS such as a flavor of BSD or Linux without the desktop and other generalized packages. I don't really consider this a well done test, and the results are only surface-level... certainly nothing you should base a serious decision on.

taeric 12y ago
JIT'ed Java code is lacking a major facility that truly performant native code has, and that is cache locality. That is, it isn't enough that the code to perform the algorithm is "well tuned" if the data structures are not. (Well, unless I am just wrong on the feats that JIT'ed code can perform nowdays.)
- pkolaczk 12y ago
  While I agree Java makes writing cache-friendly code much harder than C, it is no-way impossible. Nor is any C or C++ code implicitly cache-friendly. Generally, in both cases you must know what you're doing. I also hope some of those problems are going to be addressed in Java 9.
  - taeric 12y ago
    Fair enough. My point was supposed to be more limited, saying that a JIT'ed solution is not a cure all for performance considerations. Sure, it is great that there is a JIT compiler helping out. It just is not necessarily enough on its own to even the performance gap between native and not.
    - Alupis 12y ago
      It is enough. You can perf test the same non-trivial program (such as a biz application) in C and Java, and given you allow the JVM to warm up, discover hot spots, and JIT your code... they will run just about the same performance level... every time. The JVM is impressive.
      - sixbrx 12y ago
        JVM is impressive until closures are involved, in which case when inlining is not done (either because it cannot be or ever-changing heuristics are not met), you can drop back to virtual calls and performance suffers a great deal. Compare to C++ where closures can be used with virtually no penalty and aggressively optimized. Even C with its trick of passing an extra *void for environment comes out shining performance-wise in comparison. Edit: There's a good discussion of the problems that JVMs have in this area at http://www.azulsystems.com/blog/cliff/2011-04-04-fixing-the-inlining-problem http://www.azulsystems.com/blog/cliff/2011-04-04-fixing-the-.... "The Problem is a hard one to solve; (i.e. it’s not the case that we’ll wave our wands and do a quick fix & rebuild HotSpot and the problem will go away). John Rose, Doug Lea and I all agree that it’s a Major Problem facing JVMs with long and far reaching implications." Maybe progress will be made soon though, with more pressure caused by increasing use of closures in the new streams library and with anonymous function syntax available now.
jeremyjh 12y ago
They also didn't account for the possiblity that a Java programmer would use an ArrayList<Double> instead of double[]. You can't JIT that away; JIT does not change the semantics of the Java code so you have to code Java in a sympathetic way. I can imagine this may be more idiomatic in numeric intensive fields though.
maxlybbert 12y ago
The authors do have a comment on the Github project that code needs to be allowed to run for a while so the JIT can kick in, although I didn't see any statement of how they did that. The paper also mentions a decision to turn a loop body into a function to allow a JIT to optimize things. But, generally, I'm not interested in whether "for" loops run faster in language X or language Y. I am interested in whether some languages encourage code styles that happen to be easily optimized (I'm also interested in whether some languages encourage code styles that are easily maintained, easily extensible, or easily run into a ditch). This paper is meaningful to somebody, but it's clearly written by people who spend most of their thinking about things other than programming. That's not an insult; it's a simple fact that a lot of programs are written by people who are experts in other fields. I happen to work with a former physicist who has experience writing programs to run on one of the nation's largest supercomputers. The depressing thing (to me) is that while he knows how to write numerical programs that run close to a computer's theoretical maximum speed, that information is only well-known among a small group of people (namely people who have to show that if they're allowed to use a supercomputer, they won't make it run wasteful code). I wish it were shared more broadly.
gd1 12y ago
>JIT'ed java code will run at native C performance... because it essentially is native C once it's been JIT'ed No way. It may be native C, but it's not native C you've hand-coded yourself.
- fulafel 12y ago
  C doesn't enter into it, any way you look at it. Unless you are using some obscure Java -> C compiler.
  - Alupis 12y ago
    No, the optimizing compiler inside the JVM does produce a lot of native code for your system... c/assembler. It just has to discover hot spots then JIT them.

grayclhn 12y ago

> One of they authors addresses this in the comments on the OP: >> As we explain in the paper, this was a “design” parameter. We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. Sure, but this makes the paper less useful for answering the question, "which programming language should I use?" Or, "which programming language should I learn better?" But, yes, most economists are pretty bad programmers, so "which language should I use poorly?" is probably a useful question to answer.

igouy 12y ago
Which language should I learn to use better? is probably a useful question to answer.

Goladus 12y ago

We did not want to “adapt” the algorithm to each language to improve speed, because it will open the door to too many “uncontrolled variables” (how good a programmer in that language you are and so on). The only partial exception was with Mathematica for reasons explained in the paper. This is rather half-baked reasoning. The only ways I can think of to control for programmer skill or at least account for it is to 1. Use "idiomatic" code for all languages. 2. Use "optimized" code for all languages. 3. Use "real" code for all languages with a large and detailed sample set. In number 3 you aren't "controlling" the variables so much as diligently including them in the observations. The baseline skill of a programmer sounds to me more like an unknown than a variable-- something more worth studying first than controlling.

igouy 12y ago
The closest to #3 available seems to be -- "Plat_Forms - The web development platform comparison" https://www.plat-forms.org/ https://www.plat-forms.org/