5 ms·

A Comparison of Programming Languages in Economics [pdf]

36 points by todd8 11y ago

Hermel 11y ago
As having both, an economics and computer science background, I often feel like economists focus on the wrong metrics when evaluating programming languages. The importance of raw computing speed of a programming language pales in comparison with the importance of the applied methods, as well as the time saved in development. For example, while a speed-up by a factor of two sounds good on paper, the choice of search strategy in a constrained optimization problem can change the running time by several orders of magnitude. I am currently working on a model where the seemingly unimportant choice of price normalization (i.e. which price in the model is set to 1.0 to determine the price level) can make a difference between 2 seconds or 200 seconds of computation time.
oelang 11y ago
Meh, they dismiss java & pypy very quickly but they don't understand that these languages require warm-up. If you benchmark runs in less than 2 seconds half of the time measured is starting the runtime, profiling & compiling code. I think that interesting computational problems in economics have a much longer running time. For this kind of algorithm, basically working with arrays of doubles, java should be within 20% of c++ and normally it is just as fast.
- nicolapede 11y ago
  Would you have any reference regarding the performance of java/c++ with respect to manipulation of arrays of doubles? Just curious to know where that 20% comes from.
  - oelang 11y ago
    The 20% comes from personal experience, maybe I should have said 'close' but that can also mean anything. Basically this is an area where the hotspot heavy duty compiler (C2) does the same optimizations as gcc or llvm. I would say that I would expect gcc & llvm to be within 20% of each other. The 20% is basically different tradeoffs made in the optimization pipeline (different peephole opts, different auto-vectorisation, register allocation algorithm, ...). Java starts to seriously lags behind C++ when you need good control over the memory layout of your data. In java you can't have an array of objects, you get an array with references to objects. There are no elegant ways to solve that problem, the language just lacks the proper tools to do it elegantly. Memory layout is of course vital for cache efficiency, so that can give C & C++ a big advantage over java. Since this benchmarks uses arrays of doubles, this isn't an issue.
    - nicolapede 11y ago
      I see your point now, thanks for the explanation. I mostly do scientific calculations, so effectively manipulations of array of doubles, and I currently mostly use C#. I was under the impression that the performance is similar to C++ (GPU aside), for that the comparison with Java had got my attention.
      - cema 11y ago
        Even though C# is in many ways like Java, it allows (via unsafe blocks and pointers) to better perform memory management. Not sure how widely it is used in practice.
- east2west 11y ago
  Does Java have any way of auto vectorization? On modern CPUs SIMD can speed up numerical computation several times easily. However auto vectorization is hard in general, so compilers don't necessarily do a good job and manually optimized code is common. C++ has TMP and libraries such as Eigen use TMP to provide a nice interface to SIMD optimized code. I don't know whether Java has equivalent of compiler intrinsics for SIMD instructions, without which Java code would be far slower for many numerical computation. I believe C# has plans to add an abstraction for SIMD and Mono can already emit SIMD instructions.
  - oelang 11y ago
    Intel recently contributed improved auto-vectorisation for artithmetic to the hotspot compiler. Auto-vectorisation of move instructions has been in hotspot for a few years.
- igouy 11y ago
  >> If you benchmark runs in less than 2 seconds half of the time measured is starting the runtime, profiling & compiling code.<< Please show that it takes 1 whole second (measured after the Java program starts and before it terminates) for the JVM to profile and compile the 100 lines of code. That would suggest the JVM was pure junk.
- igouy 11y ago
  >>half of the time measured is starting the runtime, profiling & compiling code<< Here are 10 cold-run times reported by their program: Elapsed time is 3.122233018 seconds. Elapsed time is 2.970587912 seconds. Elapsed time is 3.110304658 seconds. Elapsed time is 2.886701204 seconds. Elapsed time is 2.966404493 seconds. Elapsed time is 2.905329605 seconds. Elapsed time is 2.968715703 seconds. Elapsed time is 2.922929706 seconds. Elapsed time is 2.906730019 seconds. Elapsed time is 2.919119506 seconds. Here are the stats reported by JMH after 200 samples: Result : 2.950 ±(99.9%) 0.015 s/op Statistics: (min, avg, max) = (2.932, 2.950, 2.984), stdev = 0.017 Confidence interval (99.9%): [2.935, 2.965]
- igouy 11y ago
  There is a problem with their representation of Java, and it has nothing to do with "require warm-up". The problem is that their Java program is just a single static method. Their Java program is not written as Java.
oelang 11y ago
"Java's issues with floating point arithmetic in high-performance scientific computation" Can someone explain this?
- Hermel 11y ago
  Unlike C++, Java ensures that floating point calculations always yield the exact same result, regardless of the underlying platform. This can degrate performance. I am not sure if they meant that thought, in my opinion this is an advantage of Java.
  - oelang 11y ago
    Floating point calculations are only guaranteed to be platform independent when you use the 'strictfp' modifier.
- scottlocklin 11y ago
  One of the issues is, or used to be, data locality. Arrays can be all over the place. Lots of cache stalls that way. There are tricks to deal with it, but I've never seen it in open source code. Might as well just use blas at that point.
pella 11y ago
github: https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-...
k2enemy 11y ago
https://news.ycombinator.com/item?id=7978410 https://news.ycombinator.com/item?id=7978410 Lots of discussion from a year ago.
pvitz 11y ago
I would be happy to use languages like J/K or Julia for our actuarial projection simulations instead of C++. However, despite quicker development times, a factor of 2 in the runtime for our stochastic projections would be terrible. Has anyone experience with those languages in an HPC environment and can compare it to a C++ implementation?
- scottlocklin 11y ago
  Dang, someone wanting to replace C++ with J... I'd love to say you could do this, since I am a big fan of J, but if you need to squeeze the last bit of efficiency out of a machine on a problem that can't be parallelized, C++ is probably a pretty good idea. If the "expensive in write time" logic isn't expensive in calculation time, you could wrap the expensive pieces in some verbs which call out to C. I'm often surprised though; a naive implementation of wavelets in J was quite comparable in runtime performance to a naive implementation of wavelets in C.
- Lofkin 11y ago
  Try numba in python. Its usually fortran fast and often faster than Julia.

36 points by todd8 11y ago

As having both, an economics and computer science background, I often feel like economists focus on the wrong metrics when evaluating programming languages. The importance of raw computing speed of a programming language pales in comparison with the importance of the applied methods, as well as the time saved in development. For example, while a speed-up by a factor of two sounds good on paper, the choice of search strategy in a constrained optimization problem can change the running time by several orders of magnitude. I am currently working on a model where the seemingly unimportant choice of price normalization (i.e. which price in the model is set to 1.0 to determine the price level) can make a difference between 2 seconds or 200 seconds of computation time.

oelang 11y ago

Meh, they dismiss java & pypy very quickly but they don't understand that these languages require warm-up. If you benchmark runs in less than 2 seconds half of the time measured is starting the runtime, profiling & compiling code. I think that interesting computational problems in economics have a much longer running time. For this kind of algorithm, basically working with arrays of doubles, java should be within 20% of c++ and normally it is just as fast.

nicolapede 11y ago
Would you have any reference regarding the performance of java/c++ with respect to manipulation of arrays of doubles? Just curious to know where that 20% comes from.
- oelang 11y ago
  The 20% comes from personal experience, maybe I should have said 'close' but that can also mean anything. Basically this is an area where the hotspot heavy duty compiler (C2) does the same optimizations as gcc or llvm. I would say that I would expect gcc & llvm to be within 20% of each other. The 20% is basically different tradeoffs made in the optimization pipeline (different peephole opts, different auto-vectorisation, register allocation algorithm, ...). Java starts to seriously lags behind C++ when you need good control over the memory layout of your data. In java you can't have an array of objects, you get an array with references to objects. There are no elegant ways to solve that problem, the language just lacks the proper tools to do it elegantly. Memory layout is of course vital for cache efficiency, so that can give C & C++ a big advantage over java. Since this benchmarks uses arrays of doubles, this isn't an issue.
  - nicolapede 11y ago
    I see your point now, thanks for the explanation. I mostly do scientific calculations, so effectively manipulations of array of doubles, and I currently mostly use C#. I was under the impression that the performance is similar to C++ (GPU aside), for that the comparison with Java had got my attention.
    - cema 11y ago
      Even though C# is in many ways like Java, it allows (via unsafe blocks and pointers) to better perform memory management. Not sure how widely it is used in practice.
east2west 11y ago
Does Java have any way of auto vectorization? On modern CPUs SIMD can speed up numerical computation several times easily. However auto vectorization is hard in general, so compilers don't necessarily do a good job and manually optimized code is common. C++ has TMP and libraries such as Eigen use TMP to provide a nice interface to SIMD optimized code. I don't know whether Java has equivalent of compiler intrinsics for SIMD instructions, without which Java code would be far slower for many numerical computation. I believe C# has plans to add an abstraction for SIMD and Mono can already emit SIMD instructions.
- oelang 11y ago
  Intel recently contributed improved auto-vectorisation for artithmetic to the hotspot compiler. Auto-vectorisation of move instructions has been in hotspot for a few years.
igouy 11y ago
>> If you benchmark runs in less than 2 seconds half of the time measured is starting the runtime, profiling & compiling code.<< Please show that it takes 1 whole second (measured after the Java program starts and before it terminates) for the JVM to profile and compile the 100 lines of code. That would suggest the JVM was pure junk.
igouy 11y ago
>>half of the time measured is starting the runtime, profiling & compiling code<< Here are 10 cold-run times reported by their program: Elapsed time is 3.122233018 seconds. Elapsed time is 2.970587912 seconds. Elapsed time is 3.110304658 seconds. Elapsed time is 2.886701204 seconds. Elapsed time is 2.966404493 seconds. Elapsed time is 2.905329605 seconds. Elapsed time is 2.968715703 seconds. Elapsed time is 2.922929706 seconds. Elapsed time is 2.906730019 seconds. Elapsed time is 2.919119506 seconds. Here are the stats reported by JMH after 200 samples: Result : 2.950 ±(99.9%) 0.015 s/op Statistics: (min, avg, max) = (2.932, 2.950, 2.984), stdev = 0.017 Confidence interval (99.9%): [2.935, 2.965]
igouy 11y ago
There is a problem with their representation of Java, and it has nothing to do with "require warm-up". The problem is that their Java program is just a single static method. Their Java program is not written as Java.

oelang 11y ago

"Java's issues with floating point arithmetic in high-performance scientific computation" Can someone explain this?

Hermel 11y ago
Unlike C++, Java ensures that floating point calculations always yield the exact same result, regardless of the underlying platform. This can degrate performance. I am not sure if they meant that thought, in my opinion this is an advantage of Java.
- oelang 11y ago
  Floating point calculations are only guaranteed to be platform independent when you use the 'strictfp' modifier.
scottlocklin 11y ago
One of the issues is, or used to be, data locality. Arrays can be all over the place. Lots of cache stalls that way. There are tricks to deal with it, but I've never seen it in open source code. Might as well just use blas at that point.

pella 11y ago

github: https://github.com/jesusfv/Comparison-Programming-Languages-Economics https://github.com/jesusfv/Comparison-Programming-Languages-...

k2enemy 11y ago

https://news.ycombinator.com/item?id=7978410 https://news.ycombinator.com/item?id=7978410 Lots of discussion from a year ago.

pvitz 11y ago

I would be happy to use languages like J/K or Julia for our actuarial projection simulations instead of C++. However, despite quicker development times, a factor of 2 in the runtime for our stochastic projections would be terrible. Has anyone experience with those languages in an HPC environment and can compare it to a C++ implementation?

scottlocklin 11y ago
Dang, someone wanting to replace C++ with J... I'd love to say you could do this, since I am a big fan of J, but if you need to squeeze the last bit of efficiency out of a machine on a problem that can't be parallelized, C++ is probably a pretty good idea. If the "expensive in write time" logic isn't expensive in calculation time, you could wrap the expensive pieces in some verbs which call out to C. I'm often surprised though; a naive implementation of wavelets in J was quite comparable in runtime performance to a naive implementation of wavelets in C.
Lofkin 11y ago
Try numba in python. Its usually fortran fast and often faster than Julia.