7 ms·

Measuring the Haskell Gap [pdf]

148 points by dons 13y ago

6ren 13y ago
Scribd got better! It's comparable, perhaps better, than google's viewer: https://docs.google.com/viewer?url=http%3A%2F%2Fwww.leafpetersen.com%2Fleaf%2Fpublications%2Fhs2013%2Fhaskell-gap.pdf https://docs.google.com/viewer?url=http%3A%2F%2Fwww.leafpete...
- sambeau 13y ago
  Did Scribd get 'better' in an empirical performance comparison with google's viewer? For me to take such a comparison seriously, you need to present evidence to us transparently, with the understanding that every such comparison inevitably relies on making choices and is hence only meaningful insofar as those choices can be seen and understood by the reader
  - DanWaterworth 13y ago
    I'm not sure why you were down voted. I thought what you said was very witty, but I suppose it only works if you read the paper.
- jkldotio 13y ago
  Isn't Google's viewer really the default PDF viewer in Chrome? The one which doesn't add a toolbar or any tooltips that are popping up despite my mouse not hovering over them. Scribd is for people who want to share a PDF but don't know how to do it in any other way. It's pretty much never been welcome on HN because that's the only problem it solves and everything else it does is inferior to just having a native PDF you can view with no problems and save with no problems. We are not the target market so I never understood why it was pushed on HN at all.
  - soganess 13y ago
    Chrome use an unbranded version of foxit[1][2]. [1]http://googlesystem.blogspot.com/2010/08/google-chromes-pdf-plugin-uses-foxit.html http://googlesystem.blogspot.com/2010/08/google-chromes-pdf-... [2]independently verified to me by a foxit employee.
  - omaranto 13y ago
    > Isn't Google's viewer really the default PDF viewer in Chrome? The one which doesn't add a toolbar or any tooltips that are popping up despite my mouse not hovering over them. No, it's not, unless, Google figured out a way to install their PDF viewer plugin in my Firefox without me noticing. ;) The link 6ren posted is to Google Docs's file viewing component, now sold separately: https://docs.google.com/viewer https://docs.google.com/viewer I'm not completely sure, but I think that for PDFs what the Google Docs viewer does is (1) generate a PNG image of each page, (2) extract the text and put in on the page as invisible HTML, positioned carefully to correspond to the PNG. The point of (2) is that it makes text selectable.
  - j_s 13y ago
    > why it was pushed on HN at all I agree scribd is terrible/useless (personal opinion) and does catch a lot of flack here, but it's YC S06. https://news.ycombinator.com/item?id=1326047 https://news.ycombinator.com/item?id=1326047
- venomsnake 13y ago
  Thanks. PDF is PITA to read on anything but paper anyway, but every little bit helps I suppose.
  - seanmcdirmid 13y ago
    PDFs are fairly easy to read on computers, especially high res iPads.
berkut 13y ago
It's a good conclusion I feel: this is always the issue with language benchmarks - who wrote the code, and how good were they with each of the languages. Similarly as the article points out, the compiler matters a lot: ICC can in certain cases be more than 200% faster than GCC with similar flags, and is generally 15-20% faster anyway, mainly due to more intelligent inlining and much faster (and more accurate with fpmath=fast) math libs.
- copx 13y ago
  ..only on Intel chips. It deliberately generates code that runs slow on non-Intel CPUs: http://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Criticism http://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Criticis... As an AMD user I really hope most programmers know this by now. If you make a build for the general public as opposed to only targeting Intel machines, please don't use ICC.
  - berkut 13y ago
    Not since 2010 it doesn't: http://www.hardware.fr/articles/847-1/impact-compilateurs-architectures-cpu-x86-x64.html http://www.hardware.fr/articles/847-1/impact-compilateurs-ar... It can generate code which when run on AMD is faster than GCC and MSVC.
    - copx 13y ago
      I don't read French but Intel's current official compiler documentation.. http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-win/index.htm http://software.intel.com/sites/products/documentation/docli... ..suggests nothing has changed. Search for "non-Intel". Maybe ICC generates code which beats GCC even when run on an AMD chip in one particular benchmark but that doesn't mean it generates better code in general. Personally I will never trust the Intel compiler, because it's part of their business strategy to generate bad code for AMD processors. Even if the claim in the original post about being "generally 15-20% faster" were true for Intel chips, it wouldn't be 15-20% faster on AMD or Intel's documentation - which clearly states the compiler generates inferior code for non-Intel chips - is wrong.
joelthelion 13y ago
This is a nicely done benchmark, and an impressive demonstration of HRC. Thanks for posting!
crncosta 13y ago
Does any one know if the authors are sharing the benchmark's source code? tks.
arocks 13y ago
It is interesting to read how Haskell optimized the algorithm based on the instrinsic properties of the data structures. In contrast, C compilers leveraged on the knowledge of the underlying machine. It is amazing how far Haskell compilers have come.
mhaymo 13y ago
I'm surprised by how dramatic the difference is between the speed of C and Haskell. One of my professors (at The University of Glasgow, so appropriately a Haskell fan) once claimed that it had "c-like performance". I suppose that's the point of this paper though, that "c-like performance" is a terribly vague term, meaningless without knowledge of the specific comparisons being made.
- octo_t 13y ago
  For lots of algorithms, fairly naive Haskell can get very close in performance (within 10%) of pretty decent C or C++. For example[1] shows that for very advanced algorithms (such as BLAS), Haskell can be very performant - with the optimisation being reusable and transparent to the programmer. [1] - http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/haskell-beats-C.pdf http://research.microsoft.com/en-us/um/people/simonpj/papers...
  - strmpnk 13y ago
    That's kind of an odd comparison, using unfused C code compared to fused Haskell code. Their point seems to focus on stream fusion's advantages and possibly that optimizing C takes more effort. To quote the paper: "Clearly “properly”-written C++ can outperform Haskell. The challenge is in ﬁguring out what “proper” means."
- igouy 13y ago
  >> a terribly vague term, meaningless without knowledge of the specific comparisons being made << otoh http://benchmarksgame.alioth.debian.org/u32/program.php?test=nbody&lang=gcc&id=1 http://benchmarksgame.alioth.debian.org/u32/program.php?test... otoh http://benchmarksgame.alioth.debian.org/u32/program.php?test=nbody&lang=cint&id=1 http://benchmarksgame.alioth.debian.org/u32/program.php?test... :-)
- Paradigma11 13y ago
  Well, but my Haskell code does not have to beat the best possible c code but only my best c code ;)
breckinloggins 13y ago
This is a great paper; it is very readable and motivated and I learned quite a bit. I'm also now looking forward to perusing the 2012 Ninja C paper. One small change I would make to the preprint would be to better normalize the graph symbols. In particular, failing to read the legends of each graph carefully might cause readers to misattribute results in subsequent graphs (for example, my mind wanted to associate Intel's HRC with "the lighter gray ones with the boxes", which is not a stable representation across all graphs).
- yalue 13y ago
  I would add to this comment that I would appreciate labels for the y-axes of the graphs.
  - jfarmer 13y ago
    The y-axis is unit-less. They normalized the run times by the "Normal C" run time, so that, e.g., 2.0 means "took twice as long as Normal C."
yogsototh 13y ago
I would be interested to see how jhc [1] compare to ghc. I am not sure Repa could be easily compiled with jhc thought. [1]: http://repetae.net/computer/jhc/ http://repetae.net/computer/jhc/
- dumael 13y ago
  http://mirror.seize.it/report.html http://mirror.seize.it/report.html That report is quite old though and just compares compilers with the nofib benchmark. JHC can't compile repa as repa requires multi-parameter type classes which JHC doesn't support unfortunately.
sirspazzolot 13y ago
"DRAFT - Not for redistribution" haha Very interested in seeing where Haskell is headed in the future. Major props to Intel for the disclaimer that this isn't a definitive study.
beefman 13y ago
Summary: Intel's HRC (Haskell Research Compiler) is an optimizing compiler for GHC's (Glasgow Haskell Compiler) "Core" intermediate language.* On six common benchmarks, it improves the performance of Haskell dramatically. But Haskell is still 4 times slower than the best C implementations of these benchmarks, on average. * Core is just desugared Haskell and should not be confused with GHC's other intermediate languages, STG and C--. And there is no relation to Intel's "Core" microarchitecture.

148 points by dons 13y ago

6ren 13y ago

Scribd got better! It's comparable, perhaps better, than google's viewer: https://docs.google.com/viewer?url=http%3A%2F%2Fwww.leafpetersen.com%2Fleaf%2Fpublications%2Fhs2013%2Fhaskell-gap.pdf https://docs.google.com/viewer?url=http%3A%2F%2Fwww.leafpete...

sambeau 13y ago
Did Scribd get 'better' in an empirical performance comparison with google's viewer? For me to take such a comparison seriously, you need to present evidence to us transparently, with the understanding that every such comparison inevitably relies on making choices and is hence only meaningful insofar as those choices can be seen and understood by the reader
- DanWaterworth 13y ago
  I'm not sure why you were down voted. I thought what you said was very witty, but I suppose it only works if you read the paper.
jkldotio 13y ago
Isn't Google's viewer really the default PDF viewer in Chrome? The one which doesn't add a toolbar or any tooltips that are popping up despite my mouse not hovering over them. Scribd is for people who want to share a PDF but don't know how to do it in any other way. It's pretty much never been welcome on HN because that's the only problem it solves and everything else it does is inferior to just having a native PDF you can view with no problems and save with no problems. We are not the target market so I never understood why it was pushed on HN at all.
- soganess 13y ago
  Chrome use an unbranded version of foxit[1][2]. [1]http://googlesystem.blogspot.com/2010/08/google-chromes-pdf-plugin-uses-foxit.html http://googlesystem.blogspot.com/2010/08/google-chromes-pdf-... [2]independently verified to me by a foxit employee.
- omaranto 13y ago
  > Isn't Google's viewer really the default PDF viewer in Chrome? The one which doesn't add a toolbar or any tooltips that are popping up despite my mouse not hovering over them. No, it's not, unless, Google figured out a way to install their PDF viewer plugin in my Firefox without me noticing. ;) The link 6ren posted is to Google Docs's file viewing component, now sold separately: https://docs.google.com/viewer https://docs.google.com/viewer I'm not completely sure, but I think that for PDFs what the Google Docs viewer does is (1) generate a PNG image of each page, (2) extract the text and put in on the page as invisible HTML, positioned carefully to correspond to the PNG. The point of (2) is that it makes text selectable.
- j_s 13y ago
  > why it was pushed on HN at all I agree scribd is terrible/useless (personal opinion) and does catch a lot of flack here, but it's YC S06. https://news.ycombinator.com/item?id=1326047 https://news.ycombinator.com/item?id=1326047
venomsnake 13y ago
Thanks. PDF is PITA to read on anything but paper anyway, but every little bit helps I suppose.
- seanmcdirmid 13y ago
  PDFs are fairly easy to read on computers, especially high res iPads.

berkut 13y ago

It's a good conclusion I feel: this is always the issue with language benchmarks - who wrote the code, and how good were they with each of the languages. Similarly as the article points out, the compiler matters a lot: ICC can in certain cases be more than 200% faster than GCC with similar flags, and is generally 15-20% faster anyway, mainly due to more intelligent inlining and much faster (and more accurate with fpmath=fast) math libs.

copx 13y ago
..only on Intel chips. It deliberately generates code that runs slow on non-Intel CPUs: http://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Criticism http://en.wikipedia.org/wiki/Intel_C%2B%2B_Compiler#Criticis... As an AMD user I really hope most programmers know this by now. If you make a build for the general public as opposed to only targeting Intel machines, please don't use ICC.
- berkut 13y ago
  Not since 2010 it doesn't: http://www.hardware.fr/articles/847-1/impact-compilateurs-architectures-cpu-x86-x64.html http://www.hardware.fr/articles/847-1/impact-compilateurs-ar... It can generate code which when run on AMD is faster than GCC and MSVC.
  - copx 13y ago
    I don't read French but Intel's current official compiler documentation.. http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-win/index.htm http://software.intel.com/sites/products/documentation/docli... ..suggests nothing has changed. Search for "non-Intel". Maybe ICC generates code which beats GCC even when run on an AMD chip in one particular benchmark but that doesn't mean it generates better code in general. Personally I will never trust the Intel compiler, because it's part of their business strategy to generate bad code for AMD processors. Even if the claim in the original post about being "generally 15-20% faster" were true for Intel chips, it wouldn't be 15-20% faster on AMD or Intel's documentation - which clearly states the compiler generates inferior code for non-Intel chips - is wrong.

joelthelion 13y ago

This is a nicely done benchmark, and an impressive demonstration of HRC. Thanks for posting!

crncosta 13y ago

Does any one know if the authors are sharing the benchmark's source code? tks.

arocks 13y ago

It is interesting to read how Haskell optimized the algorithm based on the instrinsic properties of the data structures. In contrast, C compilers leveraged on the knowledge of the underlying machine. It is amazing how far Haskell compilers have come.

mhaymo 13y ago

I'm surprised by how dramatic the difference is between the speed of C and Haskell. One of my professors (at The University of Glasgow, so appropriately a Haskell fan) once claimed that it had "c-like performance". I suppose that's the point of this paper though, that "c-like performance" is a terribly vague term, meaningless without knowledge of the specific comparisons being made.

octo_t 13y ago
For lots of algorithms, fairly naive Haskell can get very close in performance (within 10%) of pretty decent C or C++. For example[1] shows that for very advanced algorithms (such as BLAS), Haskell can be very performant - with the optimisation being reusable and transparent to the programmer. [1] - http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/haskell-beats-C.pdf http://research.microsoft.com/en-us/um/people/simonpj/papers...
- strmpnk 13y ago
  That's kind of an odd comparison, using unfused C code compared to fused Haskell code. Their point seems to focus on stream fusion's advantages and possibly that optimizing C takes more effort. To quote the paper: "Clearly “properly”-written C++ can outperform Haskell. The challenge is in ﬁguring out what “proper” means."
igouy 13y ago
>> a terribly vague term, meaningless without knowledge of the specific comparisons being made << otoh http://benchmarksgame.alioth.debian.org/u32/program.php?test=nbody&lang=gcc&id=1 http://benchmarksgame.alioth.debian.org/u32/program.php?test... otoh http://benchmarksgame.alioth.debian.org/u32/program.php?test=nbody&lang=cint&id=1 http://benchmarksgame.alioth.debian.org/u32/program.php?test... :-)
Paradigma11 13y ago
Well, but my Haskell code does not have to beat the best possible c code but only my best c code ;)

breckinloggins 13y ago

This is a great paper; it is very readable and motivated and I learned quite a bit. I'm also now looking forward to perusing the 2012 Ninja C paper. One small change I would make to the preprint would be to better normalize the graph symbols. In particular, failing to read the legends of each graph carefully might cause readers to misattribute results in subsequent graphs (for example, my mind wanted to associate Intel's HRC with "the lighter gray ones with the boxes", which is not a stable representation across all graphs).

yalue 13y ago
I would add to this comment that I would appreciate labels for the y-axes of the graphs.
- jfarmer 13y ago
  The y-axis is unit-less. They normalized the run times by the "Normal C" run time, so that, e.g., 2.0 means "took twice as long as Normal C."

yogsototh 13y ago

I would be interested to see how jhc [1] compare to ghc. I am not sure Repa could be easily compiled with jhc thought. [1]: http://repetae.net/computer/jhc/ http://repetae.net/computer/jhc/

dumael 13y ago
http://mirror.seize.it/report.html http://mirror.seize.it/report.html That report is quite old though and just compares compilers with the nofib benchmark. JHC can't compile repa as repa requires multi-parameter type classes which JHC doesn't support unfortunately.

sirspazzolot 13y ago

"DRAFT - Not for redistribution" haha Very interested in seeing where Haskell is headed in the future. Major props to Intel for the disclaimer that this isn't a definitive study.

beefman 13y ago

Summary: Intel's HRC (Haskell Research Compiler) is an optimizing compiler for GHC's (Glasgow Haskell Compiler) "Core" intermediate language.* On six common benchmarks, it improves the performance of Haskell dramatically. But Haskell is still 4 times slower than the best C implementations of these benchmarks, on average. * Core is just desugared Haskell and should not be confused with GHC's other intermediate languages, STG and C--. And there is no relation to Intel's "Core" microarchitecture.