7 ms·

We have a code base of roughly 200,000 lines of Haskell code, dealing with high performance SQL query parsing, compilation and optimizations. I only remember o

by arnon 9y ago

We have a code base of roughly 200,000 lines of Haskell code, dealing with high performance SQL query parsing, compilation and optimizations. I only remember one situation over the past 5 years that we had a performance issue with Haskell, that was solved by using the profiling capabilities of GHC. I disagree that performance is hard to figure out. It could be better, yes - but it's not that different than what you'd get with other programming languages.

luispedrocoelho 9y ago
(OP here) I do say it's not that big of a deal in the end: it almost always is OK and at the end you optimize the inner loops by looking at profiles; like in any other project. But when using GHC, I have indeed sometimes ran into situations where I expect something to be fast when it is not (e.g., `ByteString.map (+ value)` is incredibly slow compared to a pseudo-C loop). I also did find a bona fides performance bug in GHC https://ghc.haskell.org/trac/ghc/ticket/11783 https://ghc.haskell.org/trac/ghc/ticket/11783
- arnon 9y ago
  GHC isn't magical. It has bugs too. We had an issue with the Data.Text package and OverloadedStrings in 7.10.2 which caused extremely slow compilation times and filed a bug report for that, which was solved for 7.10.3.
- codygman 9y ago
  > (e.g., `ByteString.map (+ value)` is incredibly slow compared to a pseudo-C loop). That situation sounds like you needed a ByteString Builder to get comparable performance.
  - luispedrocoelho 9y ago
    Maybe, but pseudo-C actually worked very well. I even have some "real C" in my Haskell code to handle some inner loop stuff.
knz42 9y ago
Hi can you tell us a bit more about this SQL-related project? I'm rather practically interested.
- arnon 9y ago
  I work for www.sqream.com, and our main product is SQream DB. SQream DB is a GPU SQL database for analytics. Everything was written in-house - from the SQL parser all the way down to the storage layer. We're designed to deal with sizes from a few terabytes to hundreds of terabytes, and we use the GPU to do most of the work. It is the compiler (written in Haskell) however, that decides about specific optimizations and deals with generating the query plan. Our SQL parser and typechecker is actually BSD3 licensed, if it's interesting: https://github.com/jakewheat/hssqlppp https://github.com/jakewheat/hssqlppp
  - shusson 9y ago
    Any chance SQream DB will be open sourced or a demo version? It seems every Column based GPU DB I find is behind a paywall and targeting enterprise levels.
    - arnon 9y ago
      I think it's unlikely at this point... We too are currently during some large projects for enterprises... We will be releasing a community version on AWS and Azure in the near future which should be cheap or free, other than the instance cost on the respective cloud provider.
  - Veedrac 9y ago
    > www.sqream.com I don't like having to be that guy, but your landing page hijacks scrolling (which always causes problems; I noticed first because it broke ctrl-scroll zooming), downloads a 30MB video in a loop, forever, takes a significant amount of CPU even when the video isn't in view and the tab is not active, and despite having almost no content manages to take far more memory than any other non-email tab I have open.
zzalpha 9y ago
but it's not that different than what you'd get with other programming languages. Until you have to solve a (maddeningly common) space leak issue. That's a problem unique to lazily evaluated languages (basically Haskell) and is godawful to debug. It reminds me of solving git problems... you suddenly find yourself having to tear back the abstraction layer that is the language and compiler and start thinking about thunks and so forth. It's jarring and incredibly frustrating.
- platz 9y ago
  here is a method that can help debug space leaks: http://neilmitchell.blogspot.com/2015/09/detecting-space-leaks.html http://neilmitchell.blogspot.com/2015/09/detecting-space-lea...
  - zzalpha 9y ago
    I wish the problem was simply detecting and isolating space leaks. My experience is that actually fixing them can be incredibly difficult, hence my comment about needing to understand the gory details about how the runtime evaluates lazy expressions. Heck, that post even uses the phrase "Attempt to fix the space leak" as it's often not an obvious slam dunk. Sometimes it even devolves to peppering !'s around until the problem goes away.
    - dllthomas 9y ago
      > My experience is that actually fixing them can be incredibly difficult My experience differs, FWIW. If you know where you're creating too many thunks, and you force them as you create them, they don't accumulate. Making sure you actually are forcing them, and not simply suspending a "force this", is probably the trickiest bit until you're used to the evaluation model.
      - zzalpha 9y ago
        If you know where you're creating too many thunks, and you force them as you create them, they don't accumulate. Translation: if you've internalized the way Haskell code is compiled and executes, so that you can easily reason about how lazy evaluation is actually implemented, you can solve these problems. If not, it devolves to throwing !'s in and praying. Which is basically my point. If I don't have a hope of solving common space leaks without deeply understanding how Haskell is evaluated, that's a real challenge for anyone trying to learn the language.
nom 9y ago
> but it's not that different than what you'd get with other programming languages Does your list of other programming languages include C/C++/asm? ;)
- arnon 9y ago
  Incidentally, yes.
Confusion 9y ago
I find that incredibly hard to believe. I wonder whether you are looking at history through rose-tinted glasses or you simply don't know because other people quietly solved those problems. I ran into such problems several times in 50 line programs and not because my code was non-idiomatic or wrong. #haskell agreed it was non-trivial to get the code to perform well.

by arnon 9y ago

luispedrocoelho 9y ago

(OP here) I do say it's not that big of a deal in the end: it almost always is OK and at the end you optimize the inner loops by looking at profiles; like in any other project. But when using GHC, I have indeed sometimes ran into situations where I expect something to be fast when it is not (e.g., `ByteString.map (+ value)` is incredibly slow compared to a pseudo-C loop). I also did find a bona fides performance bug in GHC https://ghc.haskell.org/trac/ghc/ticket/11783 https://ghc.haskell.org/trac/ghc/ticket/11783

arnon 9y ago
GHC isn't magical. It has bugs too. We had an issue with the Data.Text package and OverloadedStrings in 7.10.2 which caused extremely slow compilation times and filed a bug report for that, which was solved for 7.10.3.
codygman 9y ago
> (e.g., `ByteString.map (+ value)` is incredibly slow compared to a pseudo-C loop). That situation sounds like you needed a ByteString Builder to get comparable performance.
- luispedrocoelho 9y ago
  Maybe, but pseudo-C actually worked very well. I even have some "real C" in my Haskell code to handle some inner loop stuff.

knz42 9y ago

Hi can you tell us a bit more about this SQL-related project? I'm rather practically interested.

arnon 9y ago
I work for www.sqream.com, and our main product is SQream DB. SQream DB is a GPU SQL database for analytics. Everything was written in-house - from the SQL parser all the way down to the storage layer. We're designed to deal with sizes from a few terabytes to hundreds of terabytes, and we use the GPU to do most of the work. It is the compiler (written in Haskell) however, that decides about specific optimizations and deals with generating the query plan. Our SQL parser and typechecker is actually BSD3 licensed, if it's interesting: https://github.com/jakewheat/hssqlppp https://github.com/jakewheat/hssqlppp
- shusson 9y ago
  Any chance SQream DB will be open sourced or a demo version? It seems every Column based GPU DB I find is behind a paywall and targeting enterprise levels.
  - arnon 9y ago
    I think it's unlikely at this point... We too are currently during some large projects for enterprises... We will be releasing a community version on AWS and Azure in the near future which should be cheap or free, other than the instance cost on the respective cloud provider.
- Veedrac 9y ago
  > www.sqream.com I don't like having to be that guy, but your landing page hijacks scrolling (which always causes problems; I noticed first because it broke ctrl-scroll zooming), downloads a 30MB video in a loop, forever, takes a significant amount of CPU even when the video isn't in view and the tab is not active, and despite having almost no content manages to take far more memory than any other non-email tab I have open.

zzalpha 9y ago

but it's not that different than what you'd get with other programming languages. Until you have to solve a (maddeningly common) space leak issue. That's a problem unique to lazily evaluated languages (basically Haskell) and is godawful to debug. It reminds me of solving git problems... you suddenly find yourself having to tear back the abstraction layer that is the language and compiler and start thinking about thunks and so forth. It's jarring and incredibly frustrating.

platz 9y ago
here is a method that can help debug space leaks: http://neilmitchell.blogspot.com/2015/09/detecting-space-leaks.html http://neilmitchell.blogspot.com/2015/09/detecting-space-lea...
- zzalpha 9y ago
  I wish the problem was simply detecting and isolating space leaks. My experience is that actually fixing them can be incredibly difficult, hence my comment about needing to understand the gory details about how the runtime evaluates lazy expressions. Heck, that post even uses the phrase "Attempt to fix the space leak" as it's often not an obvious slam dunk. Sometimes it even devolves to peppering !'s around until the problem goes away.
  - dllthomas 9y ago
    > My experience is that actually fixing them can be incredibly difficult My experience differs, FWIW. If you know where you're creating too many thunks, and you force them as you create them, they don't accumulate. Making sure you actually are forcing them, and not simply suspending a "force this", is probably the trickiest bit until you're used to the evaluation model.
    - zzalpha 9y ago
      If you know where you're creating too many thunks, and you force them as you create them, they don't accumulate. Translation: if you've internalized the way Haskell code is compiled and executes, so that you can easily reason about how lazy evaluation is actually implemented, you can solve these problems. If not, it devolves to throwing !'s in and praying. Which is basically my point. If I don't have a hope of solving common space leaks without deeply understanding how Haskell is evaluated, that's a real challenge for anyone trying to learn the language.

nom 9y ago

> but it's not that different than what you'd get with other programming languages Does your list of other programming languages include C/C++/asm? ;)

arnon 9y ago
Incidentally, yes.

Confusion 9y ago

I find that incredibly hard to believe. I wonder whether you are looking at history through rose-tinted glasses or you simply don't know because other people quietly solved those problems. I ran into such problems several times in 50 line programs and not because my code was non-idiomatic or wrong. #haskell agreed it was non-trivial to get the code to perform well.