4 ms·

Serious question: Is there any reason why somebody would use Haskell for data analysis when there is also R and Python - which are perfect for that job - excep

Serious question: Is there any reason why somebody would use Haskell for data analysis when there is also R and Python - which are perfect for that job - except for that the respective person happens to be a Haskell expert anyway?

gh02t 10y ago
I use all three, depending on the task. Haskell is compiled and relatively speedy, as well as being great for writing custom parsers (plus in a lot of situations it parallelizes very easily). One thing I find Haskell to be really useful for is coercing large unstructured datasets into a format that is easier to feed into Python. For instance, I once had to write a parser for the data coming off of a digitizer being fed by a rather complicated radiation detector array. It's in an obscure and somewhat bizarre binary format that is pretty tedious to work with because it involves a lot of state in the parser, plus the files are pretty enormous (they describe every radioactive particle hitting every detector in the array over several hours of measurement). My colleague wrote a horribly tedious script in Python to parse it that was complicated and agonizingly slow, but I was able to write a very natural 80-line Haskell program in a few hours that was several orders of magnitude faster as well as much more robust. I was just massaging the data to feed into Python, but it was far far easier to do in Haskell. So I find for some tasks I want to reach for Haskell, because it's natural to express the solution in Haskell. Other stuff I wouldn't even consider it, especially for stuff like data processing and plotting Haskell is not as elegant. Right tool for the job and all.
- hellofunk 10y ago
  This 80-line solution sounds very interesting. I'd love to see more examples of this particular strength of Haskell. Are there similar parsing projects that you can point to that are open source and worth looking at to better understand this use case of Haskell?
  - PinkyThePiggy 10y ago
    Here is an example from Real World Haskell (pretty good intermediate book, although it is a bit dated): http://book.realworldhaskell.org/read/using-parsec.html#csv http://book.realworldhaskell.org/read/using-parsec.html#csv A fully featured CSV parser in 20-30 [depending on how you count] lines of non-golf code.
    - hellofunk 10y ago
      I'm curious what aspects of Haskell have changed to cause this book to be dated, i.e. are there examples in this book which would be done differently now?
      - PinkyThePiggy 10y ago
        As one example, a lot of the libraries referenced in the book have gone through major version changes (1.x to 2.x etc.) that change the functions being exported and/or architecture reworks changing which libraries you need to import which means that if you try to copy + compile a lot of the code snippets, they will not compile. It is generally trivial changes in most cases, but if you are a newbie to the language, it is likely not immediately obvious because you will think you copied from the book wrong (or that the book has a typo) instead of the libraries having been changed. There is also a lot written about historical quirks you will come across that have since been fixed.
      - luisfoliv 10y ago
        This SO thread is very informative in this regard: http://stackoverflow.com/questions/23727768/which-parts-of-real-world-haskell-are-now-obsolete-or-considered-bad-practice http://stackoverflow.com/questions/23727768/which-parts-of-r...
  - gh02t 10y ago
    I remembered a bit wrong, the final version was 155 lines (I think my first stab was ~80 lines). I have a version of it on Github as a gist https://gist.github.com/jasonmhite/c4c56d4c50fc673e658b71b8286c390d https://gist.github.com/jasonmhite/c4c56d4c50fc673e658b71b82... . Can't really claim whether or not it's "good" Haskell or that it's optimal, but it gets the job done. I also dunno how comprehensible it is without knowing the ins-and-outs of the nPOD format or multiplicity counting, but hey there it is. Not sure of what the definitive reference is, but this article is pretty good I think: https://www.schoolofhaskell.com/school/starting-with-haskell/libraries-and-frameworks/text-manipulation/attoparsec https://www.schoolofhaskell.com/school/starting-with-haskell...
- e19293001 10y ago
  Thanks for sharing your experience with Haskell. Could you please point us out some reference where you've learned in creating a parser using Haskell? Any of the blogs, tutorials, articles, books would be great.
  - albertus 10y ago
    Not in the context of data analysis, but the Write Yourself a Scheme in 48 Hours[0] wikibook is a good exercise in working with Parsec. It's written for Haskell beginners, but it wasn't tedious in that regard. [0] https://en.wikibooks.org/wiki/Write_Yourself_a_Scheme_in_48_Hours https://en.wikibooks.org/wiki/Write_Yourself_a_Scheme_in_48_...
  - wrsh07 10y ago
    Did you see this post about a partial C to Rust transpiler written in literal Haskell? https://github.com/jameysharp/corrode/blob/master/src/Language/Rust/Corrode/C.md https://github.com/jameysharp/corrode/blob/master/src/Langua...
p4wnc6 10y ago
Haskell as a language is excellent for these tasks and when you are familiar with it, you can be even more rapidly productive than with Python, and much, much more than with R. The problem for me has not so much been any technical aspect of Haskell nor any technical benefits of Python or R. Instead, it has been sociological issues with the Haskell community. I won't go into it too much, but the biggest one is that there is no cohesive way to understand what kind of progression of Haskell knowledge makes you "a beginner" or "intermediate" or "advanced" -- especially for the purposes of getting a job. You might have mastered all the basics from LYAH, and know monads inside and out, and then someone comes along and says you're totally a Haskell noob because you haven't used Template Haskell for 6 years, or you don't do everything with Monad transformers, or you don't have the API of some tool like Parsec committed to memory, or you don't use language extensions and LiquidHaskell to use the compiler as a proof system of the correctness of your code. In Haskell, you're always made to feel like you're constantly a fuck-up for not knowing the next great wrinkle of abstraction or the next great toolkit up the food chain. No joke, I've experienced being called a Haskell "beginner" because I was not deeply familiar with LiquidHaskell. That's not a common opinion, but it gives you a sense of the variety. This stresses me out so much that I don't even bother applying to Haskell jobs anymore. I don't want to get an initial phone screen and then just be made to feel like I'm a dunce because I don't know how some category theory principle is embodied by phantom types or something. Yuck. Getting Haskell jobs that will actually pay you according to your experience and ability to learn is very hard. Most places just don't want to hire people who aren't super experienced in Haskell, and they default to believing everyone is a beginner unless they wrote a math Ph.D. thesis on multiparameter type classes. On the off chance that someone will talk to you, and they think "OK, this person has a bunch of years of data analysis in Python or R under their belt, and they know enough about Haskell to work pretty quickly with Monads and basic type classes ..." that means you are a junior engineer at best, and will be paid like it, even if your data analysis skill is very high or you're a very fast learner or you have an advanced degree or many years of experience. I've never been able to figure out the impenetrable bubble around Haskell jobs, but this kind of culture of believing that about 99% of Haskellers are beginners and less than 1% are anything beyond a beginner is the biggest reason why I mostly gave up learning Haskell or searching for Haskell jobs. It's funny because a lot of people argue functional programming is too much about elegance to be pragmatic, and they can "just get stuff done" with other tools. Then lots of functional programmer enthusiasts jump in and refute that (I believe correctly) but then turn around and only hire people if they jump through all of the unpragmatic too-much-elegance hoops. It's really vexing.

gh02t 10y ago

I use all three, depending on the task. Haskell is compiled and relatively speedy, as well as being great for writing custom parsers (plus in a lot of situations it parallelizes very easily). One thing I find Haskell to be really useful for is coercing large unstructured datasets into a format that is easier to feed into Python. For instance, I once had to write a parser for the data coming off of a digitizer being fed by a rather complicated radiation detector array. It's in an obscure and somewhat bizarre binary format that is pretty tedious to work with because it involves a lot of state in the parser, plus the files are pretty enormous (they describe every radioactive particle hitting every detector in the array over several hours of measurement). My colleague wrote a horribly tedious script in Python to parse it that was complicated and agonizingly slow, but I was able to write a very natural 80-line Haskell program in a few hours that was several orders of magnitude faster as well as much more robust. I was just massaging the data to feed into Python, but it was far far easier to do in Haskell. So I find for some tasks I want to reach for Haskell, because it's natural to express the solution in Haskell. Other stuff I wouldn't even consider it, especially for stuff like data processing and plotting Haskell is not as elegant. Right tool for the job and all.

hellofunk 10y ago
This 80-line solution sounds very interesting. I'd love to see more examples of this particular strength of Haskell. Are there similar parsing projects that you can point to that are open source and worth looking at to better understand this use case of Haskell?
- PinkyThePiggy 10y ago
  Here is an example from Real World Haskell (pretty good intermediate book, although it is a bit dated): http://book.realworldhaskell.org/read/using-parsec.html#csv http://book.realworldhaskell.org/read/using-parsec.html#csv A fully featured CSV parser in 20-30 [depending on how you count] lines of non-golf code.
  - hellofunk 10y ago
    I'm curious what aspects of Haskell have changed to cause this book to be dated, i.e. are there examples in this book which would be done differently now?
    - PinkyThePiggy 10y ago
      As one example, a lot of the libraries referenced in the book have gone through major version changes (1.x to 2.x etc.) that change the functions being exported and/or architecture reworks changing which libraries you need to import which means that if you try to copy + compile a lot of the code snippets, they will not compile. It is generally trivial changes in most cases, but if you are a newbie to the language, it is likely not immediately obvious because you will think you copied from the book wrong (or that the book has a typo) instead of the libraries having been changed. There is also a lot written about historical quirks you will come across that have since been fixed.
    - luisfoliv 10y ago
      This SO thread is very informative in this regard: http://stackoverflow.com/questions/23727768/which-parts-of-real-world-haskell-are-now-obsolete-or-considered-bad-practice http://stackoverflow.com/questions/23727768/which-parts-of-r...
- gh02t 10y ago
  I remembered a bit wrong, the final version was 155 lines (I think my first stab was ~80 lines). I have a version of it on Github as a gist https://gist.github.com/jasonmhite/c4c56d4c50fc673e658b71b8286c390d https://gist.github.com/jasonmhite/c4c56d4c50fc673e658b71b82... . Can't really claim whether or not it's "good" Haskell or that it's optimal, but it gets the job done. I also dunno how comprehensible it is without knowing the ins-and-outs of the nPOD format or multiplicity counting, but hey there it is. Not sure of what the definitive reference is, but this article is pretty good I think: https://www.schoolofhaskell.com/school/starting-with-haskell/libraries-and-frameworks/text-manipulation/attoparsec https://www.schoolofhaskell.com/school/starting-with-haskell...
e19293001 10y ago
Thanks for sharing your experience with Haskell. Could you please point us out some reference where you've learned in creating a parser using Haskell? Any of the blogs, tutorials, articles, books would be great.
- albertus 10y ago
  Not in the context of data analysis, but the Write Yourself a Scheme in 48 Hours[0] wikibook is a good exercise in working with Parsec. It's written for Haskell beginners, but it wasn't tedious in that regard. [0] https://en.wikibooks.org/wiki/Write_Yourself_a_Scheme_in_48_Hours https://en.wikibooks.org/wiki/Write_Yourself_a_Scheme_in_48_...
- wrsh07 10y ago
  Did you see this post about a partial C to Rust transpiler written in literal Haskell? https://github.com/jameysharp/corrode/blob/master/src/Language/Rust/Corrode/C.md https://github.com/jameysharp/corrode/blob/master/src/Langua...

p4wnc6 10y ago

Haskell as a language is excellent for these tasks and when you are familiar with it, you can be even more rapidly productive than with Python, and much, much more than with R. The problem for me has not so much been any technical aspect of Haskell nor any technical benefits of Python or R. Instead, it has been sociological issues with the Haskell community. I won't go into it too much, but the biggest one is that there is no cohesive way to understand what kind of progression of Haskell knowledge makes you "a beginner" or "intermediate" or "advanced" -- especially for the purposes of getting a job. You might have mastered all the basics from LYAH, and know monads inside and out, and then someone comes along and says you're totally a Haskell noob because you haven't used Template Haskell for 6 years, or you don't do everything with Monad transformers, or you don't have the API of some tool like Parsec committed to memory, or you don't use language extensions and LiquidHaskell to use the compiler as a proof system of the correctness of your code. In Haskell, you're always made to feel like you're constantly a fuck-up for not knowing the next great wrinkle of abstraction or the next great toolkit up the food chain. No joke, I've experienced being called a Haskell "beginner" because I was not deeply familiar with LiquidHaskell. That's not a common opinion, but it gives you a sense of the variety. This stresses me out so much that I don't even bother applying to Haskell jobs anymore. I don't want to get an initial phone screen and then just be made to feel like I'm a dunce because I don't know how some category theory principle is embodied by phantom types or something. Yuck. Getting Haskell jobs that will actually pay you according to your experience and ability to learn is very hard. Most places just don't want to hire people who aren't super experienced in Haskell, and they default to believing everyone is a beginner unless they wrote a math Ph.D. thesis on multiparameter type classes. On the off chance that someone will talk to you, and they think "OK, this person has a bunch of years of data analysis in Python or R under their belt, and they know enough about Haskell to work pretty quickly with Monads and basic type classes ..." that means you are a junior engineer at best, and will be paid like it, even if your data analysis skill is very high or you're a very fast learner or you have an advanced degree or many years of experience. I've never been able to figure out the impenetrable bubble around Haskell jobs, but this kind of culture of believing that about 99% of Haskellers are beginners and less than 1% are anything beyond a beginner is the biggest reason why I mostly gave up learning Haskell or searching for Haskell jobs. It's funny because a lot of people argue functional programming is too much about elegance to be pragmatic, and they can "just get stuff done" with other tools. Then lots of functional programmer enthusiasts jump in and refute that (I believe correctly) but then turn around and only hire people if they jump through all of the unpragmatic too-much-elegance hoops. It's really vexing.