5 ms·

Grenade: Deep Learning in Haskell

199 points by vonnik 9y ago

dannycastonguay 9y ago
Thanks Huw! I'm sure many of us have been itching to do it for some time.
cs702 9y ago
Looks really neat: composable, "smartly typed," easy-to-specify neural nets. The main issue that keeps me from giving this a try is the lack of GPU support out-of-the-box, necessary for tackling problems of more interesting size :-(
sndean 9y ago
In the discussion of HLearn [0] the author said this [1]: "This is a bit awkward for me as I've paused the development of HLearn and emphatically do not recommend anyone use it. The main problem is that Haskell (which I otherwise love) has poor support for numerical computing." Can anyone expand on this comment, and does that also apply to Grenade? [0] https://news.ycombinator.com/item?id=14402378 https://news.ycombinator.com/item?id=14402378 [1] https://news.ycombinator.com/item?id=14409595 https://news.ycombinator.com/item?id=14409595
- nanolith 9y ago
  Haskell has a mature FFI. If there are any computational hurdles, it is possible to drop down to C/C++ to manage these. A good library can easily encapsulate this. Of course, there is the point where such a library becomes a thin wrapper around an efficient numerical library written in C, but there are other advantages to Haskell that might make such a wrapper still worth the trade: an amazing type system, libraries that take advantage of this type system, and a compiler that generates fairly efficient code. If, in exchange for that, I have to marshal numerical data into and out of Haskell's FFI, I consider it a fair trade.
  - SomeStupidPoint 9y ago
    Just to elaborate: Even if all the computation were implmented in C (eg, DAG nodes in a TF style setup), being able to compose the network that forms the computation via Haskell would be a big win for me because it lets me use the type system and various powerful libraries to design my network. (And opens a straightforward path to GPU acceleration of Haskell designed code by using the C GPU libraries.)
  - kmicklas 9y ago
    Yeah this is basically how Python is used in ML/data science work. Except Haskell lets you apply solid engineering all the way from the prototype to the production system.
  - lallysingh 9y ago
    You don't have to marshal. You can use the inline-c package to wrap all that up for you.
  - ferdterguson 9y ago
    I've used Haskell a little. I understand the benefits of the type system from the perspective of something like a web app or libraries with abstractions. However, as someone who spends their days working on numerical code in Cython for research software, how would I actually leverage the type system in my numerical code? Would I ever leverage the power of the type system and define my own types? It also seems like a terrible idea to use recursion to solve multidimensional array problems typically associated with loops and the indexing of those arrays.
    - axman6 9y ago
      Recursion is only a bad idea in languages with poor support for recursion, it's not difficult to write efficient Haskell code which produces good assembly with fairly obvious recursive algorithms. Optimisations like stream fusion and libraries like foldl also contribute to really good code generation (not guaranteed and also not generally applicable optimisations, but certainly helpful)
- deleted 9y ago
  [deleted]
- spott 9y ago
  > has poor support for numerical computing. The issue comes from being lazy by default. When you are writing lazy functional code, you essentially build up a system of thunks[0] that collapse to a value when you need it. This works great for abstractions like infinite lists, or essentially only paying for the value you need. Unfortunately, they are kind of expensive (I believe they are usually implemented as a tagged union, where they are either a value that is returned immediately or a pointer to another function or thunk) when you don't need the laziness. In any sort of numerical computing, you are usually dealing with enough values that you lose a lot of performance when dealing with these thunks. Working around this is doable with a bunch of strict annotations (so that these thunks are immediately collapsed to values), but that is only half the battle: you still need to reduce the immutable function applications to a mutable version so you don't lose out by copying your values back and forth in memory, and you don't do a bunch of unnecessary calculations. All this requires "a sufficiently intelligent compiler" or a lot of compiler hints and peering into Core: the intermediate language of GHC. It isn't for the faint of heart. There have been a bunch of work on avoiding some of this: hmatrix[1] is a lapack and blas library for finally doing the calculations once you have reduced the graph, "subhask"[2] is a standard library for Haskell that attempts to add some support for basic numerical work and "numerical"[3] is a library that I believe is trying to make a well typed linear algebra library (I'm not sure if it is going to call into hmatrix/blas/lapack or not). The issue with a lot of this is that until there are good tools for this kind of thing, no-one does serious numerical programming in Haskell, and there are only a few people working on the tooling because there aren't that many people trying to do serious numerical programming in Haskell. (note: I'm not an expert in this, so don't depend on any of this for love or money. I'm sure someone will come along and tell me how I'm wrong.) [0] https://en.wikipedia.org/wiki/Thunk https://en.wikipedia.org/wiki/Thunk, https://wiki.haskell.org/Thunk https://wiki.haskell.org/Thunk [1] http://hackage.haskell.org/package/hmatrix http://hackage.haskell.org/package/hmatrix [2] https://github.com/mikeizbicki/subhask https://github.com/mikeizbicki/subhask [3] https://github.com/wellposed/numerical https://github.com/wellposed/numerical
  - aseipp 9y ago
    I think the problem the author of HLearn and SubHask faced isn't really so much of a problem with thunking or whatever. Compiler optimizations and whatnot are always great and avoiding indirection is good (it's also not super hard either to do it manually for 99% of cases). But rather, I believe the problem they were hinting at is that the current Haskell numeric hierarchy is insufficient to capture the patterns we want for the vast kinds of numerical computing we do today. But even then, the tools we need to design the hierarchy we do want are also a bit out of reach, and implementing them will take time. SubHask for example uses hmatrix and other LAPACK/BLAS libraries under the hood, which is good of course -- but almost all of the actual work in SubHask, then, is around designing a reusable hierarchy that works for all the 'weird' cases you find with this domain. Even 'non weird' things like rank/dimension polymorphic products, etc, require a lot of type-level infrastructure to accomplish today, and the features to support that are constantly evolving. (An example of APL-style rank-polymorphic stuff I typed up recently can give you an idea of how much work it is to do some 'seemingly trivial' things, though the examples at the end are rather nice[1]) SubHask also has an extremely fine grained hierarchy, with lots of little classes to handle cases like "weird containers" with appropriate laws -- and figuring that out is a lot of work too. Haskell's type system is getting more expressive all the time, so hopefully this will become less tiresome in the future, and iterating and honing the design will be far simpler. But getting the design right is much more important and the other stuff (better underlying numeric libraries, improved performance, etc) can come after. I think you're spot on that the lack of attention here creatives a negative feedback loop, as well. Another example of this I think that was bad was that very little code took into account things like numerical stability of algorithms. The SubHask/HLearn author also tried tackling this in an interesting way, too[2]. [1] https://gist.github.com/thoughtpolice/c408773f43972e48f93377ce3e4f441f https://gist.github.com/thoughtpolice/c408773f43972e48f93377... [2] https://github.com/mikeizbicki/HerbiePlugin https://github.com/mikeizbicki/HerbiePlugin
- platz 9y ago
  The author of HLearn is actually more interested in discovering a novel way of encoding machine learning problems using new type system features, as opposed to choosing a more familiar encoding that exists in other languages, that Haskell would otherwise have no problems with. This locates the constraint on the authors ambitions relative to the desired abstraction level, rather than the performance level available
lpage 9y ago
Thought of the day: bridge Grenade with like CλaSH [1] and realize network topologies directly on an FPGA. [1] http://www.clash-lang.org/ http://www.clash-lang.org/
- eli_gottlieb 9y ago
  Ah, the wondrous things we could all do with Haskell, if only we neither had nor needed day-jobs.
  - lr4444lr 9y ago
    I can't upvote this enough.
mark_l_watson 9y ago
Great project! It was easy to build and run the tests, and, the examples are very readable. Good job. I can't wait to have time to really kick the tires.

199 points by vonnik 9y ago

Thanks Huw! I'm sure many of us have been itching to do it for some time.

Looks really neat: composable, "smartly typed," easy-to-specify neural nets. The main issue that keeps me from giving this a try is the lack of GPU support out-of-the-box, necessary for tackling problems of more interesting size :-(

sndean 9y ago

In the discussion of HLearn [0] the author said this [1]: "This is a bit awkward for me as I've paused the development of HLearn and emphatically do not recommend anyone use it. The main problem is that Haskell (which I otherwise love) has poor support for numerical computing." Can anyone expand on this comment, and does that also apply to Grenade? [0] https://news.ycombinator.com/item?id=14402378 https://news.ycombinator.com/item?id=14402378 [1] https://news.ycombinator.com/item?id=14409595 https://news.ycombinator.com/item?id=14409595

nanolith 9y ago
Haskell has a mature FFI. If there are any computational hurdles, it is possible to drop down to C/C++ to manage these. A good library can easily encapsulate this. Of course, there is the point where such a library becomes a thin wrapper around an efficient numerical library written in C, but there are other advantages to Haskell that might make such a wrapper still worth the trade: an amazing type system, libraries that take advantage of this type system, and a compiler that generates fairly efficient code. If, in exchange for that, I have to marshal numerical data into and out of Haskell's FFI, I consider it a fair trade.
- SomeStupidPoint 9y ago
  Just to elaborate: Even if all the computation were implmented in C (eg, DAG nodes in a TF style setup), being able to compose the network that forms the computation via Haskell would be a big win for me because it lets me use the type system and various powerful libraries to design my network. (And opens a straightforward path to GPU acceleration of Haskell designed code by using the C GPU libraries.)
- kmicklas 9y ago
  Yeah this is basically how Python is used in ML/data science work. Except Haskell lets you apply solid engineering all the way from the prototype to the production system.
- lallysingh 9y ago
  You don't have to marshal. You can use the inline-c package to wrap all that up for you.
- ferdterguson 9y ago
  I've used Haskell a little. I understand the benefits of the type system from the perspective of something like a web app or libraries with abstractions. However, as someone who spends their days working on numerical code in Cython for research software, how would I actually leverage the type system in my numerical code? Would I ever leverage the power of the type system and define my own types? It also seems like a terrible idea to use recursion to solve multidimensional array problems typically associated with loops and the indexing of those arrays.
  - axman6 9y ago
    Recursion is only a bad idea in languages with poor support for recursion, it's not difficult to write efficient Haskell code which produces good assembly with fairly obvious recursive algorithms. Optimisations like stream fusion and libraries like foldl also contribute to really good code generation (not guaranteed and also not generally applicable optimisations, but certainly helpful)
deleted 9y ago
[deleted]
spott 9y ago
> has poor support for numerical computing. The issue comes from being lazy by default. When you are writing lazy functional code, you essentially build up a system of thunks[0] that collapse to a value when you need it. This works great for abstractions like infinite lists, or essentially only paying for the value you need. Unfortunately, they are kind of expensive (I believe they are usually implemented as a tagged union, where they are either a value that is returned immediately or a pointer to another function or thunk) when you don't need the laziness. In any sort of numerical computing, you are usually dealing with enough values that you lose a lot of performance when dealing with these thunks. Working around this is doable with a bunch of strict annotations (so that these thunks are immediately collapsed to values), but that is only half the battle: you still need to reduce the immutable function applications to a mutable version so you don't lose out by copying your values back and forth in memory, and you don't do a bunch of unnecessary calculations. All this requires "a sufficiently intelligent compiler" or a lot of compiler hints and peering into Core: the intermediate language of GHC. It isn't for the faint of heart. There have been a bunch of work on avoiding some of this: hmatrix[1] is a lapack and blas library for finally doing the calculations once you have reduced the graph, "subhask"[2] is a standard library for Haskell that attempts to add some support for basic numerical work and "numerical"[3] is a library that I believe is trying to make a well typed linear algebra library (I'm not sure if it is going to call into hmatrix/blas/lapack or not). The issue with a lot of this is that until there are good tools for this kind of thing, no-one does serious numerical programming in Haskell, and there are only a few people working on the tooling because there aren't that many people trying to do serious numerical programming in Haskell. (note: I'm not an expert in this, so don't depend on any of this for love or money. I'm sure someone will come along and tell me how I'm wrong.) [0] https://en.wikipedia.org/wiki/Thunk https://en.wikipedia.org/wiki/Thunk, https://wiki.haskell.org/Thunk https://wiki.haskell.org/Thunk [1] http://hackage.haskell.org/package/hmatrix http://hackage.haskell.org/package/hmatrix [2] https://github.com/mikeizbicki/subhask https://github.com/mikeizbicki/subhask [3] https://github.com/wellposed/numerical https://github.com/wellposed/numerical
- aseipp 9y ago
  I think the problem the author of HLearn and SubHask faced isn't really so much of a problem with thunking or whatever. Compiler optimizations and whatnot are always great and avoiding indirection is good (it's also not super hard either to do it manually for 99% of cases). But rather, I believe the problem they were hinting at is that the current Haskell numeric hierarchy is insufficient to capture the patterns we want for the vast kinds of numerical computing we do today. But even then, the tools we need to design the hierarchy we do want are also a bit out of reach, and implementing them will take time. SubHask for example uses hmatrix and other LAPACK/BLAS libraries under the hood, which is good of course -- but almost all of the actual work in SubHask, then, is around designing a reusable hierarchy that works for all the 'weird' cases you find with this domain. Even 'non weird' things like rank/dimension polymorphic products, etc, require a lot of type-level infrastructure to accomplish today, and the features to support that are constantly evolving. (An example of APL-style rank-polymorphic stuff I typed up recently can give you an idea of how much work it is to do some 'seemingly trivial' things, though the examples at the end are rather nice[1]) SubHask also has an extremely fine grained hierarchy, with lots of little classes to handle cases like "weird containers" with appropriate laws -- and figuring that out is a lot of work too. Haskell's type system is getting more expressive all the time, so hopefully this will become less tiresome in the future, and iterating and honing the design will be far simpler. But getting the design right is much more important and the other stuff (better underlying numeric libraries, improved performance, etc) can come after. I think you're spot on that the lack of attention here creatives a negative feedback loop, as well. Another example of this I think that was bad was that very little code took into account things like numerical stability of algorithms. The SubHask/HLearn author also tried tackling this in an interesting way, too[2]. [1] https://gist.github.com/thoughtpolice/c408773f43972e48f93377ce3e4f441f https://gist.github.com/thoughtpolice/c408773f43972e48f93377... [2] https://github.com/mikeizbicki/HerbiePlugin https://github.com/mikeizbicki/HerbiePlugin
platz 9y ago
The author of HLearn is actually more interested in discovering a novel way of encoding machine learning problems using new type system features, as opposed to choosing a more familiar encoding that exists in other languages, that Haskell would otherwise have no problems with. This locates the constraint on the authors ambitions relative to the desired abstraction level, rather than the performance level available

lpage 9y ago

Thought of the day: bridge Grenade with like CλaSH [1] and realize network topologies directly on an FPGA. [1] http://www.clash-lang.org/ http://www.clash-lang.org/

eli_gottlieb 9y ago
Ah, the wondrous things we could all do with Haskell, if only we neither had nor needed day-jobs.
- lr4444lr 9y ago
  I can't upvote this enough.

mark_l_watson 9y ago

Great project! It was easy to build and run the tests, and, the examples are very readable. Good job. I can't wait to have time to really kick the tires.