4 ms·

> For example in my case I ran into problem when reading all files lazily. This caused my program to have too many open file handles. It was easily solved thoug

> For example in my case I ran into problem when reading all files lazily. This caused my program to have too many open file handles. It was easily solved though, by hacking a bit to force the complete file to be read directly This kind of thing is exactly what I'm afraid of and makes me not wanting to commit to a purely functional style. IMO FP is simply the wrong tool for handling expensive resources - e.g. I/O or large memory regions. This is why I think an imperative shell handling these resources, around a functional core (potentially around another tiny imperative innermost core for handling caches), is overall a cleaner approach for performant code.

wereHamster 10y ago
Laziness (lazy IO, in this case) has nothing to do with a language being functional or not. It has everything to do with a language being strict or lazy. There are strict functional languages. Most are, in fact.
- m_mueller 10y ago
  Lazy/strict is just one property that can be problematic in a high performance context. The biggest one to me is mutability. Basically it's a simple test: Can I swap pointers? (e.g. 'model_current' and 'model_next_timestep'). If not, I can't use it, it would slow down numerical solvers tremendously to allocate and free the required memory for each step. However, if I can just have the time iteration in an imperative shell that allows pointer swapping, while keeping the interesting mathematics in a purely functional core, that would be an interesting architecture (because it could make use of inherent parallelism in a better way). So far I haven't seen anything like that becoming truly competitive with Fortran/C/C++ in the HPC space, which I find a shame. Programming for HPC should be like programming in the future, as Alan Kay likes to say, but it seems to me there is a stark (and IMO unnecessary) disconnect between the worlds of HPC and desktop/web programming today.
  - fulafel 10y ago
    There are various ways get the effect (with similar space and performance chacteristic), but there are different approaches for different FP languages. For example, persistent data structures, non-pure primitives (eg Clojure's atoms), monads, etc. For Haskell, these sound relevant if you're set on using strict dense matrices: https://wiki.haskell.org/Monad/ST https://wiki.haskell.org/Monad/ST https://hackage.haskell.org/package/bed-and-breakfast https://hackage.haskell.org/package/bed-and-breakfast I suspect HPC will always be more gnarly than elegant, since by definition HPC is about spending money on making code go fast in very specialized apps.
    - m_mueller 10y ago
      could you expand on persistent data structures? How do they give me 'free' memory with neither allocation nor mutable data? Or is that outside of pure FP?
      - fulafel 10y ago
        So persistent data structures are conceptually just an extension of the classic lisp cells: You can have a pointer to (a, b, c) and a pointer to (a, b, c, d) simultaneously, without using twice the memory or time. This can be extended to trees. And you can for example make an efficient tree-backed vector, with chunks of values at the nodes. With some thought you can even get good atomic properties so you can have safe parallel operations on the same persistent vector. Here's a good start about Clojure's persistent vector: http://hypirion.com/musings/understanding-persistent-vector-pt-1 http://hypirion.com/musings/understanding-persistent-vector-...
  - mbrock 10y ago
    Pointer swapping would probably just be swapping the arguments to your loop function.
    - m_mueller 10y ago
      Well one needs to become the output, so if you only have immutable data structures it's afaik impossible.
jfoutz 10y ago
So... a Haskell approach?
- m_mueller 10y ago
  I'm not very literate with Haskell, but didn't OP have to do a quirky hack with reading the length of the file into the void in order to get rid of open file handles? That doesn't seem to me like Haskell is well designed at letting me take over resource management if the need arises.
  - DasIch 10y ago
    This is a problem with that specific API and and not an issue with Haskell in general. Conduit and Pipes both solve this problem.
  - lomnakkus 10y ago
    It's easily avoided: Just never use the lazy I/O functions. (Not sure if hlint perhaps has a warning you can apply here. It probably should.) There are real non-brittle solutions to streaming I/O, pipes[1], conduit[2], or io-streams[3] being the most popular AFAICT. (The latter is probably the easiest to use, but is somewhat less 'composable' than the other two, especially if you have monad transformer stacks, etc.) Either that, or just use strict I/O if you're just working with smallish files. Essentially lazy I/O was a mistake that's unfortunately very hard to fix nowadays without presumably breaking a lot of working-but-potentially-brittle code. Perhaps the time is ripe for starting to actually deprecate them officially... at least almost everyone in the Haskell community seems to agree that it was a mistake and they are a constant trap for newbies. [1] http://hackage.haskell.org/package/pipes http://hackage.haskell.org/package/pipes [2] https://hackage.haskell.org/package/conduit https://hackage.haskell.org/package/conduit [3] https://hackage.haskell.org/package/io-streams https://hackage.haskell.org/package/io-streams
    - paulajohnson 10y ago
      I wouldn't want to see lazy IO disappear altogether because if you are just processing a file line by line then its a quick and dirty solution that will work fine in practice. However I agree it should come with health warnings and pointers to more scaleable solutions.

wereHamster 10y ago

Laziness (lazy IO, in this case) has nothing to do with a language being functional or not. It has everything to do with a language being strict or lazy. There are strict functional languages. Most are, in fact.

m_mueller 10y ago
Lazy/strict is just one property that can be problematic in a high performance context. The biggest one to me is mutability. Basically it's a simple test: Can I swap pointers? (e.g. 'model_current' and 'model_next_timestep'). If not, I can't use it, it would slow down numerical solvers tremendously to allocate and free the required memory for each step. However, if I can just have the time iteration in an imperative shell that allows pointer swapping, while keeping the interesting mathematics in a purely functional core, that would be an interesting architecture (because it could make use of inherent parallelism in a better way). So far I haven't seen anything like that becoming truly competitive with Fortran/C/C++ in the HPC space, which I find a shame. Programming for HPC should be like programming in the future, as Alan Kay likes to say, but it seems to me there is a stark (and IMO unnecessary) disconnect between the worlds of HPC and desktop/web programming today.
- fulafel 10y ago
  There are various ways get the effect (with similar space and performance chacteristic), but there are different approaches for different FP languages. For example, persistent data structures, non-pure primitives (eg Clojure's atoms), monads, etc. For Haskell, these sound relevant if you're set on using strict dense matrices: https://wiki.haskell.org/Monad/ST https://wiki.haskell.org/Monad/ST https://hackage.haskell.org/package/bed-and-breakfast https://hackage.haskell.org/package/bed-and-breakfast I suspect HPC will always be more gnarly than elegant, since by definition HPC is about spending money on making code go fast in very specialized apps.
  - m_mueller 10y ago
    could you expand on persistent data structures? How do they give me 'free' memory with neither allocation nor mutable data? Or is that outside of pure FP?
    - fulafel 10y ago
      So persistent data structures are conceptually just an extension of the classic lisp cells: You can have a pointer to (a, b, c) and a pointer to (a, b, c, d) simultaneously, without using twice the memory or time. This can be extended to trees. And you can for example make an efficient tree-backed vector, with chunks of values at the nodes. With some thought you can even get good atomic properties so you can have safe parallel operations on the same persistent vector. Here's a good start about Clojure's persistent vector: http://hypirion.com/musings/understanding-persistent-vector-pt-1 http://hypirion.com/musings/understanding-persistent-vector-...
- mbrock 10y ago
  Pointer swapping would probably just be swapping the arguments to your loop function.
  - m_mueller 10y ago
    Well one needs to become the output, so if you only have immutable data structures it's afaik impossible.

jfoutz 10y ago

So... a Haskell approach?

m_mueller 10y ago
I'm not very literate with Haskell, but didn't OP have to do a quirky hack with reading the length of the file into the void in order to get rid of open file handles? That doesn't seem to me like Haskell is well designed at letting me take over resource management if the need arises.
- DasIch 10y ago
  This is a problem with that specific API and and not an issue with Haskell in general. Conduit and Pipes both solve this problem.
- lomnakkus 10y ago
  It's easily avoided: Just never use the lazy I/O functions. (Not sure if hlint perhaps has a warning you can apply here. It probably should.) There are real non-brittle solutions to streaming I/O, pipes[1], conduit[2], or io-streams[3] being the most popular AFAICT. (The latter is probably the easiest to use, but is somewhat less 'composable' than the other two, especially if you have monad transformer stacks, etc.) Either that, or just use strict I/O if you're just working with smallish files. Essentially lazy I/O was a mistake that's unfortunately very hard to fix nowadays without presumably breaking a lot of working-but-potentially-brittle code. Perhaps the time is ripe for starting to actually deprecate them officially... at least almost everyone in the Haskell community seems to agree that it was a mistake and they are a constant trap for newbies. [1] http://hackage.haskell.org/package/pipes http://hackage.haskell.org/package/pipes [2] https://hackage.haskell.org/package/conduit https://hackage.haskell.org/package/conduit [3] https://hackage.haskell.org/package/io-streams https://hackage.haskell.org/package/io-streams
  - paulajohnson 10y ago
    I wouldn't want to see lazy IO disappear altogether because if you are just processing a file line by line then its a quick and dirty solution that will work fine in practice. However I agree it should come with health warnings and pointers to more scaleable solutions.