3 ms·

Libraries, languages, the cathedral and the bazaar: a social Haskell

34 points by dons 16y ago

mapleoin 16y ago
Wow, I started reading this without looking at the bottom to see what the proposed solution was. It turns out there is none. Still, it was pretty interesting for the quick analysis of the various ways that an open source programming language's basic libraries are currently developed.
- dons 16y ago
  Well, this is an exposition of the solution space that is currently being explored. Hackage 2.0 will go live soon, which exhibits many of these characterstics, * "batteries"/cathedral-style core: the Haskell Platform * bazaar/CPAN-style cloud: hackage * distribution mechanism: cabal * social aggregation: hackage 2.0 voting and ranking
cageface 16y ago
I think it may turn out that the real genius of Scala (and Clojure) may be in discovering the only truly viable model of real-world adoption for a functional language - piggybacking on a bigger, dumber, mainstream success. Just consider what Scala gets for free that Haskell has to build itself: a fast, robust VM, libraries for any mundane thing under the sun, and a foot in the door at any common dev shop. The real cathedral vs bazaar issue here is deeper than the library issue. It's the runtime that counts and sharing a runtime with something like Java delegates all the grunt work to the legions of Java programmers and lets Scala hitch along for the ride picking up all the interesting work on the top of the heap. The future of functional languages is in symbiosis.
- tkahn6 16y ago
  This may be orthogonal to your point, but Clojure is not a purely functional language and Scala is not as functional as Clojure. In regards to a VM, Haskell can compile to LLVM.
  - cageface 16y ago
    Purity is overrated. The future of functional techniques lies on shared platforms, IMO. I don't count LLVM here because that doesn't confer the same advantage.
- dons 16y ago
  Your comment seems to imply that a lot of resources are wasted developing the runtime. However, that's not the case. E.g. the Haskell runtime is kind of "done" -- its not like that's the weak spot, going by any metric. And it is robust and has great performance. http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Having a runtime doesn't free up any grunt programmers -- since they're not involved in writing the runtime in the first place. ------- Edit I really don't understand "sharing a runtime with something like Java delegates all the grunt work to the legions of Java programmers" sharing the libraries helps delegate gruntwork. Sharing the runtime does very little for a mature language.
  - cageface 16y ago
    Haskell does seem to have a good runtime, although I've read lots of posts talking about problems with high memory load. How much could you have accomplished if you hadn't had to spend resources on building that runtime though? How much energy are you going to expend going forward maintaining parity with the JVM, with 1/100th of the resources? And I really do mean runtime. You can share libraries in any language but sharing a runtime with Java means I get things like bulletproof Oracle drivers, Hadoop, Netty, Mina for free and I can use Scala at the top of a very rich food chain. It took me all of about 15 minutes to get Hadoop queries running in Scala. What's the Haskell story there?
    - dons 16y ago
      > get things like bulletproof Oracle drivers, Hadoop, Netty, Mina No, that means sharing libraries. You're using Java libraries now. The JVM has nothing to do with Oracle. > read lots of posts talking about problems with high memory load Look at the shootout stats. The JVM is a memory hog in comparison.
      - cageface 16y ago
        No, that means sharing libraries. You're using Java libraries now. The JVM has nothing to do with Oracle. I'm sharing libraries with Java. I don't have to wait for somebody to write a Scala version. Big difference. The JVM is a hog but it also holds up really high well under high memory and CPU load. Much better than most other VMs.
jrockway 16y ago
This article got me thinking a little. I know base pretty well; whenever I'm writing some code, I always end up hitting the "source code" link and checking out the implementation. I never really wonder "how does this work" as I'm programming anymore. The question is, how can one go from knowing a library to contributing to the library? One of the problems with HackageDB is that you can see everything, but you can't see where you can make changes. I had a problem with C2HS a few months or so ago; it didn't work on Windows (because DLLs use "stdcall", not "ccall", and the "ccall" string was hard-coded). I clicked the darcs link on Hackage to get the source. Dead. So I just edited the source on my local machine and moved on. A few months later I noticed that darcs was working again and there was a new maintainer, and I did get to contribute my patch. But it was kind of dumb luck, I could have easily had never noticed, and this bug never would have been fixed. So right now, it's easy to find libraries, it's easy to understand libraries, but it's not always easy to improve the libraries. And incidentally, the biggest problem with libraries in Haskell is that they never get out of the "rough draft" stage, by getting contributions from people other than the author. Incidentally, I don't have this problem with Perl, because everyone happens to use Github. This has created an interesting effect where libraries that aren't on Github become very difficult to work on, and people seem to favor rewriting instead of fixing things "the old way". (diff + an email to the author.)
jewbacca 16y ago
What sucks is that Hackage has been down all day (and is down not infrequently) and I've needed it desperately. Also, Don Stewart's (evangelized by OP) 'Download' package won't build on OS X, with a different cryptic error message than it gave me last week. Yes, these are piddly little whines of personal anecdote, and objectively it's absurd to suggest should be flawlessly and continuously taken care of by the Haskell establishment at this stage in its development, but Haskell is pretty quickly becoming my language now: I love it, I think in it, it is a daily struggle to fight the urge to evangelize like a douchebag. With the problems I've just mentioned, I'm writing what I need to in Python, hideously functional Python that would make Guido cry, but prototyping everything in Haskell. If this stuff is pissing me off, a momentarily curious Ruby bro sticking his neck out and cautiously circling the >>= in the stone is going to get bit, fuck off and not look back. Please build the cathedral soon.
- dons 16y ago
  > package won't build on OS X There is a binary installer. We're discussing how to work with MacPorts, but at the moment it only has 3rd party support status.
  - jewbacca 16y ago
    ... whoa, that's pretty cool. Also, I feel like a douchebag. Thanks for building and sharing this stuff at all. Where's that binary at? Also, what's this about MacPorts? I only ever tried using cabal (as installed with Haskell Platform) and have no understanding of the structure of the underlying installation. As perhaps the prototypical semi-clueless amateur, potential vanguard of the article's "mainstream", who's stumbled early onto this serious and enormous thing, I am scared and confused by the idea of having to install MacPorts again to muck up my pretty filesystem.
    - danielmueller 16y ago
      http://hackage.haskell.org/platform/ http://hackage.haskell.org/platform/ Works like a charm! Only hitch I had was that the prebuilt libraries are 32bit, and if you want to link something to a native library under Snow Leopard, and said library happens to be available only in 64bit, you're out of luck. Case in point: I tried to customize XMonad under X11.app/10.6 with a custom xmonad.hs. Running straight XMonad worked fine, but trying to use the recompilation failed... -> Would be nice to have a 64bit Haskell Platform build. But that's probably a more exotic usecase as you'd usually either stay in Haskell-land (which compiles in a compatible way), or link against *nix style libraries, which you probably fetch via MacPorts, and there you can control the compilation as well. A big thank you to Don for the packaging effort, it's very much appreciated by newbies (incl. me).

34 points by dons 16y ago

Wow, I started reading this without looking at the bottom to see what the proposed solution was. It turns out there is none. Still, it was pretty interesting for the quick analysis of the various ways that an open source programming language's basic libraries are currently developed.

dons 16y ago
Well, this is an exposition of the solution space that is currently being explored. Hackage 2.0 will go live soon, which exhibits many of these characterstics, * "batteries"/cathedral-style core: the Haskell Platform * bazaar/CPAN-style cloud: hackage * distribution mechanism: cabal * social aggregation: hackage 2.0 voting and ranking

cageface 16y ago

I think it may turn out that the real genius of Scala (and Clojure) may be in discovering the only truly viable model of real-world adoption for a functional language - piggybacking on a bigger, dumber, mainstream success. Just consider what Scala gets for free that Haskell has to build itself: a fast, robust VM, libraries for any mundane thing under the sun, and a foot in the door at any common dev shop. The real cathedral vs bazaar issue here is deeper than the library issue. It's the runtime that counts and sharing a runtime with something like Java delegates all the grunt work to the legions of Java programmers and lets Scala hitch along for the ride picking up all the interesting work on the top of the heap. The future of functional languages is in symbiosis.

tkahn6 16y ago
This may be orthogonal to your point, but Clojure is not a purely functional language and Scala is not as functional as Clojure. In regards to a VM, Haskell can compile to LLVM.
- cageface 16y ago
  Purity is overrated. The future of functional techniques lies on shared platforms, IMO. I don't count LLVM here because that doesn't confer the same advantage.
dons 16y ago
Your comment seems to imply that a lot of resources are wasted developing the runtime. However, that's not the case. E.g. the Haskell runtime is kind of "done" -- its not like that's the weak spot, going by any metric. And it is robust and has great performance. http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=ghc http://shootout.alioth.debian.org/u64q/benchmark.php?test=al... Having a runtime doesn't free up any grunt programmers -- since they're not involved in writing the runtime in the first place. ------- Edit I really don't understand "sharing a runtime with something like Java delegates all the grunt work to the legions of Java programmers" sharing the libraries helps delegate gruntwork. Sharing the runtime does very little for a mature language.
- cageface 16y ago
  Haskell does seem to have a good runtime, although I've read lots of posts talking about problems with high memory load. How much could you have accomplished if you hadn't had to spend resources on building that runtime though? How much energy are you going to expend going forward maintaining parity with the JVM, with 1/100th of the resources? And I really do mean runtime. You can share libraries in any language but sharing a runtime with Java means I get things like bulletproof Oracle drivers, Hadoop, Netty, Mina for free and I can use Scala at the top of a very rich food chain. It took me all of about 15 minutes to get Hadoop queries running in Scala. What's the Haskell story there?
  - dons 16y ago
    > get things like bulletproof Oracle drivers, Hadoop, Netty, Mina No, that means sharing libraries. You're using Java libraries now. The JVM has nothing to do with Oracle. > read lots of posts talking about problems with high memory load Look at the shootout stats. The JVM is a memory hog in comparison.
    - cageface 16y ago
      No, that means sharing libraries. You're using Java libraries now. The JVM has nothing to do with Oracle. I'm sharing libraries with Java. I don't have to wait for somebody to write a Scala version. Big difference. The JVM is a hog but it also holds up really high well under high memory and CPU load. Much better than most other VMs.

jrockway 16y ago

This article got me thinking a little. I know base pretty well; whenever I'm writing some code, I always end up hitting the "source code" link and checking out the implementation. I never really wonder "how does this work" as I'm programming anymore. The question is, how can one go from knowing a library to contributing to the library? One of the problems with HackageDB is that you can see everything, but you can't see where you can make changes. I had a problem with C2HS a few months or so ago; it didn't work on Windows (because DLLs use "stdcall", not "ccall", and the "ccall" string was hard-coded). I clicked the darcs link on Hackage to get the source. Dead. So I just edited the source on my local machine and moved on. A few months later I noticed that darcs was working again and there was a new maintainer, and I did get to contribute my patch. But it was kind of dumb luck, I could have easily had never noticed, and this bug never would have been fixed. So right now, it's easy to find libraries, it's easy to understand libraries, but it's not always easy to improve the libraries. And incidentally, the biggest problem with libraries in Haskell is that they never get out of the "rough draft" stage, by getting contributions from people other than the author. Incidentally, I don't have this problem with Perl, because everyone happens to use Github. This has created an interesting effect where libraries that aren't on Github become very difficult to work on, and people seem to favor rewriting instead of fixing things "the old way". (diff + an email to the author.)

jewbacca 16y ago

What sucks is that Hackage has been down all day (and is down not infrequently) and I've needed it desperately. Also, Don Stewart's (evangelized by OP) 'Download' package won't build on OS X, with a different cryptic error message than it gave me last week. Yes, these are piddly little whines of personal anecdote, and objectively it's absurd to suggest should be flawlessly and continuously taken care of by the Haskell establishment at this stage in its development, but Haskell is pretty quickly becoming my language now: I love it, I think in it, it is a daily struggle to fight the urge to evangelize like a douchebag. With the problems I've just mentioned, I'm writing what I need to in Python, hideously functional Python that would make Guido cry, but prototyping everything in Haskell. If this stuff is pissing me off, a momentarily curious Ruby bro sticking his neck out and cautiously circling the >>= in the stone is going to get bit, fuck off and not look back. Please build the cathedral soon.

dons 16y ago
> package won't build on OS X There is a binary installer. We're discussing how to work with MacPorts, but at the moment it only has 3rd party support status.
- jewbacca 16y ago
  ... whoa, that's pretty cool. Also, I feel like a douchebag. Thanks for building and sharing this stuff at all. Where's that binary at? Also, what's this about MacPorts? I only ever tried using cabal (as installed with Haskell Platform) and have no understanding of the structure of the underlying installation. As perhaps the prototypical semi-clueless amateur, potential vanguard of the article's "mainstream", who's stumbled early onto this serious and enormous thing, I am scared and confused by the idea of having to install MacPorts again to muck up my pretty filesystem.
  - danielmueller 16y ago
    http://hackage.haskell.org/platform/ http://hackage.haskell.org/platform/ Works like a charm! Only hitch I had was that the prebuilt libraries are 32bit, and if you want to link something to a native library under Snow Leopard, and said library happens to be available only in 64bit, you're out of luck. Case in point: I tried to customize XMonad under X11.app/10.6 with a custom xmonad.hs. Running straight XMonad worked fine, but trying to use the recompilation failed... -> Would be nice to have a 64bit Haskell Platform build. But that's probably a more exotic usecase as you'd usually either stay in Haskell-land (which compiles in a compatible way), or link against *nix style libraries, which you probably fetch via MacPorts, and there you can control the compilation as well. A big thank you to Don for the packaging effort, it's very much appreciated by newbies (incl. me).