13 ms·

I wouldn't recommend following the Haskell approach. It hasn't worked well for us. (I took part in creating the Haskell Platform and the process used to add pac

by tibbe 10y ago

I wouldn't recommend following the Haskell approach. It hasn't worked well for us. (I took part in creating the Haskell Platform and the process used to add packages to it. I also used to maintain a few of our core libraries, like our containers packages and networking). Small vs large standard library: A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries. Here's a number of examples of problems we currently have: - Conversions between our 5(!) string types are very common. - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle. - Standard library cannot use containers, other than lists, for the same reason. - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation. - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances. - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen. Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.

saysjonathan 10y ago
One of the common mantras I've heard among Rust core devs is "std is where code goes to die". Where do you feel the line should be drawn between standard lib and external libraries?
- skybrian 10y ago
  Maybe a change in attitude? Stability can be a good thing. Go's standard library doesn't change that much, and that's a strength. How about: "std is where code goes when it's done". As is, really done. The API's won't need changing.
  - steveklabnik 10y ago
    In the Ruby world, very few people use the standard library because it's got so many flaws, and they can't be fixed. So you end up with Nokogiri rather than REXML, all the various HTTP libs rather than net/*, etc. So it just ends up being bytes sent over the wire, wasting disk and bandwidth...
    - skybrian 10y ago
      Well, the trick is to actually get it right before standardizing it - much easier said than done. Keeping the standard library small helps with that since the bar is higher.
      - pcwalton 10y ago
        > Well, the trick is to actually get it right before standardizing it - much easier said than done. And that's what we're doing. > Keeping the standard library small helps with that since the bar is higher. But weren't you just advocating for a large standard library?
        skybrian 10y ago
        I guess I didn't understand what was meant by "where code goes to die". Graduation != dying.
    - saysjonathan 10y ago
      I wonder if identifying the atomic aspects of what you intend your language to be used for ultimately helps in narrowing down what should be in std lib. Go prioritizes network programming and bundles the necessary components, like http & rpc servers and json. The http libraries are extensible enough to allow for customization where it's wanted (like http mux) while still creating a canonical implementation that'a still viable. Has Rust identified the core demographics of who they're targeting in order to provide the most applicable platform? Is the target everyone and all application type, therefore there is no default platform? Edit: To put it another way, is there a set of packages that is either necessary for rust, rust development, or most development in rust? If std lib includes everything necessary, then who are you targeting with the default platform?
pcwalton 10y ago
I don't think most of these are applicable to Rust. > - Conversions between our 5(!) string types are very common. > - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle. We have one string type defined in std, and nobody is defining new ones (modulo special cases for legacy encodings which would not be worth polluting the default string type with). > - Standard library cannot use containers, other than lists, for the same reason. > - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation. Hash maps and trees are in the standard library already. Everyone uses them. > - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances. This is true, but this hasn't been much of a problem in Rust thus far. > - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen. That only matters if you're breaking public APIs, right? That seems orthogonal to the small-versus-large-standard-library debate. Even if you have a large standard library, if you promised it's stable you still can't break APIs.
- rtpg 10y ago
  But if you have a large standard library and want to break the API, you can. If you have 100 different libs that are basically "standard" (who doesn't have `mtl` in their applications at this point), now you have to coordinate 100 different library updates roughly at the same time. If you forget even one of them, then you've broken everything. I think the argument for a large Prelude/standard lib is similar to Google's "single repo" argument: Easy to catch usages and fix them all at once. Plus you're making the language more useful out of the box. People coming from python can understand this feeling of opening a python shell and being productive super quickly form the get-go. Arguments for small std lib exist, of course. But Giant standard libraries are more useful than not. EDIT: I think the failure of the Haskell Platform has a lot more to do with how Haskell deals with dependencies, and the difficulties it entails, than with the "batteries included" approach itself.
  - hyperhopper 10y ago
catnaroek 10y ago
Haskell's actual problem isn't the lack of a comprehensive standard library, but rather the presence of core language features that actively hinder large-scale modular programming. Type classes, type families, orphan instances and flexible instances all conspire to make as difficult as possible to determine whether two modules can be safely linked. Making things worse, whenever two alternatives are available for achieving roughly the same thing (say, type families and functional dependencies), the Haskell community consistently picks the worse one (in this case, type families, because, you know, why not punch a big hole on parametricity and free theorems?). Thanks to GHC's extensions, Haskell has become a ridiculously powerful language in exactly the same way C++ has: by sacrificing elegance. The principled approach would've been to admit that, while type classes are good for a few use cases, (say, overloading numeric literals, string literals and sequences), they have unacceptable limitations as a large-scale program structuring construct. And instead use an ML-style module system for that purpose. But it's already too late to do that.
- tibbe 10y ago
  There can be more than one problem, including having a small standard library.
  - catnaroek 10y ago
    The lack of a standard library can be fixed relatively easily: write libraries! OTOH, the existence of anti-modular language features that are extensively used in several major libraries, is a more serious problem, because: (0) It means that libraries in general won't play nicely with each other, unless they're explicitly designed to do so. (1) It can't be fixed without throwing away code.
    - tibbe 10y ago
      This whole thread is exactly about how "write libraries!" (if done outside the standard library) doesn't work (see my top post). I do agree that lack of modularity features certainly doesn't help though.
- 10y ago
kibwen 10y ago
> Empirically, languages that have large standard > libraries (e.g. Java, Python, Go) seem to do better than > their competitors. You seem to be overlooking the ultimate counterexample: C. :P
- saghm 10y ago
  JS, too, right? Forget "large" standard library, there really isn't any standard library at all
  - brandonbloom 10y ago
    You don't consider the dynamic, rich document presentation engine that is HTML to be a standard "library"? Seems like it is to me.
    - kibwen 10y ago
      HTML doesn't do anything for JS other than provide a way to create visual interfaces. It might be comparable to the role that `tkinter` plays for Python's stdlib, but HTML alone is emphatically not a standard library.
      - brandonbloom 10y ago
        My point is that most languages are totally useless on their own. JavaScript the _language_ doesn't offer any FFI or other mechanism to call outside services. Without a browser or something like Node's libuv, JavaScript wouldn't be useful at all. The capabilities provided in the box are part of the language in terms of what actually matters in motivating people to choose to use the language, no matter what form those capabilities come in.
  - tibbe 10y ago
    You don't need a standard library to win if you don't have any competitors (in the browser). :) Fitness for purpose is relative to the other options.
bitmadness 10y ago
This is exactly the main problem with Haskell. A stunning language with a lousy standard library. In my opinion, Haskell should offer arrays and maps as built-ins (like Go) and ship with crypto, networking, and serialization in the standard library (I know serialization is already there, but everyone seems to prefer Cereal, so...)
- tibbe 10y ago
  Agreed, except for the "like Go" part, which is unnecessarily ad-hoc.
- wyager 10y ago
  >Haskell should offer arrays and maps as built-ins Why? What does that gain? The standard platform provides Data.Map for maps, Data.Vector for arrays, and Data.Sequence for fast-edit sequences. It's not even clear what a "built-in" array or map in Haskell would even look like, or what semantics it should have. Especially in a pure functional language, you need to be clearer about what your intentions are. A regular mutable packed array won't work most of the time.
- lmm 10y ago
  > (I know serialization is already there, but everyone seems to prefer Cereal, so...) This is precisely why shipping things in the standard library is a bad idea. It ends up full of cruft that no-one uses because there are better alternatives.
- runeks 10y ago
  > This is exactly the main problem with Haskell. > A stunning language with a lousy standard library. I dream of the day where we can say that the main problem of Haskell is which libraries are included in the standard library. To me, we would already have reached programming nirvana at that point.
rcfox 10y ago
Out of curiousity: what do the 5 string types do differently?
- geekingfrog 10y ago
  String: Linked list of Char. Nice for teaching, horrible in every other aspects. Text and lazy Text: modern strings, with unicode handling and so on. ByteString and lazy ByteString: these are actually arrays of bytes. Used to represent binary data. Because haskell is lazy by default, and sometimes you want strictness (mostly for performances), there are two variants of Text and ByteString, and going from one flavor to the other requires manual conversion.
  - tibbe 10y ago
    Risking to go off-topic a bit, I think the lazy versions of Text and ByteString wouldn't have been needed if we had nice abstractions for streams (lists are not, they cause allocation we cannot get rid of) so that you don't need to implement a concrete stream type (e.g. lazy Text and lazy ByteString) for every data type. Rust does this well with iterators, for example.
    - wyager 10y ago
      The problem is that streams actually have very complicated semantics when they interact with the real world. What does it mean to traverse an effectful stream multiple times? Can you even do that? Data.Vector provides a very efficient stream implementation for vector operation fusion, but it's unsuitable for iterators/streams that interact with the real world. Pipes, on the other hand, combined with FreeT, provides good, reasonable semantics for effectful streams. As with many other things, Haskell forces you to be honest with what your code is actually doing (e.g. streaming things from a network) and this means that there's no one-size-fits-all implementation we can stuff everything into.
      - tibbe 10y ago
        Just sticking with the pure types there's currently no generic stream model that works well. No stream fusion system fuses all cases (even in theory) and they also fail to fuse the cases they're supposed to handle too often in practice. I haven't looked at pipes, but I'm guessing it doesn't all fuse away either.
wyager 10y ago
>Conversions between our 5(!) string types are very common. All five of those string types do different things. This isn't a problem; we just have increased expressivity. We couldn't fix this by having a more coordinated standard library. 5 is also a very manageable number IMO. >It's too difficult to make larger changes as we cannot atomically update all the packages at once. That's what Stack is for, no?

by tibbe 10y ago

saysjonathan 10y ago

One of the common mantras I've heard among Rust core devs is "std is where code goes to die". Where do you feel the line should be drawn between standard lib and external libraries?

skybrian 10y ago
Maybe a change in attitude? Stability can be a good thing. Go's standard library doesn't change that much, and that's a strength. How about: "std is where code goes when it's done". As is, really done. The API's won't need changing.
- steveklabnik 10y ago
  In the Ruby world, very few people use the standard library because it's got so many flaws, and they can't be fixed. So you end up with Nokogiri rather than REXML, all the various HTTP libs rather than net/*, etc. So it just ends up being bytes sent over the wire, wasting disk and bandwidth...
  - skybrian 10y ago
    Well, the trick is to actually get it right before standardizing it - much easier said than done. Keeping the standard library small helps with that since the bar is higher.
    - pcwalton 10y ago
      > Well, the trick is to actually get it right before standardizing it - much easier said than done. And that's what we're doing. > Keeping the standard library small helps with that since the bar is higher. But weren't you just advocating for a large standard library?
      - skybrian 10y ago
        I guess I didn't understand what was meant by "where code goes to die". Graduation != dying.
  - saysjonathan 10y ago
    I wonder if identifying the atomic aspects of what you intend your language to be used for ultimately helps in narrowing down what should be in std lib. Go prioritizes network programming and bundles the necessary components, like http & rpc servers and json. The http libraries are extensible enough to allow for customization where it's wanted (like http mux) while still creating a canonical implementation that'a still viable. Has Rust identified the core demographics of who they're targeting in order to provide the most applicable platform? Is the target everyone and all application type, therefore there is no default platform? Edit: To put it another way, is there a set of packages that is either necessary for rust, rust development, or most development in rust? If std lib includes everything necessary, then who are you targeting with the default platform?

pcwalton 10y ago

I don't think most of these are applicable to Rust. > - Conversions between our 5(!) string types are very common. > - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle. We have one string type defined in std, and nobody is defining new ones (modulo special cases for legacy encodings which would not be worth polluting the default string type with). > - Standard library cannot use containers, other than lists, for the same reason. > - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation. Hash maps and trees are in the standard library already. Everyone uses them. > - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances. This is true, but this hasn't been much of a problem in Rust thus far. > - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen. That only matters if you're breaking public APIs, right? That seems orthogonal to the small-versus-large-standard-library debate. Even if you have a large standard library, if you promised it's stable you still can't break APIs.

rtpg 10y ago
But if you have a large standard library and want to break the API, you can. If you have 100 different libs that are basically "standard" (who doesn't have `mtl` in their applications at this point), now you have to coordinate 100 different library updates roughly at the same time. If you forget even one of them, then you've broken everything. I think the argument for a large Prelude/standard lib is similar to Google's "single repo" argument: Easy to catch usages and fix them all at once. Plus you're making the language more useful out of the box. People coming from python can understand this feeling of opening a python shell and being productive super quickly form the get-go. Arguments for small std lib exist, of course. But Giant standard libraries are more useful than not. EDIT: I think the failure of the Haskell Platform has a lot more to do with how Haskell deals with dependencies, and the difficulties it entails, than with the "batteries included" approach itself.
- hyperhopper 10y ago

catnaroek 10y ago

Haskell's actual problem isn't the lack of a comprehensive standard library, but rather the presence of core language features that actively hinder large-scale modular programming. Type classes, type families, orphan instances and flexible instances all conspire to make as difficult as possible to determine whether two modules can be safely linked. Making things worse, whenever two alternatives are available for achieving roughly the same thing (say, type families and functional dependencies), the Haskell community consistently picks the worse one (in this case, type families, because, you know, why not punch a big hole on parametricity and free theorems?). Thanks to GHC's extensions, Haskell has become a ridiculously powerful language in exactly the same way C++ has: by sacrificing elegance. The principled approach would've been to admit that, while type classes are good for a few use cases, (say, overloading numeric literals, string literals and sequences), they have unacceptable limitations as a large-scale program structuring construct. And instead use an ML-style module system for that purpose. But it's already too late to do that.

tibbe 10y ago
There can be more than one problem, including having a small standard library.
- catnaroek 10y ago
  The lack of a standard library can be fixed relatively easily: write libraries! OTOH, the existence of anti-modular language features that are extensively used in several major libraries, is a more serious problem, because: (0) It means that libraries in general won't play nicely with each other, unless they're explicitly designed to do so. (1) It can't be fixed without throwing away code.
  - tibbe 10y ago
    This whole thread is exactly about how "write libraries!" (if done outside the standard library) doesn't work (see my top post). I do agree that lack of modularity features certainly doesn't help though.
10y ago

kibwen 10y ago

> Empirically, languages that have large standard > libraries (e.g. Java, Python, Go) seem to do better than > their competitors. You seem to be overlooking the ultimate counterexample: C. :P

saghm 10y ago
JS, too, right? Forget "large" standard library, there really isn't any standard library at all
- brandonbloom 10y ago
  You don't consider the dynamic, rich document presentation engine that is HTML to be a standard "library"? Seems like it is to me.
  - kibwen 10y ago
    HTML doesn't do anything for JS other than provide a way to create visual interfaces. It might be comparable to the role that `tkinter` plays for Python's stdlib, but HTML alone is emphatically not a standard library.
    - brandonbloom 10y ago
      My point is that most languages are totally useless on their own. JavaScript the _language_ doesn't offer any FFI or other mechanism to call outside services. Without a browser or something like Node's libuv, JavaScript wouldn't be useful at all. The capabilities provided in the box are part of the language in terms of what actually matters in motivating people to choose to use the language, no matter what form those capabilities come in.
- tibbe 10y ago
  You don't need a standard library to win if you don't have any competitors (in the browser). :) Fitness for purpose is relative to the other options.

bitmadness 10y ago

This is exactly the main problem with Haskell. A stunning language with a lousy standard library. In my opinion, Haskell should offer arrays and maps as built-ins (like Go) and ship with crypto, networking, and serialization in the standard library (I know serialization is already there, but everyone seems to prefer Cereal, so...)

tibbe 10y ago
Agreed, except for the "like Go" part, which is unnecessarily ad-hoc.
wyager 10y ago
>Haskell should offer arrays and maps as built-ins Why? What does that gain? The standard platform provides Data.Map for maps, Data.Vector for arrays, and Data.Sequence for fast-edit sequences. It's not even clear what a "built-in" array or map in Haskell would even look like, or what semantics it should have. Especially in a pure functional language, you need to be clearer about what your intentions are. A regular mutable packed array won't work most of the time.
lmm 10y ago
> (I know serialization is already there, but everyone seems to prefer Cereal, so...) This is precisely why shipping things in the standard library is a bad idea. It ends up full of cruft that no-one uses because there are better alternatives.
runeks 10y ago
> This is exactly the main problem with Haskell. > A stunning language with a lousy standard library. I dream of the day where we can say that the main problem of Haskell is which libraries are included in the standard library. To me, we would already have reached programming nirvana at that point.

rcfox 10y ago

Out of curiousity: what do the 5 string types do differently?

geekingfrog 10y ago
String: Linked list of Char. Nice for teaching, horrible in every other aspects. Text and lazy Text: modern strings, with unicode handling and so on. ByteString and lazy ByteString: these are actually arrays of bytes. Used to represent binary data. Because haskell is lazy by default, and sometimes you want strictness (mostly for performances), there are two variants of Text and ByteString, and going from one flavor to the other requires manual conversion.
- tibbe 10y ago
  Risking to go off-topic a bit, I think the lazy versions of Text and ByteString wouldn't have been needed if we had nice abstractions for streams (lists are not, they cause allocation we cannot get rid of) so that you don't need to implement a concrete stream type (e.g. lazy Text and lazy ByteString) for every data type. Rust does this well with iterators, for example.
  - wyager 10y ago
    The problem is that streams actually have very complicated semantics when they interact with the real world. What does it mean to traverse an effectful stream multiple times? Can you even do that? Data.Vector provides a very efficient stream implementation for vector operation fusion, but it's unsuitable for iterators/streams that interact with the real world. Pipes, on the other hand, combined with FreeT, provides good, reasonable semantics for effectful streams. As with many other things, Haskell forces you to be honest with what your code is actually doing (e.g. streaming things from a network) and this means that there's no one-size-fits-all implementation we can stuff everything into.
    - tibbe 10y ago
      Just sticking with the pure types there's currently no generic stream model that works well. No stream fusion system fuses all cases (even in theory) and they also fail to fuse the cases they're supposed to handle too often in practice. I haven't looked at pipes, but I'm guessing it doesn't all fuse away either.

wyager 10y ago

>Conversions between our 5(!) string types are very common. All five of those string types do different things. This isn't a problem; we just have increased expressivity. We couldn't fix this by having a more coordinated standard library. 5 is also a very manageable number IMO. >It's too difficult to make larger changes as we cannot atomically update all the packages at once. That's what Stack is for, no?