7 ms·

This is close to what Rust is doing, and it's working pretty well, apart from shocking newcomers who expect libstd to be useful on its own. In Rust, libstd is

by pornel 7y ago

This is close to what Rust is doing, and it's working pretty well, apart from shocking newcomers who expect libstd to be useful on its own. In Rust, libstd is mainly for interfacing with the compiler and providing interoperability between packages (crates). The wider crate ecosystem is the real standard library, since external crates are as easy to use as the standard library. For example, the libstd doesn't even have support for random number generation. There's a rand crate, which is now on 6th major breaking version. That's perfectly fine, because multiple versions can coexist in one program, and every user can upgrade (or not) at their own pace. And the crate was able to refine its interface six times, instead of being stuck with the first try forever.

icxa 7y ago
This is interesting. Go seems to have the complete opposite stance. The stdlib are some of the most useful and well written packages you can use in the Go ecosystem, and then you have the "extended" standard lib which isn't 100% in the language yet, and even further sometimes concepts from useful community packages make it into the std lib. As to why this is the case, I think maybe this is enabled by Go's backward's compatibility focus and encouragement to upgrade early and often, and the community's focus to utilize small interfaces sometimes from the stdlib itself, like io.Reader and io.Writer, or http.Handler. Added to that w.r.t. using the latest and greatest, most Go users frequently are using the latest version of Go even in production (per the go experience surveys). I am sure it also helps that Google pays people to develop, maintain, and improve the stdlib.
- rbanffy 7y ago
  This approach is useful for a while, but once something is in a stdlib, its interface is frozen forever.
  - icxa 7y ago
    (I am trying to tie the two concepts together from your reply, so I am not saying this to come off as combative but understand your objection) - What would you define as a while? I think Go has done well with it, given that it is almost 10 years old now at this point, and in reality, was in development internally at Google years before that. - Do you feel like an interface being frozen forever is particularly a bad thing in of itself? What if the interface does a really good job describing the thing, whatever it may be? For example, Go's io.Reader/io.Writer interfaces. I agree sometimes an interface being frozen is bad, but for example when Go standardized the context package, it simply added "Context" to the existing functions that now take a context (e.g. in the database/sql package, you have the old, Exec, Query, QueryRow functions, and after 1.8 you have ExecContext, QueryContext, QueryRowContext, which some people may view as a reason to have method overloading, but I view as adding better clarity.
    - DougBTX 7y ago
      The way to answer your two questions is to combine them, the definition of “a while” depends on how good a job the interface does. If the interface is really good, then it can last a really long time (perhaps as long as the language lasts). I read some discussion the other day about some ways in which the Any type in Rust isn’t as flexible as it could be, since it is frozen the only way to improve it would be to introduce a new name, such as Unknown. Similarly in C# along with adding async support, the standard library added async versions of many methods, eg Read now also has a ReadAsync partner. It does seem that having multiple names for basically the same thing adds a small but tolerable level of overhead to a language. At least if as much as possible is moved out, then projects can choose to only use the latest versions, and live in a world as if past versions never existed.
    - jerf 7y ago
      Go was and is by design a "boring" language. The core designers didn't have much trouble looking at decades of prior art and getting it mostly right. Rust libraries should be expected to take a few tries to get right, especially earlier in its lifecycle. There's more possibilities and less experience in the language. You can see a similar effect in Haskell, which has iterated many basic bits of functionality many times over.
- pornel 7y ago
  No doubt Go's stdlib is useful, and there's plenty of things it got right. However, it's not immune to making some mistakes and having to freeze them forever. The more functionality you add, the harder it gets to get it perfect on the first (and only) try. Search for "deprecated site:https://golang.org/pkg/" https://golang.org/pkg/" finds various issues ranging from cosmetic mistakes to entire packages being deprecated. CompressedSize uint32 // Deprecated: Use CompressedSize64 instead. CompressedSize64 uint64 // Go 1.1 // Deprecated: HeaderMap exists for historical compatibility // and should not be used. Requirements may change over time, so even getting something perfect now is not a guarantee it will last (e.g. pre-UTF-8 languages froze byte-oriented or UCS-2 strings, even though these were good decisions at the time). Sometimes improvements are not worth the cost of deprecation and replacement, so things are just left as they are. For example, an HTTP interface designed for request-response HTTP/1 works for stream-oriented HTTP/2, but support for push, prioritization and custom frames is bolted on. Packet-oriented HTTP/3 will add even more stuff that will have to be retrofitted somehow to the old model. Libraries can come and go, but std can't just throw away an old interface and start over.
mkesper 7y ago
But then you have six versions to have security fixes, how do you do that?
- pornel 7y ago
  You release fixes for older versions according to semver.
gmueckl 7y ago
Having incompatible libraries solving the same basic problems is absolutely not fine. Over time, this will become a huge issue for composability. C++, for instance, already is in this kind of mess with things like STL containers versus Qt comstainers vs. homegrown special case (optimized containers) and heaps of additional libraries building upon each. There are other good examples in other programming languages as well, mostly older ones. The example of random number generators is a good one, too. There are a lot of applications that require (reproducable!) PRNG sequences and sometimes you have to share PRNGs between modules. Now, looking at the rand crate, I see that the prng part of it was recently mucked around with. If I have two 3rd party modules that I require to share a PRNG that I control (say, noise generators for procedural textures), I cannot compose them if one of them uses the old and one of them uses the new version of the library.
- pornel 7y ago
  When a crate is part of a public interface, that's harder, indeed. Crates solve it in a few ways: • crates that expect to be used for interoperability are often split into smaller crates (like API and back-end, or low-level API and high-level API), so that they can evolve some parts without breaking others. • sometimes breaking changes are technically breaking, but easy to upgrade (e.g. methods renamed, args reordered). In that case most users catch up quickly. • in desperate cases, a new version can import its own old version and re-export old structs and interfaces that haven't changed, so they're compatible across major versions. • proper sharing and composition should be done via traits, so that you can implement a trait for any number generator, not just one version of one implementation.
rmtech 7y ago
I've never coded Rust - is there any distinction between a really important crate used by millions of people and something really obscure with 3 users? Are all the crates subject to security audit?
- steveklabnik 7y ago
  Anyone can upload to crates.io without review.
- pornel 7y ago
  There is no technical distinction. The community is working on a WoT/review tool (cargo-crev), but in the meantime you can see who has published the crate and who uses it. The de-facto standard crates are maintained by Rust team members or well-known authors.
  - rmtech 7y ago
    ok. So it's kind of informal at the moment. Maybe in the future we'll see more hacking of libraries (people managing to deliberately sneak exploits in) and in response stronger lockdowns on important library code.

by pornel 7y ago

icxa 7y ago

This is interesting. Go seems to have the complete opposite stance. The stdlib are some of the most useful and well written packages you can use in the Go ecosystem, and then you have the "extended" standard lib which isn't 100% in the language yet, and even further sometimes concepts from useful community packages make it into the std lib. As to why this is the case, I think maybe this is enabled by Go's backward's compatibility focus and encouragement to upgrade early and often, and the community's focus to utilize small interfaces sometimes from the stdlib itself, like io.Reader and io.Writer, or http.Handler. Added to that w.r.t. using the latest and greatest, most Go users frequently are using the latest version of Go even in production (per the go experience surveys). I am sure it also helps that Google pays people to develop, maintain, and improve the stdlib.

rbanffy 7y ago
This approach is useful for a while, but once something is in a stdlib, its interface is frozen forever.
- icxa 7y ago
  (I am trying to tie the two concepts together from your reply, so I am not saying this to come off as combative but understand your objection) - What would you define as a while? I think Go has done well with it, given that it is almost 10 years old now at this point, and in reality, was in development internally at Google years before that. - Do you feel like an interface being frozen forever is particularly a bad thing in of itself? What if the interface does a really good job describing the thing, whatever it may be? For example, Go's io.Reader/io.Writer interfaces. I agree sometimes an interface being frozen is bad, but for example when Go standardized the context package, it simply added "Context" to the existing functions that now take a context (e.g. in the database/sql package, you have the old, Exec, Query, QueryRow functions, and after 1.8 you have ExecContext, QueryContext, QueryRowContext, which some people may view as a reason to have method overloading, but I view as adding better clarity.
  - DougBTX 7y ago
    The way to answer your two questions is to combine them, the definition of “a while” depends on how good a job the interface does. If the interface is really good, then it can last a really long time (perhaps as long as the language lasts). I read some discussion the other day about some ways in which the Any type in Rust isn’t as flexible as it could be, since it is frozen the only way to improve it would be to introduce a new name, such as Unknown. Similarly in C# along with adding async support, the standard library added async versions of many methods, eg Read now also has a ReadAsync partner. It does seem that having multiple names for basically the same thing adds a small but tolerable level of overhead to a language. At least if as much as possible is moved out, then projects can choose to only use the latest versions, and live in a world as if past versions never existed.
  - jerf 7y ago
    Go was and is by design a "boring" language. The core designers didn't have much trouble looking at decades of prior art and getting it mostly right. Rust libraries should be expected to take a few tries to get right, especially earlier in its lifecycle. There's more possibilities and less experience in the language. You can see a similar effect in Haskell, which has iterated many basic bits of functionality many times over.
pornel 7y ago
No doubt Go's stdlib is useful, and there's plenty of things it got right. However, it's not immune to making some mistakes and having to freeze them forever. The more functionality you add, the harder it gets to get it perfect on the first (and only) try. Search for "deprecated site:https://golang.org/pkg/" https://golang.org/pkg/" finds various issues ranging from cosmetic mistakes to entire packages being deprecated. CompressedSize uint32 // Deprecated: Use CompressedSize64 instead. CompressedSize64 uint64 // Go 1.1 // Deprecated: HeaderMap exists for historical compatibility // and should not be used. Requirements may change over time, so even getting something perfect now is not a guarantee it will last (e.g. pre-UTF-8 languages froze byte-oriented or UCS-2 strings, even though these were good decisions at the time). Sometimes improvements are not worth the cost of deprecation and replacement, so things are just left as they are. For example, an HTTP interface designed for request-response HTTP/1 works for stream-oriented HTTP/2, but support for push, prioritization and custom frames is bolted on. Packet-oriented HTTP/3 will add even more stuff that will have to be retrofitted somehow to the old model. Libraries can come and go, but std can't just throw away an old interface and start over.

mkesper 7y ago

But then you have six versions to have security fixes, how do you do that?

pornel 7y ago
You release fixes for older versions according to semver.

gmueckl 7y ago

Having incompatible libraries solving the same basic problems is absolutely not fine. Over time, this will become a huge issue for composability. C++, for instance, already is in this kind of mess with things like STL containers versus Qt comstainers vs. homegrown special case (optimized containers) and heaps of additional libraries building upon each. There are other good examples in other programming languages as well, mostly older ones. The example of random number generators is a good one, too. There are a lot of applications that require (reproducable!) PRNG sequences and sometimes you have to share PRNGs between modules. Now, looking at the rand crate, I see that the prng part of it was recently mucked around with. If I have two 3rd party modules that I require to share a PRNG that I control (say, noise generators for procedural textures), I cannot compose them if one of them uses the old and one of them uses the new version of the library.

pornel 7y ago
When a crate is part of a public interface, that's harder, indeed. Crates solve it in a few ways: • crates that expect to be used for interoperability are often split into smaller crates (like API and back-end, or low-level API and high-level API), so that they can evolve some parts without breaking others. • sometimes breaking changes are technically breaking, but easy to upgrade (e.g. methods renamed, args reordered). In that case most users catch up quickly. • in desperate cases, a new version can import its own old version and re-export old structs and interfaces that haven't changed, so they're compatible across major versions. • proper sharing and composition should be done via traits, so that you can implement a trait for any number generator, not just one version of one implementation.

rmtech 7y ago

I've never coded Rust - is there any distinction between a really important crate used by millions of people and something really obscure with 3 users? Are all the crates subject to security audit?

steveklabnik 7y ago
Anyone can upload to crates.io without review.
pornel 7y ago
There is no technical distinction. The community is working on a WoT/review tool (cargo-crev), but in the meantime you can see who has published the crate and who uses it. The de-facto standard crates are maintained by Rust team members or well-known authors.
- rmtech 7y ago
  ok. So it's kind of informal at the moment. Maybe in the future we'll see more hacking of libraries (people managing to deliberately sneak exploits in) and in response stronger lockdowns on important library code.