12 ms·
Opinions on this are a dime a dozen. You often see the reverse of it too, for example, you might have heard that "Python's standard library is where things go t
by burntsushi 7y ago
Opinions on this are a dime a dozen. You often see the reverse of it too, for example, you might have heard that "Python's standard library is where things go to die." You could just as easily call that a "terrible error." The fact that Python's standard library has an HTTP client in it, for example, doesn't stop everyone from using requests (and, consequently, urllib3) for all their HTTP client needs. So despite the fact the standard library provides a lot of the same functionality as a third party dependency, folks are still using the third party dependency.
I think the size of the standard library is just one of possibly many contributing factors that leads to a large number of dependencies. I think a part of it is culture, but another part of it is that the tooling _enables_ it. It's so incredibly easy to write some code, push it to crates.io and let everyone else use it. That's generally a good thing, but it winds up creating this spiral where there's almost no backpressure _against_ including a dependency in a project. This means there's very little standing in the way of letting the fullest expression of DRY run wild. There are some notable examples in the NPM ecosystem where it reaches ridiculous levels. But putting the extremes aside, there's a ton of grey area and it can be pretty difficult to convince someone to write a bit more code when something else might work off the shelf. (And I mean this in the most charitable way possible. I find myself in that situation.)
I do hope we can turn the Rust ecosystem around and stop regularly having dependency trees with hundreds of crates, but it's going to be a long and difficult road. For example, not everyone even agrees with my perspective that this is actually a bad thing.
- Amezarak 7y agoThere are a lot of us out there with dev and prod environments that aren’t allowed to be connected to the Internet or otherwise strictly controlled. Languages that rely on me downloading packages are mostly dead in the water at my workplace.
- steveklabnik 7y agoFirefox, Debian, and many other build systems aren’t either. That’s orthogonal from all of this; the use case is well supported and has been for years.
- stouset 7y agoVirtually no modern language or package system requires this. All of them allow you to cache dependencies locally, so they can be committed to version control or otherwise put in place during deployment however you see fit.
- Amezarak 7y agoMaybe I didn't communicate my point clearly. I cannot just go out and get whatever dependencies I want to use. It isn't possible. What I have on the system is what I've got, and I can't add anything. I cannot put the dependencies on the system by any means. Thus a lot of things get done in Python and Java, because of the ample standard libraries. I was able, after a year of lobbying and procedures and approvals, to get a Rust compiler, but there is zero chance of me ever getting anything off crates.io.
- jchw 7y agoYou mention Python, but I am pretty sure you've written some popular Go packages, so what is your opinion on the Go standard library? I think it's a good example of 'batteries included' in the right sense. Though, some of the reason its good may just be virtue of the fact that it's newer and there's less rotting packages; I guess only time will tell for sure, but it definitely feels right to me in many cases.
- sansnomme 7y agoA better choice would be enterprise titans like Java and .NET, more bases are covered. Go's GUI story for example is just endless fragmentation compared to e.g. Java.
- jchw 7y agoApples and oranges. It is a feature that Go, Rust don't have "enterprise" standard libraries like Java and .NET.
- pcwalton 7y agoThis comment indicates the problem! Taking the scare-quotes "enterprise" as a synonym for "bloat", one person's necessary feature is another person's bloat. .NET is an interesting case, because of WinForms, which is a de facto part of the standard library. I think few C# developers think of WinForms as bloat; it's very convenient for making simple UIs. Yet putting, say, GTK+, in the Go standard library would doubtless be considered bloat. I don't think there are easy answers to these questions.
- raphlinus 7y agoIn my post, I specifically call on proc-macro support (syn and quote) plus rand (not all of rand though, just the "give me a random number" functionality that comprises 99% of the use cases but 10% of the implementation complexity) to be added to the Rust standard library. But I feel these are particularly justified because they're already present, just not accessible. Overall, I think Rust's "batteries not included" approach is a good tradeoff.
- orf 7y ago> not all of rand though, just the "give me a random number" functionality that comprises 99% of the use cases but 10% of the implementation complexity Are you suggesting including the core `rand` crate with some traits and a basic pseudo-random generator in the stdlib, and allowing other crates to use those traits to implement the many other[1] RNGs? I'm honestly not sure this is a good idea, it seems the rand crate has undergone a few iterations, crystalizing this inside the stdlib might lead to problems? 1. https://rust-random.github.io/book/guide-rngs.html https://rust-random.github.io/book/guide-rngs.html
- raphlinus 7y agoI admit, design of a proper random number library API is tricky, and experience from deploying rand will no doubt be invaluable. The point I was trying to make is that some use cases require sophistication such as being able to choose different algorithms, but a lot of the times rand shows up in a build time histogram, it's just because the user wanted some pretty good random numbers; a much simpler API would suffice.
- anp 7y agoI agree that in many cases a much simpler API would suffice, but it strikes me as odd to suggest adding a convenient but insufficient-for-security API to the same standard library that protects HashMap against DoS.
- Dylan16807 7y ago
- quickthrower2 7y agoThe culture of the programmers comes into it too. In the Java/.NET world devs are happier to take what the core libraries provide. A case in point is how the ORM Entity Framework that comes with .NET has made the older NHibernate (a separate package) obsolete. .NET developers love using the standard libs, but OTOH Microsoft has a lot of resources to create very complete libraries.
- chc 7y agoEven in Java, if you're looking to do anything with dates, everyone will tell you to use Joda-Time over java.util.Date.
- darksaints 7y agoThat's true if you're still on Java 7. java.time.* is the go-to now, and it is very high quality.
- sanderjd 7y agoI think not anymore since java.time - which is incredibly similar to Joda - came around? This is actually an illustrative example of the process I like and hope rust will develop over time: let the community reach a consensus on the best third party libraries things, then consider pulling them in to or at least taking the best parts of their APIs for the standard library.
- bluejekyll 7y agoBut won’t the community converge on that anyway, and in a similar amount of time? So why add it to the stdlib?
- quantummkv 7y agoStability? Guarantee that the api will continue to work and that no one will wake up one morning and left-pad the entire ecosystem?
- 7y ago
- dhbradshaw 7y agoThe interesting thing about the way things have developed with using third party dependencies early on is that these dependencies provide us with data. We don't have to just guess or make biased claims about what would be useful to move to the standard library. We can look at the numbers and see what people are actually using.
- ende 7y agoI feel like a reasonable middle ground to these issues is for communities to perhaps embrace “metapackages” that serve as community maintained “standard libraries”. In the R community we have tidyverse, which is practically a full mirror of R’s standard libs at this point. These metapackage communities can then focus on interoperability of constituent packages w/o overburdening the standard libraries, and core language can remain lean and concise.
- steveklabnik 7y agoSeveral attempts have been made at this in Rust in the past, but few people use them. It just adds even more dependencies that you’re not actually using.
- ende 7y agoSure, but that's why it's optional.
- eden_h 7y agoTidyverse is written by a strong contributor to R, but it isn't used regularly by a large percentage of the community, so I'd be averse to calling it part of the standard libraries - although I appreciate this is what you're referring to with the speech marks. On point of the post, Tidyverse is a great deal of bloat by loading unnecessary packages instead of specifics and creating namespace issues where two functions have the same name. It's generally fine for interactive work, but causes so many issues in development as you can't pick and choose what's loaded into the environment. In line with what you're saying, meta-libraries make sense in terms of developing a line of packages towards a singular vision, but only make sense when the core libraries aren't expanded regularly. Maybe in these cases more of push needs to be made to supporting the existing tools? R is an odd example though, as the standard libraries are loaded by default and I think only really comprise of basic stats/graphics/data.frame tools. I don't really believe a great deal of extra tools have been added to base R in the last few years, just improved in terms of speed and memory use (e.g 3.5.0's change to compiled packages)
- mantap 7y agoPython's standard library is where things go to die because of the terrible adhoc versioning system (the module name is the version number) and dynamic typing means they are afraid to change anything. But even then it's still better than having no standard library at all. The advantage of a standard library is that you only need to learn one API instead of a dozen different APIs for doing the same thing, which means you can develop a degree of mastery over it. It also reduces the friction for using better abstractions. e.g. Every professional Python programmer knows defaultdict, whereas I rarely see that data structure used in other programming languages, it's too much of a leap to install a dependency to save a few if statements, but it all adds up.
- swsieber 7y ago> The advantage of a standard library is that you only need to learn one API instead of a dozen different APIs for doing the same thing, which means you can develop a degree of mastery over it. The rust ecosystem has done well to converge on certain crates as sort of replacement for missing std features. In practice (at least in the rust ecosystem), I only need to learn one interface for: * regex (regex) * serialization (serde) * network requests (request) There are de-facto base crates in the ecosystem.
- pjmlp 7y agoDo they also compile and run across all platforms supported by rust compiler? Because that is the biggest asset from stuff being in the standard library.
- chrstphrhrt 7y agoI only know Node and Python well enough. I find with NPM that dependencies are indeed hell because anything goes. With Python it's slightly less bad because packages have a fixed tree of versions (although I don't know what's up with Conda or others yet), not to mention the more extensive standard lib. I suppose if there are real conflict workarounds required you could always use virtual environments all the way down, but I have generally just done whatever upgrading is required to resolve incompatibilities (which can sometimes preclude using third-party deps or require forking and upgrading their own deps). I also remember from years ago that Bundler in Ruby allowed version mismatches to coexist and would install both deps in the same tree, punting any runtime issues (same as NPM). Any Gentoo or NIX users might have something to add here, as I can remember being regaled at conferences by them about this topic. All that said, it would seem that Rust could have some kind of super-intelligence about dependencies due to all the static goodness. So my questions are: a) who cares how much is userland and how much is std lib if everything is equally safe and documentable? b) if "too much" becomes std lib, could any feelings of overwhelmingness not be mitigated with more namespacing?
- sgentle 7y agoI feel like the role of a standard library has become kind of overloaded. Back when you had to manage all your dependencies yourself, having that baseline of functionality bundled with the language and maintained by its authors was a logistical necessity. You just don't have time to chase deps for every little thing. But in the modern world, I agree with the small language people: the logistical problem is dead. Almost every modern language has push-button dependency management, so the difficulty overhead of using a third-party library is basically zero. And, as you pointed out, this means that the stdlib now competes with third-party libraries, and the third-party libraries proliferate wildly. So there's still a problem, it's just not logistical anymore. The other problem a stdlib solves is making decisions. This is the npm nightmare: which of these fifty libraries do I use? They're easy to install, but hard to evaluate. This compounds because every library is also making those decisions with their dependencies and so on, so you can end up with 8 different implementations of basically the same thing. You don't have this problem with a stdlib because if there's a std::string, everyone expects your code to work with std::string. So perhaps the modern stdlib is better off as a standards library: no code, just a mapping from name (ie "option_parser") to package@version (ie "clap@1.1.0"). The work of the stdlib would then be to curate this mapping in a cohesive way, so the packages are high quality individually, but also work well together and reflect the general direction of the language and its community. Whether or not these packages are actually bundled with the compiler is really just an implementation detail. The library isn't code; it's decisions. Updates would be pinned to release versions, so backwards-incompatible changes coincide with a major version/edition/etc of the language. This would be an expected process, because, as with urllib{n+1}, the first answer isn't always the best. Third-party packages are just as available as ever, of course, and a destandardised package is only a (trivially automated) rename away. Aside from providing a layer of simplicity and stability over the third-party ecosystem, the standardised libraries would serve as reference implementations for a common interface, like Node's express/connect, or Rust's tokio/futures. In other words, it could help concentrate community effort around emerging standards. I grant that this is a very, uh, political approach to what has historically been a code problem, but if anything I think the current situation owes itself largely to treating community problems ("how do we agree on a foundational layer of library code?") with technical solutions (package registry + sort by stars).
- 7y ago
- atoav 7y agoI am a Rust user myself and I think one major problem is that beeing on crates.io say nothing about the quality of code. I never published a single crate myself because in my eyes to be worth published a crate should work decently. That beeing said, I think the whole Rust userbase would benefit from having some sort of collection of well tested crates and strictly divide between private, work in progress and production crates.
- ajxs 7y agoThe canonical example I was thinking of was this: https://doc.rust-lang.org/nomicon/ffi.html https://doc.rust-lang.org/nomicon/ffi.html We only get as far as the second paragraph before the official manual is recommending that we use a 3rd party crate. I understand that this crate is made by the Rust developers, but this doesn't seem like a good approach to me.