10 ms·

Perhaps my biggest critique is that crates.io has no namespacing. Anyone can just claim a global and generic package name and we mostly have to deal with it (un

by chungy 3y ago

Perhaps my biggest critique is that crates.io has no namespacing. Anyone can just claim a global and generic package name and we mostly have to deal with it (unless you avoid using the crates.io repository, but then you'll probably have more problems...). Some of these globally-claimed generic packages are not really the best package to use. Maybe it was a reaction against the Java-style reverse DNS notation, which is verbose and annoying, but a more GitHub-style user/group namespace prefixing package names would have been the nice middle ground.

rochak 3y ago
Go made a great decision to namespace packages via this Github style.
- treyd 3y ago
  And the horrible decision to not make library-level ("module") and code-unit-level ("package") namespacing orthogonal. The former was an afterthought tacked on since the package system was designed to be used only within Google's monorepo and little care was paid to how it would work when it was released to the public and used more generally.
  - starttoaster 3y ago
    I want to understand what you just said, but I fear watering your language down a bit might be a tall ask with some people. Would you be willing to eli5 what you believe Go did that was a horrible decision with regards to module/package namespacing?
    - cdelsolar 3y ago
      I think he wants them to be treated differently for some reason.
    - ezrast 3y ago
      I think what they mean is: if you see a line like `import git.example.com/foo/bar/baz`, that could be package `baz` inside module `git.example.com/foo/bar`, or it could be package `bar/baz` inside module `git.example.com/foo`. Also, even if you know it's the latter, package namespacing isn't strictly related to directory structure, so `bar/baz` has no specific meaning outside of the context of a go import. They could have used any other separator for package components - `git.example.com/foo:bar:baz` - but instead they chose the slash, making the scheme both technically ambiguous and easy to confuse for an HTTP URL.
      - starttoaster 3y ago
        Ah that makes sense. I think Go did somewhat stumble a bit in the early days due to this, especially with repositories in GitLab, where GitLab allows essentially a directory tree where your repository can be nested indefinitely in directories like `https://gitlab.com/mygroup/subgroup1/subgroup2/repository https://gitlab.com/mygroup/subgroup1/subgroup2/repository`. I still don't think this is a huge issue, to be honest. Not one big enough for me to complain about, for sure. But it's definitely not ideal.
- baby 3y ago
  How is this a good decision? You have to alias packages when names collide
  - tapirl 3y ago
    What is the better decision to avoid name colliding?
- pjmlp 3y ago
  Go packages is yet another design item that I dislike on the language, exposing SCM URLs directly on the source code, and no story for binary caching.
  - yencabulator 3y ago
    > exposing SCM URLs directly on the source code Incorrect. You're able to use any URL you control, regardless of where your SCM is located.
    - pjmlp 3y ago
      Thanks for pointing out you didn't got what the whole point of the wrong design is about.
      - yencabulator 3y ago
        Thanks for confirming you'd rather be a hater than someone with justified opinions.
        pjmlp 3y ago
        I am quite open about my opinion regarding most of Go's design decisions.
jrockway 3y ago
URLs for packages makes a lot of sense. It works well in the land of Go. It also conveniently eliminates the need for the language to have a global packages database. Upload your package to example.com/your-thing and it's released! (You can, of course, still offer a cache and search engine if you want to.)
- alpaca128 3y ago
  Cargo does support URLs to git repos for dependencies. But crates.io is the official platform and almost every search I do on it returns at least one generically named entry with an empty repository that someone snatched away and never used.
- swatcoder 3y ago
  No, URL's don't make sense because your application shouldn't care where on the internet your dependency happened to be hosted when you integrated it. It's location has nothing to do with what it is. By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project. Your project shouldn't need to care whether GitHub fell out of fashion and the project moved to GitLab, and definitely shouldn't be relying on GitHub being available when you need to build, test, deploy, or scale. That's a completely unnecessary failure point for you to introduce. Systems that use URL-identified packages can work around some of this, but just reinforce terrible habits.
  - jrockway 3y ago
    It worked for the rest of the Internet.
    - ethbr1 3y ago
      There is something to be said for separating {unique piece of content} and {hosted location}. E.g. doi's https://en.m.wikipedia.org/wiki/Digital_object_identifier https://en.m.wikipedia.org/wiki/Digital_object_identifier
dhosek 3y ago
Agreed. The other thing I don’t really like is that you can’t split up things in the rust namespace hierarchy between crates (something that’s natural with jars in JVM). I would have liked to have defined things so that I could have the unicode handlers for finl live in finl::unicode, the parser in finl::parser, etc. but because they’re in separate crates rust gets upset about finl being defined twice and there’s no workaround for it. There are likely pitfalls I don’t see in what I want, so I live with it.
- ramranch 3y ago
  Would re-exporting partially solve your problem?
  - dhosek 3y ago
    As I recall, I could do something where I could have a common root crate that would import and re-export the other crates with the namespaces modified on the exports and then control which crates are exported through feature gates, but it just seemed more hassle than it was worth.
- epage 3y ago
  We have an RFC for this that has fairly broad support that I just need to bug some people some more so we can approve it. https://github.com/rust-lang/rfcs/pull/3243 https://github.com/rust-lang/rfcs/pull/3243
lolinder 3y ago
> Maybe it was a reaction against the Java-style reverse DNS notation I suspect it was less a reaction against anything and more just following the norms established by most other package managers. NPM, PyPI, RubyGems, Elixir's Hex, Haskell's Cabal... I'm having a hard time thinking of a non-Java package manager that was around at the time Rust came out that didn't have a single, global namespace. Some have tried to fix that since then, but it was just the way package managers worked in 2014/2015.
- jiggawatts 3y ago
  Yes, and all of those have had major security issues caused by their lack of foresight. "We're pretending security is not an issue." has been the feedback every time this is raised with the Cargo team. To be honest, it's turned me off Rust a little bit. The attitude of "Rust is memory-safe, so we don't need any other form of security." is not a good one.
  - muldvarp 3y ago
    > "We're pretending security is not an issue." has been the feedback every time this is raised with the Cargo team. Do you have a specific link where I can read this response, because this is not at all the responses I have read.
    - jiggawatts 3y ago
      Just some random Cargo security-related issues I noticed: - No strong link between the repo and the published code. - Many crates were spammed that were just a wrapper around a popular C/C++ library. There's no indication of this, so... "surprise!"... your compiled app is now more unsafe C/C++ code than Rust. - Extensive name squatting, to the point that virtual no library uses the obvious name, because someone else got to it first. The aformentioned C/C++ libraries were easy to spit out, so they often grabbed the name before a Rust rewrite could be completed and published. So you now go to Cargo to find a Rust library for 'X' and you instead have to use 'X-rs' because... ha-ha, it's actually a C/C++ package manager with some Rust crates in there also. - Transitive dependencies aren't shown in the web page. - No enforcement or indication of safe/unsafe libs, nostd, etc... - No requirement for MFA, which was a successful attack vector on multiple package managers in the past. DISCLAIMER: Some of the above may have been resolved since I last looked. Other package managers also do these things (but that's not a good thing). In my opinion, any package manager that just lets any random person upload "whatever" is outright dangerous and useless to the larger ecosystem of developers in a hurry who don't have the time to vet every single transitive dependency every month. Package managers need to grow up and start requiring a direct reference to a specific Git commit -- that they store themselves -- and compile from scratch with an instrumented compiler that spits out metadata such as "connects to the Internet, did you know?" or "is actually 99% C++ code, by the way".
yarg 3y ago
This needs to be resolved by every damned language. Just make signed dependencies a universal default, point to an https page for the package vendor and use the signing key from there. Neither node nor maven ever bothered to solve this, so we end up wandering the Wild West wondering when it will be that HR, or legal, or architecture comes knocking on the door to ask what we were thinking having a dependency on a dynamic version of Left Pad. I'd kinda like to see what Cloudflare and Let's Encrypt could come up with if they worked together on at very least a white paper and an MVP POC.
- lesuorac 3y ago
  I don't think that's the real solution. Pay somebody either internally or externally to maintain a repo of all your dependencies and point your code at that. You won't get a left-pad incident. You won't get a malicious .so incident (unless you mirror binaries instead of source code). Like if you ran out of screws to make your product with do you walk around the street and scrounge up some? No, you go to a trusted vendor and buy the screws.
  - yarg 3y ago
    I'm not suggesting scrounging. I'm suggesting that the guys who've repeatedly proven themselves technically competent around security at scale might have a couple of useful ideas regarding how the industry might go about crawling its way out of this little security clusterfuck. And perhaps even stop treating something as simple as a BOM as an enterprise feature, given that the overhead on such things is damned near zilch and the security implications are staggering. https://www.cisa.gov/sbom https://www.cisa.gov/sbom
  - cmrdporcupine 3y ago
    There's reasons why Google projects don't go out on the internet to get their 3rd party deps. They're all checked into Google3 (or chromium, etc.). One version only. With internal maintainers responsible for bringing it in and multiple people vetting it, and clear ownership over its management. E.g. you don't just get to willy nilly depend on a new version -- if you want to upgrade what's there, you gotta put a ring on it. If you upgrade it, you're likely going to be upgrading it for everyone, and the build system will run through the dependent tests for them all, etc. And the consequence is more responsible use of third party deps and less sprawling dependency trees and less complexity. And additional less security concerns as the code is checked in, its license vetted, and build systems are hunting around on the Internet for artifacts.
cetra3 3y ago
I think most people are in support of namespacing but it's a big change and will take a while to see through.
Jalad 3y ago
This is a common critique, and although I don't have insight into why the original decision to not have namespaces was made, the current outlook is that until issues related to continuity are resolved, it's a no go: https://samsieber.tech/posts/2020/09/registry-structure-influence/ https://samsieber.tech/posts/2020/09/registry-structure-infl...
- sigwinch28 3y ago
  That article starts with the premise that “it’s a feature, not a bug” then goes on to describe a whole bunch of things I consider to be anti-features of a packaging system that has a flat namespace. The first section says it discourages forking. I consider this to be bad. Nobody’s code should be more important purely because it squatted a better name. The Identity section actually makes the case that flat registries make naming harder. The section on Continuity is “we’ve tried nothing and we’re all out of ideas”. Make up an org name and grandfather all packages in the flat namespace into that special org. Also this is already a problem because packages in the flat namespace do get abandoned, then forked, and then we have the associated issues. The section on Stability seems to take it as a given that crates.io should be the only registry. I don’t. It also seems to conflate cargo with rustc for the benefit of the argument. The squatting section describes only anti-features and I don’t consider the author’s legitimate use cases to be legitimate reasons to squat. I think the only legitimate problems that need addressing are the ergonomics of accessing namespaced packages throughout transient dependencies and backwards compatibility with non-namespaced code. But the fact that these are real problems does not, to me, make a flat namespace a “feature”. It’s just easier to implement. It’s okay for it to be a mistake that takes effort and time to fix.
  - kevincox 3y ago
    Another option would be to grandfather all packages into their own org. So serde becomes serde/serde. This way you don't need to manage permission rules in the legacy "all" namespace. You get some oddities such as serde-derive/serde-derive but the package owners can choose if they want to move to serde/derive or leave it in a separate namespace.
fawadasaurus 3y ago
I did some analysis on crates.io to find the top name squatters. Then I did some calculations and found that the top name squatter created their crates at a rate of about one ever 30 seconds for a period of a week straight. I send the analysis to the crates.io team and pointed that they have a no-automation policy. They told me that it was not sufficient proof that someone was squatting those names. That's my problem with crates.io is that they have a clear policy and they don't enforce it so all the short/easy to remember names for crates are already taken and there is nothing you can do to get it.
- echelon 3y ago
  There's a secret effort in the Rust community to supplant Crates.io and create an entirely new package ecosystem with proper namespacing, security, and much better community. Not naming names, but I know several people working to put Crates.io out to pasture. There's a level of playing nice with them for the time being (eg. build reproducibility), but it's only KTLO. Crates.io needs to die for Rust to thrive. They're a bungled, mismanaged liability. New code, new leadership.
  - solardev 3y ago
    Why does something like that need to be secret...? Isn't it in the community's best interest?
    - TylerE 3y ago
      Drama avoidance and avoiding bikeshedding seem obvious. Much easier to present a working system than a design that will get nitpicked into irrelevancy.
      - solardev 3y ago
        Yeah, that makes sense.
      - sapling-ginger 3y ago
        That quite funny. Just like when some people formed a violent militant group to take down a violent tyranical dictatorship. Of course they would promise that they would absolutely de-arm right after the dictatorship has been overthrown, and immediately establish a peaceful democratic government with fair election. They absolutely would, would they? They would never turn into what they were formed to replace, would they?
SCLeo 3y ago
If you have a namespace, can't people just globally-claim namespaces instead? like serde/serde or something similar. I feel if you really don't want people claim whatever they want, you have to do the Java package style where namespaces are tied to domain names.
WatchDog 3y ago
Maven and Java really don’t get enough credit for how well it’s dependency management works. So many inferior dependency management systems for other languages have come along later, and learned nothing from those that came before it.
- baby 3y ago
  Is this a joke?
  - theendisney2 3y ago
    I think the correct approach is to do full-real-name_good-package-name it might not be practical but it would be legendary.
  - bsder 3y ago
    Not even remotely a joke. The inverse-style domain name thing does a really good job of removing the whole issue of squatting from the ecosystem. You have to demonstrate some level of commitment and identity through control of a domain name in order to publish. I would also say that this puts just enough friction so that people don't publish dogshit. crates.io demonstrates quite clearly that you either have to go all the way and take responsibility for curation or you have to get completely out of the way. There is no inbetween.
  - speed_spread 3y ago
    Having tried Java and other languages, no, it's not a joke. Other than XML Maven got a lot of things right.
    - chii 3y ago
      and i dont particularly think that using xml is that bad. The schema is well defined, and gives you good autocompletion in any competent IDE (such as intellij). It took some iterations before maven 3 became "good", so people forget that it wasn't as nice before now! Unfortunately, it seems that the lessons learned there is never really disseminated to other ecosystems - perhaps due to prejudice against "enterprisey" java. Yet, these package managers are now facing the sorts of problems solved in java.
baby 3y ago
I love this lack of namespacing personally, because it means that whatever crate you see in a project is going to be the same as the crate your see in another one. Never need to alias crate names. It happens in Golang all the time and I really think namespacing packages was a mistake there.
- lolinder 3y ago
  Golang's problems aren't due to using namespaces, they're due to delaying too many decisions until too late. Go has namespacing mostly because for a long time it didn't have a package manager at all, so people just used a bunch of ad hoc URL-based solutions mostly revolving around GitHub, which happens to have namespaces and also happened to lend itself to aliasing (because a whole GitHub URL is too long). If you want to look at an actual example of namespacing done well, Maven/Java is the place to look. There is no aliasing—the same imports always work across projects.
Grimburger 3y ago
Yet to see any proof that namespacing has made things better in other ecosystems, are go style links or other types of namespaced imports any less prone to supply chain risks? It's definitely a good thing that people choose new unique names for crates rather than dijan/base64 vs dljan/base64 Do understand the desire of having a crate for audio manipulation called "audio" but at the same time how often do we end up with "audio2" anyway? It's an imperfect solution for an imperfect world and I personally think the crates team got it right on this one.
- chungy 3y ago
  > Yet to see any proof that namespacing has made things better in other ecosystems It's really as simple as this: many libraries are generic enough implementing something that already exists. Let's say you want a library to manage the SMTP protocol. On crates.io, of course someone has already taken the "smtp" crate (ironically, this one is abandoned, but has the highest download counts, because it's the most obvious name). Let's say you disagree with the direction this smtp crate has gone, and you make your own. What do you call it? Namespaces solve this problem. You'd instead of have user1/smtp and user2/smtp competing in feature sets. You can even be user3/smtp if you don't like the first two. This is precisely what Java enables too. The standard library is in com.java.*; if you don't like how the standard library does something, you can make com.grimburger.smtp and do it yourself. If you choose to publish to the world, all the more power to you. It doesn't conflict with the standard library's smtp implementation.
corytheboyd 3y ago
I don’t work with rust on the regular, but this is so annoying with package repositories in general. No don’t use http-server, it’s bad, instead you have to use MuffinTop, it’s better. And then you just have to know that. The concept of sanctioned package names would be interesting, but probably chaotic in practice as the underlying code behind this sort of alias changes over time. This will remain a part of being a domain expert in any given ecosystem forever I think, hooray!
- Conscat 3y ago
  I think if you get three to five developers who are enthusiastic about X language, their collective knowledge will select good packages.
- ReactiveJelly 3y ago
  (You probably agree with me but I'm going to just write one big comment instead of replying to every slightly incorrect comment in the thread) Naming things really is one of the hardest problems. This crates thing is a special case of Zooko's Triangle: https://en.wikipedia.org/wiki/Zooko%27s_triangle https://en.wikipedia.org/wiki/Zooko%27s_triangle Crates.io names are human-meaningful and everyone sees the same names, but it's vulnerable to squatting, spamming, and Sybil attacks. You could tie a name to a public key, like onion addresses do, but it's unwieldy for humans. (NB, nothing stops you from doing this inside of crates.io if you really wanted) You could use pet names where "http-server" and "http-client" locally map to "hyper" and "reqwest", but nobody likes those, because they don't solve the bootstrap problem. It's a problem with all repos because when you say "http-server should simply be the best server that everyone likes right now", you have to decide who is the authority of "best", and "everyone", and "now". Don't forget how much useless crap is left in the Python stdlib marked as "don't use this as of 2018, use the 3rd-party lib that does it way better." So yeah... probably will be a problem forever. As a bit of fun here are some un-intuitive names, and my proposed improvements: - Rename Apache to "http-server" - Rename Nginx to "http-server-2" - Rename Caddy to "http-server-2-golang" - Rename libcurl to "http-client" - Rename GTK+ to "linux-gui" - Rename Qt to "linux-gui-2" - Rename xfce4 to "linux-desktop-3" Then you only need to remember which numbers are good and which numbers are bad! Like how IPv4 is the best, IPv6 is okay, but IPV5 isn't real, and HTTP 1.1 and 3 are great but 2 kinda sucked. Very simple. If a company as big as Apple can have simple names like "WebKit", "CoreGraphics", and "CoreAudio" then surely a million hackers competing in a free marketplace can do the same thing.
thayne 3y ago
> Some of these globally-claimed generic packages are not really the best package to use. Or it is just a placeholder from a squatter.
throwawaaarrgh 3y ago
CPAN has the best model IMHO. Hierarchy that starts with categories. You build on top of the base stuff and extend it, rather than reinvent/fork something with a random name. Result is a lot more improvement and reuse, and more functionality with less duplication. Plus it's easy to find stuff. Perl's whole ecosystem is amazing compared to other languages. It's a shame nobody knows it.
lifthrasiir 3y ago
I don't think a lack of namespace is that much problem. Sure, it is often annoying, but people are creative enough to create a short enough and still available crate name for most cases. Namespacing only makes sense for a large group of related crates---and it wouldn't give much benefit over a flat namespace. As other mentioned though, a typosquatting is a much bigger problem and namespacing offers only a partial solution. (You can still create a lookalike organization name, both in npm and in Github.)
- rat87 3y ago
  Id argue typesquatting a namespace is arguably easier then type squatting a package name since you will often ignore the namespace name
rat87 3y ago
I don't see the problem. Even with namespaces you'll have brandonq/xml vs parsers/xml with no clue if one is better then another. Also possibly with some confusion over whether things with the same name are forks or not. May make it a little more difficult to Google. Why not just have brandon_xml vs xml-parser and have a community list of best and most popular libraries? I guess the only issue is that some generic/obvious package names are bad packages. That can have been avoided if they banned/self-squated most of the names. I suppose if you use dns namespaces and actually tie it to ownership of the domain name it might make sense but that would also cause issues(what if you forgot to renew the domain?).
- pdimitar 3y ago
  > brandonq/xml vs parsers/xml with no clue if one is better then another. You will have this problem only once. After an initial research you settle on one and move on with life. I don't get how having to vet a dependency is going to be more difficult than before. The process is 99% the same, you still have to do the research work initially in both cases.
- xtremegoose 3y ago
  The advantage is one of trust. If the `abc` developers build well known library `abc.pqr` are well trusted then I know I can use `abc.xyz` and everything else under the same namespace without (much) vetting. We could even have `rust.xyz` for crates that are decoupled from `std` but still maintained by rust core devs such as `regex`.
pjmlp 3y ago
While Java made that notation famous, it was already used in NeXTSTEP and Objective-C, hence why you will see this all over the place in macOS and derived platforms, on configuration files and services.
satvikpendem 3y ago
I like that there are no namespaces, it forces people to come up with unique names rather than naming their calendar crate to be `calendar`.
littlestymaar 3y ago
> Perhaps my biggest critique is that crates.io has no namespacing. Anyone can just claim a global and generic package name and we mostly have to deal with it (unless you avoid using the crates.io repository, but then you'll probably have more problems...). Some of these globally-claimed generic packages are not really the best package to use. This is true that with no namespace anyone can end up squatting a cool name, but with namespace you end up in an even worse place: no-one end up with the cool names, and it makes discoverability miserable, because instead of having people come up with unique names like serde, everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json" (and remember not to use "json/json" because indeed namespace squatting is now a thing), and of course this makes it completely ungoogleable. We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse. Having a centralized public and permissionless repository opens tons of tricky questions, but namespaces are a solution to none of them.
- pdimitar 3y ago
  > everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json" What's the problem with that? You will have to explicitly state intention which is always a good thing. > and of course this makes it completely ungoogleable You are aware that just pasting username/projectname gives you exactly what you are looking for in the several top search engines, correct? > We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse. OK, if you say so. Is this worse state of things documented somewhere?
- yencabulator 3y ago
  > and of course this makes it completely ungoogleable Do you really think nobody has ever been able to google a Go import path? Some of these arguments are ridiculous.

by chungy 3y ago

rochak 3y ago

Go made a great decision to namespace packages via this Github style.

treyd 3y ago
And the horrible decision to not make library-level ("module") and code-unit-level ("package") namespacing orthogonal. The former was an afterthought tacked on since the package system was designed to be used only within Google's monorepo and little care was paid to how it would work when it was released to the public and used more generally.
- starttoaster 3y ago
  I want to understand what you just said, but I fear watering your language down a bit might be a tall ask with some people. Would you be willing to eli5 what you believe Go did that was a horrible decision with regards to module/package namespacing?
  - cdelsolar 3y ago
    I think he wants them to be treated differently for some reason.
  - ezrast 3y ago
    I think what they mean is: if you see a line like `import git.example.com/foo/bar/baz`, that could be package `baz` inside module `git.example.com/foo/bar`, or it could be package `bar/baz` inside module `git.example.com/foo`. Also, even if you know it's the latter, package namespacing isn't strictly related to directory structure, so `bar/baz` has no specific meaning outside of the context of a go import. They could have used any other separator for package components - `git.example.com/foo:bar:baz` - but instead they chose the slash, making the scheme both technically ambiguous and easy to confuse for an HTTP URL.
    - starttoaster 3y ago
      Ah that makes sense. I think Go did somewhat stumble a bit in the early days due to this, especially with repositories in GitLab, where GitLab allows essentially a directory tree where your repository can be nested indefinitely in directories like `https://gitlab.com/mygroup/subgroup1/subgroup2/repository https://gitlab.com/mygroup/subgroup1/subgroup2/repository`. I still don't think this is a huge issue, to be honest. Not one big enough for me to complain about, for sure. But it's definitely not ideal.
baby 3y ago
How is this a good decision? You have to alias packages when names collide
- tapirl 3y ago
  What is the better decision to avoid name colliding?
pjmlp 3y ago
Go packages is yet another design item that I dislike on the language, exposing SCM URLs directly on the source code, and no story for binary caching.
- yencabulator 3y ago
  > exposing SCM URLs directly on the source code Incorrect. You're able to use any URL you control, regardless of where your SCM is located.
  - pjmlp 3y ago
    Thanks for pointing out you didn't got what the whole point of the wrong design is about.
    - yencabulator 3y ago
      Thanks for confirming you'd rather be a hater than someone with justified opinions.
      - pjmlp 3y ago
        I am quite open about my opinion regarding most of Go's design decisions.

jrockway 3y ago

URLs for packages makes a lot of sense. It works well in the land of Go. It also conveniently eliminates the need for the language to have a global packages database. Upload your package to example.com/your-thing and it's released! (You can, of course, still offer a cache and search engine if you want to.)

alpaca128 3y ago
Cargo does support URLs to git repos for dependencies. But crates.io is the official platform and almost every search I do on it returns at least one generically named entry with an empty repository that someone snatched away and never used.
swatcoder 3y ago
No, URL's don't make sense because your application shouldn't care where on the internet your dependency happened to be hosted when you integrated it. It's location has nothing to do with what it is. By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project. Your project shouldn't need to care whether GitHub fell out of fashion and the project moved to GitLab, and definitely shouldn't be relying on GitHub being available when you need to build, test, deploy, or scale. That's a completely unnecessary failure point for you to introduce. Systems that use URL-identified packages can work around some of this, but just reinforce terrible habits.
- jrockway 3y ago
  It worked for the rest of the Internet.
  - ethbr1 3y ago
    There is something to be said for separating {unique piece of content} and {hosted location}. E.g. doi's https://en.m.wikipedia.org/wiki/Digital_object_identifier https://en.m.wikipedia.org/wiki/Digital_object_identifier

dhosek 3y ago

Agreed. The other thing I don’t really like is that you can’t split up things in the rust namespace hierarchy between crates (something that’s natural with jars in JVM). I would have liked to have defined things so that I could have the unicode handlers for finl live in finl::unicode, the parser in finl::parser, etc. but because they’re in separate crates rust gets upset about finl being defined twice and there’s no workaround for it. There are likely pitfalls I don’t see in what I want, so I live with it.

ramranch 3y ago
Would re-exporting partially solve your problem?
- dhosek 3y ago
  As I recall, I could do something where I could have a common root crate that would import and re-export the other crates with the namespaces modified on the exports and then control which crates are exported through feature gates, but it just seemed more hassle than it was worth.
epage 3y ago
We have an RFC for this that has fairly broad support that I just need to bug some people some more so we can approve it. https://github.com/rust-lang/rfcs/pull/3243 https://github.com/rust-lang/rfcs/pull/3243

lolinder 3y ago

> Maybe it was a reaction against the Java-style reverse DNS notation I suspect it was less a reaction against anything and more just following the norms established by most other package managers. NPM, PyPI, RubyGems, Elixir's Hex, Haskell's Cabal... I'm having a hard time thinking of a non-Java package manager that was around at the time Rust came out that didn't have a single, global namespace. Some have tried to fix that since then, but it was just the way package managers worked in 2014/2015.

jiggawatts 3y ago
Yes, and all of those have had major security issues caused by their lack of foresight. "We're pretending security is not an issue." has been the feedback every time this is raised with the Cargo team. To be honest, it's turned me off Rust a little bit. The attitude of "Rust is memory-safe, so we don't need any other form of security." is not a good one.
- muldvarp 3y ago
  > "We're pretending security is not an issue." has been the feedback every time this is raised with the Cargo team. Do you have a specific link where I can read this response, because this is not at all the responses I have read.
  - jiggawatts 3y ago
    Just some random Cargo security-related issues I noticed: - No strong link between the repo and the published code. - Many crates were spammed that were just a wrapper around a popular C/C++ library. There's no indication of this, so... "surprise!"... your compiled app is now more unsafe C/C++ code than Rust. - Extensive name squatting, to the point that virtual no library uses the obvious name, because someone else got to it first. The aformentioned C/C++ libraries were easy to spit out, so they often grabbed the name before a Rust rewrite could be completed and published. So you now go to Cargo to find a Rust library for 'X' and you instead have to use 'X-rs' because... ha-ha, it's actually a C/C++ package manager with some Rust crates in there also. - Transitive dependencies aren't shown in the web page. - No enforcement or indication of safe/unsafe libs, nostd, etc... - No requirement for MFA, which was a successful attack vector on multiple package managers in the past. DISCLAIMER: Some of the above may have been resolved since I last looked. Other package managers also do these things (but that's not a good thing). In my opinion, any package manager that just lets any random person upload "whatever" is outright dangerous and useless to the larger ecosystem of developers in a hurry who don't have the time to vet every single transitive dependency every month. Package managers need to grow up and start requiring a direct reference to a specific Git commit -- that they store themselves -- and compile from scratch with an instrumented compiler that spits out metadata such as "connects to the Internet, did you know?" or "is actually 99% C++ code, by the way".

yarg 3y ago

This needs to be resolved by every damned language. Just make signed dependencies a universal default, point to an https page for the package vendor and use the signing key from there. Neither node nor maven ever bothered to solve this, so we end up wandering the Wild West wondering when it will be that HR, or legal, or architecture comes knocking on the door to ask what we were thinking having a dependency on a dynamic version of Left Pad. I'd kinda like to see what Cloudflare and Let's Encrypt could come up with if they worked together on at very least a white paper and an MVP POC.

lesuorac 3y ago
I don't think that's the real solution. Pay somebody either internally or externally to maintain a repo of all your dependencies and point your code at that. You won't get a left-pad incident. You won't get a malicious .so incident (unless you mirror binaries instead of source code). Like if you ran out of screws to make your product with do you walk around the street and scrounge up some? No, you go to a trusted vendor and buy the screws.
- yarg 3y ago
  I'm not suggesting scrounging. I'm suggesting that the guys who've repeatedly proven themselves technically competent around security at scale might have a couple of useful ideas regarding how the industry might go about crawling its way out of this little security clusterfuck. And perhaps even stop treating something as simple as a BOM as an enterprise feature, given that the overhead on such things is damned near zilch and the security implications are staggering. https://www.cisa.gov/sbom https://www.cisa.gov/sbom
- cmrdporcupine 3y ago
  There's reasons why Google projects don't go out on the internet to get their 3rd party deps. They're all checked into Google3 (or chromium, etc.). One version only. With internal maintainers responsible for bringing it in and multiple people vetting it, and clear ownership over its management. E.g. you don't just get to willy nilly depend on a new version -- if you want to upgrade what's there, you gotta put a ring on it. If you upgrade it, you're likely going to be upgrading it for everyone, and the build system will run through the dependent tests for them all, etc. And the consequence is more responsible use of third party deps and less sprawling dependency trees and less complexity. And additional less security concerns as the code is checked in, its license vetted, and build systems are hunting around on the Internet for artifacts.

cetra3 3y ago

I think most people are in support of namespacing but it's a big change and will take a while to see through.

Jalad 3y ago

This is a common critique, and although I don't have insight into why the original decision to not have namespaces was made, the current outlook is that until issues related to continuity are resolved, it's a no go: https://samsieber.tech/posts/2020/09/registry-structure-influence/ https://samsieber.tech/posts/2020/09/registry-structure-infl...

sigwinch28 3y ago
That article starts with the premise that “it’s a feature, not a bug” then goes on to describe a whole bunch of things I consider to be anti-features of a packaging system that has a flat namespace. The first section says it discourages forking. I consider this to be bad. Nobody’s code should be more important purely because it squatted a better name. The Identity section actually makes the case that flat registries make naming harder. The section on Continuity is “we’ve tried nothing and we’re all out of ideas”. Make up an org name and grandfather all packages in the flat namespace into that special org. Also this is already a problem because packages in the flat namespace do get abandoned, then forked, and then we have the associated issues. The section on Stability seems to take it as a given that crates.io should be the only registry. I don’t. It also seems to conflate cargo with rustc for the benefit of the argument. The squatting section describes only anti-features and I don’t consider the author’s legitimate use cases to be legitimate reasons to squat. I think the only legitimate problems that need addressing are the ergonomics of accessing namespaced packages throughout transient dependencies and backwards compatibility with non-namespaced code. But the fact that these are real problems does not, to me, make a flat namespace a “feature”. It’s just easier to implement. It’s okay for it to be a mistake that takes effort and time to fix.
- kevincox 3y ago
  Another option would be to grandfather all packages into their own org. So serde becomes serde/serde. This way you don't need to manage permission rules in the legacy "all" namespace. You get some oddities such as serde-derive/serde-derive but the package owners can choose if they want to move to serde/derive or leave it in a separate namespace.

fawadasaurus 3y ago

I did some analysis on crates.io to find the top name squatters. Then I did some calculations and found that the top name squatter created their crates at a rate of about one ever 30 seconds for a period of a week straight. I send the analysis to the crates.io team and pointed that they have a no-automation policy. They told me that it was not sufficient proof that someone was squatting those names. That's my problem with crates.io is that they have a clear policy and they don't enforce it so all the short/easy to remember names for crates are already taken and there is nothing you can do to get it.

echelon 3y ago
There's a secret effort in the Rust community to supplant Crates.io and create an entirely new package ecosystem with proper namespacing, security, and much better community. Not naming names, but I know several people working to put Crates.io out to pasture. There's a level of playing nice with them for the time being (eg. build reproducibility), but it's only KTLO. Crates.io needs to die for Rust to thrive. They're a bungled, mismanaged liability. New code, new leadership.
- solardev 3y ago
  Why does something like that need to be secret...? Isn't it in the community's best interest?
  - TylerE 3y ago
    Drama avoidance and avoiding bikeshedding seem obvious. Much easier to present a working system than a design that will get nitpicked into irrelevancy.
    - solardev 3y ago
      Yeah, that makes sense.
    - sapling-ginger 3y ago
      That quite funny. Just like when some people formed a violent militant group to take down a violent tyranical dictatorship. Of course they would promise that they would absolutely de-arm right after the dictatorship has been overthrown, and immediately establish a peaceful democratic government with fair election. They absolutely would, would they? They would never turn into what they were formed to replace, would they?

SCLeo 3y ago

If you have a namespace, can't people just globally-claim namespaces instead? like serde/serde or something similar. I feel if you really don't want people claim whatever they want, you have to do the Java package style where namespaces are tied to domain names.

WatchDog 3y ago

Maven and Java really don’t get enough credit for how well it’s dependency management works. So many inferior dependency management systems for other languages have come along later, and learned nothing from those that came before it.

baby 3y ago
Is this a joke?
- theendisney2 3y ago
  I think the correct approach is to do full-real-name_good-package-name it might not be practical but it would be legendary.
- bsder 3y ago
  Not even remotely a joke. The inverse-style domain name thing does a really good job of removing the whole issue of squatting from the ecosystem. You have to demonstrate some level of commitment and identity through control of a domain name in order to publish. I would also say that this puts just enough friction so that people don't publish dogshit. crates.io demonstrates quite clearly that you either have to go all the way and take responsibility for curation or you have to get completely out of the way. There is no inbetween.
- speed_spread 3y ago
  Having tried Java and other languages, no, it's not a joke. Other than XML Maven got a lot of things right.
  - chii 3y ago
    and i dont particularly think that using xml is that bad. The schema is well defined, and gives you good autocompletion in any competent IDE (such as intellij). It took some iterations before maven 3 became "good", so people forget that it wasn't as nice before now! Unfortunately, it seems that the lessons learned there is never really disseminated to other ecosystems - perhaps due to prejudice against "enterprisey" java. Yet, these package managers are now facing the sorts of problems solved in java.

baby 3y ago

I love this lack of namespacing personally, because it means that whatever crate you see in a project is going to be the same as the crate your see in another one. Never need to alias crate names. It happens in Golang all the time and I really think namespacing packages was a mistake there.

lolinder 3y ago
Golang's problems aren't due to using namespaces, they're due to delaying too many decisions until too late. Go has namespacing mostly because for a long time it didn't have a package manager at all, so people just used a bunch of ad hoc URL-based solutions mostly revolving around GitHub, which happens to have namespaces and also happened to lend itself to aliasing (because a whole GitHub URL is too long). If you want to look at an actual example of namespacing done well, Maven/Java is the place to look. There is no aliasing—the same imports always work across projects.

Grimburger 3y ago

Yet to see any proof that namespacing has made things better in other ecosystems, are go style links or other types of namespaced imports any less prone to supply chain risks? It's definitely a good thing that people choose new unique names for crates rather than dijan/base64 vs dljan/base64 Do understand the desire of having a crate for audio manipulation called "audio" but at the same time how often do we end up with "audio2" anyway? It's an imperfect solution for an imperfect world and I personally think the crates team got it right on this one.

chungy 3y ago
> Yet to see any proof that namespacing has made things better in other ecosystems It's really as simple as this: many libraries are generic enough implementing something that already exists. Let's say you want a library to manage the SMTP protocol. On crates.io, of course someone has already taken the "smtp" crate (ironically, this one is abandoned, but has the highest download counts, because it's the most obvious name). Let's say you disagree with the direction this smtp crate has gone, and you make your own. What do you call it? Namespaces solve this problem. You'd instead of have user1/smtp and user2/smtp competing in feature sets. You can even be user3/smtp if you don't like the first two. This is precisely what Java enables too. The standard library is in com.java.*; if you don't like how the standard library does something, you can make com.grimburger.smtp and do it yourself. If you choose to publish to the world, all the more power to you. It doesn't conflict with the standard library's smtp implementation.

corytheboyd 3y ago

I don’t work with rust on the regular, but this is so annoying with package repositories in general. No don’t use http-server, it’s bad, instead you have to use MuffinTop, it’s better. And then you just have to know that. The concept of sanctioned package names would be interesting, but probably chaotic in practice as the underlying code behind this sort of alias changes over time. This will remain a part of being a domain expert in any given ecosystem forever I think, hooray!

Conscat 3y ago
I think if you get three to five developers who are enthusiastic about X language, their collective knowledge will select good packages.
ReactiveJelly 3y ago
(You probably agree with me but I'm going to just write one big comment instead of replying to every slightly incorrect comment in the thread) Naming things really is one of the hardest problems. This crates thing is a special case of Zooko's Triangle: https://en.wikipedia.org/wiki/Zooko%27s_triangle https://en.wikipedia.org/wiki/Zooko%27s_triangle Crates.io names are human-meaningful and everyone sees the same names, but it's vulnerable to squatting, spamming, and Sybil attacks. You could tie a name to a public key, like onion addresses do, but it's unwieldy for humans. (NB, nothing stops you from doing this inside of crates.io if you really wanted) You could use pet names where "http-server" and "http-client" locally map to "hyper" and "reqwest", but nobody likes those, because they don't solve the bootstrap problem. It's a problem with all repos because when you say "http-server should simply be the best server that everyone likes right now", you have to decide who is the authority of "best", and "everyone", and "now". Don't forget how much useless crap is left in the Python stdlib marked as "don't use this as of 2018, use the 3rd-party lib that does it way better." So yeah... probably will be a problem forever. As a bit of fun here are some un-intuitive names, and my proposed improvements: - Rename Apache to "http-server" - Rename Nginx to "http-server-2" - Rename Caddy to "http-server-2-golang" - Rename libcurl to "http-client" - Rename GTK+ to "linux-gui" - Rename Qt to "linux-gui-2" - Rename xfce4 to "linux-desktop-3" Then you only need to remember which numbers are good and which numbers are bad! Like how IPv4 is the best, IPv6 is okay, but IPV5 isn't real, and HTTP 1.1 and 3 are great but 2 kinda sucked. Very simple. If a company as big as Apple can have simple names like "WebKit", "CoreGraphics", and "CoreAudio" then surely a million hackers competing in a free marketplace can do the same thing.

thayne 3y ago

> Some of these globally-claimed generic packages are not really the best package to use. Or it is just a placeholder from a squatter.

throwawaaarrgh 3y ago

CPAN has the best model IMHO. Hierarchy that starts with categories. You build on top of the base stuff and extend it, rather than reinvent/fork something with a random name. Result is a lot more improvement and reuse, and more functionality with less duplication. Plus it's easy to find stuff. Perl's whole ecosystem is amazing compared to other languages. It's a shame nobody knows it.

lifthrasiir 3y ago

I don't think a lack of namespace is that much problem. Sure, it is often annoying, but people are creative enough to create a short enough and still available crate name for most cases. Namespacing only makes sense for a large group of related crates---and it wouldn't give much benefit over a flat namespace. As other mentioned though, a typosquatting is a much bigger problem and namespacing offers only a partial solution. (You can still create a lookalike organization name, both in npm and in Github.)

rat87 3y ago
Id argue typesquatting a namespace is arguably easier then type squatting a package name since you will often ignore the namespace name

rat87 3y ago

I don't see the problem. Even with namespaces you'll have brandonq/xml vs parsers/xml with no clue if one is better then another. Also possibly with some confusion over whether things with the same name are forks or not. May make it a little more difficult to Google. Why not just have brandon_xml vs xml-parser and have a community list of best and most popular libraries? I guess the only issue is that some generic/obvious package names are bad packages. That can have been avoided if they banned/self-squated most of the names. I suppose if you use dns namespaces and actually tie it to ownership of the domain name it might make sense but that would also cause issues(what if you forgot to renew the domain?).

pdimitar 3y ago
> brandonq/xml vs parsers/xml with no clue if one is better then another. You will have this problem only once. After an initial research you settle on one and move on with life. I don't get how having to vet a dependency is going to be more difficult than before. The process is 99% the same, you still have to do the research work initially in both cases.
xtremegoose 3y ago
The advantage is one of trust. If the `abc` developers build well known library `abc.pqr` are well trusted then I know I can use `abc.xyz` and everything else under the same namespace without (much) vetting. We could even have `rust.xyz` for crates that are decoupled from `std` but still maintained by rust core devs such as `regex`.

pjmlp 3y ago

While Java made that notation famous, it was already used in NeXTSTEP and Objective-C, hence why you will see this all over the place in macOS and derived platforms, on configuration files and services.

satvikpendem 3y ago

I like that there are no namespaces, it forces people to come up with unique names rather than naming their calendar crate to be `calendar`.

littlestymaar 3y ago

> Perhaps my biggest critique is that crates.io has no namespacing. Anyone can just claim a global and generic package name and we mostly have to deal with it (unless you avoid using the crates.io repository, but then you'll probably have more problems...). Some of these globally-claimed generic packages are not really the best package to use. This is true that with no namespace anyone can end up squatting a cool name, but with namespace you end up in an even worse place: no-one end up with the cool names, and it makes discoverability miserable, because instead of having people come up with unique names like serde, everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json" (and remember not to use "json/json" because indeed namespace squatting is now a thing), and of course this makes it completely ungoogleable. We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse. Having a centralized public and permissionless repository opens tons of tricky questions, but namespaces are a solution to none of them.

pdimitar 3y ago
> everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json" What's the problem with that? You will have to explicitly state intention which is always a good thing. > and of course this makes it completely ungoogleable You are aware that just pasting username/projectname gives you exactly what you are looking for in the several top search engines, correct? > We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse. OK, if you say so. Is this worse state of things documented somewhere?
yencabulator 3y ago
> and of course this makes it completely ungoogleable Do you really think nobody has ever been able to google a Go import path? Some of these arguments are ridiculous.