6 ms·

Something that I'm surprised this page doesn't talk about, and which is very important considering the recent hubbub over left-pad, is that any dependency you g

by eridius 11y ago

Something that I'm surprised this page doesn't talk about, and which is very important considering the recent hubbub over left-pad, is that any dependency you get on Cargo can be relied upon to continue to exist forever (well, as long as the crates.io site still exists, but if that goes away so does the Cargo index). The reason is because you can't ever remove a published version of your crate from crates.io. You can yank a version, which tells Cargo not to allow any projects to form new dependencies on that version, but the version isn't actually deleted and any projects that have existing dependencies on that version will continue to be allowed to use it. This is documented at http://doc.crates.io/crates-io.html#cargo-yank http://doc.crates.io/crates-io.html#cargo-yank.

waffle_ss 11y ago
> You can yank a version, which tells Cargo not to allow any projects to form new dependencies on that version, but the version isn't actually deleted That's how RubyGems used to work, except you could contact the support team and ask them to permanently delete your gem version if something really sensitive and irrevocable was put into it. They had to change that due to their support log getting too big: http://blog.rubygems.org/2015/04/13/permadelete-on-yank.html http://blog.rubygems.org/2015/04/13/permadelete-on-yank.html
- kibwen 11y ago
  And of course, a centralized site can still be compelled to remove a package via legal action if push comes to shove.
- eridius 11y ago
  Cargo's policy is that if you upload secrets, you need to change the secrets because the code can't be deleted. From the aforelinked page: > A yank does not delete any code. This feature is not intended for deleting accidentally uploaded secrets, for example. If that happens, you must reset those secrets immediately.
  - waffle_ss 11y ago
    Even if they're given a court order to take a version down? It may be their policy, but I guarantee it will happen at some point.
    - steveklabnik 11y ago
      We will comply with the law where required.
    - deleted 11y ago
      [deleted]
    - dkersten 11y ago
      Court orders will be a lot less frequent than normal user requests so the overhead to the team won't be as high.
      - waffle_ss 11y ago
        Of course, and I totally agree with their approach. I'm just saying the narrative eridius is pushing that somehow they can make assurances about things never being deleted totally false. For example, I'm sure if someone somehow put child pornography in a Rust crate it would rightly be taken down pretty fast (and not require a court order). Also I just wanted to give a little history because eridius's original comment made it sound like it was a novel concept.
  - cfallin 11y ago
    This seems a bit... inflexible. I definitely understand the arguments for not breaking builds and for reducing administrative overhead and such, but not every bit of secret data can be revoked like a key/credential can. What if you accidentally include user data, or proprietary business-logic code, or...? (Yes, with proper data hygiene and processes you'd never even come close to doing any of that, but it seems there should still be an escape hatch.)
fzzzy 11y ago
It does mention it, briefly: > This is enough information to uniquely identify source code from crates.io, because the registry is append only (no changes to already-published packages are allowed).
kibwen 11y ago
> if that goes away so does the Cargo index Not so, the location of the index is independent of crates.io. Currently the index itself is just hosted on Github, whereas all of crates.io is hosted on S3. Which is actually kind of a pain sometimes, since if Github goes down it means that Cargo won't be able to find the index and I don't know if there's an easy way to override the index check. In the future I expect Cargo will gracefully continue if the index can't be updated due to connection failure, though I'd think it would prompt the user to make sure they're aware that their local copy of the index might be out of date.
- eridius 11y ago
  Ah, good to know. I assumed the index was hosted as part of crates.io but I didn't actually bother to check. The point I was trying to make (even if I didn't communicate it properly) was that if crates.io goes away for good, the index will too and so it won't really matter that the code is inaccessible because cargo won't know where to find it anyway. Of course, based on what you said, it's certainly possible for everyone involved in crates.io to get hit by a bus and crates.io vanish when the S3 bill goes unpaid while still leaving the index up on GitHub, but if crates.io is taken down intentionally then presumably so will the index.
incepted 11y ago
> Something that I'm surprised this page doesn't talk about, and which is very important considering the recent hubbub over left-pad, is that any dependency you get on Cargo can be relied upon to continue to exist forever Maybe the reason why it's hardly talked about is because it's common sense and pretty much all dependency managers support it? Except node of course, because they have no idea what they're doing.
- tatterdemalion 11y ago
  This is incorrect and very rude. Rubygems has also implemented an automated delete feature. Rubygems and NPM have orders of magnitude more users than crates.io; once crates.io approaches that scale the team will have to allow users to delete crates or somehow find funding to field these kinds of support tickets.
  - riffraff 11y ago
    the parent comment might have been worded better, but it has a point, i.e. removing published code generally has to be supported in some way to handle special situations, e.g. * people publish stuff inadvertently (i.e. private information/keys) * people publish stuff they are not allowed to (i.e. copyright and trademark violations) * people publish stuff you do not want to see published (i.e. stuff related to breaking legal or ethical laws) It can be argued that users must not do that or that cargo.io doesn't have to oblige, but if they, for example, get a DMCA notice they'll still have to.
    - gnoway 11y ago
      I think the key is that crates.io (and rubygems.org, nuget.org, etc.) as repo owners have to own the removal operation themselves and not delegate that to the package owner/maintainer. The repo owner is better positioned to take package consumers' needs into account and make good decisions about when and how to remove a package. As far as satisfying support requests: obviously exceptional stuff like DMCA gets handled, but package owners publishing keys, etc. is still their responsibility and shouldn't accelerate or even guarantee removal IMO. If you publish secrets, it's not crates.io's responsibility to help you hide that mistake, and you need to be changing those secrets anyway.
weinzierl 11y ago
> well, as long as the crates.io site still exists, but if that goes away so does the Cargo index That makes me wonder: Is it easy or possible to replace crates.io with a self hosted repository? Background of the question is that I know of a company where access to the standard public maven repo is forbidden. They use a commercial repository provider but I don't know if it is hosted on premises.
- steveklabnik 11y ago
  It's not easy, but it's possible. Everything is open source, and Cargo can easily be pointed at whatever host you want. The feature that I'd like to see but haven't found the time to implement is delegation: "look at this index first, but if it's not here, go look at this other one". Right now, if you spin up your own crates.io and point Cargo at it, it won't have any packages... which works for some people, but not others.
  - pyre 11y ago
    What would be useful is a sort of "caching proxy" that could have various knobs to handle situations like: - crates.io is down - crates.io says this cached package doesn't exist. - etc...
    - ecnahc515 11y ago
      This is already possible today with existing caching proxies. This is a great way to make your CI/builds more reliable and quicker.
      - pyre 11y ago
        I meant a caching proxy that functioned as a mini Crates.io in the absence of actual crates.io being up. Depending on the crates.io protocol, just caching HTTP requests might not be enough, but also acting as a (offline-able) middle man that knows the protocol gives rise to other knobs and such (e.g. a configurable blacklist of packages).
  - 11y ago
jalami 11y ago
I know it wouldn't really fix issues like left-pad, but I would really like namespaced packages similar to the way Github does them. I think group ownership would be more explicit and understood. Top level packages encourages sqatting and small/old/unmaintained packages getting names that people would misunderstand for something else. I understand a lot of that is on the user of the library to research before downloading, but intuitiveness is a virtue. It's true packages could just be called eg. reactjs-node-bridge or something as opposed to reactjs/node-bridge, but anyone can prefix deceptively. Cargo addresses the namespacing concern here http://internals.rust-lang.org/t/crates-io-package-policies/1041 http://internals.rust-lang.org/t/crates-io-package-policies/...
viraptor 11y ago
But that's really a crates.io feature, not cargo's. You could easily deploy your own repository which serves everyone a random file when you pull "some_crate-1.2.3".
airless_bar 11y ago
Let's just see what happens when they get their first court order, which they certainly will (due to their global name-spacing).
- steveklabnik 11y ago
  As I said below, we will comply with the law. (And I'm not sure what that has to do with namespacing.)
  - eyko 11y ago
    I believe he means that packages aren't namespaced (owner/package-name), so if I publish a package called nike, Nike could come and want to take over the package name.
    - kibwen 11y ago
      Global namespacing has nothing to do with that. Even with per-user namespacing, there's nothing to stop a user from making their username "nike" and publishing all their code under "nike/package-name", which would be subject to just as much potential legal action as a package named "nike" itself. Likewise there's also nothing to stop Bob from uploading "bob/nike".
      - airless_bar 11y ago
        If course global name-spacing has something to do with it! What's the difference between a "username" resulting in nike/package-name and a package called nike-package-name? Not much in practice. Name-spacing doesn't mean "add another made up name entry somewhere". That's why people who thought about these issues out-sourced this concern completely: Prove your ownership of nike.com and you can publish as nike.com. Suddenly 99.9% of the trademark issues are gone, handled by registrars. I think this approach is pretty obvious and I find the lack of thought coming from rust devs deeply concerning.
        11y ago
themartorana 11y ago
I think this is the scenario that has me feeling warm and fuzzy about vendoring. Sure, eventually even vendored dependencies will become stale, but I don't have to rely on a package manager past the initial delivery. It's not perfect, but I don't sweat code disappearing from a black box I don't own.

by eridius 11y ago

waffle_ss 11y ago

> You can yank a version, which tells Cargo not to allow any projects to form new dependencies on that version, but the version isn't actually deleted That's how RubyGems used to work, except you could contact the support team and ask them to permanently delete your gem version if something really sensitive and irrevocable was put into it. They had to change that due to their support log getting too big: http://blog.rubygems.org/2015/04/13/permadelete-on-yank.html http://blog.rubygems.org/2015/04/13/permadelete-on-yank.html

kibwen 11y ago
And of course, a centralized site can still be compelled to remove a package via legal action if push comes to shove.
eridius 11y ago
Cargo's policy is that if you upload secrets, you need to change the secrets because the code can't be deleted. From the aforelinked page: > A yank does not delete any code. This feature is not intended for deleting accidentally uploaded secrets, for example. If that happens, you must reset those secrets immediately.
- waffle_ss 11y ago
  Even if they're given a court order to take a version down? It may be their policy, but I guarantee it will happen at some point.
  - steveklabnik 11y ago
    We will comply with the law where required.
  - deleted 11y ago
    [deleted]
  - dkersten 11y ago
    Court orders will be a lot less frequent than normal user requests so the overhead to the team won't be as high.
    - waffle_ss 11y ago
      Of course, and I totally agree with their approach. I'm just saying the narrative eridius is pushing that somehow they can make assurances about things never being deleted totally false. For example, I'm sure if someone somehow put child pornography in a Rust crate it would rightly be taken down pretty fast (and not require a court order). Also I just wanted to give a little history because eridius's original comment made it sound like it was a novel concept.
- cfallin 11y ago
  This seems a bit... inflexible. I definitely understand the arguments for not breaking builds and for reducing administrative overhead and such, but not every bit of secret data can be revoked like a key/credential can. What if you accidentally include user data, or proprietary business-logic code, or...? (Yes, with proper data hygiene and processes you'd never even come close to doing any of that, but it seems there should still be an escape hatch.)

fzzzy 11y ago

It does mention it, briefly: > This is enough information to uniquely identify source code from crates.io, because the registry is append only (no changes to already-published packages are allowed).

kibwen 11y ago

> if that goes away so does the Cargo index Not so, the location of the index is independent of crates.io. Currently the index itself is just hosted on Github, whereas all of crates.io is hosted on S3. Which is actually kind of a pain sometimes, since if Github goes down it means that Cargo won't be able to find the index and I don't know if there's an easy way to override the index check. In the future I expect Cargo will gracefully continue if the index can't be updated due to connection failure, though I'd think it would prompt the user to make sure they're aware that their local copy of the index might be out of date.

eridius 11y ago
Ah, good to know. I assumed the index was hosted as part of crates.io but I didn't actually bother to check. The point I was trying to make (even if I didn't communicate it properly) was that if crates.io goes away for good, the index will too and so it won't really matter that the code is inaccessible because cargo won't know where to find it anyway. Of course, based on what you said, it's certainly possible for everyone involved in crates.io to get hit by a bus and crates.io vanish when the S3 bill goes unpaid while still leaving the index up on GitHub, but if crates.io is taken down intentionally then presumably so will the index.

incepted 11y ago

> Something that I'm surprised this page doesn't talk about, and which is very important considering the recent hubbub over left-pad, is that any dependency you get on Cargo can be relied upon to continue to exist forever Maybe the reason why it's hardly talked about is because it's common sense and pretty much all dependency managers support it? Except node of course, because they have no idea what they're doing.

tatterdemalion 11y ago
This is incorrect and very rude. Rubygems has also implemented an automated delete feature. Rubygems and NPM have orders of magnitude more users than crates.io; once crates.io approaches that scale the team will have to allow users to delete crates or somehow find funding to field these kinds of support tickets.
- riffraff 11y ago
  the parent comment might have been worded better, but it has a point, i.e. removing published code generally has to be supported in some way to handle special situations, e.g. * people publish stuff inadvertently (i.e. private information/keys) * people publish stuff they are not allowed to (i.e. copyright and trademark violations) * people publish stuff you do not want to see published (i.e. stuff related to breaking legal or ethical laws) It can be argued that users must not do that or that cargo.io doesn't have to oblige, but if they, for example, get a DMCA notice they'll still have to.
  - gnoway 11y ago
    I think the key is that crates.io (and rubygems.org, nuget.org, etc.) as repo owners have to own the removal operation themselves and not delegate that to the package owner/maintainer. The repo owner is better positioned to take package consumers' needs into account and make good decisions about when and how to remove a package. As far as satisfying support requests: obviously exceptional stuff like DMCA gets handled, but package owners publishing keys, etc. is still their responsibility and shouldn't accelerate or even guarantee removal IMO. If you publish secrets, it's not crates.io's responsibility to help you hide that mistake, and you need to be changing those secrets anyway.

weinzierl 11y ago

> well, as long as the crates.io site still exists, but if that goes away so does the Cargo index That makes me wonder: Is it easy or possible to replace crates.io with a self hosted repository? Background of the question is that I know of a company where access to the standard public maven repo is forbidden. They use a commercial repository provider but I don't know if it is hosted on premises.

steveklabnik 11y ago
It's not easy, but it's possible. Everything is open source, and Cargo can easily be pointed at whatever host you want. The feature that I'd like to see but haven't found the time to implement is delegation: "look at this index first, but if it's not here, go look at this other one". Right now, if you spin up your own crates.io and point Cargo at it, it won't have any packages... which works for some people, but not others.
- pyre 11y ago
  What would be useful is a sort of "caching proxy" that could have various knobs to handle situations like: - crates.io is down - crates.io says this cached package doesn't exist. - etc...
  - ecnahc515 11y ago
    This is already possible today with existing caching proxies. This is a great way to make your CI/builds more reliable and quicker.
    - pyre 11y ago
      I meant a caching proxy that functioned as a mini Crates.io in the absence of actual crates.io being up. Depending on the crates.io protocol, just caching HTTP requests might not be enough, but also acting as a (offline-able) middle man that knows the protocol gives rise to other knobs and such (e.g. a configurable blacklist of packages).
- 11y ago

jalami 11y ago

I know it wouldn't really fix issues like left-pad, but I would really like namespaced packages similar to the way Github does them. I think group ownership would be more explicit and understood. Top level packages encourages sqatting and small/old/unmaintained packages getting names that people would misunderstand for something else. I understand a lot of that is on the user of the library to research before downloading, but intuitiveness is a virtue. It's true packages could just be called eg. reactjs-node-bridge or something as opposed to reactjs/node-bridge, but anyone can prefix deceptively. Cargo addresses the namespacing concern here http://internals.rust-lang.org/t/crates-io-package-policies/1041 http://internals.rust-lang.org/t/crates-io-package-policies/...

viraptor 11y ago

But that's really a crates.io feature, not cargo's. You could easily deploy your own repository which serves everyone a random file when you pull "some_crate-1.2.3".

airless_bar 11y ago

Let's just see what happens when they get their first court order, which they certainly will (due to their global name-spacing).

steveklabnik 11y ago
As I said below, we will comply with the law. (And I'm not sure what that has to do with namespacing.)
- eyko 11y ago
  I believe he means that packages aren't namespaced (owner/package-name), so if I publish a package called nike, Nike could come and want to take over the package name.
  - kibwen 11y ago
    Global namespacing has nothing to do with that. Even with per-user namespacing, there's nothing to stop a user from making their username "nike" and publishing all their code under "nike/package-name", which would be subject to just as much potential legal action as a package named "nike" itself. Likewise there's also nothing to stop Bob from uploading "bob/nike".
    - airless_bar 11y ago
      If course global name-spacing has something to do with it! What's the difference between a "username" resulting in nike/package-name and a package called nike-package-name? Not much in practice. Name-spacing doesn't mean "add another made up name entry somewhere". That's why people who thought about these issues out-sourced this concern completely: Prove your ownership of nike.com and you can publish as nike.com. Suddenly 99.9% of the trademark issues are gone, handled by registrars. I think this approach is pretty obvious and I find the lack of thought coming from rust devs deeply concerning.
      - 11y ago

themartorana 11y ago

I think this is the scenario that has me feeling warm and fuzzy about vendoring. Sure, eventually even vendored dependencies will become stale, but I don't have to rely on a package manager past the initial delivery. It's not perfect, but I don't sweat code disappearing from a black box I don't own.