6 ms·

Signing the OPAM Repository: TUF Meets Git

42 points by hannesm 11y ago

amirmc 11y ago
I've recently become really interested in things around the periphery of secure systems keys etc. For example in this scenario, what happens if one of key-holders dies? Presumably the remaining are enough for a quorum, but say something bad happens at a conference that many of them are at? It makes me think of why the President and Vice-president aren't in the same place at the same time. In this case, I expect it would be possible to start again and select a new group of keyholders but what downstream effects would that have? In any case, I'm looking forward to seeing this progress. (Incidentally, one of the contributors is also an author of the OCaml TLS stack.)
- hannesm 11y ago
  The quorum definitively helps. If too few keys/people are remaining, there are two options: download and install an updated opam (with a new set of keys) out of band OR using another opam-repository (with e.g. trust-on-first-use) for an update of opam itself (which then ships new keys for the main opam repository).
justinsb 11y ago
(Edit: I think it is great that we're working on signing software. I think that is very important, so important that I want to see it done right. So I have questions, in particular about TUF vs the more knee-jerk alternatives. But I would like to see signing, whether through TUF or the more well-known options.) It seems that in order to add a new package or developer, a quorum of administrators must go to their secure signing machines and approve the key for that delegation. I don't see how that scales to active communities with lots of new developers. Despite that administrative work and delay, it doesn't seem that the end result offers any guarantees about the repository other than that files haven't been tampered with. The repositories can still include malicious people or malicious software, because the administrators aren't doing any form of validation. I also wonder what happens if people start squatting names and how the administrators will determine who has the legitimate claim to names. This feels very similar to the CA system, except that we have pinned a particular CA (our "trusted administrators"). We have to compare that to choosing a CA (e.g. LetsEncrypt.org) and requiring that all keys be signed by that. I'm no fan of the CA system, but it does seem that the CA system (with pinning) is more tested, and has already implemented stronger procedures and guarantees than what we are getting here. The timestamp signing system is nice, in that it allows for easy mirroring without requiring TLS. Given the timestamp signing key is online, it does seem we could more easily just serve the timestamp file from a central server though, over https, and then allow everything else to be mirrored (the https://security.debian.org https://security.debian.org model, I believe)
- AltGr 11y ago
  Not at all: adding new packages or developers is automatically signed (by the snapshot bot). Administrators may later sign the keys and delegations, individually or in batch, after verifying the developer identity out of band (the specification doesn't yet include the mechanisms for this verification, but with a git repo based on GitHub, the GitHub login gives a good start).
  - justinsb 11y ago
    Thank you for the clarification. Can I install these "not yet trusted" packages? Presumably I can, I just get a big warning. And how do you deal with squatters - can I (as the "real developer") submit a second claim on an unverified package name (that an impostor has claimed)? Edit: And does this mean that the snapshot bot automatically merges (well-formed) pull requests for new packages?
    - AltGr 11y ago
      The pull-requests for new packages are manually merged (as it the case now), automatically signed. Names uniquely identify one package. A repository maintainer can replace the owner of a package, and clear the existing package contents. This shouldn't be done for a different package with the same name if any users still have the package installed, though, as they will consider that an up/down-grade. Hope that clarifies a little!
      - justinsb 11y ago
        That does help me a lot - thanks! The manual "pre-approval process" via accepting the PR is what I was missing, and that does seem to fix most of the shortcomings of the underlying system (assuming a reliable human!) So you have a human that looks at incoming requests to prevent abuse, and to arbitrate problems. And then the snapshot server signs that result to allow distribution over HTTP, and then periodically it gets signed by the quorum of "gods" to indicate that these packages have the full seal of approval. I think it is pretty specific to OCaml's current workflow and volume, but signing is better than not signing!
amirmc 11y ago
For those who are interested in following the discussion that the maintainers/devs are having, you'll find it starting at: http://lists.ocaml.org/pipermail/platform/2015-June/000595.html http://lists.ocaml.org/pipermail/platform/2015-June/000595.h...
allendoerfer 11y ago
What I do not get about these repositories is, why they have no namespacing or a built in democratic mechanism to transfer ownership. Because this does not exist, two related problems occur: Outdated packages squatting their own name and otherwise worse packages squatting a generic name. The adoption problems Python 3 still has at this point, could be mitigated entirely if package owners could lose the name for outdated packages and it would be transferred to a Python 3 fork. npm has this problem on an even larger scale. The Python community can at least slowly move away from PIL and use Pillow instead, because the libraries are reasonably big. This is much harder to do with all of npms tiny packages. I doubt that the best solution always wins, because the best name is just an enormous advantage. If names would be democratically assigned, we would get some kind of extended standard library. Newbies that want the best library for their problem that works with the current version of the language would not have to search for Beautifulsoup, voluptuous or requests, but could just use xml, validation and http. Of course, this is a hard problem and you have to be careful. We do not want Java, we do not want broken dependencies. So my ideas a solution could be based on are these (example is for pip, but applies to others as well): # No fork exists pip install flask >Downloading/unpacking mitshuko.flask >… # A fork exists pip install flask >Multiple users have created a package named "flask". Please specify the owner >of the package or use "--most-popular" to install the one with the most >installations in the last 3 months. > >***************************************************************************** >Attention: pip does not check if the package actually does the same thing, be >sure to check that before you switch to a package from another author. >***************************************************************************** > >mitsuhiko.flask (10000 installations) >forker.flask (3 installations) pip install --most-popular flask >Downloading/unpacking mitsuhiko.flask >… pip install -U flask # On upgrade, this message is only shown, when the used package is # already at the current version, has not been updated for a long time # and a more popular option exists. >You have installed the current version of mitsuhiko.flask, however >mitsuhiko.flask has not been updated since 7 months and there is a more >popular package available of the same name from another author. > >***************************************************************************** >Attention: pip does not check if the package actually does the same thing, be >sure to check that before you switch to a package from another author. >***************************************************************************** > >forker.flask (5000 installations) >mitsuhiko.flask (4000 installations) pip install -U forker.flask >Downloading/unpacking mitshuko.flask >…

42 points by hannesm 11y ago

I've recently become really interested in things around the periphery of secure systems keys etc. For example in this scenario, what happens if one of key-holders dies? Presumably the remaining are enough for a quorum, but say something bad happens at a conference that many of them are at? It makes me think of why the President and Vice-president aren't in the same place at the same time. In this case, I expect it would be possible to start again and select a new group of keyholders but what downstream effects would that have? In any case, I'm looking forward to seeing this progress. (Incidentally, one of the contributors is also an author of the OCaml TLS stack.)

hannesm 11y ago
The quorum definitively helps. If too few keys/people are remaining, there are two options: download and install an updated opam (with a new set of keys) out of band OR using another opam-repository (with e.g. trust-on-first-use) for an update of opam itself (which then ships new keys for the main opam repository).

justinsb 11y ago

(Edit: I think it is great that we're working on signing software. I think that is very important, so important that I want to see it done right. So I have questions, in particular about TUF vs the more knee-jerk alternatives. But I would like to see signing, whether through TUF or the more well-known options.) It seems that in order to add a new package or developer, a quorum of administrators must go to their secure signing machines and approve the key for that delegation. I don't see how that scales to active communities with lots of new developers. Despite that administrative work and delay, it doesn't seem that the end result offers any guarantees about the repository other than that files haven't been tampered with. The repositories can still include malicious people or malicious software, because the administrators aren't doing any form of validation. I also wonder what happens if people start squatting names and how the administrators will determine who has the legitimate claim to names. This feels very similar to the CA system, except that we have pinned a particular CA (our "trusted administrators"). We have to compare that to choosing a CA (e.g. LetsEncrypt.org) and requiring that all keys be signed by that. I'm no fan of the CA system, but it does seem that the CA system (with pinning) is more tested, and has already implemented stronger procedures and guarantees than what we are getting here. The timestamp signing system is nice, in that it allows for easy mirroring without requiring TLS. Given the timestamp signing key is online, it does seem we could more easily just serve the timestamp file from a central server though, over https, and then allow everything else to be mirrored (the https://security.debian.org https://security.debian.org model, I believe)

AltGr 11y ago
Not at all: adding new packages or developers is automatically signed (by the snapshot bot). Administrators may later sign the keys and delegations, individually or in batch, after verifying the developer identity out of band (the specification doesn't yet include the mechanisms for this verification, but with a git repo based on GitHub, the GitHub login gives a good start).
- justinsb 11y ago
  Thank you for the clarification. Can I install these "not yet trusted" packages? Presumably I can, I just get a big warning. And how do you deal with squatters - can I (as the "real developer") submit a second claim on an unverified package name (that an impostor has claimed)? Edit: And does this mean that the snapshot bot automatically merges (well-formed) pull requests for new packages?
  - AltGr 11y ago
    The pull-requests for new packages are manually merged (as it the case now), automatically signed. Names uniquely identify one package. A repository maintainer can replace the owner of a package, and clear the existing package contents. This shouldn't be done for a different package with the same name if any users still have the package installed, though, as they will consider that an up/down-grade. Hope that clarifies a little!
    - justinsb 11y ago
      That does help me a lot - thanks! The manual "pre-approval process" via accepting the PR is what I was missing, and that does seem to fix most of the shortcomings of the underlying system (assuming a reliable human!) So you have a human that looks at incoming requests to prevent abuse, and to arbitrate problems. And then the snapshot server signs that result to allow distribution over HTTP, and then periodically it gets signed by the quorum of "gods" to indicate that these packages have the full seal of approval. I think it is pretty specific to OCaml's current workflow and volume, but signing is better than not signing!

amirmc 11y ago

For those who are interested in following the discussion that the maintainers/devs are having, you'll find it starting at: http://lists.ocaml.org/pipermail/platform/2015-June/000595.html http://lists.ocaml.org/pipermail/platform/2015-June/000595.h...

allendoerfer 11y ago

What I do not get about these repositories is, why they have no namespacing or a built in democratic mechanism to transfer ownership. Because this does not exist, two related problems occur: Outdated packages squatting their own name and otherwise worse packages squatting a generic name. The adoption problems Python 3 still has at this point, could be mitigated entirely if package owners could lose the name for outdated packages and it would be transferred to a Python 3 fork. npm has this problem on an even larger scale. The Python community can at least slowly move away from PIL and use Pillow instead, because the libraries are reasonably big. This is much harder to do with all of npms tiny packages. I doubt that the best solution always wins, because the best name is just an enormous advantage. If names would be democratically assigned, we would get some kind of extended standard library. Newbies that want the best library for their problem that works with the current version of the language would not have to search for Beautifulsoup, voluptuous or requests, but could just use xml, validation and http. Of course, this is a hard problem and you have to be careful. We do not want Java, we do not want broken dependencies. So my ideas a solution could be based on are these (example is for pip, but applies to others as well): # No fork exists pip install flask >Downloading/unpacking mitshuko.flask >… # A fork exists pip install flask >Multiple users have created a package named "flask". Please specify the owner >of the package or use "--most-popular" to install the one with the most >installations in the last 3 months. > >***************************************************************************** >Attention: pip does not check if the package actually does the same thing, be >sure to check that before you switch to a package from another author. >***************************************************************************** > >mitsuhiko.flask (10000 installations) >forker.flask (3 installations) pip install --most-popular flask >Downloading/unpacking mitsuhiko.flask >… pip install -U flask # On upgrade, this message is only shown, when the used package is # already at the current version, has not been updated for a long time # and a more popular option exists. >You have installed the current version of mitsuhiko.flask, however >mitsuhiko.flask has not been updated since 7 months and there is a more >popular package available of the same name from another author. > >***************************************************************************** >Attention: pip does not check if the package actually does the same thing, be >sure to check that before you switch to a package from another author. >***************************************************************************** > >forker.flask (5000 installations) >mitsuhiko.flask (4000 installations) pip install -U forker.flask >Downloading/unpacking mitshuko.flask >…