7 ms·

I've enabled "trusted publishing" as it is called for python packages (publishing to cheeseshop/PyPI). However, what they call trusted publishing is just a con

I've enabled "trusted publishing" as it is called for python packages (publishing to cheeseshop/PyPI). However, what they call trusted publishing is just a configuration where PyPI tells github that publishing from a particular workflow name on a particular repository is ok, without further tokens. So PyPI "trusts" github actions and the maintainer is out of the loop. All good? Well, if you trust Github! It would be a lot better to me if both the maintainer and github were involved, something like - the maintainer signs off on some artifacts, and the github action verifies they are reproducible exactly by a workflow, and /then/ it's published.

londons_explore 3y ago
I do trust github with anything where any dishonesty on their part would be discoverable from the end result. Just a single piece of strong evidence that github the company had injected malware into something (rather than a github user) would end their business.
- perihelions 3y ago
  I think we should have learned this lesson by now. The revelation that GitHub deliberately injects malware into things would be unlikely to "end their business": it would contentiously divide their userbase between purists who believe repo hosts should not be injecting malware into people's FOSS projects, and pragmatists who defend the trust-and-safety reasons for injecting malware into FOSS (say, to assist US intelligence preventing terror attacks). We've had this recurring discussion in many forms, and the typical result is that peoples' capacity for outrage wears out quickly. Who still remembers, for instance, that the GitHub entity sabotaged its users by giving the NSA backdoors to its private infrastructure [0]? How much business is Microsoft missing out on today, in 2024, because of that betrayal of trust? Essentially none, right? They're bigger than ever. GitHub sneaking NSA malware onto their platform—maybe selectively serving it to certain targets—would be perfectly in character with their past behavior. It would be smart business sense (they're a major DoD vendor). It'd be totally irrational to have a faith-based trust that they wouldn't do this. (There's a precedent: the FOSS host SourceForge destroyed itself by making the error of judgement of injecting code into people's repos [1]. But that was a decade ago: the Overton Window's shifted a lot. What SourceForge lost their business for doing in 2013—injecting ads—is something Microsoft comfortably gets away with today, on Windows desktops). [0] https://news.ycombinator.com/item?id=6027779 https://news.ycombinator.com/item?id=6027779 ("How Microsoft handed the NSA access to encrypted messages" (2013)) [1] https://en.wikipedia.org/wiki/SourceForge#Adware_controversy https://en.wikipedia.org/wiki/SourceForge#Adware_controversy
  - hyperpape 3y ago
    Whether or not you draw a distinction, it seems plausible that the world as a whole would draw a distinction between sharing encrypted messages with the NSA and deliberately injecting malware into widely distributed software. Corporations can look at the former and think it is unlikely to affect them. So regardless of how bad its prior behavior is, it makes sense that Microsoft has more of a reason to try to keep software distributed through GitHub as secure as it reasonably can.
    - peteradio 3y ago
      How about they just put a checkbox in the settings (checked by default of course) to install the malware. If you have a problem with it you can navigate to the settings and uncheck the install malware box. Now of course you may need to check from time to time if the box has reverted back to its natural checked state because of some backend update.
    - tsimionescu 3y ago
      Still, if the NSA wants to target a "rogue state" with malware through GitHub, they are much much more likely than not to comply. Even if it were ever discovered that they did so, and if it were ever widely publicized, it would not affect the majority of GitHub paying customers (which are US or EU companies, which by and large don't care if their geopolitical rivals are served malware).
  - lupusreal 3y ago
    It may seem inconceivable that github would infect downloads with malware, but 20 years ago sourceforge doing it would have seemed inconceivable. They did though. You're saying that it happened before so it won't happen again, but that's silly. Anybody with sense back then would have already known that sourceforge's reputation would be ruined by it, just as everybody knows githubs reputation would be ruined. But you can't count on businesses being run on common sense.
    - perihelions 3y ago
      You misread my comment: I'm saying it absolutely could happen and there's no sound reason to assume it won't.
- Aachen 3y ago
  "oops, our bad, we'll improve security" It won't end their business and it's currently a single point of failure. Having and checking pgp signatures would mean the developer and Microsoft/GitHub have to both get compromised, which is a risk I'd much more readily take than github never ever getting compromised for security-critical software like ssh or its dependencies Edit: realised this isn't true unless github starts doing code reviews. The developer can still always push anything. But by having digital signatures, you could require both devs to OK a change or release. Or, at least not everything is compromised if github is compromised: every developer would have to be compromised separately to forge all signatures. Using github as the single point of trust seems like an exceedingly large problem waiting to happen at least once in our lifetimes
  - londons_explore 3y ago
    Since the code, the build process, and the final signed tarball would all be made available to the public in the case of these 'verified builds', it would only take one person to rebuild and diff the results to uncover any malpractice by github. Obviously in most cases, any differences will be bits of the build process that are nondeterministic like including timestamps in the final binary.
  - woodruffw 3y ago
    One of the key observations from PyPI’s removal of PGP support was that virtually nobody actually verified the signatures uploaded to the index (when they were actual signatures, and not just garbage or error files), and that most of their backing keys were expired, weak, or otherwise impossible to publicly establish. Independent signatures do mean that the author could stymie a compromise of their source host. But on net, users are worse maintainers of cryptographic key materials (and worse users of obtuse ecosystems like PGP) than big companies are. That doesn’t mean we shouldn’t enable user signing; only that we need better, more misuse-resistant tools for it.
    - Aachen 3y ago
      In response to the last sentence (copying function broke here, idk why but so I can't easily quote): agreed, though it seems odd to throw out the existing solution before having a new one, preferring no solution over a partial one
orf 3y ago
For Python releases, this is quite a big problem. Python packages consist of many different downloadable files - for example, the Datadog APM library has over 60 different files[1] depending on the platform and Python version. There's nothing to enforce that each of these files contains the same code - and indeed, they shouldn't: each platform-specific .whl file contains platform-specific binaries. But for the python and other platform-agnostic code, it can and does differ if the publishing process is done manually. One example of this is an intel project called "devcloud" - a specific file was included only in the .whl release (and not the source distribution) that contained an AWS access key[2]. The Amazon pay SDK also fell victim to this - the developer accidentally included an integration test that contained a key[3]. Someone else included a markdown file containing ~400 OpenAI keys in a similar manner, only present in a single .whl file out of several. There are _lots_ of examples of this, so many that I created a project to attempt to quantify this, prevent accidental credential leaks and bring greater visibility to the contents of PyPI by publishing all the contents of PyPI releases to Github and indexing them: https://py-code.org/ https://py-code.org/ There's a hosted clickhouse dataset available here: https://py-code.org/datasets https://py-code.org/datasets. On my to-do list is to write a query to find differences in files within the same package version, where they should be the same. This is almost always an indication of something malicious or accidental. "Package integrity" is a really interesting + complex area when you add a platform-specific dimension to it. For Python specifically, I wonder if the "separate whl per platform" approach is a good one. Maybe a layered approach would be better - the "source" is the source of truth, and pre-built binaries _only_ contain pre-built binaries and not the duplicated "source"? 1. https://pypi.org/project/ddtrace/#files https://pypi.org/project/ddtrace/#files 2. https://inspector.pypi.io/project/devcloud/0.0.2/packages/9f/59/739768bdffb30404afe359bea1975664e78d2d272124a08e98591ef3e8a0/devcloud-0.0.2-py3-none-any.whl/devcloud/result.py#line.16 https://inspector.pypi.io/project/devcloud/0.0.2/packages/9f... 3. https://inspector.pypi.io/project/amazon-pay/2.5.1/packages/db/20/cc6965de19ba9839823043d50de44ececffeeaa6712c77b16721224ad9c3/amazon_pay-2.5.1.tar.gz/amazon_pay-2.5.1/amazon_pay/integration_test.py#line.6 https://inspector.pypi.io/project/amazon-pay/2.5.1/packages/...
Retr0id 3y ago
Reproducible builds could help here, perhaps. If GitHub's CI runners and GitLab's CI runners (for example) are producing the same tarballs, that's probably a good sign.
woodruffw 3y ago
> However, what they call trusted publishing is just a configuration where PyPI tells github that publishing from a particular workflow name on a particular repository is ok, without further tokens. So PyPI "trusts" github actions and the maintainer is out of the loop. That isn’t quite how trusted publishing works: it’s morally equivalent to trusting GitHub with your manually configured API token, except GitHub mints the token using its own publicly verifiable key material. In other words: it’s no more trusting of GitHub than manually configuring a secret would be, but is significantly more misuse and compromise resistant (since the minted API token is auto-expiring and minimally scoped to whatever project the repo is linked to). You’re right that this does not itself involve reproducibility or maintainer signoff. But signoff is something you can configure for yourself on GitHub releases (which PyPI cannot meaningfully enforce), and reproducibility is a distant ecosystem-sized goal at this point. Lashing trusting publishers to either would have meant making the perfect the enemy of the good. (It’s also something we’re working on, by exposing cryptographic publish attestations tied to the trusted publisher. That is still being designed, but I hope to have more to share publicly soon.) Source: I implemented PyPI’s trusted publishing.
- guappa 3y ago
  > but is significantly more misuse and compromise resistant I don't think it is at all. Without it, I have a supersecret token.txt file on my machine. With it, I have a supersecret .ssh/ directory, which grants access to do a git tag, which will publish without needing to know the pypi secrets at all. You just moved the secret into another file. Am I wrong? Can you explain how am I wrong if you think so?
  - woodruffw 3y ago
    > With it, I have a supersecret .ssh/ directory, which grants access to do a git tag, which will publish without needing to know the pypi secrets at all. That isn't how trusted publishing works. It uses OpenID Connect, which both requires a privileged workflow (in the case of GitHub) and is itself self-expiring. There's a lot of information available in PyPI's documentation about how it works[1]. But the basic version is that short-lived tokens always reduce attacker exposure, since they prevent a temporary compromise from becoming a permanent one (i.e., via theft of a long-lived credential). Similarly, auto-scoped credentials always reduce credential misuse, since they prevent users from over-scoping their API tokens to get things to "just work." Both were observed issues on PyPI prior to trusted publishing, and were a huge part of both the original API token and 2FA rollouts. [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/
guappa 3y ago
> All good? Well, if you trust Github! And if you trust that your secret ssh keys don't get hacked away from your machine, giving someone else the rights to push a tag and automatically make a release on pypi.

londons_explore 3y ago

I do trust github with anything where any dishonesty on their part would be discoverable from the end result. Just a single piece of strong evidence that github the company had injected malware into something (rather than a github user) would end their business.

perihelions 3y ago
I think we should have learned this lesson by now. The revelation that GitHub deliberately injects malware into things would be unlikely to "end their business": it would contentiously divide their userbase between purists who believe repo hosts should not be injecting malware into people's FOSS projects, and pragmatists who defend the trust-and-safety reasons for injecting malware into FOSS (say, to assist US intelligence preventing terror attacks). We've had this recurring discussion in many forms, and the typical result is that peoples' capacity for outrage wears out quickly. Who still remembers, for instance, that the GitHub entity sabotaged its users by giving the NSA backdoors to its private infrastructure [0]? How much business is Microsoft missing out on today, in 2024, because of that betrayal of trust? Essentially none, right? They're bigger than ever. GitHub sneaking NSA malware onto their platform—maybe selectively serving it to certain targets—would be perfectly in character with their past behavior. It would be smart business sense (they're a major DoD vendor). It'd be totally irrational to have a faith-based trust that they wouldn't do this. (There's a precedent: the FOSS host SourceForge destroyed itself by making the error of judgement of injecting code into people's repos [1]. But that was a decade ago: the Overton Window's shifted a lot. What SourceForge lost their business for doing in 2013—injecting ads—is something Microsoft comfortably gets away with today, on Windows desktops). [0] https://news.ycombinator.com/item?id=6027779 https://news.ycombinator.com/item?id=6027779 ("How Microsoft handed the NSA access to encrypted messages" (2013)) [1] https://en.wikipedia.org/wiki/SourceForge#Adware_controversy https://en.wikipedia.org/wiki/SourceForge#Adware_controversy
- hyperpape 3y ago
  Whether or not you draw a distinction, it seems plausible that the world as a whole would draw a distinction between sharing encrypted messages with the NSA and deliberately injecting malware into widely distributed software. Corporations can look at the former and think it is unlikely to affect them. So regardless of how bad its prior behavior is, it makes sense that Microsoft has more of a reason to try to keep software distributed through GitHub as secure as it reasonably can.
  - peteradio 3y ago
    How about they just put a checkbox in the settings (checked by default of course) to install the malware. If you have a problem with it you can navigate to the settings and uncheck the install malware box. Now of course you may need to check from time to time if the box has reverted back to its natural checked state because of some backend update.
  - tsimionescu 3y ago
    Still, if the NSA wants to target a "rogue state" with malware through GitHub, they are much much more likely than not to comply. Even if it were ever discovered that they did so, and if it were ever widely publicized, it would not affect the majority of GitHub paying customers (which are US or EU companies, which by and large don't care if their geopolitical rivals are served malware).
- lupusreal 3y ago
  It may seem inconceivable that github would infect downloads with malware, but 20 years ago sourceforge doing it would have seemed inconceivable. They did though. You're saying that it happened before so it won't happen again, but that's silly. Anybody with sense back then would have already known that sourceforge's reputation would be ruined by it, just as everybody knows githubs reputation would be ruined. But you can't count on businesses being run on common sense.
  - perihelions 3y ago
    You misread my comment: I'm saying it absolutely could happen and there's no sound reason to assume it won't.
Aachen 3y ago
"oops, our bad, we'll improve security" It won't end their business and it's currently a single point of failure. Having and checking pgp signatures would mean the developer and Microsoft/GitHub have to both get compromised, which is a risk I'd much more readily take than github never ever getting compromised for security-critical software like ssh or its dependencies Edit: realised this isn't true unless github starts doing code reviews. The developer can still always push anything. But by having digital signatures, you could require both devs to OK a change or release. Or, at least not everything is compromised if github is compromised: every developer would have to be compromised separately to forge all signatures. Using github as the single point of trust seems like an exceedingly large problem waiting to happen at least once in our lifetimes
- londons_explore 3y ago
  Since the code, the build process, and the final signed tarball would all be made available to the public in the case of these 'verified builds', it would only take one person to rebuild and diff the results to uncover any malpractice by github. Obviously in most cases, any differences will be bits of the build process that are nondeterministic like including timestamps in the final binary.
- woodruffw 3y ago
  One of the key observations from PyPI’s removal of PGP support was that virtually nobody actually verified the signatures uploaded to the index (when they were actual signatures, and not just garbage or error files), and that most of their backing keys were expired, weak, or otherwise impossible to publicly establish. Independent signatures do mean that the author could stymie a compromise of their source host. But on net, users are worse maintainers of cryptographic key materials (and worse users of obtuse ecosystems like PGP) than big companies are. That doesn’t mean we shouldn’t enable user signing; only that we need better, more misuse-resistant tools for it.
  - Aachen 3y ago
    In response to the last sentence (copying function broke here, idk why but so I can't easily quote): agreed, though it seems odd to throw out the existing solution before having a new one, preferring no solution over a partial one

orf 3y ago

For Python releases, this is quite a big problem. Python packages consist of many different downloadable files - for example, the Datadog APM library has over 60 different files[1] depending on the platform and Python version. There's nothing to enforce that each of these files contains the same code - and indeed, they shouldn't: each platform-specific .whl file contains platform-specific binaries. But for the python and other platform-agnostic code, it can and does differ if the publishing process is done manually. One example of this is an intel project called "devcloud" - a specific file was included only in the .whl release (and not the source distribution) that contained an AWS access key[2]. The Amazon pay SDK also fell victim to this - the developer accidentally included an integration test that contained a key[3]. Someone else included a markdown file containing ~400 OpenAI keys in a similar manner, only present in a single .whl file out of several. There are _lots_ of examples of this, so many that I created a project to attempt to quantify this, prevent accidental credential leaks and bring greater visibility to the contents of PyPI by publishing all the contents of PyPI releases to Github and indexing them: https://py-code.org/ https://py-code.org/ There's a hosted clickhouse dataset available here: https://py-code.org/datasets https://py-code.org/datasets. On my to-do list is to write a query to find differences in files within the same package version, where they should be the same. This is almost always an indication of something malicious or accidental. "Package integrity" is a really interesting + complex area when you add a platform-specific dimension to it. For Python specifically, I wonder if the "separate whl per platform" approach is a good one. Maybe a layered approach would be better - the "source" is the source of truth, and pre-built binaries _only_ contain pre-built binaries and not the duplicated "source"? 1. https://pypi.org/project/ddtrace/#files https://pypi.org/project/ddtrace/#files 2. https://inspector.pypi.io/project/devcloud/0.0.2/packages/9f/59/739768bdffb30404afe359bea1975664e78d2d272124a08e98591ef3e8a0/devcloud-0.0.2-py3-none-any.whl/devcloud/result.py#line.16 https://inspector.pypi.io/project/devcloud/0.0.2/packages/9f... 3. https://inspector.pypi.io/project/amazon-pay/2.5.1/packages/db/20/cc6965de19ba9839823043d50de44ececffeeaa6712c77b16721224ad9c3/amazon_pay-2.5.1.tar.gz/amazon_pay-2.5.1/amazon_pay/integration_test.py#line.6 https://inspector.pypi.io/project/amazon-pay/2.5.1/packages/...

Retr0id 3y ago

Reproducible builds could help here, perhaps. If GitHub's CI runners and GitLab's CI runners (for example) are producing the same tarballs, that's probably a good sign.

woodruffw 3y ago

> However, what they call trusted publishing is just a configuration where PyPI tells github that publishing from a particular workflow name on a particular repository is ok, without further tokens. So PyPI "trusts" github actions and the maintainer is out of the loop. That isn’t quite how trusted publishing works: it’s morally equivalent to trusting GitHub with your manually configured API token, except GitHub mints the token using its own publicly verifiable key material. In other words: it’s no more trusting of GitHub than manually configuring a secret would be, but is significantly more misuse and compromise resistant (since the minted API token is auto-expiring and minimally scoped to whatever project the repo is linked to). You’re right that this does not itself involve reproducibility or maintainer signoff. But signoff is something you can configure for yourself on GitHub releases (which PyPI cannot meaningfully enforce), and reproducibility is a distant ecosystem-sized goal at this point. Lashing trusting publishers to either would have meant making the perfect the enemy of the good. (It’s also something we’re working on, by exposing cryptographic publish attestations tied to the trusted publisher. That is still being designed, but I hope to have more to share publicly soon.) Source: I implemented PyPI’s trusted publishing.

guappa 3y ago
> but is significantly more misuse and compromise resistant I don't think it is at all. Without it, I have a supersecret token.txt file on my machine. With it, I have a supersecret .ssh/ directory, which grants access to do a git tag, which will publish without needing to know the pypi secrets at all. You just moved the secret into another file. Am I wrong? Can you explain how am I wrong if you think so?
- woodruffw 3y ago
  > With it, I have a supersecret .ssh/ directory, which grants access to do a git tag, which will publish without needing to know the pypi secrets at all. That isn't how trusted publishing works. It uses OpenID Connect, which both requires a privileged workflow (in the case of GitHub) and is itself self-expiring. There's a lot of information available in PyPI's documentation about how it works[1]. But the basic version is that short-lived tokens always reduce attacker exposure, since they prevent a temporary compromise from becoming a permanent one (i.e., via theft of a long-lived credential). Similarly, auto-scoped credentials always reduce credential misuse, since they prevent users from over-scoping their API tokens to get things to "just work." Both were observed issues on PyPI prior to trusted publishing, and were a huge part of both the original API token and 2FA rollouts. [1]: https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/

guappa 3y ago

> All good? Well, if you trust Github! And if you trust that your secret ssh keys don't get hacked away from your machine, giving someone else the rights to push a tag and automatically make a release on pypi.