5 ms·

Python 3.12.0 from a supply chain security perspective

120 points by zdw 3y ago

r9295 3y ago
Correct me if I'm wrong but SLSA would only prevent artifact tampering (eg. Account takeover on pypi) instead of typo squatting or build script abuse for example?
- SethMLarson 3y ago
  You've got it right, SLSA build provenance in particular only tells you that the artifact you have came from X software repo, at Y commit/tag, built using Z workflow. SLSA doesn't make any mention of what is actually in the artifact (but you can now safely verify the correct commit knowing it was used as input). Typosquatting is an interesting one, because if you've made a typo in one place but not the other (ie installing package name "requestss", but repo is "psf/requests") then SLSA would "save" you by erroring on the mismatch. But that doesn't stop you from typoing in /both/ parameters.
MichaelMug 3y ago
My biggest takeaway from this article is the Supply chain Levels for Software Artifacts (SLSA) security framework: https://github.com/slsa-framework/slsa-verifier https://github.com/slsa-framework/slsa-verifier
- Terretta 3y ago
  > biggest takeaway from this article is the Supply chain Levels for Software Artifacts (SLSA) security framework See also GUAC from Kusari, Google, Citi, and others: “GUAC (Graph for Understanding Artifact Composition) aims to fill in the gaps by ingesting software metadata, like SBOMs, and mapping out relationships between software. When you know how one piece of software affects another, you’ll be able to fully understand your software security position and act as needed.” https://guac.sh https://guac.sh https://www.kusari.dev https://www.kusari.dev
carreau 3y ago
As a curiosity, what would it entail to make the two tgz byte-for-byte identical ? There was/is some discussion in setuptools about how to normalize the tarball (https://github.com/pypa/setuptools/issues/2133#issuecomment-1691158410 https://github.com/pypa/setuptools/issues/2133#issuecomment-...) coudl something similar be applied to Building Python itself ?
- Denvercoder9 3y ago
  > As a curiosity, what would it entail to make the two tgz byte-for-byte identical ? It can't be that complicated. The tarballs autogenerated by GitHub (using `git archive`) were byte-for-byte identical for years, until GitHub upgraded git and things broke because entire ecosystems had started to rely on that. [1] https://news.ycombinator.com/item?id=34586917 https://news.ycombinator.com/item?id=34586917
- stabbles 3y ago
  The suggestion there (uid = gid = 1000; uname = user; gname = users) isn't great. Just use uid = gid = 0, and omit uname/gname. If you're distributing software via a tarball, the uid/gid bits are meaningless. They only make sense when you archive / backup a directory and plan to extract on the same system. If you set them to anything other than 0, it may happen that when the tarball is extracted as root user, ownership is changed to the uid/gid of the tarinfo provided those exist on the system. That's a lot of fun! Python itself in fact tries to chown files when extracting a tarfile (under sudo). If you set uid = gid = 0, then at least when extracting as root, the files remain owned by root.
  - carreau 3y ago
    Thanks for advice, and I assume you are the one who commented on the upstream issue. This show it is not trivial, and it would be nice to be done automatically by default.
- SethMLarson 3y ago
  I believe the only differences were uid/gid and username/groupname values between the two tarballs. One had the information of Thomas Wouters, the release manager of 3.12, and the other had generic GitHub Action usernames/groups. Normalizing these values to something known like 0/0 would have done the trick.
wallrat 3y ago
Is there any effort to integrate SLSA with PyPI? GitHub recently announced[1] that npm support for SLSA is GA now. [1] https://github.blog/changelog/2023-09-26-npm-provenance-general-availability/ https://github.blog/changelog/2023-09-26-npm-provenance-gene...
- SethMLarson 3y ago
  Great question! PyPI already supports Trusted Publishers [1], which gets you most of the benefits of SLSA build provenance (provable link between artifacts and a public software repository). Implementing Trusted Publishers is the recommended first step for ecosystems looking to implement build provenance [2]. [1] https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/ [2] https://github.com/ossf/wg-securing-software-https://docs.pypi.org/trusted-publishers/repos/blob/main/docs/build-provenance-for-all-package-registries.md https://github.com/ossf/wg-securing-software-https://docs.py... I don't think there's a big effort /right now/ to implement complete SLSA build provenance for PyPI and expose it for users to verify.
crabbone 3y ago
In what situation, when it comes to deployed products is any of this relevant? Having used Python for decades, across multiple organizations starting from mega-corps and down to five programmers I've never used built Python binaries for any project that required Python. It's not hard to build your own, and it gives you better control of what's included (Python has a handful of optional compile-time dependencies, which most projects don't need). ---- Also, on personal level, I don't think I ever use built binaries from Python.org. I either build them myself, or use whatever the distro maintainers built. Maybe if you develop on Mac / Windows then it matters... but you already chose to suffer by not having control of your tools -- another drop in a bucket, does it even matter? NB. Also, official Docker images of Python don't use binaries from Python.org. So, nobody who deploys in containers is likely to either build themselves, or to use something other than Python.org binaries.
- evrimoztamur 3y ago
  I think it is. There are still millions of devices which either use their own distribution's provided Python package or some form of prepackaged container.
  - crabbone 3y ago
    Why would what was described in the article be of any use to these devices you describe? If it's built by distribution -- the article is irrelevant. If they use prepackaged container -- the article is irrelevant :|
- tmpX7dMeXU 3y ago
  As someone that’s done both, you are certainly overstating the suffering endured by not compiling your own Python. I very much believe that this is a consequence of your ideology rather than any indication of frequency.
  - crabbone 3y ago
    Suffering doesn't come from not compiling. Suffering comes from inability to trust your system. Not everyone suffers from this. Most don't care. But then the question is why would they care in the case described by the article, if they don't give a rats ass in general?
ealexhudson 3y ago
I have to say it makes sense to me that the binaries would be built from the source tarballs rather than git, seems a bit odd that you'd have separate pipelines and binaries that are produced from a (slightly) different source.
syrusakbary 3y ago
This is great. On a related note, we recently compiled Python 3.12.0 to WebAssembly: using it should shield almost any use case of malicious behavior even on external untrusted modules. https://wasmer.io/python/python https://wasmer.io/python/python Run it locally and see that filesystem access and network are completely sandboxed by default! :) wasmer run python/python
- adolph 3y ago
  > recently compiled Python 3.12.0 to WebAssembly Any chain is as strong as its weakest link.
  - gizmo686 3y ago
    But not all composition is a chain. A sandboxed solution is at least as strong as the sandbox.
- miohtama 3y ago
  What’s benefit of this over running Python in a container like Docker? I know Docker is not 100% watertight, but it’s very unlikely a normal user encounters trojaned Python code that tries to break out from a container. The run-time penalty for using WebAssembly for Python is pretty severe at the moment, at least what I have tried.
  - deleted 3y ago
    [deleted]
  - chalsprhebaodu 3y ago
    I’ve personally never been hit by a Trojaned Python project, why use any container or jail at all?
    - kjok 3y ago
      It is just a matter of time: https://pytorch.org/blog/compromised-nightly-dependency/ https://pytorch.org/blog/compromised-nightly-dependency/
zitterbewegung 3y ago
This sounds great if you want the PSF to make these changes I think the next step would be a PEP that describes it.
- SethMLarson 3y ago
  This doesn't change the Python language or packaging so wouldn't require a PEP, I'm working with release managers on this GitHub repo: https://github.com/python/release-tools https://github.com/python/release-tools
amelius 3y ago
Didn't Anaconda show that the whole Pip approach is flawed and we need a language-agnostic/multi-language approach?
- maxloh 3y ago
  npm works great on JavaScript.
  - underclocked 3y ago
    Are you being serious? https://www.cpomagazine.com/cyber-security/open-source-developer-intentionally-corrupts-npm-libraries-suspected-hack-turns-out-to-be-mischief/ https://www.cpomagazine.com/cyber-security/open-source-devel... https://therecord.media/malware-found-in-npm-package-with-millions-of-weekly-downloads https://therecord.media/malware-found-in-npm-package-with-mi... ... and the list goes on
  - xkcd-sucks 3y ago
    You can even use npm to manage and publish Python code, or could a few years ago if they haven't added constraints on the registry since then

120 points by zdw 3y ago

Correct me if I'm wrong but SLSA would only prevent artifact tampering (eg. Account takeover on pypi) instead of typo squatting or build script abuse for example?

SethMLarson 3y ago
You've got it right, SLSA build provenance in particular only tells you that the artifact you have came from X software repo, at Y commit/tag, built using Z workflow. SLSA doesn't make any mention of what is actually in the artifact (but you can now safely verify the correct commit knowing it was used as input). Typosquatting is an interesting one, because if you've made a typo in one place but not the other (ie installing package name "requestss", but repo is "psf/requests") then SLSA would "save" you by erroring on the mismatch. But that doesn't stop you from typoing in /both/ parameters.

MichaelMug 3y ago

My biggest takeaway from this article is the Supply chain Levels for Software Artifacts (SLSA) security framework: https://github.com/slsa-framework/slsa-verifier https://github.com/slsa-framework/slsa-verifier

Terretta 3y ago
> biggest takeaway from this article is the Supply chain Levels for Software Artifacts (SLSA) security framework See also GUAC from Kusari, Google, Citi, and others: “GUAC (Graph for Understanding Artifact Composition) aims to fill in the gaps by ingesting software metadata, like SBOMs, and mapping out relationships between software. When you know how one piece of software affects another, you’ll be able to fully understand your software security position and act as needed.” https://guac.sh https://guac.sh https://www.kusari.dev https://www.kusari.dev

carreau 3y ago

As a curiosity, what would it entail to make the two tgz byte-for-byte identical ? There was/is some discussion in setuptools about how to normalize the tarball (https://github.com/pypa/setuptools/issues/2133#issuecomment-1691158410 https://github.com/pypa/setuptools/issues/2133#issuecomment-...) coudl something similar be applied to Building Python itself ?

Denvercoder9 3y ago
> As a curiosity, what would it entail to make the two tgz byte-for-byte identical ? It can't be that complicated. The tarballs autogenerated by GitHub (using `git archive`) were byte-for-byte identical for years, until GitHub upgraded git and things broke because entire ecosystems had started to rely on that. [1] https://news.ycombinator.com/item?id=34586917 https://news.ycombinator.com/item?id=34586917
stabbles 3y ago
The suggestion there (uid = gid = 1000; uname = user; gname = users) isn't great. Just use uid = gid = 0, and omit uname/gname. If you're distributing software via a tarball, the uid/gid bits are meaningless. They only make sense when you archive / backup a directory and plan to extract on the same system. If you set them to anything other than 0, it may happen that when the tarball is extracted as root user, ownership is changed to the uid/gid of the tarinfo provided those exist on the system. That's a lot of fun! Python itself in fact tries to chown files when extracting a tarfile (under sudo). If you set uid = gid = 0, then at least when extracting as root, the files remain owned by root.
- carreau 3y ago
  Thanks for advice, and I assume you are the one who commented on the upstream issue. This show it is not trivial, and it would be nice to be done automatically by default.
SethMLarson 3y ago
I believe the only differences were uid/gid and username/groupname values between the two tarballs. One had the information of Thomas Wouters, the release manager of 3.12, and the other had generic GitHub Action usernames/groups. Normalizing these values to something known like 0/0 would have done the trick.

wallrat 3y ago

Is there any effort to integrate SLSA with PyPI? GitHub recently announced[1] that npm support for SLSA is GA now. [1] https://github.blog/changelog/2023-09-26-npm-provenance-general-availability/ https://github.blog/changelog/2023-09-26-npm-provenance-gene...

SethMLarson 3y ago
Great question! PyPI already supports Trusted Publishers [1], which gets you most of the benefits of SLSA build provenance (provable link between artifacts and a public software repository). Implementing Trusted Publishers is the recommended first step for ecosystems looking to implement build provenance [2]. [1] https://docs.pypi.org/trusted-publishers/ https://docs.pypi.org/trusted-publishers/ [2] https://github.com/ossf/wg-securing-software-https://docs.pypi.org/trusted-publishers/repos/blob/main/docs/build-provenance-for-all-package-registries.md https://github.com/ossf/wg-securing-software-https://docs.py... I don't think there's a big effort /right now/ to implement complete SLSA build provenance for PyPI and expose it for users to verify.

crabbone 3y ago

In what situation, when it comes to deployed products is any of this relevant? Having used Python for decades, across multiple organizations starting from mega-corps and down to five programmers I've never used built Python binaries for any project that required Python. It's not hard to build your own, and it gives you better control of what's included (Python has a handful of optional compile-time dependencies, which most projects don't need). ---- Also, on personal level, I don't think I ever use built binaries from Python.org. I either build them myself, or use whatever the distro maintainers built. Maybe if you develop on Mac / Windows then it matters... but you already chose to suffer by not having control of your tools -- another drop in a bucket, does it even matter? NB. Also, official Docker images of Python don't use binaries from Python.org. So, nobody who deploys in containers is likely to either build themselves, or to use something other than Python.org binaries.

evrimoztamur 3y ago
I think it is. There are still millions of devices which either use their own distribution's provided Python package or some form of prepackaged container.
- crabbone 3y ago
  Why would what was described in the article be of any use to these devices you describe? If it's built by distribution -- the article is irrelevant. If they use prepackaged container -- the article is irrelevant :|
tmpX7dMeXU 3y ago
As someone that’s done both, you are certainly overstating the suffering endured by not compiling your own Python. I very much believe that this is a consequence of your ideology rather than any indication of frequency.
- crabbone 3y ago
  Suffering doesn't come from not compiling. Suffering comes from inability to trust your system. Not everyone suffers from this. Most don't care. But then the question is why would they care in the case described by the article, if they don't give a rats ass in general?

ealexhudson 3y ago

I have to say it makes sense to me that the binaries would be built from the source tarballs rather than git, seems a bit odd that you'd have separate pipelines and binaries that are produced from a (slightly) different source.

syrusakbary 3y ago

This is great. On a related note, we recently compiled Python 3.12.0 to WebAssembly: using it should shield almost any use case of malicious behavior even on external untrusted modules. https://wasmer.io/python/python https://wasmer.io/python/python Run it locally and see that filesystem access and network are completely sandboxed by default! :) wasmer run python/python

adolph 3y ago
> recently compiled Python 3.12.0 to WebAssembly Any chain is as strong as its weakest link.
- gizmo686 3y ago
  But not all composition is a chain. A sandboxed solution is at least as strong as the sandbox.
miohtama 3y ago
What’s benefit of this over running Python in a container like Docker? I know Docker is not 100% watertight, but it’s very unlikely a normal user encounters trojaned Python code that tries to break out from a container. The run-time penalty for using WebAssembly for Python is pretty severe at the moment, at least what I have tried.
- deleted 3y ago
  [deleted]
- chalsprhebaodu 3y ago
  I’ve personally never been hit by a Trojaned Python project, why use any container or jail at all?
  - kjok 3y ago
    It is just a matter of time: https://pytorch.org/blog/compromised-nightly-dependency/ https://pytorch.org/blog/compromised-nightly-dependency/

zitterbewegung 3y ago

This sounds great if you want the PSF to make these changes I think the next step would be a PEP that describes it.

SethMLarson 3y ago
This doesn't change the Python language or packaging so wouldn't require a PEP, I'm working with release managers on this GitHub repo: https://github.com/python/release-tools https://github.com/python/release-tools

amelius 3y ago

Didn't Anaconda show that the whole Pip approach is flawed and we need a language-agnostic/multi-language approach?

maxloh 3y ago
npm works great on JavaScript.
- underclocked 3y ago
  Are you being serious? https://www.cpomagazine.com/cyber-security/open-source-developer-intentionally-corrupts-npm-libraries-suspected-hack-turns-out-to-be-mischief/ https://www.cpomagazine.com/cyber-security/open-source-devel... https://therecord.media/malware-found-in-npm-package-with-millions-of-weekly-downloads https://therecord.media/malware-found-in-npm-package-with-mi... ... and the list goes on
- xkcd-sucks 3y ago
  You can even use npm to manage and publish Python code, or could a few years ago if they haven't added constraints on the registry since then