7 ms·
PyPI Blog: Project Quarantine
- toomuchtodo 2y agoAwesome work, kudos to the PyPI team. Will it be possible to receive notifications of projects quarantine as a member of the public?
- HanClinto 2y agoYour comment also has me dreaming about a Dependabot-esque utility that opens Github issues on repositories that have quarantined projects in their requirements.txt. Quarantining would prevent anyone from building / installing new copies of the compromised software, so this utility would only help people who were a) monitoring the project, and b) had a local version installed pre-quarantine. That's a pretty narrow scope of users, so now that I type all this out, I'm realizing that the juice is likely not worth the squeeze.
- toomuchtodo 2y agoOne of my responsibilities is software supply chain security in a financial services org, so this signal would be valuable for vulnerability management of dependencies. I wouldn't call it "threat hunting" per se, but ground truth around threat actor patterns helps us build better defensive systems in this regard. Keeping the bad bits out is way easier than remediating once they've been ingested into systems. > Your comment also has me dreaming about a Dependabot-esque utility that opens Github issues on repositories that have quarantined projects in their requirements.txt. It's not a bad idea, let Github know! Their security team is very good from my interactions with them.
- intelVISA 2y agoThat sounds quite daunting, Python and supply chain security are almost at odds with each other these days. Lowkey surprised that any well-resourced org would use it given the outsized risk profile and poor performance.
- toomuchtodo 2y agoIt’s not used in the core or for anything load bearing, but has some ancillary uses, and we strive for total coverage (as much as practical). If we use something, we want to secure it as best we can.
- alsodumb 2y agoGiven how widespread PyPI usage is, I'm surprised they only have one full time security staff. I mean I guess it makes sense, usage doesn't always mean they get more donations/money, but damn.
- spencerchubb 2y agocompanies that actually care about security have a more secure solution and don't allow devs to use pypi
- cjalmeida 2y agoYou’d be surprised by the amount of companies handling critical infrastructure that are OK with using PyPI directly
- LtWorf 2y agoHe said companies that care, not companies that should care but do not.
- f1shy 2y agoThat is somewhat terrifying
- spencerchubb 2y agoreally depends on the company. my company cares a lot about security because it's a huge fortune 50 company with sensitive data and a lot of reputation could be lost with a security scandal
- davidshepherd7 2y agoCould you give some examples of more secure solutions?
- spencerchubb 2y agojfrog is the one my company uses
- IshKebab 2y agoThe still don't even have a way to avoid dependency confusion attacks when using private package repos (other than also registering every single private package name you use on pypi.org). Blows my mind.
- woodruffw 2y agoWho is "they"? PyPI is an index; it doesn't control your installing client. (This is a larger issue - or feature, depending on your perspective - with Python packaging. But it's important to understand that PyPI itself can't force `pip` or any other client to pick any particular resolution order between indices.)
- LtWorf 2y agoFor all intents and purposes "pip" is the official client. It is referenced in the official documentation https://docs.python.org/3/installing/index.html https://docs.python.org/3/installing/index.html
- woodruffw 2y agoThe fact that pip is the official client isn’t in dispute. The point was that pip and PyPI are different entities, per a larger pattern of devolved ownership/control/standards-over-tools in Python packaging. PyPI has little to no say over how pip and other tools choose to handle resolutions across multiple indices.
- xgstation 2y agothe fact that `pip install` just runs whatever is in `setup.py` is still mind baffling, even if the author weren't mallicious the `setup.py` can still do harm (say delete a file by mistake), there really needs to be an official way of sandbox its running.
- woodruffw 2y agoIt's not good, but it should also not be baffling: it's the exact same thing other ecosystems do (npm with install hooks/scripts, Rust with build.rs, Ruby with gemspecs, etc).
- xgstation 2y agoI know other ecosystems do the same and those are baffling too, especially for the newer created languages like rust, which is why https://internals.rust-lang.org/t/pre-rfc-sandboxed-deterministic-reproducible-efficient-wasm-compilation-of-proc-macros/19359 https://internals.rust-lang.org/t/pre-rfc-sandboxed-determin... exists
- woodruffw 2y agoSandboxing is a great idea. But the fact that this is a near-universal feature of language packaging reveals a preference that's going to be hard to counter: users do want effectively-arbitrary system access at build time, because that's the paradigm that's supported by the million-and-one different ways in which a build environment can be valid.
- f1shy 2y agoNotably also common lisp (quicklisp)
- ogrisel 2y agoNote that it's possible to disable that behavior with `pip install --only-binary :all:`. This way, pip will fail if a dependency does not provide a `.whl` package, instead of automatically falling back to the "build from source" mode that can lead to arbitrary code execution at install time (via setuptools' `setup.py` or any other build backend mechanism). However, installing from wheels just protects from arbitrary code execution at install time. If you do not trust the source and integrity of the package you install, you would still be subject to arbitrary code execution at import time. Therefore, tools and processes to improve package provenance tracing and integrity checking are useful for both kinds of installations.
- f1shy 2y agoI see some comments about the lack of security of Pypi. And they are totally right, I’m also concerned. But to be fair, many other languages don’t fare better in that arena. I don’t want to give examples, but everyone knows horror histories with other languages. Again, is not that because others are worse, is ok, but I would cut a little slack. Specially for the fact that having all packages somehow signed/audited would be a titanic task. And I guess I’m not willing to pay for it.
- nathanmills 2y agoQuarantining projects is just a band-aid. If you’re worried about malware, maybe stop letting random people upload code to the official package index. Or just write better docs so people stop using random packages in the first place.
- openrisk 2y agoIts always an interesting dynamic: assuming a high trust society pays dividends - Python would be nowhere close the success it has been without PyPI. But then success attracts trust abusers and forces raising the fences (which comes with higher costs, both direct and indirect). Direct costs in the people and infrastructure that must be dedicated to the task. Indirect costs in the frictions generated by complicating workflows. It all points to the need for open source ecosystems to be taken more seriously by the economically able users who most benefit from this amazing development.
- LtWorf 2y agoThey won't pay anything unless they are forced to do so. Basic capitalism brings to externalise costs to society
- NeutralCrane 2y agoPerhaps, but can you explain how an alternative to capitalism wouldn’t result in people no paying for a service they don’t have to pay for?
- LtWorf 2y agoIn an alternative system you can get a salary from the government to work on open source software, and the companies pay for that in taxes. Of course you must embargo Malta, Netherlands and all the other countries that thrive on grabbing taxes from other countries.
- yencabulator 2y agoPeople in more communally oriented societies pay for things they "don't have to" pay for because there's a social obligation.
- me_vinayakakv 2y agohttps://socket.dev/ https://socket.dev/ does a good job in detecting malicious packages in npm. In their FAQ[1], they mention that they have plans to expand to PyPI as well. [1]: https://docs.socket.dev/docs/faq https://docs.socket.dev/docs/faq
- oefrha 2y ago> The one project cleared was a project containing obfuscated code, in violation of the PyPI Acceptable Use Policy. Interesting, I didn’t know that. While I haven’t released anything obfuscated on PyPI, I’ve certainly written Python projects that include obfuscated code by necessity, namely scrapers packing duktape (embedded JS interpreter) and third party obfuscated JS blobs to generate signatures and stuff. I know for a fact there are projects like that on PyPI. I wonder if those are allowed. (Come to think of it, those probably can be DMCAed if the targeted service provider is sufficiently motivated.)