5 ms·

Anyone can upload anything to PyPI. This is kind of like saying that you detected malicious packages on GitHub - the question is whether anyone actually ran it.

by geofft 5y ago

Anyone can upload anything to PyPI. This is kind of like saying that you detected malicious packages on GitHub - the question is whether anyone actually ran it. They say that the packages were downloaded 30,000 times, but automated processes like mirrors can easily inflate this. (As can people doing the exact sort of research they were doing - they themselves downloaded the files from PyPI!) Quoting PyPI maintainer Dustin Ingram https://twitter.com/di_codes/status/1421415135743254533 https://twitter.com/di_codes/status/1421415135743254533 : > *And here's your daily reminder that download statistics for PyPI are hugely inflated by mirrors & scrapers. Publish a new package today and you'll get 1000 'downloads' in 24 hours without even telling anyone about it.*

yonixw 5y ago
If anyone wonders about the same for NPM, it is around 400. That what happened to my almost empty package. https://i.imgur.com/Ryr2voN.png https://i.imgur.com/Ryr2voN.png
Frost1x 5y ago
Part of the issue is that FOSS, libraries, and independendent package managers and their specific repositories have exploded in about every domain. No longer are there a handful of places where software and libraries exist. Pick an ecosystem and there's probably a sub or additional levels of package/library management ecosystems below it. Developers have really bought into grabbing a package for everything and leveraging hundreds and thousands of packages, most of which have limited to no sort of vetting. We've had software complexity growing over the years, but the one benefit in previous years is that it was in fairly concentrated areas where many eyes were often watching. You could somewhat rely on the fact that someone had looked through and approved such additions to a package repo. It's a naive security but there were more professional eyes you could leverage, lowering overall risk. Not anymore, it's more of this breakneck speed, leverage every package you can to save resources and glue them together without looking at them in detail, because the entire reason you're using them is because you don't have time. It's not all shops, plenty of teams vet or roll their own functionality to avoid this but there's a large world of software out there that just blindly trusts everything down the chain in an era where there should be less trust. Some software shops have never seen a package or library they didn't like and will use even trivial to implement packages (the benefit of your own implementation being you know it's secure and won't change under your feet unless an inside threat makes the change). There's a tradeoff to externalizing costs and tech debt for maintainance you pass on using these systems, the cost being you take on more risk in various forms.
tyingq 5y ago
Doesn't installing a python package from PyPI (optionally) run some of the code in the package? Like "setup.py" ? I'd take advantage of that if I were injecting malicious code in a module.
- codefined 5y ago
  Downloading a Python package (as done by scrapers, mirrors and security analysts) does not run setup.py. Only if the module is installed is this run. It's analagous to downloading vs. running an executable.
  - tyingq 5y ago
    Ah, sure. Just making the distinction that you don't have to actually use a module within your code. That installing the module, even if you never use it in your own code, runs some of the code in that module.
  - Majestic121 5y ago
    Doing a pip install actually runs the setup.py of the package for source dist, which means running an executable. It's not the case for wheels though, so you can protect yourself by restricting to binary : --only-binary. Also doing a pip download is not sensible to this issue, but most people do pip install
- chriswarbo 5y ago
  Yep. In fact, I recently had to deal with this monstrosity https://pypi.org/project/awslambdaric https://pypi.org/project/awslambdaric whose setup.py invokes a shell script https://github.com/aws/aws-lambda-python-runtime-interface-client/blob/56eb1ca4894955bd6e017f92164d4e0f4a56cd26/setup.py#L18 https://github.com/aws/aws-lambda-python-runtime-interface-c... That shell script runs 'make && make install' on a couple of bundled dependencies, but in principle it could do anything https://github.com/aws/aws-lambda-python-runtime-interface-client/blob/56eb1ca4894955bd6e017f92164d4e0f4a56cd26/scripts/preinstall.sh https://github.com/aws/aws-lambda-python-runtime-interface-c...
- dane-pgp 5y ago
rlpb 5y ago
> Anyone can upload anything to PyPI. This is kind of like saying that you detected malicious packages on GitHub - the question is whether anyone actually ran it. There's a bigger social problem here. In many communities it has become completely normalized for any dependency to be just added from these types of "anyone can upload" repositories without any kind of due diligence as to provenance or security. It's as if these communities have just given up on that. For example, if I suggest that a modern web app only use dependencies that ship in Debian (a project that does actually take this kind of thing seriously), many would laugh me out of the building. The only practical alternative in many cases is to give up trying. It's now rare for projects to properly audit their dependencies because the community at large isn't rallying around doing it. It's a vicious circle. This kind of incident serves as a valuable regular reminder of the risks that these communities are taking. Dismissing this by saying "anyone can upload" misses the point.
- oefrha 5y ago
  > It's now rare for projects to properly audit their dependencies In the Python ecosystem, it is at least pretty easy to limit yourself to a handful of developers you trust (e.g. Django developers, Pallets developers, etc.). In the npm ecosystem however, for instance I just ran `npx create-react-app` and got a node_modules with 1044 subdirectories, a 11407-line yarn.lock, and "207 vulnerabilities found" from `yarn audit`. Well what can you possibly do.
- toss1 5y ago
  >> if I suggest that a modern web app only use dependencies that ship in Debian (a project that does actually take this kind of thing seriously), many would laugh me out of the building And you are more right, and they are more wrong than they know. Not only are malicious inserts in code a problem in themselves, if you have failed to properly vet your dependencies and it causes real losses for one of your users/customers, YOU have created a liability for yourself. Sure, many customers may never figure it out, and it might take them a while to prove it in court, but if it even gets to the point where someone is damaged and notices, and decides to do something about it, you have defense costs. The "whatever" attitude has no place in serious engineering of any kind, and anyone with a "move fast and break things" attitude (unless these tests are properly sandboxed) shows that they are not engaged in any serious engineering.

by geofft 5y ago

yonixw 5y ago

If anyone wonders about the same for NPM, it is around 400. That what happened to my almost empty package. https://i.imgur.com/Ryr2voN.png https://i.imgur.com/Ryr2voN.png

Frost1x 5y ago

Part of the issue is that FOSS, libraries, and independendent package managers and their specific repositories have exploded in about every domain. No longer are there a handful of places where software and libraries exist. Pick an ecosystem and there's probably a sub or additional levels of package/library management ecosystems below it. Developers have really bought into grabbing a package for everything and leveraging hundreds and thousands of packages, most of which have limited to no sort of vetting. We've had software complexity growing over the years, but the one benefit in previous years is that it was in fairly concentrated areas where many eyes were often watching. You could somewhat rely on the fact that someone had looked through and approved such additions to a package repo. It's a naive security but there were more professional eyes you could leverage, lowering overall risk. Not anymore, it's more of this breakneck speed, leverage every package you can to save resources and glue them together without looking at them in detail, because the entire reason you're using them is because you don't have time. It's not all shops, plenty of teams vet or roll their own functionality to avoid this but there's a large world of software out there that just blindly trusts everything down the chain in an era where there should be less trust. Some software shops have never seen a package or library they didn't like and will use even trivial to implement packages (the benefit of your own implementation being you know it's secure and won't change under your feet unless an inside threat makes the change). There's a tradeoff to externalizing costs and tech debt for maintainance you pass on using these systems, the cost being you take on more risk in various forms.

tyingq 5y ago

Doesn't installing a python package from PyPI (optionally) run some of the code in the package? Like "setup.py" ? I'd take advantage of that if I were injecting malicious code in a module.

codefined 5y ago
Downloading a Python package (as done by scrapers, mirrors and security analysts) does not run setup.py. Only if the module is installed is this run. It's analagous to downloading vs. running an executable.
- tyingq 5y ago
  Ah, sure. Just making the distinction that you don't have to actually use a module within your code. That installing the module, even if you never use it in your own code, runs some of the code in that module.
- Majestic121 5y ago
  Doing a pip install actually runs the setup.py of the package for source dist, which means running an executable. It's not the case for wheels though, so you can protect yourself by restricting to binary : --only-binary. Also doing a pip download is not sensible to this issue, but most people do pip install
chriswarbo 5y ago
Yep. In fact, I recently had to deal with this monstrosity https://pypi.org/project/awslambdaric https://pypi.org/project/awslambdaric whose setup.py invokes a shell script https://github.com/aws/aws-lambda-python-runtime-interface-client/blob/56eb1ca4894955bd6e017f92164d4e0f4a56cd26/setup.py#L18 https://github.com/aws/aws-lambda-python-runtime-interface-c... That shell script runs 'make && make install' on a couple of bundled dependencies, but in principle it could do anything https://github.com/aws/aws-lambda-python-runtime-interface-client/blob/56eb1ca4894955bd6e017f92164d4e0f4a56cd26/scripts/preinstall.sh https://github.com/aws/aws-lambda-python-runtime-interface-c...
dane-pgp 5y ago

rlpb 5y ago

> Anyone can upload anything to PyPI. This is kind of like saying that you detected malicious packages on GitHub - the question is whether anyone actually ran it. There's a bigger social problem here. In many communities it has become completely normalized for any dependency to be just added from these types of "anyone can upload" repositories without any kind of due diligence as to provenance or security. It's as if these communities have just given up on that. For example, if I suggest that a modern web app only use dependencies that ship in Debian (a project that does actually take this kind of thing seriously), many would laugh me out of the building. The only practical alternative in many cases is to give up trying. It's now rare for projects to properly audit their dependencies because the community at large isn't rallying around doing it. It's a vicious circle. This kind of incident serves as a valuable regular reminder of the risks that these communities are taking. Dismissing this by saying "anyone can upload" misses the point.

oefrha 5y ago
> It's now rare for projects to properly audit their dependencies In the Python ecosystem, it is at least pretty easy to limit yourself to a handful of developers you trust (e.g. Django developers, Pallets developers, etc.). In the npm ecosystem however, for instance I just ran `npx create-react-app` and got a node_modules with 1044 subdirectories, a 11407-line yarn.lock, and "207 vulnerabilities found" from `yarn audit`. Well what can you possibly do.
toss1 5y ago
>> if I suggest that a modern web app only use dependencies that ship in Debian (a project that does actually take this kind of thing seriously), many would laugh me out of the building And you are more right, and they are more wrong than they know. Not only are malicious inserts in code a problem in themselves, if you have failed to properly vet your dependencies and it causes real losses for one of your users/customers, YOU have created a liability for yourself. Sure, many customers may never figure it out, and it might take them a while to prove it in court, but if it even gets to the point where someone is damaged and notices, and decides to do something about it, you have defense costs. The "whatever" attitude has no place in serious engineering of any kind, and anyone with a "move fast and break things" attitude (unless these tests are properly sandboxed) shows that they are not engaged in any serious engineering.