4 ms·

The problem with Pypi is the transitive dependencies. You may trust a high profile package but it could pull in any number of projects that are exploited by bad

The problem with Pypi is the transitive dependencies. You may trust a high profile package but it could pull in any number of projects that are exploited by bad actors.

freeqaz 4y ago
This is a real problem that I believe will become increasingly commonly used by attackers to break into companies. Unfortunately, it's not an easy problem to automate the detection. There are some heuristics, but you're basically writing something like Norton Antivirus for Python packages lol. A sufficiently skilled attacker can make their code look totally innocent[0]. :) Source: Former security engineer at Uber/Snap that's now working on tooling in this space[1]. 0: "Underhanded C Competition" http://www.underhanded-c.org/ http://www.underhanded-c.org/ 1: Open Source "Supply Chain Security" platform https://github.com/lunasec-io/lunasec https://github.com/lunasec-io/lunasec
- jollofricepeas 4y ago
  It’s already extremely common in the Python, Ruby and Node ecosystem (see sources below). The only company that has a real chance at making a dent in the issue is GitHub and to a degree they already have. SOURCES: - https://www.mandiant.com/resources/blog/supply-chain-node-js https://www.mandiant.com/resources/blog/supply-chain-node-js - https://www.zdnet.com/article/backdoor-found-in-ruby-library-for-checking-for-strong-passwords/ https://www.zdnet.com/article/backdoor-found-in-ruby-library... - https://www.reversinglabs.com/blog/mining-for-malicious-ruby-gems https://www.reversinglabs.com/blog/mining-for-malicious-ruby... - https://news.sophos.com/en-us/2021/10/24/node-poisoning-hijacked-package-delivers-coin-miner-and-credential-stealing-backdoor/amp/ https://news.sophos.com/en-us/2021/10/24/node-poisoning-hija... - https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/amp/ https://www.bleepingcomputer.com/news/security/dev-corrupts-... - https://portswigger.net/daily-swig/amp/malicious-python-library-ctx-removed-from-pypi-repo https://portswigger.net/daily-swig/amp/malicious-python-libr...
raverbashing 4y ago
And a problem which is some 5x worse in npm
- marcosdumay 4y ago
  5x worse on each level, maybe. That composes quite quickly. Npm packages have really crazy dependency graphs. It's usually on the 1000s of times larger than what you get on most languages, 100s of times larger than python or ruby.
  - pitsnatch 4y ago
    Why is this the case?
    - raverbashing 4y ago
      Because JS had a very poor stdlib and attracted people that had little to no experience in best-practices in older languages Hence left-pad. And this pet-peeve of mine https://nodejs.org/docs/latest/api/process.html#processargv https://nodejs.org/docs/latest/api/process.html#processargv (who cares about the node fullpath in argv[0]?)
      - SpeedilyDamage 4y ago
        Why oh why do people cite left-pad like this as an example of package bloat? It’s functionality was so valuable it was included in ECMAScript 2017! The problem with left pad was a security one, not a package bloat one.
        deleted 4y ago
        [deleted]
        gfxgirl 4y ago
        No, the problem is bad developers pulling in dependencies for trivial functionally. If there was a `for-loop` npm package bad devs would be pulling it in instead of writing their own for loops. Padding on the left is something if it doesn't exist you write it in a few lines of code yourself. You don't add a package for such trivial functionality.
pydry 4y ago
I find that this is more of an npm problem - thanks to the lack of a standard library and a cultural affectation towards small packages with a narrow focus. A high level python package is more likely to depend upon 3-30 well known packages.
- danenania 4y ago
  I think the biggest factor is just a much larger overall number of packages in the ecosystem. If there are, on average, 20 popular packages for solving a given problem instead of 3, you end up with exponentially more transitive dependencies.
wil421 4y ago
The company I work for hosts their own PyPi mirror internally and will attempt to ban packages or dependencies before PyPi does.
- nerdponx 4y ago
  I think more companies should do this.
jlarocco 4y ago
But that's how trust works. If you don't trust them to pull in safe dependencies, then you don't really trust them.
- warkdarrior 4y ago
  There are various degrees of trust. One may trust a package maintainer to write correct, well-designed code and to maintain it. But should one trust that package maintainer to have solid security standards in their code repo, their CI/CD, and their distribution mechanism?
Fatnino 4y ago
So, actually you don't really trust them...

The problem with Pypi is the transitive dependencies. You may trust a high profile package but it could pull in any number of projects that are exploited by bad actors.

freeqaz 4y ago

This is a real problem that I believe will become increasingly commonly used by attackers to break into companies. Unfortunately, it's not an easy problem to automate the detection. There are some heuristics, but you're basically writing something like Norton Antivirus for Python packages lol. A sufficiently skilled attacker can make their code look totally innocent[0]. :) Source: Former security engineer at Uber/Snap that's now working on tooling in this space[1]. 0: "Underhanded C Competition" http://www.underhanded-c.org/ http://www.underhanded-c.org/ 1: Open Source "Supply Chain Security" platform https://github.com/lunasec-io/lunasec https://github.com/lunasec-io/lunasec

jollofricepeas 4y ago
It’s already extremely common in the Python, Ruby and Node ecosystem (see sources below). The only company that has a real chance at making a dent in the issue is GitHub and to a degree they already have. SOURCES: - https://www.mandiant.com/resources/blog/supply-chain-node-js https://www.mandiant.com/resources/blog/supply-chain-node-js - https://www.zdnet.com/article/backdoor-found-in-ruby-library-for-checking-for-strong-passwords/ https://www.zdnet.com/article/backdoor-found-in-ruby-library... - https://www.reversinglabs.com/blog/mining-for-malicious-ruby-gems https://www.reversinglabs.com/blog/mining-for-malicious-ruby... - https://news.sophos.com/en-us/2021/10/24/node-poisoning-hijacked-package-delivers-coin-miner-and-credential-stealing-backdoor/amp/ https://news.sophos.com/en-us/2021/10/24/node-poisoning-hija... - https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/amp/ https://www.bleepingcomputer.com/news/security/dev-corrupts-... - https://portswigger.net/daily-swig/amp/malicious-python-library-ctx-removed-from-pypi-repo https://portswigger.net/daily-swig/amp/malicious-python-libr...

raverbashing 4y ago

And a problem which is some 5x worse in npm

marcosdumay 4y ago
5x worse on each level, maybe. That composes quite quickly. Npm packages have really crazy dependency graphs. It's usually on the 1000s of times larger than what you get on most languages, 100s of times larger than python or ruby.
- pitsnatch 4y ago
  Why is this the case?
  - raverbashing 4y ago
    Because JS had a very poor stdlib and attracted people that had little to no experience in best-practices in older languages Hence left-pad. And this pet-peeve of mine https://nodejs.org/docs/latest/api/process.html#processargv https://nodejs.org/docs/latest/api/process.html#processargv (who cares about the node fullpath in argv[0]?)
    - SpeedilyDamage 4y ago
      Why oh why do people cite left-pad like this as an example of package bloat? It’s functionality was so valuable it was included in ECMAScript 2017! The problem with left pad was a security one, not a package bloat one.
      - deleted 4y ago
        [deleted]
      - gfxgirl 4y ago
        No, the problem is bad developers pulling in dependencies for trivial functionally. If there was a `for-loop` npm package bad devs would be pulling it in instead of writing their own for loops. Padding on the left is something if it doesn't exist you write it in a few lines of code yourself. You don't add a package for such trivial functionality.

pydry 4y ago

I find that this is more of an npm problem - thanks to the lack of a standard library and a cultural affectation towards small packages with a narrow focus. A high level python package is more likely to depend upon 3-30 well known packages.

danenania 4y ago
I think the biggest factor is just a much larger overall number of packages in the ecosystem. If there are, on average, 20 popular packages for solving a given problem instead of 3, you end up with exponentially more transitive dependencies.

wil421 4y ago

The company I work for hosts their own PyPi mirror internally and will attempt to ban packages or dependencies before PyPi does.

nerdponx 4y ago
I think more companies should do this.

jlarocco 4y ago

But that's how trust works. If you don't trust them to pull in safe dependencies, then you don't really trust them.

warkdarrior 4y ago
There are various degrees of trust. One may trust a package maintainer to write correct, well-designed code and to maintain it. But should one trust that package maintainer to have solid security standards in their code repo, their CI/CD, and their distribution mechanism?

Fatnino 4y ago

So, actually you don't really trust them...