6 ms·

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I

by danjoc 9y ago

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosquatting problems. That's not a coincidence. That's also a strong reason why Java still dominates the enterprise. If you (the reader) are a PyPi/NPM user, I challenge you to watch this, https://www.youtube.com/watch?v=pBJafU0p_Nk https://www.youtube.com/watch?v=pBJafU0p_Nk and tell me why you shouldn't use a repository manager like Sonatype Nexus which validates package signatures, checks licenses, and does vulnerability scanning. By itself, package signatures aren't the holy grail? That's not the point. Security is achieved in layers. Too bad they aren't mature enough to understand that.

tclancy 9y ago
>I get the impression they don't care about security at all. They seem like children I had the absolute pleasure of working with dstufft for 6 or 7 months years ago and learned a ton from his breadth of knowledge and willingness to teach (and I had more than a decade of experience at the time). I can appreciate being unhappy with the situation and not agreeing with the approach, but personal attacks and assuming ill intent don't help. It is, of course, possible the developers actually do pay attention to the details and just see the issue better than you do. I read that link at the time and I do wonder how you would address the issues he raises. The concern seems not so much not wanting to apply security but actually thinking about the underlying issues. Which is worse: an open but dangerous Pypi or one with a bunch of security theater around keys and hashes and other stuff everyone pays lip service to but never actually checks?
cookiecaper 9y ago
Typosquatting and package signatures are separate issues. Package signing only prevents typosquatting insofar as either the user or some intermediate layer resolves the typo to the intended package. If someone was going to this effort, they'd probably go to the effort of double-checking the package name before installation anyway. PyPi needs moderators to sit in the middle and remove anything that is obviously malicious, whether the packages are signed or not. Bad guys can sign packages just as easily as good guys. Software should also be used to correct likely typos, perhaps including checking against a blacklist of known-bad package hashes, before the package is installed. Yes, these approaches are imperfect, but they are better than doing nothing. "Perfect is the enemy of good".
- xapata 9y ago
  > needs moderators Like Anaconda and Enthought? And countless internal departments? Or are you suggesting folks donate to the PSF and they hire a team?
  - cookiecaper 9y ago
    I'm suggesting that whoever currently has admin rights on the PyPi packaging servers, and it's someone, take responsibility for this and physically remove the typosquatting libs from the lookup mechanism. "We need donations before we can do that" doesn't pass muster as far as I'm concerned; leaving this unaddressed is an existential issue for PyPI. There are already privately-maintained repositories and that's great, but IMO it's not an excuse for PyPi to leave this vulnerability open.
    - xapata 9y ago
      Ah, so you're going to refuse to use PyPI until someone volunteers. I'm not certain that's an existential issue for PyPI. I plan to continue using it.
zzzeek 9y ago
> In contrast, Maven central requires signing. forgive my ignorance and my lack of 56 minutes to watch the entire youtube video, but who are the identities behind these signatures? The blog post you reference discusses the problem both of users signing their own packages (anyone can make a signature and any malicious package author can point people at a maliciously-owned signature as well) as well as having a central key (used by organizations with employees and known contributors, does not scale to pypi's model). I'm also ignorant of a vulnerability scanner for Python (haven't looked). Does such a tool exist and have you proposed it as part of pypi's infrastructure? I am sure they'd be interested in that. I'm not sure how the license file of a product impacts the issue of it being malware or not. Took a look at https://www.sonatype.com/ https://www.sonatype.com/ and it appears to be a closed-source, commercial product - it appears to have a database of vulnerabilities in some format, but it appears to use hashes of some kind. I'm not sure how that would work against arbitrary Python source code, but again, I am ignorant. I would encourage you to write a comprehensive rebuttal to the blog post you refer towards.
- danjoc 9y ago
  >who are the identities behind these signatures? https://maven.apache.org/guides/mini/guide-central-repository-upload.html https://maven.apache.org/guides/mini/guide-central-repositor... "we require you to provide PGP signatures for all your artifacts (all files except checksums), and distribute your public key to a key server like http://pgp.mit.edu." http://pgp.mit.edu." >anyone can make a signature The article flip flops on this. any hacker can do it it's too much burden for developers >any malicious package author can point people at a maliciously-owned signature as well Anyone can also verify ownership of the key before accepting packages signed by it. This is something professionals do. This is something institutions do. This is something three letter agencies do. >Does such a tool exist There are CVEs for python. One could even scan a repository using those. Java has prebuilt tools for this. OWASP has the dependency-check plugin for Maven. Nexus uses the same information in their repository health checks. >have you proposed it as part of pypi's infrastructure? I am sure they'd be interested in that. Why would I? Given their response to signed packages, I would expect a response along the lines of "Too much burden. Too hard. Not perfect. Not worth it. Security theater. Go away. Ur dumb." >I'm not sure how the license file of a product impacts the issue of it being malware or not. It's one of those nice features of good repository management. Do python packages even list licenses? I mean, I assume they would, but then, they actively resist implementing other basic things which I would just assume they could do. Licenses change over time. Some enterprises treat GPL like a virus. Knowing ReactJS changes from Apache to BSD + Patents in a new version is as important to someone in the business as knowing if a package is compromised. >it appears to be a closed-source, commercial product Nexus OSS is open source, Nexus Professional is commercially licensed. The later has a few nice features the former does not. Both can manage PyPi, NPM, Ruby, Docker, Maven, and Nuget repos to name a few. https://www.sonatype.com/nexus-repository-oss https://www.sonatype.com/nexus-repository-oss >I would encourage you to write a comprehensive rebuttal to the blog post you refer towards It's easier to fool people than to convince them that they have been fooled. -- Mark Twain
donaldstufft 9y ago
Package signing doesn't achieve anything without a trust model behind it, which is exactly what that post states. Too many people go "we need to add some crypto to this thing!" without developing a threat model and that ends up making the crypto pointless wankery to act as a security blanket without actually solving any problems. Maven Central, to my knowledge, does not have typo squatting problems because Sonatype has a manual review process for all new projects. It has absolutely nothing to do with the fact that they allow projects to upload PGP signatures and it could not have anything to do with that, because PGP does not provide any mechanism to prevent that. For example, there may be `urllib3` which is a valid project that must be signed by key X. We'll ignore how a tool like pip would find out that key X is the right key (although this is actually the most important part of a package signing solution) and just grant that we've solved that problem. Someone then comes and registers another project, `urlib3` which must be signed by key Y. The attack that is being described here is that a user would erroneously say ``pip install urlib3`` when they meant to type ``pip install urllib3`` and pip would then fetch that and download the package and install it. I think it is pretty obvious that signing doesn't help here, because pip doesn't know that the user really wanted urllib3 and not urlib3, so it can only determine that urlib3 is supposed to be signed by key Y (which of course, the hypothetical malicious person controlling urlib3 would have), fetch the package and verify it's signature. There is only one tried and true method for preventing across the board this kind of human introduced error collision (aka typo squatting), and that is manual review of all new projects. The problem with manual review then becomes one scale. There are as of this time of writing 117,226 unique projects on PyPI with an average growth of around 100 new projects a day. In addition there are zero full time developers or operations or support people working on PyPI. There is one part time paid person (me), plus my unpaid time, plus one other part time unpaid developer/ops person who do the vast bulk of the work. There is simply not enough available bandwidth to process 100 new projects every day and to validate them for typo squatting/confusion possibilities. Beyond that, there are a number of possible heuristic based approaches that can try to reduce the chance of this from happening such as using levinstein distance, unicode confusables, attempting develop "reputation", etc. Most of these are either so broad as to catch a lot of projects which are not typo squatting but are real, actual different things or are so narrow as to be trivially defeated. That's not to say they aren't worthwhile or there isn't an idea that would make sense but focusing on that has not been a priority for a largely volunteer based organization because there are lower hanging fruit that are more impactful , because at the end of the day without a manual review system individual end users are still ultimately responsible for ensuring they're asking for the correct thing (and even beyond that, they're responsible for ensuring that the thing they're asking be installed is something that satisfies their own security constraints). Security is achieved by layering multiple secure systems on top of each other, not by randomly rubbing crypto on things because it makes you feel good to have crypto involved.
Ajedi32 9y ago
> https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://caremad.io/posts/2013/07/packaging-signing-not-holy-... Thanks, that's actually a great article that explains very well why you can't just throw signatures at the problem and claim that fixes everything. As other commenters have pointed out, the reason Maven central doesn't have this problem has nothing to do with signatures, and everything to do with the fact that all new packages must undergo manual review, which is unfortunately a solution that doesn't scale. (See the "Linux Has Packaging Signing, Let’s Steal Theirs" section from the article you linked.)

by danjoc 9y ago

tclancy 9y ago

>I get the impression they don't care about security at all. They seem like children I had the absolute pleasure of working with dstufft for 6 or 7 months years ago and learned a ton from his breadth of knowledge and willingness to teach (and I had more than a decade of experience at the time). I can appreciate being unhappy with the situation and not agreeing with the approach, but personal attacks and assuming ill intent don't help. It is, of course, possible the developers actually do pay attention to the details and just see the issue better than you do. I read that link at the time and I do wonder how you would address the issues he raises. The concern seems not so much not wanting to apply security but actually thinking about the underlying issues. Which is worse: an open but dangerous Pypi or one with a bunch of security theater around keys and hashes and other stuff everyone pays lip service to but never actually checks?

cookiecaper 9y ago

Typosquatting and package signatures are separate issues. Package signing only prevents typosquatting insofar as either the user or some intermediate layer resolves the typo to the intended package. If someone was going to this effort, they'd probably go to the effort of double-checking the package name before installation anyway. PyPi needs moderators to sit in the middle and remove anything that is obviously malicious, whether the packages are signed or not. Bad guys can sign packages just as easily as good guys. Software should also be used to correct likely typos, perhaps including checking against a blacklist of known-bad package hashes, before the package is installed. Yes, these approaches are imperfect, but they are better than doing nothing. "Perfect is the enemy of good".

xapata 9y ago
> needs moderators Like Anaconda and Enthought? And countless internal departments? Or are you suggesting folks donate to the PSF and they hire a team?
- cookiecaper 9y ago
  I'm suggesting that whoever currently has admin rights on the PyPi packaging servers, and it's someone, take responsibility for this and physically remove the typosquatting libs from the lookup mechanism. "We need donations before we can do that" doesn't pass muster as far as I'm concerned; leaving this unaddressed is an existential issue for PyPI. There are already privately-maintained repositories and that's great, but IMO it's not an excuse for PyPi to leave this vulnerability open.
  - xapata 9y ago
    Ah, so you're going to refuse to use PyPI until someone volunteers. I'm not certain that's an existential issue for PyPI. I plan to continue using it.

zzzeek 9y ago

> In contrast, Maven central requires signing. forgive my ignorance and my lack of 56 minutes to watch the entire youtube video, but who are the identities behind these signatures? The blog post you reference discusses the problem both of users signing their own packages (anyone can make a signature and any malicious package author can point people at a maliciously-owned signature as well) as well as having a central key (used by organizations with employees and known contributors, does not scale to pypi's model). I'm also ignorant of a vulnerability scanner for Python (haven't looked). Does such a tool exist and have you proposed it as part of pypi's infrastructure? I am sure they'd be interested in that. I'm not sure how the license file of a product impacts the issue of it being malware or not. Took a look at https://www.sonatype.com/ https://www.sonatype.com/ and it appears to be a closed-source, commercial product - it appears to have a database of vulnerabilities in some format, but it appears to use hashes of some kind. I'm not sure how that would work against arbitrary Python source code, but again, I am ignorant. I would encourage you to write a comprehensive rebuttal to the blog post you refer towards.

danjoc 9y ago
>who are the identities behind these signatures? https://maven.apache.org/guides/mini/guide-central-repository-upload.html https://maven.apache.org/guides/mini/guide-central-repositor... "we require you to provide PGP signatures for all your artifacts (all files except checksums), and distribute your public key to a key server like http://pgp.mit.edu." http://pgp.mit.edu." >anyone can make a signature The article flip flops on this. any hacker can do it it's too much burden for developers >any malicious package author can point people at a maliciously-owned signature as well Anyone can also verify ownership of the key before accepting packages signed by it. This is something professionals do. This is something institutions do. This is something three letter agencies do. >Does such a tool exist There are CVEs for python. One could even scan a repository using those. Java has prebuilt tools for this. OWASP has the dependency-check plugin for Maven. Nexus uses the same information in their repository health checks. >have you proposed it as part of pypi's infrastructure? I am sure they'd be interested in that. Why would I? Given their response to signed packages, I would expect a response along the lines of "Too much burden. Too hard. Not perfect. Not worth it. Security theater. Go away. Ur dumb." >I'm not sure how the license file of a product impacts the issue of it being malware or not. It's one of those nice features of good repository management. Do python packages even list licenses? I mean, I assume they would, but then, they actively resist implementing other basic things which I would just assume they could do. Licenses change over time. Some enterprises treat GPL like a virus. Knowing ReactJS changes from Apache to BSD + Patents in a new version is as important to someone in the business as knowing if a package is compromised. >it appears to be a closed-source, commercial product Nexus OSS is open source, Nexus Professional is commercially licensed. The later has a few nice features the former does not. Both can manage PyPi, NPM, Ruby, Docker, Maven, and Nuget repos to name a few. https://www.sonatype.com/nexus-repository-oss https://www.sonatype.com/nexus-repository-oss >I would encourage you to write a comprehensive rebuttal to the blog post you refer towards It's easier to fool people than to convince them that they have been fooled. -- Mark Twain

donaldstufft 9y ago

Package signing doesn't achieve anything without a trust model behind it, which is exactly what that post states. Too many people go "we need to add some crypto to this thing!" without developing a threat model and that ends up making the crypto pointless wankery to act as a security blanket without actually solving any problems. Maven Central, to my knowledge, does not have typo squatting problems because Sonatype has a manual review process for all new projects. It has absolutely nothing to do with the fact that they allow projects to upload PGP signatures and it could not have anything to do with that, because PGP does not provide any mechanism to prevent that. For example, there may be `urllib3` which is a valid project that must be signed by key X. We'll ignore how a tool like pip would find out that key X is the right key (although this is actually the most important part of a package signing solution) and just grant that we've solved that problem. Someone then comes and registers another project, `urlib3` which must be signed by key Y. The attack that is being described here is that a user would erroneously say ``pip install urlib3`` when they meant to type ``pip install urllib3`` and pip would then fetch that and download the package and install it. I think it is pretty obvious that signing doesn't help here, because pip doesn't know that the user really wanted urllib3 and not urlib3, so it can only determine that urlib3 is supposed to be signed by key Y (which of course, the hypothetical malicious person controlling urlib3 would have), fetch the package and verify it's signature. There is only one tried and true method for preventing across the board this kind of human introduced error collision (aka typo squatting), and that is manual review of all new projects. The problem with manual review then becomes one scale. There are as of this time of writing 117,226 unique projects on PyPI with an average growth of around 100 new projects a day. In addition there are zero full time developers or operations or support people working on PyPI. There is one part time paid person (me), plus my unpaid time, plus one other part time unpaid developer/ops person who do the vast bulk of the work. There is simply not enough available bandwidth to process 100 new projects every day and to validate them for typo squatting/confusion possibilities. Beyond that, there are a number of possible heuristic based approaches that can try to reduce the chance of this from happening such as using levinstein distance, unicode confusables, attempting develop "reputation", etc. Most of these are either so broad as to catch a lot of projects which are not typo squatting but are real, actual different things or are so narrow as to be trivially defeated. That's not to say they aren't worthwhile or there isn't an idea that would make sense but focusing on that has not been a priority for a largely volunteer based organization because there are lower hanging fruit that are more impactful , because at the end of the day without a manual review system individual end users are still ultimately responsible for ensuring they're asking for the correct thing (and even beyond that, they're responsible for ensuring that the thing they're asking be installed is something that satisfies their own security constraints). Security is achieved by layering multiple secure systems on top of each other, not by randomly rubbing crypto on things because it makes you feel good to have crypto involved.

Ajedi32 9y ago

> https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://caremad.io/posts/2013/07/packaging-signing-not-holy-... Thanks, that's actually a great article that explains very well why you can't just throw signatures at the problem and claim that fixes everything. As other commenters have pointed out, the reason Maven central doesn't have this problem has nothing to do with signatures, and everything to do with the fact that all new packages must undergo manual review, which is unfortunately a solution that doesn't scale. (See the "Linux Has Packaging Signing, Let’s Steal Theirs" section from the article you linked.)