8 ms·

Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last

by hannob 9y ago

Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last year someone wrote his thesis describing python typosquatting and standard library name squatting: http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana... However after that the packages used in this thesis - the most successful one being urllib2 - weren't blocked, they were deleted. Benjamin Bach was able to register urllib2 afterwards. Benjamin and I decided that we'd now try to register as many stdlib names as possible. See also: https://www.pytosquatting.org/ https://www.pytosquatting.org/

edraferi 9y ago
I appreciate the proactive approach. Is your project the author of the packages identified by NBU? If so: (1) Why is the tracking pingback obfuscated? (2) Why does the code include a cheeky hello instead of a link to https://www.pytosquatting.org/ https://www.pytosquatting.org/ ? (3) Why is there not a visible warning when installing one of these packages? ================= edit: Reading through the linked blog post [0], it appears these researchers used different code that DID provide visible warning and an cleartext pingback. It also collected command history and hardware information. [0] http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...
- hannob 9y ago
  We're not the authors of those packages. But we own many others. 1. We're not obfuscating pingbacks. 2./3. We're raising an exception with an explanation and a link. Just look at the code of one of our packages: https://pypi.python.org/pypi/codecs https://pypi.python.org/pypi/codecs The research in 2016 was done by someone else. The kinda crazy thing is: Some of the package names he used were made available again after that instead of being blocked... And now we own them.
  - edraferi 9y ago
    Man. You're right - that's a mess.
    - chupapuma 9y ago
      And just to be super clear, this is the code. Hard from scary or obfuscated: html = urllib_request.urlopen( "https://www.pytosquatting.org/pingback/pypi/{}/".format(package_name) ) raise Exception( "This is a bogus package that should not be installed\n\n" "Please read https://www.pytosquatting.org" )
      - deleted 9y ago
        [deleted]
      - sounds 9y ago
        "Hard from scary or obfuscated" Maybe you are typing on mobile? "Far from scary or obfuscated" reads more clearly.
hodgesrm 9y ago
This is a scary attack. One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. I can't think of a good reason why code in any well-behaved application should be allowed to make random outbound network calls. I think it's also on app developers to rethink the culture of randomly grabbing packages to build applications quickly. This is already a security problem even with approved repos. Having a rat's nest of packages makes it hard to upgrade quickly when those repos post updates to address vulnerabilities. Edit: Removed confusing statement about return connections
- zokier 9y ago
  > One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. That breaks down very quickly with the combination of public CDNs and TLS. I suppose you could do SNI based firewalling, but that is bit ugly, and afaik you can't do that easily with common firewalls (like netfilter).
  - katastic 9y ago
    This. If it was that easy we'd already be doing it on all our desktops and servers.
  - hodgesrm 9y ago
    As I mentioned above some of this goes back to application design. Ideally if you are layered correctly the application components of the system that own data should not talk to the outside world except in very narrowly circumscribed ways. Unfortunately most real world systems that have internet access are created in far from ideal conditions.
- philsnow 9y ago
  This is probably a good mitigation anyway, but it's not enough: a malicious package could include an (obfuscated) API key that allows it to exfil whatever data it wants to an s3 bucket or similar. Unless you're super gung-ho about using Service Control Policies _and_ your service doesn't otherwise need to access s3, then there isn't really a way to block this.
tclancy 9y ago
>both the pypi devs and the python security team decided to ignore it. Without trying to downplay the seriousness, I think that's a less-than-charitable take on the bind they felt they were in. It also sounds like there's a crack team of dedicated devs sitting around waiting for something to work on.
- danjoc 9y ago
  When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosquatting problems. That's not a coincidence. That's also a strong reason why Java still dominates the enterprise. If you (the reader) are a PyPi/NPM user, I challenge you to watch this, https://www.youtube.com/watch?v=pBJafU0p_Nk https://www.youtube.com/watch?v=pBJafU0p_Nk and tell me why you shouldn't use a repository manager like Sonatype Nexus which validates package signatures, checks licenses, and does vulnerability scanning. By itself, package signatures aren't the holy grail? That's not the point. Security is achieved in layers. Too bad they aren't mature enough to understand that.
  - tclancy 9y ago
    >I get the impression they don't care about security at all. They seem like children I had the absolute pleasure of working with dstufft for 6 or 7 months years ago and learned a ton from his breadth of knowledge and willingness to teach (and I had more than a decade of experience at the time). I can appreciate being unhappy with the situation and not agreeing with the approach, but personal attacks and assuming ill intent don't help. It is, of course, possible the developers actually do pay attention to the details and just see the issue better than you do. I read that link at the time and I do wonder how you would address the issues he raises. The concern seems not so much not wanting to apply security but actually thinking about the underlying issues. Which is worse: an open but dangerous Pypi or one with a bunch of security theater around keys and hashes and other stuff everyone pays lip service to but never actually checks?
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
jwilk 9y ago
> Being total jerks, we have a pingback in the setup.py of all packages Ugh. Yes, you are being jerks. The ethical way to collect statistics would be to ask the victim to click the pingback link.

by hannob 9y ago

edraferi 9y ago

I appreciate the proactive approach. Is your project the author of the packages identified by NBU? If so: (1) Why is the tracking pingback obfuscated? (2) Why does the code include a cheeky hello instead of a link to https://www.pytosquatting.org/ https://www.pytosquatting.org/ ? (3) Why is there not a visible warning when installing one of these packages? ================= edit: Reading through the linked blog post [0], it appears these researchers used different code that DID provide visible warning and an cleartext pingback. It also collected command history and hardware information. [0] http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...

hannob 9y ago
We're not the authors of those packages. But we own many others. 1. We're not obfuscating pingbacks. 2./3. We're raising an exception with an explanation and a link. Just look at the code of one of our packages: https://pypi.python.org/pypi/codecs https://pypi.python.org/pypi/codecs The research in 2016 was done by someone else. The kinda crazy thing is: Some of the package names he used were made available again after that instead of being blocked... And now we own them.
- edraferi 9y ago
  Man. You're right - that's a mess.
  - chupapuma 9y ago
    And just to be super clear, this is the code. Hard from scary or obfuscated: html = urllib_request.urlopen( "https://www.pytosquatting.org/pingback/pypi/{}/".format(package_name) ) raise Exception( "This is a bogus package that should not be installed\n\n" "Please read https://www.pytosquatting.org" )
    - deleted 9y ago
      [deleted]
    - sounds 9y ago
      "Hard from scary or obfuscated" Maybe you are typing on mobile? "Far from scary or obfuscated" reads more clearly.

hodgesrm 9y ago

This is a scary attack. One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. I can't think of a good reason why code in any well-behaved application should be allowed to make random outbound network calls. I think it's also on app developers to rethink the culture of randomly grabbing packages to build applications quickly. This is already a security problem even with approved repos. Having a rat's nest of packages makes it hard to upgrade quickly when those repos post updates to address vulnerabilities. Edit: Removed confusing statement about return connections

zokier 9y ago
> One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. That breaks down very quickly with the combination of public CDNs and TLS. I suppose you could do SNI based firewalling, but that is bit ugly, and afaik you can't do that easily with common firewalls (like netfilter).
- katastic 9y ago
  This. If it was that easy we'd already be doing it on all our desktops and servers.
- hodgesrm 9y ago
  As I mentioned above some of this goes back to application design. Ideally if you are layered correctly the application components of the system that own data should not talk to the outside world except in very narrowly circumscribed ways. Unfortunately most real world systems that have internet access are created in far from ideal conditions.
philsnow 9y ago
This is probably a good mitigation anyway, but it's not enough: a malicious package could include an (obfuscated) API key that allows it to exfil whatever data it wants to an s3 bucket or similar. Unless you're super gung-ho about using Service Control Policies _and_ your service doesn't otherwise need to access s3, then there isn't really a way to block this.

tclancy 9y ago

>both the pypi devs and the python security team decided to ignore it. Without trying to downplay the seriousness, I think that's a less-than-charitable take on the bind they felt they were in. It also sounds like there's a crack team of dedicated devs sitting around waiting for something to work on.

danjoc 9y ago
When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosquatting problems. That's not a coincidence. That's also a strong reason why Java still dominates the enterprise. If you (the reader) are a PyPi/NPM user, I challenge you to watch this, https://www.youtube.com/watch?v=pBJafU0p_Nk https://www.youtube.com/watch?v=pBJafU0p_Nk and tell me why you shouldn't use a repository manager like Sonatype Nexus which validates package signatures, checks licenses, and does vulnerability scanning. By itself, package signatures aren't the holy grail? That's not the point. Security is achieved in layers. Too bad they aren't mature enough to understand that.
- tclancy 9y ago
  >I get the impression they don't care about security at all. They seem like children I had the absolute pleasure of working with dstufft for 6 or 7 months years ago and learned a ton from his breadth of knowledge and willingness to teach (and I had more than a decade of experience at the time). I can appreciate being unhappy with the situation and not agreeing with the approach, but personal attacks and assuming ill intent don't help. It is, of course, possible the developers actually do pay attention to the details and just see the issue better than you do. I read that link at the time and I do wonder how you would address the issues he raises. The concern seems not so much not wanting to apply security but actually thinking about the underlying issues. Which is worse: an open but dangerous Pypi or one with a bunch of security theater around keys and hashes and other stuff everyone pays lip service to but never actually checks?

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

jwilk 9y ago

> Being total jerks, we have a pingback in the setup.py of all packages Ugh. Yes, you are being jerks. The ethical way to collect statistics would be to ask the victim to click the pingback link.