18 ms·

Malicious software libraries found in PyPI posing as well known libraries

475 points by nariinano 9y ago

a3n 9y ago
Dry run?
- b101010 9y ago
  The "malicious" code at the end of the advisory looks like nothing more than a beacon announcing it was installed? edit: get current working directory get username get hostname concatenate the last 3 together obfuscate(/encrypt?) this string send the result as a http request to 121.42.217.44 (the value of the base64 string)
  - julianwachholz 9y ago
    # Welcome Here! :) # just toy, no harm :)
rantanplan 9y ago
The regex they have for identifying fake/harmful packages is wrong. `pip list –format=legacy | egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib) '` This incorrectly lists `urllib3` or the `cryptography` package for example, which are perfectly valid packages. [UPDATE] Read "tobltobs" comment below. I incorrectly removed a trailing space from the regex.
- nariinano 9y ago
  I believe urllib3 is built-in. So if you have installed it from PyPI you've gotten a malicious version.
  - rantanplan 9y ago
    https://pypi.python.org/pypi/urllib3 https://pypi.python.org/pypi/urllib3
  - cpburns2009 9y ago
    urllib and urllib2 are built-in for Python 2, and were merged and reorganized as just urllib in Python 3. urllib3 is a third-party module.
    - haikuginger 9y ago
      This is correct. In general, though, most packages don't rely on urllib3 directly, but on `requests`, which uses urllib3 but provides a friendlier API and built-in SSL cert verification.
  - wyldfire 9y ago
    It's not generally true that built-in packages which also appear on PyPI are malicious. Many batteries-included packages are also maintained outside of CPython. This is because: (1) in many cases they existed outside prior to being included in CPython, (2) they can experiment with new features before they're included in the CPython version of their package.
- deleted 9y ago
  [deleted]
- tobltobs 9y ago
  Not for me. There is space at the end between the closing bracket and the apostrophe. Maybe you did remove this space when you corrected the smart apostrophes.
  - rantanplan 9y ago
    You're right. It seems I did remove the space. When I put it back in it doesn't print anything.
- julianj 9y ago
  xml should be added to this list. https://pypkg.com/pypi/xml/f/setup.py https://pypkg.com/pypi/xml/f/setup.py
- osteele 9y ago
  Conda users: Here's a script that runs this check against each environment: https://gist.github.com/osteele/198b50a2a208e5bc7e5fb8d010cf590c https://gist.github.com/osteele/198b50a2a208e5bc7e5fb8d010cf...
- jastr 9y ago
  pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'
  - pmoriarty 9y ago
    When running that command, I get output like this: grep: alabaster: No such file or directory grep: appdirs: No such file or directory grep: arandr: No such file or directory for dozens and dozens of packages. Are those errors benign?
    - jwilk 9y ago
      No, jastr's command is wrong.
      - jastr 9y ago
        Whoops, remove the xargs
sdiepend 9y ago
Related to this? http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...
- xnyhps 9y ago
  That one is even more malicious, it uploads contents of your ~/.bash_history and system profile. But at least it notifies you afterwards...
  - edraferi 9y ago
    Yes, but they do filter bash_history client side, only transmitting pip-related commands. They did this to find additional common typos. The relevant code: def get_command_history(): if os.name == 'nt': # handle windows # http://serverfault.com/questions/95404/ #is-there-a-global-persistent-cmd-history # apparently, there is no history in windows :( return '' elif os.name == 'posix': # handle linux and mac cmd = 'cat {}/.bash_history | grep -E "pip[23]? install"' return os.popen(cmd.format(os.path.expanduser('~'))).read()
dpflan 9y ago
This is interesting in conjunction with the recent post about Python's popularity because that may be a weakness exploited here [1.]. It's easy to use and install and get libraries for anything, and apparently libraries for infecting your machine :(. [1.] https://news.ycombinator.com/item?id=15249348 https://news.ycombinator.com/item?id=15249348
- nariinano 9y ago
  The problem is that the vetting process of PyPI is completely inexistant. This has happened many times in the past, the last time I remember they uploaded a few libraries called "bs4" and stuff like that.
thearn4 9y ago
It looks like the code phones home to a server in China: IP: 121.42.217.44 Decimal: 2032851244 Hostname: 121.42.217.44 ASN: 37963 ISP: Hangzhou Alibaba Advertising Co.,Ltd. Organization: Hangzhou Alibaba Advertising Co.,Ltd. Services: None detected Type: Broadband Assignment: Static IP Blacklist: Click to Check Blacklist Status Continent: Asia Country: China cn flag State/Region: Zhejiang City: Hangzhou Latitude: 30.2936 (30° 17′ 36.96″ N) Longitude: 120.1614 (120° 9′ 41.04″ E)
- raverbashing 9y ago
  I wonder what would happen if the return payload had some data that would trigger the GFoC
- sdiepend 9y ago
  When you go this adress http://121.42.217.44:8080/ http://121.42.217.44:8080/ "Hi bro :) Welcome Here! Leave Messages via HTTP Log Please :)"
  - a_participant 9y ago
    Perhaps they have a zero day telnet client or browser exploit. :)
chatmasta 9y ago
Package managers seem to be an increasingly popular attack vector. It's only luck that none of the attacks have been particularly malicious yet. Considering how many package manager downloads go to a server in a datacenter, a widely distributed malicious package could control a botnet with extremely high throughput, or wreak havoc on any databases it comes into contact with. It's only a matter of time before something like this happens. A big part of the problem is that application package managers, like pip or npm, are far less sophisticated than those of operating systems, like aptitude or yum. It needs to be easy for developers to open source their code, and to mark dependencies with precise commit hashes, but the download also needs to be secure and verifiable. There are many difficult tradeoffs to consider in terms of usability, centralization, security and trust.
- raesene6 9y ago
  Another fun fact to consider is that with many package formats, you can execute arbitrary code at install time so if a malicious package can get into a repository, it's very likely to start compromising systems quickly. Whilst a package manager repo. compromise would be the biggest bang in terms of attack, compromising the credentials of the developers of popualar libraries would be an easier attack (and indeed is already happening https://twitter.com/chrispederick/status/892768218162487300 https://twitter.com/chrispederick/status/892768218162487300)
  - ams6110 9y ago
    Doubly so since when installed on a server very often it will be done as "sudo pip install ....." (/s/pip/other-package-manager/ as needed)
    - tekromancr 9y ago
      I almost never see this. Even on systems that are only running a single python project, I only ever see folks use virtualenv. The only time I ever see things installed with sudo is when the package is being installed in a docker container.
VMG 9y ago
how likely is it that npm and other package managers that do not use digital signatures by default are unaffected?
IgorPartola 9y ago
This to me is the nightmare scenario. Well one of the two, the other one being that a developer of an obscure library I use has their password to PyPI compromised and a bad actor uploads a backdoored version of the library. Fundamentally, the reason this is different from how thinks like Linux distos work is because Linux distros have maintainers who are in charge of making sure every new update to one of their packages is legit. I am sure you can try to sneak malicious code in, but it isn't going to be easy. I am not advocating that PyPI (and npm) adopt the same model. That would be too restrictive. But maybe just showing the number of downloads isn't the best way to assure whether the package is legit. Perhaps some kind of built in review system would be nice.
- raesene6 9y ago
  A review system unfortunately isn't likely to be practicable with current development models. npm alone has over 500,000 packages (http://www.modulecounts.com/ http://www.modulecounts.com/) so even a one time review isn't going to happen. If people want a more trusted solution the likely outcome is that they'll need to use a smaller more static set of libraries and then either do the audits themselves, or outsource that to a 3rd party. Ofc with current speeds of change and deployments, it doesn't seem likely that many companies will adopt that model.
  - mschuster91 9y ago
    > npm alone has over 500,000 packages (http://www.modulecounts.com/ http://www.modulecounts.com/) so even a one time review isn't going to happen. But at least the modules with the most downloads (webpack, react, or stuff like left-pad) could be vetted, and especially npm could implement a 2-or-more person model - basically, everyone with publish access can upload a new artifact, but to actually have it distributed to endusers, a second person would be required to sign off.
    - raesene6 9y ago
      Indeed it's not impossible to do (although full code review would be expensive/tricky/slow). The fact it hasn't been done despite the obvious risks indicates how much demand there is for this feature...
jamespo 9y ago
I wonder if it's worthwhile having a check that compares closeness of the name to existing popular packages and if so does some extended vetting.
fruiapps 9y ago
Curious to know something similar happening for Scala.
Sir_Cmpwn 9y ago
I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human being verifying the upstream and keeping malware out, and get more consistency across packages as a bonus. If you want your package added it's as simple as sending an email and provides a new avenue for people to contribute to the success of the ecosystem as package maintainers. When you make the next big thing, consider this approach.
- drdaeman 9y ago
  Maybe you're right, but I see one possible downside that is quite important. I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice. With the intermediate maintainers, that's going to get worse. I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.
  - nyrikki 9y ago
    PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file. I do wish that `--user` was the default for pip. It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression. I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.
raesene6 9y ago
This isn't, in any way, a new problem. I did a presentation on this topic for OWASP AppSecEU 2015 (https://www.youtube.com/watch?v=Wn190b4EJWk&list=PLpr-xdpM8wG-ZTcHhFfAeBthNVZVEtkg9&index=10 https://www.youtube.com/watch?v=Wn190b4EJWk&list=PLpr-xdpM8w...) and when doing the research for that I encountered cases of repo. attacks and compromise. IME the problem will continue unless the customers (e.g. companies making use of the libraries hosted) are willing to pay more for a service with higher levels of assurance. The budget required to implement additional security at scale is quite high, and probably not a good match with a free (at point of use) service.
- cdnsteve 9y ago
  I'm sure companies would pay for it. The service needs to be part of the main package service, not some third party.
  - raesene6 9y ago
    Interesting if you think that npm/Rubygems/PyPI are leaving a load of money on the table, why do you think they haven't introduced those services so far...
    - cdnsteve 9y ago
      Because their mission isn't to generate income like a traditional business. But if the income went back to the foundations, like Python Foundation, I think that would make sense.
      - raesene6 9y ago
        npm is a commercial organisation, they offer paid subscriptions but don't offer a curated package signed option...
        cdnsteve 9y ago
        Sort of a critical feature they are missing
EstDelenda 9y ago
Any method of software distribution which is not rooted in cryptographic author verification against a fine-grained, user-manageable trust store should be put bellow the sanity waterline, 20 years ago.
justinsaccount 9y ago
well shit, I guess I should have followed up on this after I noticed it 2 months ago. https://twitter.com/JustinAzoff/status/881163562739277824 https://twitter.com/JustinAzoff/status/881163562739277824
- andrew3726 9y ago
  Yup, same: https://gist.github.com/Spotlight0xff/829b7ebf32c4feec60ec44eb86b0fb3f https://gist.github.com/Spotlight0xff/829b7ebf32c4feec60ec44...
defined 9y ago
Here's something that contributes to typosquatting: the lack of responsiveness by package management organizations to claims on orphaned or unmaintainable packages. People who upload packages often leave organizations, who are then stuck with a package they can't update because the password went with the person, and the email reset link points to a now-defunct email address. Petitioning the package management team is sometimes fruitless, forcing a needless new instance of typosquatting.
- chupapuma 9y ago
  I have found the PyPI group of people to be very helpful in these cases. You also should probably, as an organization, have more than one owner of your packages. That way, unless two people leave, things aren't orphaned. We have gone as far to have a 'meta-user' that is on all packages. It is only ever used to recover a fully abandoned package.
  - defined 9y ago
    I understand you are trying to be helpful, and of course you are right, but the fact is that sometimes things fall between the cracks, especially in, say, hard-pressed startups. There are so many shoulds in the world that don't make it to dids, it reminds me of the joke about the salesman trying to sell farming improvement techniques and being turned down by the old farmer, who says, "Son, I don't farm half as good as I know how to already." Unfortunately, I have not found the PyPI group as helpful as you have. Perhaps I have been looking in the wrong places.
    - defined 9y ago
      UPDATE: I was helped out by a very nice person from PyPI, so kudos to them.
ehnto 9y ago
Part of my dislike for the Node ecosystem in particular and I am sure others have a similar problem, is the dependency trees are super complex. Because packages tend to be small and many, and each of those has their own dependencies, you can end up with hundreds of packages installed which is simply impractical to manually review. It is not node, but we do in fact manually review each package we utilize for our given language because it's feasible and worthwhile as the dependency tree is small in this ecosystem. Each and every package is a possible attack vector whether that be intentionally or just because it's poorly written and we can't simply ignore that because it's the done thing and "the community reviews them".
singularity2001 9y ago
"Success of the attack relies on negligence of the developer" How about package manager managers accept their enourmous responsabilty? urllib vs urllib2, one is a virus? Sorry but that is not "negligence of the developer"
- singularity2001 9y ago
  The least they can do is create an alias system for common libs or disallow some lib names. Another easy thing to implement would be a popularity check: "This package was only installed nnn times. Did you mean xxx, or do you want to proceed with the installation of yyy by author dev@g00gle.com?" Email verification is a must.
- bughunter3 9y ago
  There are over 100,000 packages and PyPI is run by volunteers. This is not practical. PyPI is not a curated distribution.
  - singularity2001 9y ago
    there are many ways to reduce the likelihood of malicious packages. not all of them require active curation. some can be systemic.
- zokier 9y ago
  Managing supply chain is one the basic principles of good engineering. Not properly vetting your sources is negligence. The problem of course is that computers are really good at amplifying work, including mistakes. So small mistake, like a typo, could have catastrophic impact, like injecting malware that can take over the whole system.
  - deleted 9y ago
    [deleted]
ConfucianNardin 9y ago
Relatedly: https://github.com/pypa/pypi-legacy/issues/644#issuecomment-305134745 https://github.com/pypa/pypi-legacy/issues/644#issuecomment-... Also http://evilpackage.fatezero.org/ http://evilpackage.fatezero.org/ / https://github.com/fate0/cookiecutter-evilpy-package https://github.com/fate0/cookiecutter-evilpy-package That one has neutered the call-home code by now, though.
cdnsteve 9y ago
I'm all for security but this hit a nerve with me: "Success of the attack relies on negligence of the developer, or system administrator, who does not check the name of the package thoroughly." Package managers need to do more. If they had an enterprise version that you could subscribe to monthly/annually invoice that you would get enterprises onboard, they are concerned about security and will pay. Developers like us will help encourage it. I'd rather not see some third-party "secure" package managers but make them part of PyPi and send funding to the Python foundation. They are seeking donations but that doesn't work well with businesses. Make it a monthly/yearly service.
- underko 9y ago
  There's written to check the package name and not to go through whole source code.
kasabali 9y ago
Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly. They solved the issue 2 decades ago by heavily vetting packages before accepting them into repositories. Users are allowed to add and use packages from 3rd party repositories. Maybe solution to this is creating curated repositories based on publicly open ones and using them by default (and requiring opt-in for using other repositories). Conda for Python and Stackage for Haskell seems like relevant solutions.
- mikehollinger 9y ago
  There's a certain amount of work (and therefore money) required to do this. That incremental difference is small for a well designed application, but - someone must actually vet and curate the contents of the repo. That tends to slow down execution, leading to scenarios where things like docker a year or two ago from the canonical "trusty" repo were hopelessly behind the "real" docker since docker was evolving so quickly and trusty was by design slowing down. Each commit that went into trusty required a team to submit and a team to approve. That costs money. ;-)
  - kasabali 9y ago
    It is a matter of distribution and release policy and not an inherent limitation of the model. Stable/lts/enterprise distributions have other concerns like preventing regressions and configuration or behavior changes during lifetime of release. Rolling distributions like Arch and OpenSuse Tumbleweed on the other hand can move a lot faster but still provide basic vetting wrt security and sanity of new/updated packages.
- disconnected 9y ago
  > Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly. https://www.schneier.com/blog/archives/2008/05/random_number_b.html https://www.schneier.com/blog/archives/2008/05/random_number... A.K.A., the Debian openssl Fiasco. Just one example of distros fucking up the packages from upstream and causing major havoc.
bhouston 9y ago
I bet there are quite a few malicious NPM packages that we do not know about. Is Node is used in government and military solutions? If so then the NPM ecosystem is likely targeted by state actors, and it is a sitting duck.
- thehardsphere 9y ago
  State actors do not limit themselves to government and military targets; many of them target civilians for all sorts of purposes.
kumarvvr 9y ago
Whoa urlib & urllib3. Those are pretty popular packages, especially to newbies. Hundreds of websites that teach web-scraping use those libraries. Wonder what is an effective form of protection against such attack vectors? Do digitally signed certificates fit into this usage scenario??
- zapt02 9y ago
  > Do digitally signed certificates fit into this usage scenario?? No, because either the package author would have to sign them, in which case you have to choose to trust each package author, or the repository would sign them, in which case there would be no improvement for this current issue, since the repo would sign the fake packages as well.
  - kumarvvr 9y ago
    Any way in which blockchain technology can be used? Like, the transaction becomes the act of the author uploading the code and the repo and user verify the transaction in some form?
    - peterwwillis 9y ago
      Nope. You can't solve phishing with technological means. You have to curate either a whitelist or blacklist. The best way to handle this is whitelists of trusted package maintainers and/or code authors.
      - kumarvvr 9y ago
        Ohh ok. Thanks. Great to know.
      - zapt02 9y ago
        To be fair you can do a lot with simple heurestics. New package + large number of downloads || new package || new package author = show users a warning message before installing the package.
  - 9y ago
EGreg 9y ago
This is why I am not a huge fan of using package managers. I like to understand the code we put into our platform, and vet it. And not have it change under us automatically, after that, but review the changes manually before accepting it. I felt a bit curmudgeonly but we have a responsibility at https://qbix.com/platform https://qbix.com/platform for all our apps being secure. I wanted to use repos for each package and manually git pull or hg pull them when they changed. I was finally convinced by our developers to just use package managers with version pinning. Honestly it's really hard to avoid package managers, especially for all the newer functionality such as Payment Requests or Web Push. Luckily there is version pinning. We want our clients to feel secure that we vetted ALL the code that went into the platform. So our package json (and composer.json) uses version pinning. We'd rather take a bug report and manually fix it than NO bug report and have a SHTF moment.
wiradikusuma 9y ago
Anyone know if this is also an issue for Java? I've used Maven repository for ages, and I know many big cos depend on it.
- 1ba9115454 9y ago
  It's absolutely an issue. I'm pretty sure no one is looking at every jar file added to maven to see if there's an issue. In your POM file do you have a checksum?
- thehardsphere 9y ago
  It's less of an issue, but it still could be an issue. Deployed Maven artifiacts from Central are to required to be signed with a PGP key and are only supposed to come from approved hosts. I don't know how strictly that is enforced and how hard it is to become a host, but at least there is some kind of process. Maven Central also doesn’t allow the removal of artifacts after they've been published, and every artifact requires a unique version and name. And the names are namespaced. So you don't have the issues that you see with npm, where someone can pull a package and break everything people are using, and then some third party can come in and publish anything under the exact same name. Is this model perfectly secure? No, you still have to trust that the artifact was signed by a non-malicious person from a host that was not compromised.
EGreg 9y ago
Here is the general problem with dependencies: When a dependency changes, all the projects that directly depend on it should get notified immediately and their maintainers should rush to test the new changes, to see if they break anything. There is no shortcut around this, because if B1, B2, ... Bn depend on A1, the consequences may be different for each Bk. The only real secure optimization that can be done is realizing that some of the Bk use A1 the exact same limited way and thus make an intermediate A1b that depends on A1 which those Bk's depend on. These "projection" builds may be automated by eg the set of methods called by the B's. Anyway, this is the way that iOS does it before iOS 11 comes out to users. They release a beta to all developers. And they even fix bugs in the beta before releasing to the public. Without beta testing periods, you can get laziness and just auto-accepting of whatever cane out. There is be an "alpha release" feature in git where maintainers might put out the next version to be tested by all who depend on it. THIS FEATURE SHOULD NOTIFY THE MAINTAINERS SUBSCRIBED TO THE REPO. THE BUILD ITSELF SHOULD GET ISSUES AND RATINGS FROM MAINTAINERS AS THEY TEST THE NEW BUILD. And releases should not be too frequent. This is the way to prevent bad things from happening. But that also means that the deeper the dependency is, the more levels this process could take to propagate to end-users.
julianj 9y ago
Looks like they missed one: https://pypkg.com/pypi/xml/f/setup.py https://pypkg.com/pypi/xml/f/setup.py Dork: site:https://pypkg.com https://pypkg.com intext:"just toy, no harm"
kevin_thibedeau 9y ago
Python needs a way to run 2to3 during package installation that doesn't use setup.py (setup.cfg or wheels). As it stands now, you have the hassle of building a release four times if you want to support all combos of Py2, Py3, 32-bit, and 64-bit platforms. The absent support for 2 to 3 migration in the safer alternatives is why I stick with setup.py.
- ThiefMaster- 9y ago
  No, what you need to do is fix your package's code to work on Python 2 and 3 without running 2to3 on it. The only case where this doesn't work is if you have binary extensions - but then you need separate wheels in any case.
topbagsvk 9y ago
If you want your package added it's as simple as sending an email to http://www.lvbagsgb.com http://www.lvbagsgb.com
amykhar 9y ago
some of this could be helped by intelligent naming of packages. If something is called urllib, name the package urllib because that's what people are going to look for.
hannob 9y ago
Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last year someone wrote his thesis describing python typosquatting and standard library name squatting: http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana... However after that the packages used in this thesis - the most successful one being urllib2 - weren't blocked, they were deleted. Benjamin Bach was able to register urllib2 afterwards. Benjamin and I decided that we'd now try to register as many stdlib names as possible. See also: https://www.pytosquatting.org/ https://www.pytosquatting.org/
- edraferi 9y ago
  I appreciate the proactive approach. Is your project the author of the packages identified by NBU? If so: (1) Why is the tracking pingback obfuscated? (2) Why does the code include a cheeky hello instead of a link to https://www.pytosquatting.org/ https://www.pytosquatting.org/ ? (3) Why is there not a visible warning when installing one of these packages? ================= edit: Reading through the linked blog post [0], it appears these researchers used different code that DID provide visible warning and an cleartext pingback. It also collected command history and hardware information. [0] http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...
  - hannob 9y ago
    We're not the authors of those packages. But we own many others. 1. We're not obfuscating pingbacks. 2./3. We're raising an exception with an explanation and a link. Just look at the code of one of our packages: https://pypi.python.org/pypi/codecs https://pypi.python.org/pypi/codecs The research in 2016 was done by someone else. The kinda crazy thing is: Some of the package names he used were made available again after that instead of being blocked... And now we own them.
    - edraferi 9y ago
      Man. You're right - that's a mess.
Aissen 9y ago
I'm glad Go completely sidestepped the name-rush induced by this type of package managers (composer, cpan, rvm, pypi, npm…). Just provide an URL. Done.
- eeZah7Ux 9y ago
  ...which is even less secure.
  - Aissen 9y ago
    May I ask why ? If anything, it's more secure, since you know exactly who's publishing what. Yes, it might put a higher burden on the publisher if they don't host on github/gitlab, etc. But it strips the "magic" part and makes sure the dev knows where the code is coming from.
    - eeZah7Ux 9y ago
      With packages identified by full URLs it's more likely to make a typo, or a misremember a part of the URL, or search for it on a search engine and pick a fork instead of the right one, or paste one from stack overflow or other forums that is plain false or even look legitimate due to unicode tricks. Also DNS MiTM/hijack can be used to inject a backdoor. Or the expiration of a legitimate domain.
      - krapp 9y ago
        You can also make a typo when all you need is a package name - as long as human beings have to type thigns out, that's going to be a problem. On the other hand, with a URL, you can actually inspect the code directly and (if it's hosted on Github or somewhere similar) see whether it's starred, forked or has any issues. It's not a case of URLs being less secure, it's just a tradeoff that pushes some of the security work to the community itself, rather than a dedicated staff of curators. And since most package managers eventually resolve packages to a URL somewhere, the issues you mention are probably present in other package managers, albeit hidden behind abstractions.
thedonkeycometh 9y ago
This seems to be getting more of a threat. Can't see how you prevent this sort of malicious spoofing without explicitly curating the list. Personally, I prefer the uncurated, caveat emptor approach, but this could do witha third party stepping up. (moz seems keen to do anything to get us to like them) Ultimately, sounds like something that could be solved with a blockchain of legitimisation. (ran by moz) ;)
mwerty 9y ago
How about a Levenshtein distance threshold for new package names to be accepted? I.e only allow names that are different enough from the existing set to avoid typos (or whatever errors we are trying to guard against)?
- andrewfong 9y ago
  You don't need a strict ban for this to work either. Maybe just an end-user warning if distance < N and the relative popularity of the two modules is very high. You could also allow users or organizations to explicitly whitelist some names.
elcapitan 9y ago
Would it be possible to have a general package manager (like apt) as reusable base for the individual language specific package managers? I know that npm and pip and gem etc all do some additional stuff, but at the core they all do the same (pull packages from repo, do some postinstall, resolve dependencies, maybe in some cases even check if the package is legit). So we could implement and check that once and then just reuse it like we do with many other libraries for image processing etc.
- eeZah7Ux 9y ago
  +1 to this. There's no need to reinvent the wheel a million times. At least having a shared standard on how to do packaging.
- zokier 9y ago
  See also: PackageKit.
- takluyver 9y ago
  The packaging technology is not the issue here, it's about the repository. If you made an apt repo where anyone could claim an unused name and start uploading packages, you'd have exactly the same issue.
1ba9115454 9y ago
Unless your package manager enforces signatures and you trust the person that signed the package. Then this is an attack vector for you. That includes Java (Maven), Ruby (Gems, Bundler), Node (npm), Haskel (stack) etc etc. Installing code via package managers is the coders equivelant of opening up an exe sent to you in an email. Code downloaded from the internet is not to be trusted.
- thehardsphere 9y ago
  I thought Maven enforces signatures? Though that doesn't fully mitigate the risk as you still have to trust the signer.
- anentropic 9y ago
  Signatures are good, but do not help in this case (typo-squatting)
- zokier 9y ago
  Package signing is no silver bullet. Signing packages helps against typosquatting about as much as SSL certificates help against phishing. Or in other words, not at all, especially if we don't have the certificates rooted in real world identities (like EV SSL certs).
atticusberg 9y ago
to see if you have any of these deps on your python path: pip list –format=legacy | egrep -e '^acqusition$' -e '^apidev-coop$' -e '^bzip$' -e '^crypt$' -e '^django-server$' -e '^pwd$' -e '^setup-tools$' -e '^telnet$' -e '^urlib3$' -e '^urllib$' to see if you have any projects in a given directory that require them: cat $(find /path/to/dir -name 'requirements.txt') | egrep -e '^acqusition==' -e '^apidev-coop==' -e '^bzip==' -e '^crypt==' -e '^django-server==' -e '^pwd==' -e '^setup-tools==' -e '^telnet==' -e '^urlib3==' -e '^urllib=='
- jastr 9y ago
  pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'
mwexler 9y ago
Both Anaconda (for Python, https://docs.anaconda.com/anaconda/packages/pkg-docs https://docs.anaconda.com/anaconda/packages/pkg-docs) and Microsoft (for R, https://mran.microsoft.com/ https://mran.microsoft.com/) have "reviewed and audited" collections of packages for their languages. That's part of what you pay for when you buy support for the open source tools.
pishpash 9y ago
Maybe gov.sk should be vouched for too, I mean what's the chain of trust here? Why should I trust anyone?
jastr 9y ago
To check a few you different requirements.txt files (will look 3 folders deep) find . -maxdepth 3 -name requirements.txt | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)'
- jastr 9y ago
  Avoid some false positives pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'
phonkee 9y ago
Do not forget their password that worked for couple of years: nbuSR123 ...
pishpash 9y ago
Maybe packages should be signed by several trusted maintainers. Or, noticing PyPI packages list a source code link on github sometimes, along those lines, there can be a process to prove ownership of some known online identity, keybase style. Unpopular packages can also be flagged, especially one that has a near twin that is much more popular. There are many solutions.
ris 9y ago
Hooray for the "wild west" model of package repositories. Come back maintainers & packagers, all is forgiven!
teilo 9y ago
I think we need a system to prevent this instead of the wild-west that PyPi has become. For example: Developer signatures that are checked against a community rating. If someone does `pip install` pip would look up the developer signature of the package and check a community rating that would verify this is a developer who has offered legit packages in the past. It's not foolproof, but it would go a long way towards solving this.
- zokier 9y ago
  You could add the ability for well known members to vet newbie developers, maybe by signing their key. And now you have re-invented web of trust.
  - teilo 9y ago
    I was thinking more than the ratings would handle this, rather than having to sign keys. Ratings by well-respected and vetted members would have more weight.
- wongarsu 9y ago
  That sounds easy to defeat. Make some mundane, but legit packages (maybe on of those "$X but without the pointless complexity"-packages), gain trust, once trust is reached start uploading typo-squatting packages. Knowing today's internet, programmers from cheap-labour nations (India & Co.) would soon start offering "trusted PyPi accounts" for sale on hacker forums.
  - teilo 9y ago
    Yes, but keys can also be revoked, providing a way to mitigate this.
asperous 9y ago
I once tried to upload a package called "requirements.txt" (since people do pip install requirements.txt all the time forgetting the -r). Pypi actually blocks that name from being a package!
geekamongus 9y ago
Why is there no indication of any of this on the python.org website or any of their social media accounts? I checked: https://pypi.python.org/pypi https://pypi.python.org/pypi https://www.python.org/blogs/ https://www.python.org/blogs/ http://planetpython.org/ http://planetpython.org/ https://pypi.python.org/security https://pypi.python.org/security https://twitter.com/pythoninsider https://twitter.com/pythoninsider https://plus.google.com/+Python https://plus.google.com/+Python https://www.facebook.com/pythonlang?fref=ts https://www.facebook.com/pythonlang?fref=ts https://twitter.com/ThePSF https://twitter.com/ThePSF
- takluyver 9y ago
  I guess that's because it's not a surprise. This has come up before, and it's basically unavoidable with the way PyPI is designed to work: if you see an unclaimed name, you can put whatever you want there.
  - geekamongus 9y ago
    I see your point, but I don't think that it needs to be a surprise to be announced or made well-known. No one is surprised that Microsoft releases loads of patches every Patch Tuesday, but they still publicize it and make it well known and easy for people to find out about.
- haypo 9y ago
  While there is no public announcement from the PSF yet, I sent an email to the python-dev mailing list at least to announce the issue but also try to discuss how to mitigate/prevent it. https://mail.python.org/pipermail/python-dev/2017-September/149569.html https://mail.python.org/pipermail/python-dev/2017-September/... Honestly, I am impressed that the information gone so quick! The National Security Authority of Slovakia contacted the PSRT 10 days ago. All packages were removed 1h10 after we got their email. We were discussing how to communicate about this issue, while they published an advisory. A few hours after the advisory was published, I saw the information on IRC, Twitter, LWN, etc. I didn't expect that the advisory would be published so quickly. FYI last week there was also a CPython sprint attended by more than 20 Python core developers. We were busy on discussing Python enhancements.
jwilk 9y ago
python-dev thread: https://mail.python.org/pipermail/python-dev/2017-September/149569.html https://mail.python.org/pipermail/python-dev/2017-September/...
marcinkuzminski 9y ago
This is a known problem for a while. For example when we (RhodeCode) had an installer based on pip we actually rolled our own PYPI index. Having one you host is very easy and there are nice projects in existence that allowed that. It basically solved a problem of deployment when pypi wasn't available, speed up our test installer builds, and also total control over the packages we ship.
jiffyToGo 9y ago
tiny open source project for this. https://github.com/williamforbes/pypi_hacked_names https://github.com/williamforbes/pypi_hacked_names

475 points by nariinano 9y ago

a3n 9y ago

Dry run?

b101010 9y ago
The "malicious" code at the end of the advisory looks like nothing more than a beacon announcing it was installed? edit: get current working directory get username get hostname concatenate the last 3 together obfuscate(/encrypt?) this string send the result as a http request to 121.42.217.44 (the value of the base64 string)
- julianwachholz 9y ago
  # Welcome Here! :) # just toy, no harm :)

rantanplan 9y ago

nariinano 9y ago
I believe urllib3 is built-in. So if you have installed it from PyPI you've gotten a malicious version.
- rantanplan 9y ago
  https://pypi.python.org/pypi/urllib3 https://pypi.python.org/pypi/urllib3
- cpburns2009 9y ago
  urllib and urllib2 are built-in for Python 2, and were merged and reorganized as just urllib in Python 3. urllib3 is a third-party module.
  - haikuginger 9y ago
    This is correct. In general, though, most packages don't rely on urllib3 directly, but on `requests`, which uses urllib3 but provides a friendlier API and built-in SSL cert verification.
- wyldfire 9y ago
  It's not generally true that built-in packages which also appear on PyPI are malicious. Many batteries-included packages are also maintained outside of CPython. This is because: (1) in many cases they existed outside prior to being included in CPython, (2) they can experiment with new features before they're included in the CPython version of their package.
deleted 9y ago
[deleted]
tobltobs 9y ago
Not for me. There is space at the end between the closing bracket and the apostrophe. Maybe you did remove this space when you corrected the smart apostrophes.
- rantanplan 9y ago
  You're right. It seems I did remove the space. When I put it back in it doesn't print anything.
julianj 9y ago
xml should be added to this list. https://pypkg.com/pypi/xml/f/setup.py https://pypkg.com/pypi/xml/f/setup.py
osteele 9y ago
Conda users: Here's a script that runs this check against each environment: https://gist.github.com/osteele/198b50a2a208e5bc7e5fb8d010cf590c https://gist.github.com/osteele/198b50a2a208e5bc7e5fb8d010cf...
jastr 9y ago
pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'
- pmoriarty 9y ago
  When running that command, I get output like this: grep: alabaster: No such file or directory grep: appdirs: No such file or directory grep: arandr: No such file or directory for dozens and dozens of packages. Are those errors benign?
  - jwilk 9y ago
    No, jastr's command is wrong.
    - jastr 9y ago
      Whoops, remove the xargs

sdiepend 9y ago

Related to this? http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...

xnyhps 9y ago
That one is even more malicious, it uploads contents of your ~/.bash_history and system profile. But at least it notifies you afterwards...
- edraferi 9y ago
  Yes, but they do filter bash_history client side, only transmitting pip-related commands. They did this to find additional common typos. The relevant code: def get_command_history(): if os.name == 'nt': # handle windows # http://serverfault.com/questions/95404/ #is-there-a-global-persistent-cmd-history # apparently, there is no history in windows :( return '' elif os.name == 'posix': # handle linux and mac cmd = 'cat {}/.bash_history | grep -E "pip[23]? install"' return os.popen(cmd.format(os.path.expanduser('~'))).read()

dpflan 9y ago

This is interesting in conjunction with the recent post about Python's popularity because that may be a weakness exploited here [1.]. It's easy to use and install and get libraries for anything, and apparently libraries for infecting your machine :(. [1.] https://news.ycombinator.com/item?id=15249348 https://news.ycombinator.com/item?id=15249348

nariinano 9y ago
The problem is that the vetting process of PyPI is completely inexistant. This has happened many times in the past, the last time I remember they uploaded a few libraries called "bs4" and stuff like that.

thearn4 9y ago

It looks like the code phones home to a server in China: IP: 121.42.217.44 Decimal: 2032851244 Hostname: 121.42.217.44 ASN: 37963 ISP: Hangzhou Alibaba Advertising Co.,Ltd. Organization: Hangzhou Alibaba Advertising Co.,Ltd. Services: None detected Type: Broadband Assignment: Static IP Blacklist: Click to Check Blacklist Status Continent: Asia Country: China cn flag State/Region: Zhejiang City: Hangzhou Latitude: 30.2936 (30° 17′ 36.96″ N) Longitude: 120.1614 (120° 9′ 41.04″ E)

raverbashing 9y ago
I wonder what would happen if the return payload had some data that would trigger the GFoC
sdiepend 9y ago
When you go this adress http://121.42.217.44:8080/ http://121.42.217.44:8080/ "Hi bro :) Welcome Here! Leave Messages via HTTP Log Please :)"
- a_participant 9y ago
  Perhaps they have a zero day telnet client or browser exploit. :)

chatmasta 9y ago

Package managers seem to be an increasingly popular attack vector. It's only luck that none of the attacks have been particularly malicious yet. Considering how many package manager downloads go to a server in a datacenter, a widely distributed malicious package could control a botnet with extremely high throughput, or wreak havoc on any databases it comes into contact with. It's only a matter of time before something like this happens. A big part of the problem is that application package managers, like pip or npm, are far less sophisticated than those of operating systems, like aptitude or yum. It needs to be easy for developers to open source their code, and to mark dependencies with precise commit hashes, but the download also needs to be secure and verifiable. There are many difficult tradeoffs to consider in terms of usability, centralization, security and trust.

raesene6 9y ago
Another fun fact to consider is that with many package formats, you can execute arbitrary code at install time so if a malicious package can get into a repository, it's very likely to start compromising systems quickly. Whilst a package manager repo. compromise would be the biggest bang in terms of attack, compromising the credentials of the developers of popualar libraries would be an easier attack (and indeed is already happening https://twitter.com/chrispederick/status/892768218162487300 https://twitter.com/chrispederick/status/892768218162487300)
- ams6110 9y ago
  Doubly so since when installed on a server very often it will be done as "sudo pip install ....." (/s/pip/other-package-manager/ as needed)
  - tekromancr 9y ago
    I almost never see this. Even on systems that are only running a single python project, I only ever see folks use virtualenv. The only time I ever see things installed with sudo is when the package is being installed in a docker container.

VMG 9y ago

how likely is it that npm and other package managers that do not use digital signatures by default are unaffected?

IgorPartola 9y ago

This to me is the nightmare scenario. Well one of the two, the other one being that a developer of an obscure library I use has their password to PyPI compromised and a bad actor uploads a backdoored version of the library. Fundamentally, the reason this is different from how thinks like Linux distos work is because Linux distros have maintainers who are in charge of making sure every new update to one of their packages is legit. I am sure you can try to sneak malicious code in, but it isn't going to be easy. I am not advocating that PyPI (and npm) adopt the same model. That would be too restrictive. But maybe just showing the number of downloads isn't the best way to assure whether the package is legit. Perhaps some kind of built in review system would be nice.

raesene6 9y ago
A review system unfortunately isn't likely to be practicable with current development models. npm alone has over 500,000 packages (http://www.modulecounts.com/ http://www.modulecounts.com/) so even a one time review isn't going to happen. If people want a more trusted solution the likely outcome is that they'll need to use a smaller more static set of libraries and then either do the audits themselves, or outsource that to a 3rd party. Ofc with current speeds of change and deployments, it doesn't seem likely that many companies will adopt that model.
- mschuster91 9y ago
  > npm alone has over 500,000 packages (http://www.modulecounts.com/ http://www.modulecounts.com/) so even a one time review isn't going to happen. But at least the modules with the most downloads (webpack, react, or stuff like left-pad) could be vetted, and especially npm could implement a 2-or-more person model - basically, everyone with publish access can upload a new artifact, but to actually have it distributed to endusers, a second person would be required to sign off.
  - raesene6 9y ago
    Indeed it's not impossible to do (although full code review would be expensive/tricky/slow). The fact it hasn't been done despite the obvious risks indicates how much demand there is for this feature...

jamespo 9y ago

I wonder if it's worthwhile having a check that compares closeness of the name to existing popular packages and if so does some extended vetting.

fruiapps 9y ago

Curious to know something similar happening for Scala.

Sir_Cmpwn 9y ago

I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human being verifying the upstream and keeping malware out, and get more consistency across packages as a bonus. If you want your package added it's as simple as sending an email and provides a new avenue for people to contribute to the success of the ecosystem as package maintainers. When you make the next big thing, consider this approach.

drdaeman 9y ago
Maybe you're right, but I see one possible downside that is quite important. I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice. With the intermediate maintainers, that's going to get worse. I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.
- nyrikki 9y ago
  PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file. I do wish that `--user` was the default for pip. It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155 Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression. I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.

raesene6 9y ago

This isn't, in any way, a new problem. I did a presentation on this topic for OWASP AppSecEU 2015 (https://www.youtube.com/watch?v=Wn190b4EJWk&list=PLpr-xdpM8wG-ZTcHhFfAeBthNVZVEtkg9&index=10 https://www.youtube.com/watch?v=Wn190b4EJWk&list=PLpr-xdpM8w...) and when doing the research for that I encountered cases of repo. attacks and compromise. IME the problem will continue unless the customers (e.g. companies making use of the libraries hosted) are willing to pay more for a service with higher levels of assurance. The budget required to implement additional security at scale is quite high, and probably not a good match with a free (at point of use) service.

cdnsteve 9y ago
I'm sure companies would pay for it. The service needs to be part of the main package service, not some third party.
- raesene6 9y ago
  Interesting if you think that npm/Rubygems/PyPI are leaving a load of money on the table, why do you think they haven't introduced those services so far...
  - cdnsteve 9y ago
    Because their mission isn't to generate income like a traditional business. But if the income went back to the foundations, like Python Foundation, I think that would make sense.
    - raesene6 9y ago
      npm is a commercial organisation, they offer paid subscriptions but don't offer a curated package signed option...
      - cdnsteve 9y ago
        Sort of a critical feature they are missing

EstDelenda 9y ago

Any method of software distribution which is not rooted in cryptographic author verification against a fine-grained, user-manageable trust store should be put bellow the sanity waterline, 20 years ago.

justinsaccount 9y ago

well shit, I guess I should have followed up on this after I noticed it 2 months ago. https://twitter.com/JustinAzoff/status/881163562739277824 https://twitter.com/JustinAzoff/status/881163562739277824

andrew3726 9y ago
Yup, same: https://gist.github.com/Spotlight0xff/829b7ebf32c4feec60ec44eb86b0fb3f https://gist.github.com/Spotlight0xff/829b7ebf32c4feec60ec44...

defined 9y ago

Here's something that contributes to typosquatting: the lack of responsiveness by package management organizations to claims on orphaned or unmaintainable packages. People who upload packages often leave organizations, who are then stuck with a package they can't update because the password went with the person, and the email reset link points to a now-defunct email address. Petitioning the package management team is sometimes fruitless, forcing a needless new instance of typosquatting.

chupapuma 9y ago
I have found the PyPI group of people to be very helpful in these cases. You also should probably, as an organization, have more than one owner of your packages. That way, unless two people leave, things aren't orphaned. We have gone as far to have a 'meta-user' that is on all packages. It is only ever used to recover a fully abandoned package.
- defined 9y ago
  I understand you are trying to be helpful, and of course you are right, but the fact is that sometimes things fall between the cracks, especially in, say, hard-pressed startups. There are so many shoulds in the world that don't make it to dids, it reminds me of the joke about the salesman trying to sell farming improvement techniques and being turned down by the old farmer, who says, "Son, I don't farm half as good as I know how to already." Unfortunately, I have not found the PyPI group as helpful as you have. Perhaps I have been looking in the wrong places.
  - defined 9y ago
    UPDATE: I was helped out by a very nice person from PyPI, so kudos to them.

ehnto 9y ago

Part of my dislike for the Node ecosystem in particular and I am sure others have a similar problem, is the dependency trees are super complex. Because packages tend to be small and many, and each of those has their own dependencies, you can end up with hundreds of packages installed which is simply impractical to manually review. It is not node, but we do in fact manually review each package we utilize for our given language because it's feasible and worthwhile as the dependency tree is small in this ecosystem. Each and every package is a possible attack vector whether that be intentionally or just because it's poorly written and we can't simply ignore that because it's the done thing and "the community reviews them".

singularity2001 9y ago

"Success of the attack relies on negligence of the developer" How about package manager managers accept their enourmous responsabilty? urllib vs urllib2, one is a virus? Sorry but that is not "negligence of the developer"

singularity2001 9y ago
The least they can do is create an alias system for common libs or disallow some lib names. Another easy thing to implement would be a popularity check: "This package was only installed nnn times. Did you mean xxx, or do you want to proceed with the installation of yyy by author dev@g00gle.com?" Email verification is a must.
bughunter3 9y ago
There are over 100,000 packages and PyPI is run by volunteers. This is not practical. PyPI is not a curated distribution.
- singularity2001 9y ago
  there are many ways to reduce the likelihood of malicious packages. not all of them require active curation. some can be systemic.
zokier 9y ago
Managing supply chain is one the basic principles of good engineering. Not properly vetting your sources is negligence. The problem of course is that computers are really good at amplifying work, including mistakes. So small mistake, like a typo, could have catastrophic impact, like injecting malware that can take over the whole system.
- deleted 9y ago
  [deleted]

ConfucianNardin 9y ago

Relatedly: https://github.com/pypa/pypi-legacy/issues/644#issuecomment-305134745 https://github.com/pypa/pypi-legacy/issues/644#issuecomment-... Also http://evilpackage.fatezero.org/ http://evilpackage.fatezero.org/ / https://github.com/fate0/cookiecutter-evilpy-package https://github.com/fate0/cookiecutter-evilpy-package That one has neutered the call-home code by now, though.

cdnsteve 9y ago

I'm all for security but this hit a nerve with me: "Success of the attack relies on negligence of the developer, or system administrator, who does not check the name of the package thoroughly." Package managers need to do more. If they had an enterprise version that you could subscribe to monthly/annually invoice that you would get enterprises onboard, they are concerned about security and will pay. Developers like us will help encourage it. I'd rather not see some third-party "secure" package managers but make them part of PyPi and send funding to the Python foundation. They are seeking donations but that doesn't work well with businesses. Make it a monthly/yearly service.

underko 9y ago
There's written to check the package name and not to go through whole source code.

kasabali 9y ago

Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly. They solved the issue 2 decades ago by heavily vetting packages before accepting them into repositories. Users are allowed to add and use packages from 3rd party repositories. Maybe solution to this is creating curated repositories based on publicly open ones and using them by default (and requiring opt-in for using other repositories). Conda for Python and Stackage for Haskell seems like relevant solutions.

mikehollinger 9y ago
There's a certain amount of work (and therefore money) required to do this. That incremental difference is small for a well designed application, but - someone must actually vet and curate the contents of the repo. That tends to slow down execution, leading to scenarios where things like docker a year or two ago from the canonical "trusty" repo were hopelessly behind the "real" docker since docker was evolving so quickly and trusty was by design slowing down. Each commit that went into trusty required a team to submit and a team to approve. That costs money. ;-)
- kasabali 9y ago
  It is a matter of distribution and release policy and not an inherent limitation of the model. Stable/lts/enterprise distributions have other concerns like preventing regressions and configuration or behavior changes during lifetime of release. Rolling distributions like Arch and OpenSuse Tumbleweed on the other hand can move a lot faster but still provide basic vetting wrt security and sanity of new/updated packages.
disconnected 9y ago
> Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly. https://www.schneier.com/blog/archives/2008/05/random_number_b.html https://www.schneier.com/blog/archives/2008/05/random_number... A.K.A., the Debian openssl Fiasco. Just one example of distros fucking up the packages from upstream and causing major havoc.

bhouston 9y ago

I bet there are quite a few malicious NPM packages that we do not know about. Is Node is used in government and military solutions? If so then the NPM ecosystem is likely targeted by state actors, and it is a sitting duck.

thehardsphere 9y ago
State actors do not limit themselves to government and military targets; many of them target civilians for all sorts of purposes.

kumarvvr 9y ago

Whoa urlib & urllib3. Those are pretty popular packages, especially to newbies. Hundreds of websites that teach web-scraping use those libraries. Wonder what is an effective form of protection against such attack vectors? Do digitally signed certificates fit into this usage scenario??

zapt02 9y ago
> Do digitally signed certificates fit into this usage scenario?? No, because either the package author would have to sign them, in which case you have to choose to trust each package author, or the repository would sign them, in which case there would be no improvement for this current issue, since the repo would sign the fake packages as well.
- kumarvvr 9y ago
  Any way in which blockchain technology can be used? Like, the transaction becomes the act of the author uploading the code and the repo and user verify the transaction in some form?
  - peterwwillis 9y ago
    Nope. You can't solve phishing with technological means. You have to curate either a whitelist or blacklist. The best way to handle this is whitelists of trusted package maintainers and/or code authors.
    - kumarvvr 9y ago
      Ohh ok. Thanks. Great to know.
    - zapt02 9y ago
      To be fair you can do a lot with simple heurestics. New package + large number of downloads || new package || new package author = show users a warning message before installing the package.
- 9y ago

EGreg 9y ago

This is why I am not a huge fan of using package managers. I like to understand the code we put into our platform, and vet it. And not have it change under us automatically, after that, but review the changes manually before accepting it. I felt a bit curmudgeonly but we have a responsibility at https://qbix.com/platform https://qbix.com/platform for all our apps being secure. I wanted to use repos for each package and manually git pull or hg pull them when they changed. I was finally convinced by our developers to just use package managers with version pinning. Honestly it's really hard to avoid package managers, especially for all the newer functionality such as Payment Requests or Web Push. Luckily there is version pinning. We want our clients to feel secure that we vetted ALL the code that went into the platform. So our package json (and composer.json) uses version pinning. We'd rather take a bug report and manually fix it than NO bug report and have a SHTF moment.

wiradikusuma 9y ago

Anyone know if this is also an issue for Java? I've used Maven repository for ages, and I know many big cos depend on it.

1ba9115454 9y ago
It's absolutely an issue. I'm pretty sure no one is looking at every jar file added to maven to see if there's an issue. In your POM file do you have a checksum?
thehardsphere 9y ago
It's less of an issue, but it still could be an issue. Deployed Maven artifiacts from Central are to required to be signed with a PGP key and are only supposed to come from approved hosts. I don't know how strictly that is enforced and how hard it is to become a host, but at least there is some kind of process. Maven Central also doesn’t allow the removal of artifacts after they've been published, and every artifact requires a unique version and name. And the names are namespaced. So you don't have the issues that you see with npm, where someone can pull a package and break everything people are using, and then some third party can come in and publish anything under the exact same name. Is this model perfectly secure? No, you still have to trust that the artifact was signed by a non-malicious person from a host that was not compromised.

EGreg 9y ago

Here is the general problem with dependencies: When a dependency changes, all the projects that directly depend on it should get notified immediately and their maintainers should rush to test the new changes, to see if they break anything. There is no shortcut around this, because if B1, B2, ... Bn depend on A1, the consequences may be different for each Bk. The only real secure optimization that can be done is realizing that some of the Bk use A1 the exact same limited way and thus make an intermediate A1b that depends on A1 which those Bk's depend on. These "projection" builds may be automated by eg the set of methods called by the B's. Anyway, this is the way that iOS does it before iOS 11 comes out to users. They release a beta to all developers. And they even fix bugs in the beta before releasing to the public. Without beta testing periods, you can get laziness and just auto-accepting of whatever cane out. There is be an "alpha release" feature in git where maintainers might put out the next version to be tested by all who depend on it. THIS FEATURE SHOULD NOTIFY THE MAINTAINERS SUBSCRIBED TO THE REPO. THE BUILD ITSELF SHOULD GET ISSUES AND RATINGS FROM MAINTAINERS AS THEY TEST THE NEW BUILD. And releases should not be too frequent. This is the way to prevent bad things from happening. But that also means that the deeper the dependency is, the more levels this process could take to propagate to end-users.

julianj 9y ago

Looks like they missed one: https://pypkg.com/pypi/xml/f/setup.py https://pypkg.com/pypi/xml/f/setup.py Dork: site:https://pypkg.com https://pypkg.com intext:"just toy, no harm"

kevin_thibedeau 9y ago

Python needs a way to run 2to3 during package installation that doesn't use setup.py (setup.cfg or wheels). As it stands now, you have the hassle of building a release four times if you want to support all combos of Py2, Py3, 32-bit, and 64-bit platforms. The absent support for 2 to 3 migration in the safer alternatives is why I stick with setup.py.

ThiefMaster- 9y ago
No, what you need to do is fix your package's code to work on Python 2 and 3 without running 2to3 on it. The only case where this doesn't work is if you have binary extensions - but then you need separate wheels in any case.

topbagsvk 9y ago

If you want your package added it's as simple as sending an email to http://www.lvbagsgb.com http://www.lvbagsgb.com

amykhar 9y ago

some of this could be helped by intelligent naming of packages. If something is called urllib, name the package urllib because that's what people are going to look for.

hannob 9y ago

Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last year someone wrote his thesis describing python typosquatting and standard library name squatting: http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana... However after that the packages used in this thesis - the most successful one being urllib2 - weren't blocked, they were deleted. Benjamin Bach was able to register urllib2 afterwards. Benjamin and I decided that we'd now try to register as many stdlib names as possible. See also: https://www.pytosquatting.org/ https://www.pytosquatting.org/

edraferi 9y ago
I appreciate the proactive approach. Is your project the author of the packages identified by NBU? If so: (1) Why is the tracking pingback obfuscated? (2) Why does the code include a cheeky hello instead of a link to https://www.pytosquatting.org/ https://www.pytosquatting.org/ ? (3) Why is there not a visible warning when installing one of these packages? ================= edit: Reading through the linked blog post [0], it appears these researchers used different code that DID provide visible warning and an cleartext pingback. It also collected command history and hardware information. [0] http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...
- hannob 9y ago
  We're not the authors of those packages. But we own many others. 1. We're not obfuscating pingbacks. 2./3. We're raising an exception with an explanation and a link. Just look at the code of one of our packages: https://pypi.python.org/pypi/codecs https://pypi.python.org/pypi/codecs The research in 2016 was done by someone else. The kinda crazy thing is: Some of the package names he used were made available again after that instead of being blocked... And now we own them.
  - edraferi 9y ago
    Man. You're right - that's a mess.

Aissen 9y ago

I'm glad Go completely sidestepped the name-rush induced by this type of package managers (composer, cpan, rvm, pypi, npm…). Just provide an URL. Done.

eeZah7Ux 9y ago
...which is even less secure.
- Aissen 9y ago
  May I ask why ? If anything, it's more secure, since you know exactly who's publishing what. Yes, it might put a higher burden on the publisher if they don't host on github/gitlab, etc. But it strips the "magic" part and makes sure the dev knows where the code is coming from.
  - eeZah7Ux 9y ago
    With packages identified by full URLs it's more likely to make a typo, or a misremember a part of the URL, or search for it on a search engine and pick a fork instead of the right one, or paste one from stack overflow or other forums that is plain false or even look legitimate due to unicode tricks. Also DNS MiTM/hijack can be used to inject a backdoor. Or the expiration of a legitimate domain.
    - krapp 9y ago
      You can also make a typo when all you need is a package name - as long as human beings have to type thigns out, that's going to be a problem. On the other hand, with a URL, you can actually inspect the code directly and (if it's hosted on Github or somewhere similar) see whether it's starred, forked or has any issues. It's not a case of URLs being less secure, it's just a tradeoff that pushes some of the security work to the community itself, rather than a dedicated staff of curators. And since most package managers eventually resolve packages to a URL somewhere, the issues you mention are probably present in other package managers, albeit hidden behind abstractions.

thedonkeycometh 9y ago

This seems to be getting more of a threat. Can't see how you prevent this sort of malicious spoofing without explicitly curating the list. Personally, I prefer the uncurated, caveat emptor approach, but this could do witha third party stepping up. (moz seems keen to do anything to get us to like them) Ultimately, sounds like something that could be solved with a blockchain of legitimisation. (ran by moz) ;)

mwerty 9y ago

How about a Levenshtein distance threshold for new package names to be accepted? I.e only allow names that are different enough from the existing set to avoid typos (or whatever errors we are trying to guard against)?

andrewfong 9y ago
You don't need a strict ban for this to work either. Maybe just an end-user warning if distance < N and the relative popularity of the two modules is very high. You could also allow users or organizations to explicitly whitelist some names.

elcapitan 9y ago

Would it be possible to have a general package manager (like apt) as reusable base for the individual language specific package managers? I know that npm and pip and gem etc all do some additional stuff, but at the core they all do the same (pull packages from repo, do some postinstall, resolve dependencies, maybe in some cases even check if the package is legit). So we could implement and check that once and then just reuse it like we do with many other libraries for image processing etc.

eeZah7Ux 9y ago
+1 to this. There's no need to reinvent the wheel a million times. At least having a shared standard on how to do packaging.
zokier 9y ago
See also: PackageKit.
takluyver 9y ago
The packaging technology is not the issue here, it's about the repository. If you made an apt repo where anyone could claim an unused name and start uploading packages, you'd have exactly the same issue.

1ba9115454 9y ago

Unless your package manager enforces signatures and you trust the person that signed the package. Then this is an attack vector for you. That includes Java (Maven), Ruby (Gems, Bundler), Node (npm), Haskel (stack) etc etc. Installing code via package managers is the coders equivelant of opening up an exe sent to you in an email. Code downloaded from the internet is not to be trusted.

thehardsphere 9y ago
I thought Maven enforces signatures? Though that doesn't fully mitigate the risk as you still have to trust the signer.
anentropic 9y ago
Signatures are good, but do not help in this case (typo-squatting)
zokier 9y ago
Package signing is no silver bullet. Signing packages helps against typosquatting about as much as SSL certificates help against phishing. Or in other words, not at all, especially if we don't have the certificates rooted in real world identities (like EV SSL certs).

atticusberg 9y ago

to see if you have any of these deps on your python path: pip list –format=legacy | egrep -e '^acqusition$' -e '^apidev-coop$' -e '^bzip$' -e '^crypt$' -e '^django-server$' -e '^pwd$' -e '^setup-tools$' -e '^telnet$' -e '^urlib3$' -e '^urllib$' to see if you have any projects in a given directory that require them: cat $(find /path/to/dir -name 'requirements.txt') | egrep -e '^acqusition==' -e '^apidev-coop==' -e '^bzip==' -e '^crypt==' -e '^django-server==' -e '^pwd==' -e '^setup-tools==' -e '^telnet==' -e '^urlib3==' -e '^urllib=='

jastr 9y ago
pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'

mwexler 9y ago

Both Anaconda (for Python, https://docs.anaconda.com/anaconda/packages/pkg-docs https://docs.anaconda.com/anaconda/packages/pkg-docs) and Microsoft (for R, https://mran.microsoft.com/ https://mran.microsoft.com/) have "reviewed and audited" collections of packages for their languages. That's part of what you pay for when you buy support for the open source tools.

pishpash 9y ago

Maybe gov.sk should be vouched for too, I mean what's the chain of trust here? Why should I trust anyone?

jastr 9y ago

jastr 9y ago
Avoid some false positives pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'

phonkee 9y ago

Do not forget their password that worked for couple of years: nbuSR123 ...

pishpash 9y ago

Maybe packages should be signed by several trusted maintainers. Or, noticing PyPI packages list a source code link on github sometimes, along those lines, there can be a process to prove ownership of some known online identity, keybase style. Unpopular packages can also be flagged, especially one that has a near twin that is much more popular. There are many solutions.

ris 9y ago

Hooray for the "wild west" model of package repositories. Come back maintainers & packagers, all is forgiven!

teilo 9y ago

I think we need a system to prevent this instead of the wild-west that PyPi has become. For example: Developer signatures that are checked against a community rating. If someone does `pip install` pip would look up the developer signature of the package and check a community rating that would verify this is a developer who has offered legit packages in the past. It's not foolproof, but it would go a long way towards solving this.

zokier 9y ago
You could add the ability for well known members to vet newbie developers, maybe by signing their key. And now you have re-invented web of trust.
- teilo 9y ago
  I was thinking more than the ratings would handle this, rather than having to sign keys. Ratings by well-respected and vetted members would have more weight.
wongarsu 9y ago
That sounds easy to defeat. Make some mundane, but legit packages (maybe on of those "$X but without the pointless complexity"-packages), gain trust, once trust is reached start uploading typo-squatting packages. Knowing today's internet, programmers from cheap-labour nations (India & Co.) would soon start offering "trusted PyPi accounts" for sale on hacker forums.
- teilo 9y ago
  Yes, but keys can also be revoked, providing a way to mitigate this.

asperous 9y ago

I once tried to upload a package called "requirements.txt" (since people do pip install requirements.txt all the time forgetting the -r). Pypi actually blocks that name from being a package!

geekamongus 9y ago

Why is there no indication of any of this on the python.org website or any of their social media accounts? I checked: https://pypi.python.org/pypi https://pypi.python.org/pypi https://www.python.org/blogs/ https://www.python.org/blogs/ http://planetpython.org/ http://planetpython.org/ https://pypi.python.org/security https://pypi.python.org/security https://twitter.com/pythoninsider https://twitter.com/pythoninsider https://plus.google.com/+Python https://plus.google.com/+Python https://www.facebook.com/pythonlang?fref=ts https://www.facebook.com/pythonlang?fref=ts https://twitter.com/ThePSF https://twitter.com/ThePSF

takluyver 9y ago
I guess that's because it's not a surprise. This has come up before, and it's basically unavoidable with the way PyPI is designed to work: if you see an unclaimed name, you can put whatever you want there.
- geekamongus 9y ago
  I see your point, but I don't think that it needs to be a surprise to be announced or made well-known. No one is surprised that Microsoft releases loads of patches every Patch Tuesday, but they still publicize it and make it well known and easy for people to find out about.
haypo 9y ago
While there is no public announcement from the PSF yet, I sent an email to the python-dev mailing list at least to announce the issue but also try to discuss how to mitigate/prevent it. https://mail.python.org/pipermail/python-dev/2017-September/149569.html https://mail.python.org/pipermail/python-dev/2017-September/... Honestly, I am impressed that the information gone so quick! The National Security Authority of Slovakia contacted the PSRT 10 days ago. All packages were removed 1h10 after we got their email. We were discussing how to communicate about this issue, while they published an advisory. A few hours after the advisory was published, I saw the information on IRC, Twitter, LWN, etc. I didn't expect that the advisory would be published so quickly. FYI last week there was also a CPython sprint attended by more than 20 Python core developers. We were busy on discussing Python enhancements.

jwilk 9y ago

python-dev thread: https://mail.python.org/pipermail/python-dev/2017-September/149569.html https://mail.python.org/pipermail/python-dev/2017-September/...

marcinkuzminski 9y ago

This is a known problem for a while. For example when we (RhodeCode) had an installer based on pip we actually rolled our own PYPI index. Having one you host is very easy and there are nice projects in existence that allowed that. It basically solved a problem of deployment when pypi wasn't available, speed up our test installer builds, and also total control over the packages we ship.

jiffyToGo 9y ago

tiny open source project for this. https://github.com/williamforbes/pypi_hacked_names https://github.com/williamforbes/pypi_hacked_names