15 ms·
Typosquatting programming language package managers
- pmontra 10y agoProbably the maintainers of the package managers know which typos their users do, because of the 404s in the logs or equivalent errors. A preventive action could be starting to blacklist any name resolving to 404. If somebody eventually tries to upload a package in the blacklist, a maintainer should check the code and whitelist the name. Obviously people can be very crative with typos and with squattinq and there is no real protection against mistakes.
- epalmer 10y ago> Obviously people can be very crative with typos and with squattinq and there is no real protection against mistakes. I see what you did.
- utexaspunk 10y agoMight it work to mandate that the name of an uploaded package have a minimum levenshtein distance (or similar calculation) from the names of all the existing packages? Then you wouldn't have to worry about maintaining a blacklist.
- wycats 10y agoThat would mean that, for example on crates.io, you couldn't create a `libm`, because `libc` is already very popular. I don't think that works.
- utexaspunk 10y agoTrue- levenshtein isn't the best algorithm for the purpose. Is there an algorithm that takes key proximity into account? Like, 'libm' and 'libc' are sufficiently different to preclude typos, but 'lib[n/j/k]' or 'lib[x/d/f/v]' are not?
- sqeaky 10y agoKey proximity on which of the hundreds of keyboard layouts?
- utexaspunk 10y agoGood question... I'd imagine your standard QUERTY makes up a sizeable majority of programmers, but then I have no data to back that up... :)
- nikcub 10y agoThe default approach would stop automated attacks, there is no reason why the repository couldn't whitelist libm after review
- pmontra 10y agoIt seems a good idea. I used the Ruby code at the beginning of http://stackoverflow.com/questions/16323571/measure-the-distance-between-two-strings-with-ruby http://stackoverflow.com/questions/16323571/measure-the-dist... to calculate the distance between the package names at page 60 of the thesis and their typos. The maximum is 2. I checked some similar package names from a Gemfile.lock of a project of mine. Unfortunately the two gems hike and hirb are also at distance 2. Probably many short names are close with this metric. A combination of the two approaches could be ok: knowing that a name was blacklisted should be an indicator that's not a good name, despite the distance with any other name, plus an approval of the maintainers for distance 2. But a blacklist could generate another type of squatting, with people trying to pre-blacklist perfectly legit names. Only one thing is sure: there is more work to do for the maintainers and this extra friction is not good. Edit: the distance suffers from the same problem.
- hughes 10y agoSurely some troll would deploy a fleet of machines that flood package indexes with requests to available names, effectively blacklisting entire dictionaries and eventually all short names.
- pmontra 10y agoYeah, this is what I came to think too. I mentioned it in another comment. Somebody suggested to use a distance indicator, but trolls could attack that too.
- deleted 10y ago[deleted]
- ryanmarsh 10y agoSo last week my client discovered there's a gem named bunlder... sigh
- pmontra 10y agoThere is a gem called bundle which doesn't do anything but preventing a typosquat https://rubygems.org/gems/bundle https://rubygems.org/gems/bundle Total downloads 1,800,600 Source (empty) at https://github.com/will/bundle https://github.com/will/bundle and interesting README. https://rubygems.org/gems/bundler https://rubygems.org/gems/bundler Total downloads 92,116,090 It's almost the 2%.
- rspeer 10y agoI think the authors here missed an opportunity for even more effective squatting like that: cases where the name you import, name you type at the command line, or name you commonly call the package by is different from the name in the repository. In Python, "pytables" (should be "tables") and "skimage" (should be "scikit-image") come to mind.
- nathancahill 10y agoYeah. I think it's becoming a reflex for programmers when they get an import error like: Error: Cannot find module 'x' to quickly type: npm install x
- willlll 10y agoMy gem has a good downlaod/loc ratio.
- eudox 10y agoI'm a fan of the approach of personally submitting projects to the repository maintainer (e.g. through GitHub issues), and having the maintainer personally approve them. It does raise the barrier to entry, but it would prevent typosquatting and regular namesquatting. EDIT: Does any major package manager provide a "did you mean" functionality, offering a list of actual package names similar to what you typed?
- philjackson 10y agoThat's a massive burden on the poor person who has to ok the package - especially at NPM's scale, for example.
- yoo1I 10y agoWell, ideally you'd set up some sort of system where multiple people work on managing a repository, similar to maybe how linux distributions package applications and libraries.
- eudox 10y agoNPM's scale is the exception, rather than the rule.
- seldo 10y agoWe believe npm's scale is a direct result of having the lowest ceremony to publish a package. Turning the dial in the direction we did has pros and cons.
- burkaman 10y agoAPT does and others probably do too, but it obviously only gives suggestions when the package you entered doesn't exist.
- eudox 10y agoRight, it's only useful if you've prevented typosquatting. Which Debian has, because submitting a new package is a much more involved processes than sudo apt-get publish.
- zeveb 10y agoReminds me of the quote, 'there are only two hard things in computer science: naming things, cache invalidation and off-by-one errors.' I think that this clearly falls under the heading 'naming issue.' People know what they want, but do not enter it properly. I can't think of a 100% off-hand, which isn't surprising, because it's a hard problem. pmontra's suggestion to use typo blacklisting ain't a bad idea. Maybe some sort of reputation-per-name could help?
- a_t48 10y agoSure it's not an off-by-one[-key] error? :)
- blowski 10y agoBanks have a similar problem when people write cheques or set up standing orders. You have to put a name and the account number. I wonder if you could do something similar here - enter the name of the package and a code of some sort. I haven't thought this through in a lot of detail.
- PeterisP 10y agoBanks generally solve the issue with simple classic checksumming methods that guarantee that any number with a typo or swapped neighbouring characters will always result in an invalid number. That doesn't work with arbitrary names because they are, well, arbitrary.
- throwawaysocks 10y agoWhy not? Central repositories could require that all names are within a certain Levenshtein distance of one another. This could get mildly annoying every once in a while when there are legitimate non-clashing names. A better metric/typo recognition technique is probably possible. Or else some manual process for requesting exceptions (maybe with a tiny fee to help fund the overall project) would also address this problem. EDIT: Just downloaded and read the thesis abstract. The author actually suggests the first idea: "The analytical part generates ideas for countermeasures that allow repository maintainers or users to detect typosquatting attacks in the future. For this purpose potential typosquatting candidates could be generated for each legitimate package name with the help of the Levenshtein distance algorithms or Bayesian networks. Another option that can be considered is the Metaphone algorithm."
- nichochar 10y agoWow, this a very good study and explanation of what typo squatting is, and I really liked how he proved it's effectiveness. I wonder what kind of steps we can take to prevent this risk.
- trungaczne 10y agoI think we will have to rely on crypto hash in some form. Similar to download checksum. It won't be convenient, but it will be safe(r).
- bpicolo 10y agoThat doesn't really save you from typos
- trungaczne 10y agoI was thinking something along the line of a mandatory hash/checksum along with the name of the software you are trying to install from a package manager. It does not have to be very long, just enough to avoid common collisions.
- irremediable 10y agoReally cool applied research. If I get the time, I'll check out the author's thesis.
- baby 10y agoAfter watching this awesome Defcon talk https://www.youtube.com/watch?v=YqxaKGA9Lnc https://www.youtube.com/watch?v=YqxaKGA9Lnc I wondered if there was any use cases for bit/typo squating in crypto. This is a pretty cool one! Not crypto but interesting none-the-less :)
- Mizza 10y agoThis seems like pretty unethical research to me. Also, doesn't point out that the bigger threat is that this is wormable.
- throwawaysocks 10y agoThere was no actual intrusion, so this feels like fair game to me. Especially since mitigating a very possible attack vector is a direct result of running experiment. Still, hopefully the researchers got an IRB to sign off on the experiment setup...
- kbenson 10y agoWell, there was a small intrusion. It reports back a filtered command history (including just package install commands), the hardware info, and the list of installed modules (along with regular info, like system type, if there are admin privileges, etc). That's not nothing, but it is fairly benign. I was worried about the command history until I saw it was filtered, and that mostly allayed my misgivings.
- placeybordeaux 10y agoThe research got computers to execute code on them without authorization and extracted information from them. That is a crime under the CFAA in the USA. Not sure what it is in Germany/EU.
- billyhoffman 10y ago"Your honor, my client created and published a software library. The so-called victims here wrote code that specifically referenced my client's software library, by name mind you. My client in no way compelled or solicited the victims to do so. Now how can that be called 'without authorization?'"
- random28345 10y ago> "Your honor, my client created and published a software library. The so-called victims here wrote code that specifically referenced my client's software library, by name mind you. My client in no way compelled or solicited the victims to do so. Now how can that be called 'without authorization?'" The prosecuting attorney is going to tell a jury of twelve of your non-technical "peers" that it is hacking. Your client can either go to trial for seventeen thousand two hundred and eighty nine counts of felony hacking, and risk half a million years in prison, or they can plea bargain to 5 years in prison and a felony on his record. Or your client can hang himself, but I'm pretty sure a federal prosecutor counts that as a win too.
- szx 10y agoWhen you think about it, how different is the destructive potential of an npm/pip install from curl | bash that (some) people tend to froth at the mouth about? It's pretty mind blowing how big of a blindspot package installers are. I guess running everything inside a e.g. Docker container/VM would be a partial interim solution for the paranoid?
- lmm 10y ago> When you think about it, how different is the destructive potential of an npm/pip install from curl | bash that (some) people tend to froth at the mouth about? It's a bit better - there is only one possible source of compromise rather than everyone on the network path. Given that npm/pip likely keep archives of all packages uploaded, it would be much harder (perhaps impossible) to attack someone secretly this way, at least in the long term. Good package managers require signing of uploads (e.g. maven central requires every package to have a GPG signature; Debian goes further, and requires your key to be signed by an existing member of the organization). If the client checks the signatures you end up with a system that's perhaps actually secure.
- szx 10y agoSigning is definitely part of the answer but there's still the question of trust. A signed package doesn't really tell you that much. In the best case scenario it tells you the package you're installing in fact came from developer X and contains code Y (which you kinda already know since you have the source code). This works as long as you know and trust developer X, or did your due diligence reading through the code (which you can already do today). I can't think of an end solution that wouldn't have to rely on network effects and social proof, which strikes me as rather fragile. Maybe formal verification and AI can help, but that's a long way off (?)
- raesene10 10y agoFor me they're very similar. I actually did a talk last year for OWASP AppsecEU where I started with the curl|bash bit and pointed out where rubygems/npm etc aren't really a lot better in some ways https://www.youtube.com/watch?v=Wn190b4EJWk https://www.youtube.com/watch?v=Wn190b4EJWk
- bennofs 10y agoDid anyone else find it surprising the the number of total requests (45334) is so much higher than the number of unique total requests (17289)? It is more than twice the number of unique requests! Possible explainations: * Perhaps many of those are automated build systems, which would also explain the high number of systems with admin access (for example, if you use travis without docker, every build runs in a clean vm with admin access). * People download one package and install it multiple times? Seems unlikely Any other ideas?
- caseysoftware 10y agoNumerous developers and/or building multiple servers behind a single IP address aka NAT. It's pretty common.
- lighttower 10y agoThe person who ran the line, sudo pip install lumpy (instead of numpy) Ran it again because it 'didn't work'
- joepvd 10y agoAutomated testing, continuous integration/delivery, et cetera download and install packages pretty often. If the type is made in the requirements.txt or package.json or what have you, the error can be repeated very often up to and including production.
- Guillaume86 10y agoI think he forgot to define a baseline (could be wrong, I didn't read the paper). He should have generated a few packages with a completely innocent name (and maybe some packages with just a GUID as a name) to see how much downloads / installs they get too.
- cderwin 10y agoIn the case of python (not sure about the other package managers) if a valid package requires the hacked package, each project that requires that valid package will download and install the hacked package separately if you're using virtual environments. Also if you're using docker you reinstall everything when your requirements file changes.
- optimuspaul 10y agoI'm confused.. is it 17 computers or 17000 computers? inconsistent use of decimals in this article.
- PeterisP 10y agoPart of the problem is the many packages that require sudo permissions to install - IMHO that should be an exceptional case, but it isn't.
- nneonneo 10y agoPackages often require sudo in order to install to the global interpreter - it's a security hazard otherwise. Imagine a Python package which overrides the sys module. If it didn't require sudo, anyone could install it and compromise Python for everyone else (or, for instance, compromise setuid programs). The two solutions here are user-local packages (pip --user, for example) and virtual environments.
- cormacrelf 10y agoAnd 'npmjs.org' is misspelled as 'npmsjs.org' in the introduction. Nice.
- Mahn 10y ago> In the thesis itself, several powerful methods to defend against typo squatting attacks are discussed. Therefore they are not included in this blog post. http://incolumitas.com/data/thesis.pdf http://incolumitas.com/data/thesis.pdf section 5 "Practical implications". Just wanted to point out that in case you skipped it it's worth a read, some interesting proposals there that are worth discussing with package manager maintainers. I particularly like the preemptive approach of auto-blacklisting common typos by simply monitoring the number of times a specific unexisting package is requested over time (5.10). So if a lot of people regularly attempt to install the unexisting package "reqeusts", it could signal that it's a common typo and should be blacklisted to prevent malicious use in the future. False positives could always be sorted out manually by communicating with the package manager maintainers.
- nailer 10y agoYou'd Bayesian that. - The package name is something lot of people regularly attempt to install, but it doesn't exist (per above) - The package name is 1-2 chars off from the name of another package which has more than X downloads - The package is frequently installed then uninstalled in a short time
- wbond 10y agoWe've gotten flack from package developers submitting new packages to Package Control [0] because all additions to the default channel are hand reviewed. Part of this process is to prevent accidentally close package names, to try and encourage collaboration and to encourage developers to actually explain what their package does and how to use it. My hope is to be automating a large amount of the review in the next few months, however I think this is a good argument for never having it be fully automatic. Having a human sanity check submissions isn't a terrible idea if we can keep the workload down. Certainly this doesn't prevent a malicious author from posting a legitimate package and then changing the contents to be malicious, but that can be somewhat solved by turning off automatic updates. [0] https://packagecontrol.io https://packagecontrol.io
- SCdF 10y agoHey Will, Thanks for keeping Package Control high quality, I know it's highly appreciated :-)
- notduncansmith 10y agoAnother grateful Package Control user here.
- eudox 10y agoKeep fighting the good fight.
- deleted 10y ago[deleted]
- bcg1 10y agoSonatype has a manual review process as well before allowing new projects to deploy to Maven Central. [1][2] One step to mitigate things like this as well would be to have some sort of "crowd-sourcing" command in the package manager program... like "npm flag coffe-script" or something like that to alert repository maintainers of possible issues. [1]: http://central.sonatype.org/pages/ossrh-guide.html http://central.sonatype.org/pages/ossrh-guide.html [2]: http://central.sonatype.org/articles/2014/Feb/27/why-the-wait/ http://central.sonatype.org/articles/2014/Feb/27/why-the-wai...
- airless_bar 10y agoThis only seems to be an issue for languages where packages reside in a global namespace, like Python, Rust etc. I think most languages these days are a bit smarter and avoid this beginner mistake (for various reasons).
- abiox 10y agothis is yet another reason why i really wished rust had went for namespaced packaging on crates.io. i like so many of the decisions the rust team made, but not this one.
- vemv 10y agoRuby and JS package managers are un-namespaced as well.
- SixSigma 10y agoJulia too, though there is a central list. I haven't done any tests for this kind of thing.
- anentropic 10y agocouldn't you register a typo namespace?
- Klathmon 10y agoYes, but then you'd need to also register a ton of packages under that namespace. That's something that can be flagged for manual review before it gets too far.
- anentropic 10y agomaybe I don't understand the namespaces... but if you are targeting a package `someuser/popularpackage` can you not just register your own malicious `popularpackage` under a typo namespace like `smoeuser`?
- mirekrusin 10y agowith npm there should be at least an option which prompts for Y/N/A when package has preinstall hook. but even this just tries to put the problem under carpet. you could still for example have requests package which just installs request package, works as expected, just sends request/response to your own server from time to time. ie. when there's http basic auth used only.
- seldo 10y agoIt is possible to disable install hooks at install time by running npm install with --ignore-scripts. You can also make this the default, with npm config set ignore-scripts true (and then --ignore-scripts false at install time if you wish to run them).
- zanchey 10y agoSolaris did (does?) this - "this package contains installation scripts which run as superuser" or words to that effect. Unfortunately I never found a owa to inspect the scripts directly so it wasn't all that helpful.
- ysavir 10y agoInstead of blacklisting, why not respond with a "You requested package ABD, but we think you might mean package ABC. Enter 'yes' to continue or anything else to start over." That way authors can continue to use any name they want, and the emphasis is on letting installers know that they might be installing the wrong package.
- sheac 10y agoBut if ABD and ABC are both package names in the system, then in order to present that warning we have to do some sort of resolution process to determine whether one is typosquatting. Now that there's a strategy for finding fakers: 1) You have an attacker-defender arms race. The attacker will always be one step ahead of the defender. 2) You have the extra burden of keeping up in this race, otherwise your security feature is a facade. At best, this is useless. At worst, it lulls your users into a false sense of security.
- zardeh 10y agoI feel like "pick the more popular package" is a good enough solution in this case.
- sheac 10y agoCool. Attacker-defender race is on! As attacker, my next strategy is create a bunch of agents (<10K should be enough) to download my typo packages. Your move, defender ;) But seriously, my point has less to do with the particular tactics of the adversaries and more to do with how the proposed strategy of automatically detecting potential typos invites gaming.
- zardeh 10y agoPerfect, if each of those 10K hosts downloads the library 100 times you can now typo-attack the zope.event (working in python) library, which gets ~100 downloads per day, many of which are automated and so invulnerable to your attack. Your attack vector gets you, we'll say 1 new hit every 2 days at most, and likely only one a week or so (according to some math, on `requst` vs. requests)
- mbroshi 10y agoMaybe this is overly naive, but when I make a typo in the Google search bar, it doesn't even search for my typo-ed term (even if it would have gotten some hits), it searches for what I actually meant to type. Can't package managers have a similar feature?
- abstractbeliefs 10y agoThe main problem is when you really did mean to search for the typo term. There's no inherent problem in two packages having similar names. Consider the following: requests - a python package for making HTTP requests. requestr - a python package for a fictional startup that allows you to send requests to your nearest and dearest. Given they both could be typos of each other: 1) How do we determine which one to use? What if someone accidentally also tries "requestd", somewhere between the two ? 2) How do we apply the principle of least surprise - I asked to install requests, and everything installed just fine, but now I can't import it?!
- ekimekim 10y ago$ pip install requestr Package "requestr": did you mean "requests"? [Y/n] (reason for this warning: similar spelling and requests is much more popular) Pass --no-spell-warnings to disable this feature.
- sheerun 10y agoGlad to hear bower is stated to be safe in this regard :)
- tbrock 10y agoThe homebrew model where packages and changes to packages are reviewed takes care of this problem quite nicely.
- zmanian 10y agoWe need operating system vendors to give us a mechanism for easily creating and managed sandboxed dev environments. Ones dev environment should be a place where remote code execution is a high probablity and we need better tools to partition that from high value data.
- jogjayr 10y agoI thank my stars every time I get a "Package not found" error due to a typo, because I'm reminded that it could have been much worse.
- andrewstuart 10y agoOuch. This really hurts. So hard to protect against human error.
- jwilk 10y agoTrying to parse the title made my head hurt. It should be "Typosquatting software package names" or something.