7 ms·
Postmortem: Server compromised due to publicly accessible Redis
- zurn 11y agoRelated previous discussion: https://news.ycombinator.com/item?id=10537852 https://news.ycombinator.com/item?id=10537852
- kudosneste 11y agoThis is also a good reminder why running services as root is not recommended. If he had used a separate non-root account for Redis, the damage would have been limited to that single user account. That's not a very good situation either, but at least the attacker would not have been able to modify init scripts and replace system files.
- repomies69 11y ago> If he had used a separate non-root account for Redis, the damage would have been limited to that single user account. Probably not. The attacker would have gotten access to the machine. As there is a lot of software installed locally some probably have holes, which allows for full root access. So I don't really believe it makes much difference for serious attacker if the attacker gets root access or normal access.
- eeZi 11y agoIt's not that easy to get a local privilege escalation on an up-to-date system.
- antirez 11y agoAfter understanding how much this attack was used, I modified the blog post in order to prevent script kiddies from using it without a few changes, but still it caused a lot more harm than I expected. Before Redis 3.2 I'll add security features to prevent this from happening, while adding them in a form that should prevent also people from getting annoyed by the security features. It's an hard mix to get right.
- Jgrubb 11y agoI was wondering about that when I read that post ;)
- tombrossman 11y agoThe original is still archived at http://web.archive.org/web/20151103154949/http://antirez.com/news/96 http://web.archive.org/web/20151103154949/http://antirez.com..., so if the goal is to make the script kiddies work for it you need to add this to your robots.txt: User-agent: ia_archiver Disallow: / Unfortunately that excludes your whole site from the Wayback Machine, but you might try modifying the Disallow. (I haven't a clue if their rules are granular or strictly site-wide.)
- mulander 11y agoOne person wrote and linked an automatic tool exploiting the attack you detailed in the original blog posts comment section. [1] - https://github.com/matiasinsaurralde/evilredis https://github.com/matiasinsaurralde/evilredis
- tedivm 11y agoYou posted a tutorial describing how to use your software to exploit servers, and you're surprised that people used it? I really hope this was a one time lapse in judgement and not indicative of how the redis team thinks about security.
- johnmaguire2013 11y agoThis is the full disclosure vs antisec debate... if they posted it after they had a patched version out, the best thing to do is explain the issue and get people to upgrade, in my opinion. Granted, I didn't read the original blog post. Were they distributing tools to make it easy? Or were they explaining how it worked? In any case, if he hadn't written the blog post, someone else would have.
- andrewchambers 11y agoThe only fault lies with those who put the service on the open internet.
- Terretta 11y agoPopping antirez quote on the lunacy of running this "publicly accessible": > "From time to time I get security reports about Redis. It’s good to get reports, but it’s odd ... in a software which is designed (as we explain in our security page here http://redis.io/topics/security http://redis.io/topics/security) to be totally insecure if exposed to the outside world." Regardless of the readme, the Lessons Learned should include both "know the security posture of the services you run" and "don't run anything publicly facing unless you are forced to, and only then if hardened well".
- zurn 11y agoThe first paragraph says he made a mistake when uninstalling Redis, he didn't have it running on purpouse. Everyone is fallible, things tend to go wrong. Insecure defaults in software hurt even expert devs (which this guy is if you look at his blog). If your software is "designed to be totally insecure if exposed to the outside world" you should not ship it with defaults exposing it to the outside world. IOW you should not design software to be totally insecure by default.
- dijit 11y agobut it runs and binds to 127.0.0.1 by default, what better default is there for this?
- zurn 11y agoAccording to http://www.antirez.com/news/96 http://www.antirez.com/news/96 it binds to 0.0.0.0 by default, though in latest versions there is an example config file provided for binding to 127.0.0.1
- tedunangst 11y agobinding to 127.0.0.1 by default would be much better than the actual default of all addresses.
- pizza234 11y agoOf course everyone is fallible, and this is the reason why careful admins put safeguards and layers of security. In a well-prepared system, the administrator would: #1 block input traffic on all the ports, except the required ones. as routine, after uninstalling any service, open ports are reviewed; even if a service is forgotten running, it won't be able to communicate. #2 not run redis or any service as root user. Not doing so is, to say the least, reckless, and with this assumption/context, it doesn't make much sense to theorize about software design.
- techjuice 11y agoThis should have been run as a non privileged user, on private IPs, a local socket or localhost along with a long and strong password, so only authenticated connections would be able to be able to use it. Since none of these memory store solutions (memcached, redis, etc.) should be able to be connected to on public addresses it would be a nice feature to only allow them to listen on private internal IP space or sockets by default and fail to run if anything else is used. This would help reduce everyone's attack surface by default and help prevent misconfiguration by the user or 3rd party integration software.
- Terretta 11y agoYou can use arbitrary address space in a non-routed subnet. But it could refuse non RFC 1918 addresses unless overridden.
- unixhero 11y agoI think you did a very good job at dissecting this attack. Kudos
- jessaustin 11y agoInstall things through the package manager when possible. This makes it easier to uninstall the software cleanly. Of course one uses the system package manager when possible, and a specialized package manager like npm in most other cases, but not all is lost when one needs a recent version of some autotools monstrosity. In that case just install it using GNU stow. Then uninstalling is just an invocation of "stow -D" (plus deleting a single directory). But that only makes sense for a box one physically possesses. Why on earth would one reuse a VPS? Nevertheless, I'm glad that TFA did, because the investigation was very interesting.
- dgregd 11y agoThe most important thing to do after a server setup is to enable firewall. By default block all ports on public IP and then open ssh, http and https. There are tools like ufw, just few commands and your server is secure. This is much better approach than securing all services and system deamons run on a server.
- pixl97 11y ago>This is much better approach than securing all services and system deamons run on a server. Um, no. You should absolutely be doing both. Defense in depth is the only way to remain secure in failure of any other component. A firewall WILL NOT PROTECT YOU FROM THAT. For example, you just put on the firewall and leave a daemon running on localhost as root. The firewall allows connections from localhost by default (as most do). An attacker finds a hole in your webserver and is able to run scripts in the httpd context. The script makes a connection to the daemon on localhost, exploits it, and now has root privileges. With root the attacker modifies the firewall rules to allow access. You are now fully exploited. Had your daemon been secured they would have only been a takeover of httpd and daemon accounts and not a full system compromise. Privilege separation is extremely important.
- AznHisoka 11y agoIs there a Saas that helps detect these open instances for you? Sort of like the opposite of pingdom. I want an email when something is accessible because I might have changed a firewall setting that might have accidentally opened the port.
- sandstrom 11y agoThere is a tool called Detectify[1], which is quite nifty! Not sure if it's a good fit in your case, but possibly. [1] https://detectify.com/ https://detectify.com/
- yclept 11y agoSeems like any monitoring saas that allows you to define custom alarm criteria or plugins would let you do this. I know rackspace cloud has this.
- jlgaddis 11y agoAny port scanner will tell you this.
- barrkel 11y agoIn addition to what others say - first firewall, then lock down services in turn - consider running OSSEC or another intrusion detection system if you are concerned about intrusion. But really this is the OP's problem: he ran a system open to the internet without any apparent analysis of attack vectors. Server was compromised due to incompetence - not developer incompetence, but operations incompetence, a different discipline.
- rmdoss 11y agoCan't agree more with the OSSEC recommendation ( http://ossec.net http://ossec.net ) . Been using it extensively and gives an early warning system if something is going on (both in terms of files changes, backdoors and log trails).
- revelation 11y agoHow can you make a software intentionally insecure and then also bind to all interfaces by default? This isn't very difficult, just make it bind to localhost by default.
- pixl97 11y ago>How can you make a software intentionally insecure and then also bind to all interfaces by default? In the last 20 years I've been working on systems, that is the default on way too much. It's only been in very recent history that things come more secured by default. In the past popularity was strongly correlated with ability to set up. 'Works by default with no security' is really easy to setup and reduces the number of complaints that "product is too hard to use, when with $x product instead".
- KyleBrandt 11y agoConsidering the skills demonstrated in this analysis, you might discover getting started with configuration management (puppet/chef) is easier and might take less time then you expect (even in a fast moving early startup). Might have totalled less time then this hack took up.
- dethos 11y agoAs it was said in the "background section" this is not a problem with redis but a configuration/ops problem instead. Either way, you should always check the open ports and listening services in a given machine. In this case for a quick check you could use: https://www.isredisallsafe.com/ https://www.isredisallsafe.com/
- pixl97 11y ago>this is not a problem with redis but a configuration/ops problem instead. Default system wide failure is a redis problem. Any modern daemon should come with configuration files that a) don't allow the application to run as root unless uncommented by user action, and b) bind to localhost. Further configuration should be necessary to break the system as thoroughly as this user did.
- SFjulie1 11y agoKss, being compromised by a script kiddie is bad. But I am puzzled: a running process like redis is supposed to run in a chroot, and /tmp is supposed to always be mounted with no exec, the chroot and process should have been runned with seteuid/guid redis... I normally have backend technologies confined in a jails with pf authorizing only inbound & outbound traffic on the legitimate interface+port, and you need no ssh access in a jail. Looks to me there is a cascade of sysadmins wrong practice. Devops and modern "full stack devs" seems to have lost their basics.
- jlgaddis 11y agoI wear a "sysadmin" hat (among many others) but not a "dev" hat. Every sysadmin I know who's worth their paycheck does the things you've mentioned above (and others), every time. Perhaps this is what happens when you take a "dev" with no real sysadmin experience and make him/her responsible for these machines? It's like giving my neighbor's kid with a small dirt bike the key to my Harley and sending him out on the highway -- bad things are bound to happen. It reminds me of the common solution that many "amateur sysadmins" have when they encounter SELinux issues: turning it off. :/
- SFjulie1 11y agoSelinux is crap. To complex to configure, and poorly configured by default. process poorly packaged crash and nothing in the logs. You disable it, see there was a warning about a poor permission and shout at selinux for making you lose days worth of works by poorly logging. Btw, I am a dev. But I have to sometimes go in production see servers because I need to help production sysadmins troubleshoot the weirdness of todays mis-performing easy to use solutions (selinux, systemd, rsyslogd, packaging). When you have to recompile centos or debian packages to have them support vanilla features and you are a dev, it is boring.
- golergka 11y agoI'm not a server developer, but words "play around" in context of production server do seem like a red flag to me.
- jonesb6 11y agoAs always I think the answer is in the middle. You can't have a piece of software that defaults to insecure and expect everyone to have the common sense and skill to secure it. You rarely can release a piece of software with hard security defaults and expect it to have enough usability to gain critical mass / adoption / be popular. I remember when I first learned web development and I absolutely had a belief that the defaults would protect me from royally fucking up. Postgres security defaults for example. And although this is obviously not true in reality, we can't let ourselves overlook it just because.
- atmosx 11y agoThis is strange[1]. Why on earth would you have a Redis DB facing the internet?! [1] https://www.shodan.io/search?query=6379 https://www.shodan.io/search?query=6379
- kevinchen 11y agoBecause out of the box, Redis binds to 0.0.0.0 (and many people don't bother changing the defaults)
- viraptor 11y ago> "So we’ll just delete these files, kill the running process, and be done with it, right?" No, no, no, never! You can poke around the server to figure out how you got attacked, but afterwards the only way to recover is to rebuild from scratch. Someone had root on the system. For all you know, you're currently running a recompiled version of kernel/init, which will recreate all the original scripts when you're not looking. > Lessons Learned I'd also add: learn about Apparmor / Selinux and use them in production. Even if redis was running as root, you could still prevent most of the attack (no file writing, no execution, etc.)
- eeZi 11y agoTry Ansible instead of Puppet/Chef, it's much easier to use!
- sandstrom 11y agoA neat command is `sudo lsof -i -P` which will show something like this: java 1114 elasticsearch IPv6 TCP localhost:9200 nginx 1169 www-data IPv6 TCP *:80 nginx 1169 www-data IPv6 TCP *:443 dhclient 1620 root IPv4 UDP *:68 sshd 1697 root IPv6 TCP *:22 I.e. services listening on various ports. Knowing which services are running plus a firewall as an extra precaution goes a long way to avoid issues like this.