7 ms·
M64.pl – how I learned that the default settings are not production settings
- eli 12y agoDisabling password logins is good, but there are other steps you could/should take. These are all pretty simple: https://library.linode.com/securing-your-server https://library.linode.com/securing-your-server Also, I would guess the attack is fully automated.
- hackerboos 12y agoI'm scared noobs are going to blindly turn off passwords in their SSH configs without correctly setting up keys.
- eli 12y agoLinode has "console" access through the admin. But yes, good point. One should follow the linked document steps in order.
- forgottenpass 12y agoYeah, probably. But mistakes are how we learn.
- artursapek 12y agoThat's a legitimate fear. I added a few words pointing that out in the takeaway section.
- edoceo 12y agoDuh. Always lock SSH down
- scott_karana 12y agoI don't think the www user has a password in any distro, by default. Meaning one of his administrators must have set one (likely for temporary troubleshooting), kicking off the whole issue... edit: if SSH access was indeed the first cause. Running any upload-and-run script as `www` would let them set the password themselves, I think.
- euank 12y agoFor anyone wondering how to properly troubleshoot in this manner without breaking things: Run 'sudo su www -s /bin/bash'. using '-s /bin/bash' will override the usual nologin shell, and running 'su' as root will mean a passwordless account can be su'd too. This will allow you to try accessing files and directories as if you had the user 'www's privileges without having to make the 'www' account regularly usable. You should never set a real shell or password for any accounts that a real user will not be using.
- Maxious 12y agoProbably an administrator troubleshooting, www user should also have a nologin shell by default which cannot be changed by a normal user via an uploaded script. Maybe default settings are production settings after all!
- dbalan 12y agoUsing a tool like fail2ban is also favourable. Most of the bots will give up and look for other targets even if you ban them for a hour.
- kbar13 12y agofail2ban is a log prettifier, doesn't actually add anything to security.
- mp4box 12y agofrom the user's profile https://news.ycombinator.com/user?id=kbar13 https://news.ycombinator.com/user?id=kbar13 "I work for Linode, but everything I say is me being an idiot." so..
- threeseed 12y agoGiven Linode's security record being an idiot or not seems irrelevant to me.
- forgottenpass 12y ago...so what exactly is it you're trying to read into his self-deprecation? Because it's not there.
- agranig 12y agoCould you elaborate on that? fail2ban seems quite popular for blocking malicious IPs.
- pronoiac 12y agoNo, fail2ban watches logs for repeatedly failed login attempts, and then blackholes given IPs using firewall rules. I prefer denyhosts, which is more ssh-specific.
- danieldk 12y agoThe problem with fail2ban is that you have another possible vulnerable program that an external user can feed information to (although very restricted). I have always been surprised that fail2ban is so popular, since iptables can do rate limiting, etc. So, it's easy to block most attacks with the in-kernel firewall: http://www.debian-administration.org/articles/187 http://www.debian-administration.org/articles/187
- mqzaidi 12y agoYou should also run logwatch - it will show you all the attempts that keep happening on any public servers running ssh. I once had to look at a box compromised, which was then used to brute force other servers, and it contained a file with 100-200 passwords for other hosts it brute-forced.
- keeperofdakeys 12y agoAnd if you do want to use password authentication, simply whitelist the allowed logins. This assumes you are using a strong password.
- threeseed 12y agoOther options: Run it on a different port, use certificates or YubiKey for authentication, firewall it to your home computer, use Fail2ban or Denyhosts.
- keeperofdakeys 12y agoRunning it on a different port is only protection against random attacks (sweeping ip addresses looking for common passwords). This seems much more like a targeted attack, someone took the time and effort to try to get in. A small port scan isn't hard, and will get around changing the port. Of course, the other things you've listed are solutions for this. I was just posting a single option to complement the other comments.
- deckiedan 12y agoI have an ssh-access group, then the ssh server config is limited to that group. It makes it very easy to keep track of, and also works pretty well for maintenance with ansible.
- stiff 12y agoI once had a box "bot-netted" by some automatic scanner after I set up postgres on it, being in a hurry after another box crashed unexpectedly. It is stunningly idiotic to me that SSH seems to be enabled by default for all users (at least on Ubuntu server).
- zaroth 12y agoAll users with a password set. So 'setting the password' has perhaps an unclear side effect. I guess if you disable password logins, someone would need to get a pubkey into a user's ~/.ssh directory.
- tankenmate 12y agoOr more importantly, once in they insert a public key in there so if you naively disable password access they'll still be able to get back in. Change the password AND remove any keys in ~/.ssh. Also a debsums or rpm -qV is a good idea. A complete re-install is my preferred method though.
- zaroth 12y agoA couple funny things; the intruder changing the www password, and running a mining script which is going to generate basically zero satoshi before getting shut down. I guess change the password so someone else can't break in after you? I can't explain the mining script. Maybe they want to alert you the box is owned, as a means to find only targets who don't wipe their system after being infected? What's the point...?
- Kiro 12y agoEven if the script only manages to mine very little it would still be worthwhile considering this is probably 100% automated.
- nodesocket 12y agoAlso, heads up. Don't ever run Elasticsearch open to the world in iptables. This is actually very common because users want to use Kibana (http://www.elasticsearch.org/overview/kibana/ http://www.elasticsearch.org/overview/kibana/). Dynamic scripting is allowed by default (< version 1.2.0) and be exploited easily. It is especially nasty if you run Elasticsearch as root (also very common).
- x0x0 12y agoyeah, "special" people at my work did this. Along with installing / automatically starting elasticsearch on every ami so that every box could be rooted, not just the search cluster. And putting the fucking admin pam on the boxes instead of learning about agent forwarding. And not securing processes in any way -- why would you use chroot or docker, that would throw up obstacles in front of the nice chinese people who want to use your ec2 boxes. it was a very long week.
- buro9 12y agoOn the cheap Linode instances I setup for others, I always install this: http://www.configserver.com/cp/csf.html http://www.configserver.com/cp/csf.html It includes the equivalent of fail2ban, manages iptables, watches log files, can email on failed logins, processes gone mad, and a few other things through a very simple config file. It's that simplicity of the config file that I like, it means I can hand it to someone and know that they're unlikely to break something due to misconfiguration or failure to actually apply something. This is on top of obvious things like running ssh on a non-default port, turning off password auth and restricting ssh to named users only (never root). The install script is easy enough: http://www.configserver.com/free/csf/install.txt http://www.configserver.com/free/csf/install.txt so it's really the case that I can do whatever work I'm doing for someone (hello Wordpress) and this can be my simple security solution which at least means I'm sure that I haven't left every window open.
- deoxxa 12y agoWhy do you run ssh on a non-default port?
- rurounijones 12y agoBrute-forcers only hit the default port. If there is no response on the default port then they just move on to greener servers. Moving off the default port is a good way to avoid your logs getting filled up with failed login attempt messages.
- tacticus 12y agoThough something like fwknop would be a nicer way of resolving that.
- thejosh 12y agoWhy not? You can always create an entry in .ssh/config for the host with the port if needed. Host example.com IdentityFile ~/.ssh/id_rsa_example User foobar Hostname 127.0.0.1 Port 1234
- kybernetyk 12y agoI hope you have killed and reinstalled the box because once a system has been compromised you can't ever trust it again.
- michh 12y agoIn this case, if I read the article correctly, only a single account was compromised. I'm assuming the www user doesn't have sudo privileges (though I might be wrong as it shouldn't have a password either and it clearly did..). Sure there might have also been a local privilege escalation vulnerability, so rootkits are definitely something to check for. Depending on the situation, re-installing might be less work, so then it'd make sense. But especially if you're not the only one using the machine, having to reinstall the OS every time a single user fucks up and has 'password' for a password, it's going to get really tedious really quickly.
- artursapek 12y agoI made sure. www has no privileges. It can't read root's home directory or touch any of its processes. If it could, I'm sure I would be busy cleaning up a much larger mess instead of blogging about it.
- kybernetyk 12y agoI'm not trying to imply you didn't check thoroughly because I don't know the whole story and your background. But for me personally I wouldn't take the risk - even if the compromised account was 'nobody'. I certainly don't know enough about computer forensics to make sure that a system hasn't been altered (and I dissect malware for recreational purposes). So unless it's a system that hosts something completely unimportant I'd take the weekend free to wipe + reinstall the stuff. Even recently a linux local privilege escalation has been discovered[0]. So who knows what the hacked www user really had access to. [0] http://seclists.org/oss-sec/2014/q2/467 http://seclists.org/oss-sec/2014/q2/467
- jon-wood 12y ago> But especially if you're not the only one using the machine, having to reinstall the OS every time a single user fucks up and has 'password' for a password, it's going to get really tedious really quickly. This is what automated installs are for. Kill the box, fire up a new one, and run the provisioning scripts. As a bonus you've got actual documentation showing how the box should be set up.
- adders 12y agoThe script appears to be be at least part minerd, a program for bitcoin/litcoin mining. Details of minerd at https://bitcointalk.org/index.php?topic=55038.0 https://bitcointalk.org/index.php?topic=55038.0
- viraptor 12y agoWhile someone could set the password on the www user manually, I get the impression that the author is the only admin and would remember that. Since the only authenticated user is www, it's likely someone just exploited his application / webserver and set the password on www for later ssh logins. The author will soon know that if the situation repeats without the ssh access.
- mistaken 12y agoThe m64.pl executable seems to be a vanilla version of cpuminerd. By default it connects to http://127.0.0.1:9332 http://127.0.0.1:9332 and mines using the scrypt algorithm. It could have been started with different parameters, or the port was tunneled to a pool. Would be interesting to know the ip. off: does anyone know which font he uses in the terminal? Looks awesome.
- euank 12y agoAssuming you're referring to the font of the blog's text... Firefox: Right click, inspect element, click the "fonts" tab on the right. Chrome: Right click, inspect elmeent, look at "Computed" styles on the right, find Rendered Fonts. Ironically, the answer is he's not using a font. It uses Times New Roman on my Chrome and DejaVu Serif on my firefox because those are my default system fonts, not because he specified them. If you look at the source of the webpage, you'll see he has zero stylesheet links, all his styles are inline in <style> blocks... So searching font in that one source page is sufficient to find he only styles code blocks, the rest is default. If you're referring to some other font, I have no clue, sorry!
- mistaken 12y agoThanks, but I was referring to the screenshot :)
- artursapek 12y agoIt's http://www.fontsquirrel.com/fonts/M-1m http://www.fontsquirrel.com/fonts/M-1m. I don't think I'll ever switch from it. Very narrow and efficient font.
- cel1ne 12y agoBeware of this: http://unix.stackexchange.com/a/8886/11172 http://unix.stackexchange.com/a/8886/11172 To fully lock down SSH you have to check PAM config as well. My 2 cents about defending against scripters: * SSH: non-standard port, no passwords, special user-group who is allowed to login. * WWW: redirect everything to https. For some reason bot-attacks dropped from hundreds to none over night. :D
- Fuxy 12y agoI was hoping for a more detailed account of how he managed to log in as www. That looked like a pretty amateur attempt yet he managed to log in as www. Was there no password for www? Why was it even available to SSH into? This leaves a lot of questions unanswered.
- andyhmltn 12y agoSeeing a bruteforce in the logs isn't at all uncommon. I see it every time I have a new server. Turn off password authentication and disable root login via SSH. That way, you're golden!
- nathell 12y agoA similar if not same thing: http://ubuntuforums.org/showthread.php?t=2203663 http://ubuntuforums.org/showthread.php?t=2203663
- xtrumanx 12y agoI've been considering moving an app off of Heroku and to Digital Ocean so I could learn the ropes of administrating a Linux server but I feel there's so much I don't know that I should know if I wanted to effectively manage my own server. If I got hacked with m64.pl, I'd get as far as running top and killing the process. His immediate assumption was that someone got a file that file on to his server and so he went on to check the logs and found all the failed login attempts. I would have wondered what the file was, googled around and sat around confused. Luckily deploying a new instance on Digital Ocean sounds simple enough so I would probably just do that but that approach would leave me with a sense that I don't know what I'm doing at all. There was a time when my solutions to problems on Windows machines was to restart and/or format but I don't do that anymore but still meet people who do. I'd rather not start from square 1 when I make the leap to Linux. What are the things one should know to legitimately say they can manage their own Linux server? I could try to just wing it and solve every problem as I encounter them but for right now I'd just like to identify the gaps in my knowledge.