48 ms·

How I spend my first 5 minutes on a server

• anderiv 14y ago
  I like the KISS aspect of this, though for accountability purposes, I prefer each user to have their own account. One correction for you: the sshd_config lines should have no "=" symbol.
  • plusbryan 14y ago
    Good catch! Fixed
• zrail 14y ago
  The first five minutes on any of my servers involve giving it a name, installing puppet and adding the server name to my central puppet config. You seriously do this by hand for every server? That seems error prone and a huge waste of time when tools like puppet and chef exist.
  • trxblazr 14y ago
    I think the author meant for this to be a simple set of security guidelines for beginners. But yeah, any reasonably large prod stack should probably be deployed and setup with puppet and what not. It's DRY and less error prone than a human low on sleep.
  • ams6110 14y ago
    Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it.
    • WALoeIII 14y ago
      You don't have to re-learn puppet/chef/cfengine/bash script every time. You learn it once. It requires much less effort to "maintain" than manually updating more than 1 box.
    • dbarlett 14y ago
      I was in the same position - too many distinct environments for bash/Fabric, too little time to learn Chef/Puppet/CFEngine. Ansible [1] seems like a good compromise: you get the simplicity (runs over SSH) and host targeting of Fabric with the declarative nature and idempotency of the more complex tools. You can start with all-in-one "playbooks" [2], then split out tasks, handlers, Jinja2 templates, files, and variables [3]. As an aside, I think the default fail2ban config is too loose and quiet. Here's [4] an Ansible task file that configures it to be more aggressive and send notification emails. [1] http://ansible.cc/ http://ansible.cc/ [2] https://gist.github.com/dbarlett/5079802 https://gist.github.com/dbarlett/5079802 [3] https://github.com/fdavis/ansible-best-practices https://github.com/fdavis/ansible-best-practices [4] https://gist.github.com/dbarlett/5079715 https://gist.github.com/dbarlett/5079715
      • slurgfest 14y ago
        What do you usually do with these fail2ban notifications?
      • gingerlime 14y ago
        I'm using fabric quite frequently, and am trying to understand what makes a configuration management tools a much better choice. I'm currently using fabric for anything from bootstrap a new environment from scratch, via restoring a snapshot from backups, to pushing code updates stored on git. Perhaps I'm being really daft, but it always evades me why something as simple as sudo("apt-get install -y <name your packages>") needs to be replaced with - name: Install prerequisites for PPA management apt: pkg=$item state=present update_cache=yes with_items: - python-software-properties - software-properties-common I do use a pretty homogeneous environment, which makes things simpler, but this is a deliberate choice to avoid complexity. If I know all my hosts are e.g. debian 6, then what makes ansible/chef/puppet so much better than fabric? I'm not trying to be provocative or negative. I'm really trying to understand the supposedly big difference between what's labelled a deployment tool, and configuration management tools.
        • vacri 14y ago
          that same line in puppet is "package { 'packagename': ensure => installed }". Or you can use 'latest' to keep it updated. multiple packages can be done this way if you have a lot: $prep_packages = ['package1', 'package2', 'blahblah'] package { $prep_packages: ensure => installed } You can of course use extra arguments if you need them.
    • grey-area 14y ago
      Puppet and Chef are yet another thing to learn and maintain, if the guy is a part-time admin with a lot of other responsibilities and a small number of servers it may not be worth it. Speaking from experience as a part time admin, it is definitely worth it to automate the process, even with one or two servers. Any build system will repay in spades the first time you have to set up a new server or rebuild an old one, esp. if you're in a hurry after a server failure. That said Linode offers other easier options (Linode specific of course). You can take the commands usually executed, and put them in a shell script in the language of your choice which is uploaded and executed on first run (they call this StackScripts - there are lots of examples and helper scripts on their site, but the scripts are really very basic). It's perhaps easier than learning a DSL as it's just codifying the same commands you would run via ssh by hand. You can also clone nodes so you could set one up in a known good state and clone that each time. This does tie you to linode, so long term one of the other options available would be worth looking in to.
    • kelnos 14y ago
      I think being a part-time admin with other responsibilities is a perfect reason to learn something like Chef or Puppet. It'll end up saving you a lot of time that you can devote to other things.
  • baghali 14y ago
    Without Chef there would be no way for me to rollout a new server in our cluster. Investing time into Chef was one of the greatest things I ever did. Chef is the best documentation of our infrastructure. The second best tool I'm using is fpm[1] to make custom debian packages. [1] https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
  • Volpe 14y ago
    While not directly related to this article, last I checked Puppet/Chef don't have particularly useful windows support. Is there an chef like solution for windows?
    • tedchs 14y ago
      Puppet has great Windows support nowadays; not sure about Chef.
• deleted 14y ago
  [deleted]
• shuzchen 14y ago
  I dunno about this guy but apt-get update/apt-get upgrade takes upwards of 15 minutes on new installs (as does yum upgrade on rhel-compatible systems). Kinda eats up into my first 5 minutes on a server.
  • cdjk 14y ago
    Look into apt-cacher-ng. I have a slow internet connection at home and it made updates go a lot faster for multiple machines. Newer versions (in wheezy, not squeeze) even work with yum.
• philfreo 14y ago
  While I agree these practices seem pretty safe/standard, it's still enough manual work that many people will just skip half of it out of laziness for small projects. Does anyone have good reusable Puppet (or bash scripts) published that they use on all new servers?
  • vahe 14y ago
    Here's something I wrote very quickly (bash script): https://github.com/vahek/vSetup https://github.com/vahek/vSetup It does most of the stuff mentioned in the post, however doesn't setup automatic updates or Logwatch.
    • harshreality 14y ago
      (to both corin and vahe): That kind of shell scripting is the horrorshow that motivated configuration management tools* in the first place. Shell scripts require a lot of added complexity to manage multiple heterogeneous servers, or to be idempotent. * Puppet, Chef, Ansible, Salt, CFEngine, etc.
      • vahe 14y ago
        I agree. However, there is no learning curve to get started using this script and was put together in a few minutes.
  • corin_ 14y ago
    A personal script, focuses more on getting a server ready for my preferences than it does on security: http://pastie.org/pastes/6376503/text http://pastie.org/pastes/6376503/text
• deleted 14y ago
  [deleted]
• ultimoo 14y ago
  Nice writeup. Although I prefer to automatically configure using Chef, I didn't know of fail2ban and logwatch -- I will definitely look into those. Also, since this post is for beginners, you should mention about restarting the sshd over the lish tty and not over the ssh ptty.
• networked 14y ago
  Beginner or not, you should probably use visudo [1] instead of vim /etc/sudoers for the sanity checks that it provides, if nothing else. A botched edit of /etc/sudoers that locks you (along with every other user) out of administrative access is an unpleasant way to learn this. [1] http://linux.die.net/man/8/visudo http://linux.die.net/man/8/visudo
  • Maxious 14y ago
    Similarly "ufw allow from {ipaddress-you-will-access-from} to any port 22" sounds like a good way to accidentally lock yourself out unless you have an out of band backup
    • loeg 14y ago
      While true, I'd like to note that the article's author does have OOB access.
    • andrewvc 14y ago
      FWIW I never understood UFW over straight IP tables, is it really easier to read?
      • luser001 14y ago
        IMHO, yes. At least on Ubuntu, it's never been too clear to me how I should save my rules so that they come back on startup. The ufw man page is pretty decent.
        • leetrout 14y ago
          I've not tried anything complex with UFW so I still use iptables on my bastion host that handles my vpn tap. It's not terribly complex to make rules come back on startup (but probably more involved than one would hope). For anyone else that followed the thread to this point- this advice on bringing iptables back up on reboot worked for me http://rackerhacker.com/2009/11/16/automatically-loading-iptables-on-debianubuntu/ http://rackerhacker.com/2009/11/16/automatically-loading-ipt... YMMV
• cdjk 14y ago
  I've never understood the point of fail2ban if you disable ssh password authentication. Yeah, it might eliminate some spam in the logs, but if you only allow key-based authentication that doesn't really matter.
  • denrober 14y ago
    My only thought is even with password auth. turned off hostile persons can still connect to the ssh daemon and might discover and unknown vulnerability or some type of dos. If you know someone is trouble why let them in even if you are wearing armor?
    • numbsafari 14y ago
      It'd really suck if they could forge traffic from you that looked like a failed login attempt and that resulted in you getting banned.
      • anderiv 14y ago
        That is why fail2ban allows you to configure a whitelisted IP (your office, VPN, etc.) that will never get banned.
  • MichaelGG 14y ago
    fail2ban can deal with much more than SSH login attempts. I believe you can configure it to do arbitrary actions on arbitrary log events. Perhaps it comes with a block for HTTP, too? Seeing as he blocks SSH and only allows it from his IP, there won't be any SSH attempts reaching sshd at all, so fail2ban wouldn't be doing anything if it was looking at SSH only.
  • petejansson 14y ago
    Keys can be brute-forced.
    • deleted 14y ago
      [deleted]
• serf 14y ago
  don't vim sudoers. the days of the password are not over. fail2ban is next to useless with key auth.
• vahe 14y ago
  I also recommend changing the default SSH port.
  • rdl 14y ago
    I hate it when people do that, myself. Especially when you have a lot of other tools, many of which don't take port arguments easily. IMO best practice is to firewall off everything except some bastion hosts or VPN gateway.
    • plusbryan 14y ago
      Totally agree. If you stick to good security practices, working around a non-standard port is an unnecessary annoyance imho.
    • pseudonym 14y ago
      I actually do this on my personal server because more than one public wifi (and one of my previous jobs, at a public high school) disallowed outbound 22 connections.
  • afhof 14y ago
    Don't do this. It adds almost no extra security and makes it hard for routers that prioritizes port 22 traffic as interactive.
    • kristofferR 14y ago
      Sure, it'll not stop dedicated manual intrusion attempts, but it will actually prevent a ton of automated bots from even just trying to connect with common passwords through SSH.
      • corin_ 14y ago
        Which is irrelevant if you have any one of: strong passwords, no passwords, fail2ban
• petsos 14y ago
  I find it very bad practice to blindly pass -f to commands. chown deploy:deploy /home/deploy -Rf
  • X-Istence 14y ago
    "-Rf" not found... :P flag arguments should go before any other arguments to be compatible with the most Unix systems.
  • plusbryan 14y ago
    Thanks for the catch! Fixed
• throwaway503 14y ago
  And, when some dev screws up by pushing a poisonous build, there is no way to figure out who to contact to fix. I generally have an extensive pam+LDAP setup to control Logins. Having said that, if I am trying to use keys, puppet filebucket works for me.
• niggler 14y ago
  How does UFW compare to the rhel firewall configuration utilities?
• jedberg 14y ago
  To prevent having to log into root via remote console, set up a second backup account with an ssh key and sudo access, and then put that key somewhere safe. The chances of both that and the deploy user getting corrupted at the same time is unlikely.
  • plusbryan 14y ago
    That's a good alternative. At least for Linode, remote console is fairly easy, but some don't offer this feature.
• rdl 14y ago
  My big thing is making sure 1) the machine comes back up cleanly after a reboot and 2) is current on all patches 3) is running as little as possible. Also a big fan of externally verifying what ports are open, and making sure the system is in monitoring, backup, config management systems. Config management is kind of optional if you have a small number of servers which don't duplicate configurations, though, and there's often no need to back up the OS, but any data should be backed up automatically.
• plasma 14y ago
  Is there a web app that I can just have control my servers from? I'd like to do puppet/chef, but I read that its not 100% working for Windows yet (I need to manage both) and it sounds overly complicated.
• ck2 14y ago
  Your first 5 seconds should probably be w last dmesg
  • stevekemp 14y ago
    Add "free" and "top" there too.
    • cs02rm0 14y ago
      I've had df -h write out to the motd before for clients who insisted on tiny slices of disk, no automated monitoring and no manual checking of disk space even after it chewed itself up. I'm not sure it helped, but it made me feel better.
• twodayslate 14y ago
  Why install fail2ban and logwatch instead of csf?
  • shredfvz 14y ago
    http://configserver.com/free/csf/license.txt http://configserver.com/free/csf/license.txt Looks like an interesting piece of software nonetheless.
  • davidandgoliath 14y ago
    Does it still have all of those root exploits?
    • twodayslate 14y ago
      I wasn't aware of any. If a malicious user has root access though, you are fucked anyways. Please correct me if I am wrong.
• Matsta 14y ago
  https://github.com/Xeoncross/lowendscript https://github.com/Xeoncross/lowendscript I just use this bash script when setting up a new VPS. It pretty much takes care of it all for you, plus it sets up a ngnix/mysql stack already optimised for law RAM machines. Also sets up 3proxy which is handy for viewing Hulu/Netflix :)
• shizcakes 14y ago
  The premise of this thing is not good advice. 1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one. 2) Everything else listed in this document should be configured by a configuration management system. 3) "User account sync tools" should have no place in a modern infrastructure, you should use your configuration management tool to (at the bare minimum) deploy /etc/passwd and /etc/sudoers across your infrastructure. 4) You should not use shared/role accounts. The "incremental cost" is paid back immediately when someone leaves your organization; having to update everyone of a changed password or having a password change have any negative impact at all should not be a thing your company does. This stuff isn't hard. It's worth doing right.
  • mef 14y ago
    For #4, wouldn't the only change when someone leaves the organization be to remove their key from authorized_keys for the shared account? Why would anyone else have to be updated?
    • mbreese 14y ago
      But they still would know the 'deploy' password needed for sudo access. And while you could be relatively sure that they couldn't get access, you still couldn't be completely sure since they did have sudo access to begin with. So, the best thing would be to change the shared password. That could be avoided with non-shared accounts.
      • luser001 14y ago
        Are you insinuating that the user could have used sudo access to install a backdoor of some sort? If so, changing the password won't stop them either. Am I missing something?
    • joelesalas 14y ago
      It's a basic principle of security. Each account represents one person so that you have a full audit of who did what by watching the activity of a given user account. If everything is run as "devops" user for example, you have no idea who actually performed a given task. Was it Bill, or was it an automated job? PCI-DSS requirements also affect your model for user accounts (hint: shared users are often not compliant). From the perspective of a sysadmin, this article has a lot of issues and it's inadvisable to follow its recommendations. Who doesn't use a hardware firewall? Who exposes ssh to the internet (requiring fail2ban) when a VPN server is much more secure and easier to use? Setting up an LDAP server is really easy and costs nothing. There's no excuse for shared accounts.
• magnetikonline 14y ago
  I wouldn't bother with fail2ban considering password based SSH logins are disabled (which is good). Since the author is using ufw to control iptables, better to just use "ufw limit" rules for SSH port 22 to slow down the rate of any automated SSH bots trying to give your server a workout.
  • danieldk 14y ago
    Indeed. I have always been a bit worried about such approaches, since they parse log files and attackers have some control over what is written to log files (user names and host names).
• beagle3 14y ago
  1. You should do "apt-get dist-upgrade" to get new kernel packages as well, otherwise you are stuck on an old kernel. (You might want that. I prefer updated kernel for the security, firefoxen, etc.). "apt-get upgrade" will only update existing packages - but the kernel updates require new packages to be installed. 2. If you're on ubuntu, root already has no password, and your initial setup user (whether it is called "deploy" or "kilroy") is in the sudoers file. 3. Other things I install in the "5 minutes with server" are: htop molly-guard screen git-core etckeeper git-core because I prefer my etckeeper in git, but if you want it in bzr you don't need git-core. INSTALL AND CONFIGURE ETCKEEPER AS SOON AS YOU CAN, seriously. You need it. You'll thank me when you try to figure out when and how something in /etc got borked. (you need to edit /etc/etckeeper/etckeeper.conf if you use gif. You need to do "etckeeper init" and then "etckeeper commit" to establish the baseline) molly-guard stops you from rebooting the wrong server screen (or alternatively tmux) lets you keep your session open through ssh session disconnects (e.g. when moving from wifi to 3G, or between 3G towers that give you different external IP). The most useful way to use screen is "screen -xR" which also lets you share your session with someone else should you need to.
  • mercurial 14y ago
    I love etckeeper, but the use of a configuration management system (puppet/chef/salt...) tends to reduce its usefulness.
    • beagle3 14y ago
      On the contrary - it is useful exactly to figure out when and what manual changes were made outside of the automated system.
      • mercurial 14y ago
        I'm not sure I get your "on the contrary". Any configuration change on a managed configuration file is transient and bound to overwritten next time state is restored. This makes etckeeper absolutely less useful compared to a system where there is no way to restore state and an unfortunate "rm /etc/passwd" has little remedy. It does not make it "useless", but it's definitely less needed.
• bcl 14y ago
  Don't forget: netstat -ntap | less ps aux | less Also check to see what's enabled to run at boot time via whatever your flavor uses. Check for unusual daemons, ssh running on other ports (yes, the provider pre-loaded systems with a back-door ssh without disclosing it to us). This is especially important when you are taking over admin on a server you didn't setup yourself. Other folks have weird ideas on how to admin things. Like webmin for example... I also like epylog for finding unexpected stuff in the logs.
  • ck2 14y ago
    to see what's enabled to run at boot time what's good beyond this: chkconfig cat /etc/rc.local
    • majelix 14y ago
      chkconfig is redhat specific. I believe debian-varients use update-rc.d. Generically (for Linux), take a look at /etc/init.d , /etc/inittab, or /etc/rc?.d.
• bluesmoon 14y ago
  We do something similar, though we have a script that sets up the box. We use linode as well, so the script deploys a new box with a complex root password and my id_dsa.pub file. It also sets up /etc/skel and the profile files so that useradd uses them later. We don't use a single deploy user though, instead, each user with deploy perms is in the sudo group.
• nhebb 14y ago
  Looking through the responses here, I'm hopeful that someone will launch a "Sysacademy" variant for system administration training. There are tutorials scattered around the web, but (at least for those of us who don't know where to look), there doesn't seem to be one place that puts in under one umbrella.
  • bgilroy26 14y ago
    Hopefully there will be more than one! I had fun this past November with Snori74's Grep101 course. I don't think it's open right now, but when it runs, it's a great stab at exactly what you're describing. Initial blog: http://grep101.blogspot.com/ http://grep101.blogspot.com/ HN post: http://news.ycombinator.com/item?id=4726827 http://news.ycombinator.com/item?id=4726827
  • dbarlett 14y ago
    I have high hopes for Ops School (https://ops-school.readthedocs.org/en/latest/ https://ops-school.readthedocs.org/en/latest/).
    • manish_gill 14y ago
      Looks really awesome. Thanks! :)
• lhnn 14y ago
  Hey, fuck this author, right? Not even using enterprise level tooling for his deploys (Chef/puppet). Because if someone installed Chef, they wouldn't need to know this stuff!
• thaumaturgy 14y ago
  A few gentle suggestions: > The days of passwords are over. You’ll enhance security and ease of use in one fell swoop by ditching those passwords and employing public key authentication for your user accounts. ssh keys are better than passwords only because they contain (and require) more information. On the other hand, if your dev's machine is lost or stolen or compromised, so is your ssh key. This is especially a problem in environments with a shared account with full access, as you have. So, it's probably a good idea to make sure you're using a passphrase with your ssh key (during ssh-keygen), unless you need a passwordless login for a shell script or other automated remote system. > passwd deploy: Set a complex password - you can either store it somewhere secure or make it something memorable to the team. This is the password you'll use to sudo. Not necessarily. Anybody with access to the "deploy" account can use "passwd" to change its password to anything they like. (Edit: I'm wrong on this! passwd does require your current password; I've just gotten used to doing it for other accounts via sudo, which doesn't.) Changing the passwd on your own account doesn't require sudo. For this reason, I think it's better to simply give deploy nopasswd access to everything, and then delete and lock deploy's password to prevent it from being used at all (passwd -d -l deploy). You'll have effectively the same amount of security, but this way nobody will need to remember or retrieve a complex password, and you'll prevent, say, some accident in /etc/ssh/sshd_config from making deploy remotely accessible via a password. You can do something better than this though, but it takes a little effort. Deployment is often the same steps over and over again (an rsync or an occasional apachectl graceful in my case). You can give the deploy user nopasswd access to only a shell script that's writable only by root; this way, deploy can still do 90% or more of their job without ever being given system administrator rights. You do have to be a little careful writing shell scripts though -- $* and "$@" still trip me up once in a while. > Logwatch is a daemon that monitors your logs and emails them to you. This is useful for tracking and detecting intrusion. This seems of dubious security value to me -- probably better as a generic sysadmin tool, so that you get annoyed by noisy logs and seek out and fix minor problems instead of ignoring them. Thing is, if someone does get access to your server, you pretty much can't trust it at all anymore. With services like Linode, you're really better off just launching a clean new instance, re-running your setup script (if you have one), and moving your data over. I had to deal with the occasional intrusion in some pretty icky servers at an ISP once upon a time. We used rkhunter for a while, but I learned pretty quick that successful attacks against Linux servers are plenty enough sophisticated to alter all the basic tools that you would use to detect and remove the rootkit. There is one caveat: I've been playing around with the idea of setting up rsyslogd to route syslog messages to mysql, and then using mysql replication to have an up-to-the-second offsite copy. I'd combine that with Snoopy (https://github.com/a2o/snoopy https://github.com/a2o/snoopy) or something similar. The point isn't to try to clean up an intrusion, it's to see how the intrusion happened so that I could close that hole. I haven't gotten around to setting this up yet, so I can't say anything terribly smart about it. Finally: if you're going to have a problem with unauthorized access to your Linux or BSD server, it's probably going to be via one of its services, not via brute force ssh or anything similar. So, if you're concerned about this kind of stuff (and if you're being paid to be a sysadmin, you have to be), then you need to spend most of your attention making sure that your various services are set up correctly (apache/php/mysql/postfix/dovecot/spamassassin/etc. in my case).
• berlinbrown 14y ago
  It may seem like a waste of $20 a month if you aren't doing anything big with it. But you can setup a linux virtual server on linode and go through all of those steps on your own. Also test out your setup. http://library.linode.com/ http://library.linode.com/
  • voltagex_ 14y ago
    EC2 is cheaper if you're using the server for hours in a month - i.e. to learn, like I am.
  • calgaryeng 14y ago
    Or rock Vagrant. The ability to quickly setup & tear down makes for some quick experimentation.
• berlinbrown 14y ago
  What do you about all of those chinese hackers hitting your sshd server. I have 30 different ips and fail2ban doesn't seem to ban them.
  • ashray 14y ago
    Try csf. Works like a charm for me.
    • daemon13 14y ago
      Did you use it in production? Any pointers how to start? Good tutorials or walkthroughs?
  • luser001 14y ago
    Just curious. Have you enabled password logins on your servers, or do they only allow RSA key-based logins?
    • berlinbrown 14y ago
      I am new to this. I don't allow ssh logins password based. The hackers fail according to sshd but logwatch lists a list of chinese and russian attempts. I was hoping my firewall would block them. I tried entering a block of ips but some of the same ones are connecting. I don't know what this means: Illegal users from: undef: 20 times 183.60.177.246: 7 times 217.14.134.68: 7 times 219.149.30.170: 6 times
  • RobAley 14y ago
    Start by changing your SSH port from 22 to something else. It stops 99% of automated attacks. Beware that it is obscurity, not security though. That said, it does help clear some of the "noise" to focus on the other 1% of attacks where the attackers are actually a bit more sophisticated than a simple script.
• josephkern 14y ago
  Firstly, a nice checklist. Easy actionable steps, repeatable, and pretty much most of what you need. Secondly, you are about 4-5 hours away from learning puppet (or Chef) and making this checklist into actual code. Thirdly, you now have a checklist of items that you can use in a job interview if you get the oppertunity to gain a new-hire or an intern. Lastly, good on you for submitting this to a peer-review on HN. We can be a picky lot. TL;DR Checklists are a good first step for building a proper config management system.
  • laumars 14y ago
    We can be picky, but with good reason as security is an exact science and a costly one to get wrong. There's so much conflicting and out right bad advice posted online these days that sometimes it takes a picky community to help clarify the best practices. For what it's worth, some of the advice given in that article was worth mentioning (eg fail2ban, it's a great tool). But the shared account suggestion was the complete opposite of how you should be managing user accounts as you lose audit trails. And for that comment alone, I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate. That article has generated a lot of good discussion though. So even if just indirectly, it's been a valuable contribution to HN.
    • josephkern 14y ago
      Picky is good! We never grow without constructive critisim. And a community of practice may yeild better results than an individual (at least for simple things). > I'd recommend people read that article with a degree of scepticism before rushing onto any boxes they might administrate. Agreed. But the same could be said of everything; hackers and engineers, empiricists both. Some of the advice in the article would seem apporpriate to someone who had never worked with another senior engineer. No fault of the OP. System engineering and security are as much a learned craft as a science; a dialectic between sand castles (if you will).
• kbuck 14y ago
  Why install fail2ban? You already have SSH password auth disabled, and you only allow SSH connections from your office. Won't this just risk banning your own office if someone's SSH client is misconfigured?
  • vacri 14y ago
    fail2ban is useful for things other than SSH - I've seen it deal handily with people probing our asterisk server.
    • tuzakey 14y ago
      agreed, you can set up fancy jails for people scanning other services too, someone who probes SMTP/POP/IMAP doesn't need to hit SIP and SSH. Depending on the scenario you could choose to say block an entire netblock from hitting ssh after a single offensive IP probes a few services. Even a 10min jail time will cause most attackers to give up and move along to their next victim (unless you're being targeted.)
  • mich41 14y ago
    Seconded. Autobanning always causes trouble if enough people use the server from one IP address.
  • RobAley 14y ago
    If helps if a) you've misconfigured your "office only" rules, or b) someone has breached your office network/one of your local machines (but hasn't discovered any of your keys/passwords yet) and is trying to get into the server. It also helps with non-ssh services too. A belt-and-braces approach. If you accidentally lock yourself out using it, use your out-of-band access to re-enable your logins.
• jiggy2011 14y ago
  The guide recommends blocking SSH access to anything other than your own IP address. The problem is that my IP number sometimes changes at which point I end up locked out totally. So to get around this you either have to allow SSH from anywhere or you have to use some remote KVM system. Most of the remote KVM systems seem to be based on Java applets which is not really something you want to enable on your system. So what is the best way to get around this? Just open up SSH and be diligent about your SSH security? implement port knocking?
  • plusbryan 14y ago
    This may not work for you, but I have a similar situation when I'm at home or I'm travelling. To solve it I set up a VPN at the office (we use Meraki hardware, so it was literally just a click) and connect to this first - then connect to the server.
    • jiggy2011 14y ago
      I did consider that approach, but the closest thing I have to an office is my home (with a dynamic IP) anyway! I thought of setting up a VPN on a server, but all that would really do is move the problem from securing SSH to securing VPN.
  • RobAley 14y ago
    I've never tried this, but could you enable access from a given hostname rather than IP address, and then use a dynamic DNS system like http://www.noip.com/ http://www.noip.com/ to automatically update a hostname when your IP address changes? Two downsides I can see are that a) you would be vulnerable to DNS poisoning attack on the server side, but that would be the equivalent to just opening it up to all IPs, and b) dynamic DNS systems can take a short while to refresh the DNS record and/or it can get cached by your servers up-stream DNS servers. Some tinkering with your server to make it always use the authoritive DNS servers for your dynamic DNS hostname would help with that. I'm just thinking out loud, I've never tried any of that!
  • bluedino 14y ago
    Our provider allows us to modify firewall rules through their control panel. Perfect for when we have a dev working from home or a hotel room for a few days.
• waverider 14y ago
  1. I don't think it's a good idea to use the same account by multiple people. 2. And you're not using a configuration management tool (like SaltStack, which is also a remote execution engine) this will give you: - a central point to manage all your server - predictable configuration on all servers with the same role - a configuration documentation place (and even history if you git the confs) - will make managing multiple users a breeze 3. Use VPN and private services on private IP.
• itry 14y ago
  I use "adduser" instead of "useradd". Its more convinient as it does the remainings steps automatically for your. It comes with ubuntu by default. Any downsides?
  • kaoD 14y ago
    For some reason it's not working in last Arch's build. Manual "useradd" is such a pain in the ass...
• itry 14y ago
  Is there a benefit of using ufw instead of iptables?
  • cmpitg 14y ago
    It's easier to get started with UFW, and you don't need to learn much about networking to be able to setup your firewall. I haven't checked this out (I prefer iptables and never actually UFW), but it's said UFW is a front-end to configure iptables.
• plainOldText 14y ago
  The first two things I prefer to do after I log in for the first time (on ubuntu): > ufw enable > ufw default deny This way, after I log in, I will not allow anyone else to connect to my machine (I've had instances when by the time I changed my root password "bad guys" had already tried to connect to my machine). Of course after I do the server setup (which is usually a script that will change ssh ports, install packages, etc) I will allow other services in ufw.
  • stevekemp 14y ago
    Once upon a time somebody wanted to give me access to a new system - they ran "adduser steve", and set the password to be "steve". Two hours later, when I read the mail, that account had already been compromised. I knew it was a risk, but had no idea that it would happen so __quickly__.
• KevinMS 14y ago
  I was wondering about this the other day, is PasswordAuthentication no really that necessary if a strong password is used? Honestly, being a private key screw-up away from never being able to log in again scares me a little.
  • ziziyO 14y ago
    That's never the case. Any decent colo facility has some way for you to reboot in single user mode and remote in to that, IP KVM is one such tool.
  • mich41 14y ago
    Encrypt the key (ssh-keygen offers you to do so) and backup it to bazillion clouds.
• snambi 14y ago
  This is very useful
• belorn 14y ago
  It saddens me each time I see a security best practice guide that suggest turning off ssh access for root. Its a very useful feature, and the security industry should focus on the security problems rather than removing features without thinking about the actually benefits of doing so. Sysadmins with root access should be able to handle a random 8 character or longer password, and that number is large enough for a secure public accessible ssh. If your not a sysadmin or unable to remember a random password, try go with a passphrase like "correct horse battery staple". If you have too many machines and thus can't remember passwords, then use a master password locally with a bit larger password and store certs there. To do some math to illustrate the security of a random 8 character long password, someone would need to fill a 100/100 line for several hundred years to get through them all. By that time you should have noticed the constantly full connection, and be happy you haven't died of old age yet even after your 200th birthday. However, 8 character passwords are not the suggested length by security experts. They suggest using a 10 character long password, as that is also secure in the case that your password hash somehow got lost. For the several servers that I have, I have been more worried about the logs from failed attempts than I have been of anyone guessing the password. Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it. It also decrease the noise level in the server room.
  • tomp 14y ago
    > Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it. Or change the ssh port.
    • drivebyacct2 14y ago
      Another "security through obscurity" pointless trick. If I'm attacking your server and it's not running SSH on port 22, it's a very short matter of time until I've found it anyway.
      • tomp 14y ago
        Sure. It's only meant to stop automated script-kiddie attacks that target all open SSH servers on the internet. Less logs, less disk usage, ...
• dave1010uk 14y ago
  Couple of related questions: 1) Many people seem to be recommending Puppet / Chef. How many servers or installs do you need before this is a good ROI? (Over using odd bash scripts or cPanel/WHM) 2) Am I right in thinking kernel updates don't get applied until the server is rebooted? If so, how / when do you manage this?
  • stevekemp 14y ago
    I've always thought 10 was the magic cut-off. I can imagine running "apt-get update && apt-get upgrade" ten times in a row before I get annoyed. Any more than that and I'd want scripting. Really there's nothing stopping you from automating "some" machines and later pushing that out to the rest of your hosts. (Or just setting up trivial automation "configure NTP", "Disable root logins", "Append 'steve' to sudoers" - and keeping doing the rest by hand until you're confident.) FWIW I use my toy configuration system, slaughter: http://www.steve.org.uk/Software/slaughter/ http://www.steve.org.uk/Software/slaughter/
  • lmm 14y ago
    Puppet/chef are certainly worth it for 3 machines. If it's an important machine they're worth it for even 1 machine (so that you have a disaster recovery procedure).
  • justincormack 14y ago
    Re 2 yes that is true. You need to build this in to your maintenance.
    • dave1010uk 14y ago
      Do you just reboot all machines every month or so or do you keep track of when kernel security patches come out some how?
      • justincormack 14y ago
        Given that there have been real kernel security issues recently I would reboot sooner. You can do a risk assessment per issue. Monthly automated doesn't seem the best option.
• fsniper 14y ago
  I do not recommend unattended upgrades. Every upgrade without any exceptions have the possibility to destroy any working system or your applications. Test and deploy would be wiser. By the way, what the heck developers do on production systems?
  • RobAley 14y ago
    Bear in mind that in small company, developer == dev op == sys op == network manager = testers == toilet cleaner. i.e. they do everything, dev or production. As regards to unattended upgrades, again in small companies the choice is usually between them or no upgrades (no time for the test/deploy cycle across all platforms in use etc., less frequently used systems get forgotten and so on). As the article points out, its safest just to go for security updates only. If the security update breaks things, then yes, your system is down until you fix it. If a hacker breaches your system, your system is down until you re-deploy it, and then you have to deal with the PR fallout of lost credentials and data etc. Its only anecdotal, but I've had no problems with unattended security updates on my ubuntu boxes in the 5-ish years. Security breach attempts are a daily occurrence. It's nice if its not an either/or choice, but for many it is.
    • marcosdumay 14y ago
      If you don't have time to test, you have yet another reason to not automate updates. It's better to run them only when you are around and have some time to fix whatever goes wrong.
      • RobAley 14y ago
        We're talking security updates here, by the time you've found time to test, your server may well be hacked.
• bnegreve 14y ago
  > vim /home/deploy/.ssh/authorized_keys [and] Add the contents of the id_rsa.pub on your local machine and any other public keys that you want to have access to this server to this file. Cool trick: use ssh-copy-id <server> from the client machine. From the man page: ssh-copy-id - install your public key in a remote machine's authorized_keys It is much easier than editing the remote .ssh/authorized_keys file since copying and pasting the key is error prone due to extra new lines typically added by the terminal emulator.
• matwizzle 14y ago
  I'd propose using OSSEC over logwatch & fail2ban. Ossec seems to be a bit of an obscure tool, but a thoroughly functional one at that. Logwatch gives a bit too much info at once to interpret properly, while OSSEC will only alert you when something is actually up. Ossec provides (among some other things): * Log file monitoring, with severity levels, occurence counting, time-based rules and auto-response. This means (for instance) you get to watch your auth.log for failed login attempts and after 10 failures fire a script to ban him, alert sysop by email or have hubot alert your sysops. Or whatever floats your boat. * File integrity monitoring. Make sure noone's been mucking around with your files. It has real-time support (through inotify), but if you don't want use that, make sure you store the database it keeps off-server for forensics if need be. Pro-tip: FIM and auto-updates are a tad unnerving. * It can watch command output. You can use that to make sure the `netstat -tnap` output doesn't change, for instance. * For larger/more compliant instances, it has a client<->server setup available.
  • daemon13 14y ago
    I have heard good words about OSSEC but never tried it because of it's perception being: - heavyweight; - not actively developed. Are my perceptions right? If no, how would you recommend to start using it? Any good tutorials or other pointers?
    • matwizzle 14y ago
      Last version dates from 2012-11-19, not the fastest of releases - but seems ok. Trend Micro is involved (and offers commercial support). It doesn't feel heavyweight to me. It does start a bunch of daemons for all of its processing and it has the client->server bit built right in. That may make it feel heavyweight, I guess. But, you don't have to use the client->server stuff if you don't want to. It'll still do all of its magic for you. Ossec will do a lot out of the box. So, I suggest installing it with the default ruleset and the active-response stuff turned off (the installer will ask you). Then dive into the rules and ossec.conf (knowledge of regex is required). Documentation @ http://www.ossec.net/doc/index.html http://www.ossec.net/doc/index.html
    • druiid 14y ago
• martinced 14y ago
  I honestly do not understand why you need a deploy account with sudo access. I much prefer a deploy accound which DOES NOT have sudo access. I add firewall rules transparently redirecting 80/443 to non-privileged ports that the webapp is actually listening to. Hence no need to be root / sudo'ed for the deploy account. You then could get a bit fancier and set the login shell for the deploy account to /bin/false or something fancy (like a fake honeypot shell) to add a bit of "defense in depth" in case someone exploiting a hole in your web server manages tries to drop down to a shell. You'd then use another account to do the (automated) deploy/start/stop/update/patch whatever. I'd also say that during the first five minutes you should set the default firewalling rules to REJECT anything and then only whitelist what is actually allowed.
  • shabble 14y ago
    > I'd also say that during the first five minutes you should set the default firewalling rules to REJECT anything and then only whitelist what is actually allowed. At which point your 'net connection blips and suddenly you wish you'd paid extra for console access.
    • pavel_lishin 14y ago
      That's how I felt about the line where he recommends only allowing the deploy user to connect from a certain number of white-listed IPs. That's great and secure until you get a phone call while you're at the airport or on vacation, and then you hope that you can SSH into a machine at the office just so you can tunnel through from a white-listed IP.
      • shabble 14y ago
        Yeah, I've always felt nervous about doing whitelisting (and to some extent, things like fail2ban) for reasons like that. Having a couple of backup hosts / friends with shell-servers listed who you can bounce through might mitigate it somewhat though, whilst still avoiding 90% of the portscans-from-(china|mars) stuff. One idea I had was to enable something annoying and kooky like port-knocking or OTP pass/connection enabled 'backdoor entrance' for emergencies, but ended up being too lazy, and realised it was just expanding the attack surface. To my original point, we had an interesting setup for one network where the firewall changes were all manipulated via a script which required the change to be applied, and then confirmed after a short delay with a different command, otherwise after 5 minutes the ruleset would revert to the previous known-working one. It definitely saved some downtime/late night DC trips.
• jackalope 14y ago
  Maybe this is considered a preinstallation step, but the very first thing to do is set the system time on the hardware, before you even boot the OS for the first time. Then the first step after booting is to confirm the time and reset it, if necessary. This is essential for accurate and usable logs, file times, version control timestamps, etc. It's also a good idea to ensure that sshd has fresh keys that are unique to that machine. Hopefully, your images are installed without sshd keys, otherwise you'll have multiple servers with the exact same keys, which is considered bad practice. During initial configuration before deployment, you might want to remove the keys so that sshd will create fresh ones when it starts: rm -rf /etc/ssh/ssh_host_*
• sneak 14y ago
  This is terrible. Never let developers log into servers. The output from your devs are commits.
  • deleted 14y ago
    [deleted]
• outside1234 14y ago
  My take on all this is that you should spend 0 minutes on a server and be using a PaaS solution. There might be cases where you can't, but increasingly I can't find them.
• belorn 14y ago
  > No secure server is complete without a firewall. Comments like those are why I normally point people to actual security expects (like, say, Schneier), and why I recommend that new admins should ignore as much as possible the practices chanted by the industry. A secure server does not need a firewall. A firewall can be used to secure a server against a specific threat, but that's it. The days of ping of death are behind us. I would like to point out that following the article's guide and firewalling away ICMP, you can end up with a lot of trouble. (see http://serverfault.com/questions/84963/why-not-block-icmp http://serverfault.com/questions/84963/why-not-block-icmp). Some ICMP messages are not blocked by default by ufw, so I'm unsure how damaging ufw is when used like this. At any rate, a Firewall is a block. A new fresh server install won't have ports that needs to be blocked. By putting up a firewall, there is nothing to be gained. Before the firewall, the ports are closed. After adding the firewall, the ports are closed. All that is gained is a hurdle the next time one wants to install something like a monitor tool (like Munin), or a new service. It might be useful as a last line of defense against malware regarding outgoing traffic. I am normally against that kind of thing however (as focusing on the cause is better than the effect). At best, one can catch a spam malware, but any bot net, web server, ddos or other type of malware are untouched by the rules (port 80 and 443 is allowed). If the server has email sending configured so root message can be sent, then the spam malware can use that route and the firewall will just sit there. So let's take a newly installed machine. What threats can be identified and what risks are we trying to mitigate with the help of this firewall (as specified by the article)? The only thing I can think of is either a Zero day TCP/IP stack vulnerability (not a realistic threat), or that the admin doesn't trust the other admins when they install new services. Yes, if an admin installs a new email server and enables relaying to the whole world against the explicit recommendation in bold font by the install wizard and the configuration file, a firewall can block that admins' actions. Then again, that same admin could just as well have disabled the firewall to "get the mail to work", so I'm not sure it's a viable defense against bad admins.
  • ceph_ 14y ago
    ACL based firewalls are the foundation of real network security. I'm not saying every server needs to be running its own firewall. But these security measures need to be implemented at some level, whether it's at a network level or on an individual machine basis.
• alan_cx 14y ago
  Imagine you are a fairly normal windows user or even sysadmin. Imagine you are considering Linux to replace some task that a windows server performs. Now imagine the conclusion after reading this thread. As some one who can just about get something useful done in Linux, this thread makes me want to never use it again, it just looks too scary. Loads of disagreements which seems to have lots of dire consequences. OK, great discussion for deep geekery, but scary as hell for normals. Now, when ever I see those annoying posts from smug Linux users who jump in every time a Win or Mac user highlights a problem, I now have this discussion to point to as to why Mac and Win users wont generally go near Linux. Sorry chaps (and chapesses), I am on your side, Linux is a great thing, but this has to be the worst advert for Linux ever.
  • kaoD 14y ago
    This IS great advert for Linux. It's just that newbies are not this ad's target audience. Sure, you can put Ubuntu in your mom's laptop and she'll be up and running in a breeze, but hey, she's won't be reading Hacker News.
  • marcosdumay 14y ago
    There is just one action on the post that will have dire consequences if ignored: disabling the ssh password authentication. The rest are decisions that you don't need to rush into, and things you should learn when they make sense. Be advised that "the proper way to configure a server" is an elusive beast, you'll be always improving it, it's never done.
• npsimons 14y ago
  passwd Change the root password to something long and complex. And bam! Not even a full paragraph in and security fail. root login should be disabled completely, and all use of privileges should be through sudo. Debian sets this up for you automatically upon install if you supply an empty root password. Of course, disabling root is just the beginning (and the first user created needs to be locked down, as they are now essentially root).
  • marcosdumay 14y ago
    > root login should be disabled completely Would you care to explain why?
    • npsimons 14y ago
      The foremost reason is that root is carte blanche to ruin a system. This may be fine for a developer seat or even a desktop that is not critical infrastructure (shared), but on a multi-user system (and this includes things such as email and web servers), you really should have a "think twice" prompt, along with logging of who did what and when (eg, sudo). Even if you leave root as ssh key only login, then once an attacker has gained that key (or found a remote exploit), they don't need a password; they're root. Setup administrator users who are given a limited set of sudo commands and only allow them to login via SSH with keys and require complex passwords for sudo, and that's multiple layers of protection and logging. Nothing's perfect, but it has long been recognized that root is a big gaping hole in UNIX security; that's why things like SELinux and RBAC were created, and it's why Windows for so long was so insecure (ie, the main user was essentially passwordless root).
      • marcosdumay 14y ago
        The problem of sudo is that it's so easy to make a mistake in your command (and in sudoers). That's a security flaw in itself. Also, you can always sudo bash, or sudo su. You have a point about that extra password. But not about SELinux and RBAC being created because of that, nor in comparing the security of a system with root to one where eveybody is root. All said, I'm still unconvinced. The logging isn't that usefull (I've tried it), and the extra password isn't relevant enough. Also, none of them will detain a malicious user (Linux has very week defenses against you grabing your own password).
• frostiebot 14y ago
  Kinda concerned about the usage of ufw with fail2ban without the mention that you have to do a little more work for fail2ban to actually work with ufw (otherwise fail2ban causes interesting issues/clobbers ufw's rules)
  • GaryGapinski 14y ago
    I haven't seen this problem on recent versions of Ubuntu (10.04, 12.04). fail2ban bans will preempt regular ufw rules (as they should).
• giulivo 14y ago
  I think you're doing it wrong. Having a _shared_ account, with sudo privileges and a common password doesn't look smart. Also, you're forcing devs to copy their public keys around. I think you underestimate the benefits brought in by a centralized system, like LDAP, which would also allow you to manage the permissions with some more granularity
• robomartin 14y ago
  I went through the article and then read every single post on this thread. I am not a security expert so I won't even try to contribute except to say that I see a lot of people offering criticism without taking the extra step of explaining how they would go about hardening a fresh Linux install (or a pile-o-servers in a rack, whatever is applicable). It'd sure be nice for those of us who are not security experts to read alternative approaches rather than, paraphrasing and not picking on anyone, "using a firewall is dumb" or "blocking ssh is pointless". I like isolated ideas such as using a script to completely automate the provisioning of new boxes. Kind of a no-brainer if you ask me. The problem is that such recommendations are not followed by something like "Here's the script I use on Ubuntu 12.04 LTS". How about it guys? Would you care to attempt to produce a canonical HN "How to harden your server" reference? Maybe one of the security experts on HN can start a repository on Github to evolve a canonical script. I'm pretty much 100% Ubuntu 12.04 LTS, so it is my hope that this is one of the platforms that is addressed. I did some looking around and this is what I found (I am in no position to evaluate the merits of any of these at anything beyond an intermediate level): https://github.com/bluedragonz/server-shield https://github.com/bluedragonz/server-shield https://github.com/eglimi/linux_hardening https://github.com/eglimi/linux_hardening http://www.cyberciti.biz/tips/linux-security.html http://www.cyberciti.biz/tips/linux-security.html http://ubuntuforums.org/showthread.php?t=1002167 http://ubuntuforums.org/showthread.php?t=1002167 http://www.thefanclub.co.za/how-to/how-secure-ubuntu-1204-lts-server-part-1-basics http://www.thefanclub.co.za/how-to/how-secure-ubuntu-1204-lt... http://www.andrewault.net/2010/05/17/securing-an-ubuntu-server/ http://www.andrewault.net/2010/05/17/securing-an-ubuntu-serv... http://ubuntuforums.org/showthread.php?t=1919111 http://ubuntuforums.org/showthread.php?t=1919111 https://help.ubuntu.com/12.04/serverguide/security.html https://help.ubuntu.com/12.04/serverguide/security.html http://www.sans.org/score/checklists/linuxchecklist.pdf http://www.sans.org/score/checklists/linuxchecklist.pdf http://nvd.nist.gov/scap/content/stylesheet/scap-rhel5-document.htm http://nvd.nist.gov/scap/content/stylesheet/scap-rhel5-docum... http://blogs.csoonline.com/ubuntu_lts_vulnerability_scrub_against_national_vulnerability_database_nvd_nist_gov http://blogs.csoonline.com/ubuntu_lts_vulnerability_scrub_ag... http://ubuntuforums.org/showthread.php?t=510812 http://ubuntuforums.org/showthread.php?t=510812
  • tucosan 14y ago
• armored_mammal 14y ago
  So I'll agree with many of the commentators that several of the practices suggested aren't 'ideal.' However, they are easy and possibly better than having no 'practice' at all. Just as an example, the shared user account with unique SSH keys per user. Sure, it's obnoxious in some respects, but many of the criticisms I'm reading in the comments like "but they could reinstate their access with a cron job that re-adds their key when they leave" and such are silly - presumably those who are using the shared account are developers/sysadmins with sudo privileges. Regardless of whether they have a shared account they have privileges to do whatever they feel like to the systems in question. Hence I'd argue it's a fairly reasonable solution for the situation when you don't have the time/resources to configure something more complex and you have to trust all parties anyway. I think there are two larger takeaways: First - Managing multiple users across many servers and dev systems is not easy enough, particularly for smaller organizations, and only gets worse when you try to get more granular about who can do what. Second - umm... no idea anymore. Forgot what was second. Automation is good?
• cbracco 14y ago
  This will get buried but I recently posted a line-by-line guide of my first time setting up a VPS server, maybe it will help someone! http://cbracco.me/vps/ http://cbracco.me/vps/