18 ms·

My First 10 Minutes on a Server

• cleeus 10y ago
  echo "set background=dark" > /etc/vim/vimrc.local
  • _RPM 10y ago
    You should use `>>` in order not truncate the existing file if there is one.
  • JoshTriplett 10y ago
    For this and a hundred other things, I prefer to use a git home directory.
  • raimue 10y ago
    All users in your team are forced to use dark terminal backgrounds?
    • JoshTriplett 10y ago
      You can actually autodetect whether the terminal background is light or dark. For any xterm-compatible terminal, write '\x1b]11;?\x07' to the terminal, and it'll write back a string telling you the foreground color (for instance, '\x1b]11;rgb:0000/0000/0000\x07', which if written back would set the foreground color). If the color matches 'rgb/RRRR/GGGG/BBBB', compute the luminance of that color, and assume a dark background if <0.5 and light otherwise.
      • moosingin3space 10y ago
        I didn't know about this, but from now on, when writing CLIs that use color, I'm going to take this into account!
        • JoshTriplett 10y ago
          Awesome; more tools should do that. Some caveats, though: * You might not get a response from every terminal, so limit how long you wait. * If you don't already have echo turned off, turn if off before sending the sequence, because otherwise it'll be visible as though the user typed it. * You don't know that the color will use the "rgb:RRRR/GGGG/BBBB" format (a terminal can return anything XParseColor can understand); just read the string from the escape to the terminator, look for 'rgb:', and ignore formats you don't understand. * To calculate whether a color is "light" or "dark", see https://en.wikipedia.org/wiki/Luma_%28video%29 https://en.wikipedia.org/wiki/Luma_%28video%29: dark = (0.299*red + 0.587*green + 0.114*blue) < 0.5;
          • moosingin3space 10y ago
            Have you measured how long typical terminals take to respond? Regarding the third point, it might be a good idea to just feed it to XParseColor and process it from there.
            • JoshTriplett 10y ago
              > Have you measured how long typical terminals take to respond? Arbitrarily long. Consider that a user might run your application over SSH via a high-latency network connection. Better to just handle it asynchronously. Your input loop needs to watch for escape sequences anyway, so watch for that one and process it when or if you see it. Sadly, that only works for interactive screen-oriented applications, not run-and-exit command-line applications that want to use color. > Regarding the third point, it might be a good idea to just feed it to XParseColor and process it from there. That assumes you have libX11 and an X Display available. The former is a heavy dependency for a CLI application, and the latter requires you to connect to the X server. I'd suggest just manually handling the common case of "rgb:R/G/B" (where each component may use 1-4 digits and requires scaling accordingly), and then deal with anything else if your users actually encounter it in the wild.
              • moosingin3space 10y ago
• tdalaa 10y ago
  Pretty useful, thanks
• overcast 10y ago
  Very useful, most of this stuff is pretty common for anyone who has done any regular sysadmin work, but definitely good to have a checklist.
• deleted 10y ago
  [deleted]
• malingo 10y ago
  This is good advice on achieving the most secure SSH configuration: https://stribika.github.io/2015/01/04/secure-secure-shell.html https://stribika.github.io/2015/01/04/secure-secure-shell.ht... "My goal with this post here is to make NSA analysts sad."
  • codelitt 10y ago
    Hahaha great quote. I haven't seen this article before. It looks quite good. Thanks for posting it.
  • newman314 10y ago
    Actually, I've improved on this somewhat by splitting configs to 6.5+ vs. older. Corrections welcome. At some point, I will get around to publishing it. Configs OpenSSH 6.5+ Server UsePrivilegeSeparation sandbox KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group14-sha1 Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 OpenSSH Server Legacy #UsePrivilegeSeparation yes KexAlgorithms diffie-hellman-group14-sha1 Ciphers aes256-ctr,aes128-ctr MACs hmac-sha2-512,hmac-sha2-256 OpenSSH 6.5+ Client UseRoaming no IdentitiesOnly yes KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1 HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,ssh-rsa-cert-v01@openssh.com,ssh-rsa Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1 Host * IdentityFile ~/.ssh/id_ed25519 IdentityFile ~/.ssh/id_rsa HashKnownHosts yes VisualHostKey yes VerifyHostKeyDNS ask AddressFamily inet ForwardX11 no ForwardX11Trusted no OpenSSH Client Legacy UseRoaming no IdentitiesOnly yes KexAlgorithms diffie-hellman-group14-sha1 HostKeyAlgorithms ssh-rsa-cert-v01@openssh.com,ssh-rsa Ciphers aes256-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1 Host * IdentityFile ~/.ssh/id_rsa HashKnownHosts yes VisualHostKey yes VerifyHostKeyDNS ask AddressFamily inet ForwardX11 no ForwardX11Trusted no
• brokenwren 10y ago
  This one is pretty decent but if you want the ultimate guide check out this one: https://www.inversoft.com/guides/2016-guide-to-user-data-security https://www.inversoft.com/guides/2016-guide-to-user-data-sec... It covers 10x what all the other guides cover in terms of server and application security. It was posted a few weeks ago on HN but didn't make the front-page.
  • atonse 10y ago
    My worry here is that, in posting what seems to be a book, people just won't even do it because we don't have time to do it, unless this is a primary part of their jobs. If a 10 minute guide gets users 90% of the way, then they're more likely to do it. And that's good enough to cover a majority of automated attacks. Update: I take it back – they've provided scripts to run this stuff. I will explore these. Thanks for the link.
    • robotdan 10y ago
      Agreed - it is a bit long. But if you want to set something up from top to bottom it may be worth it. Cool - I saw the link to Github as well, looks like there is code to do much of what the article outlines. Awesome.
      • brokenwren 10y ago
        The hackthis application that is referenced in the guide is also in Github here: https://github.com/inversoft/passport-js-example https://github.com/inversoft/passport-js-example It uses Ember, Node.js, Express, Sequelize, MySQL and Passport User Database (https://www.inversoft.com/products/user-database-sso https://www.inversoft.com/products/user-database-sso).
    • throwanem 10y ago
      The problem isn't that it's long; the problem is that it's not navigable - there needs to be a table of contents. There's great stuff in here, but it's hard to sort out what I already know how to do from what I can actually use.
• YngwieMalware 10y ago
  I'd been using this article for a couple years when I was a Linux server neophyte and now some of these things seem obvious to me. A good article for total noobs.
  • YngwieMalware 10y ago
    Just realized I'm getting downvoted because I thought this was the first 5 minutes article. Whatever!
• jtchang 10y ago
  Why don't they disable root logins with password period and only allow SSH key authentication? Also if you put a passphase on your SSH key does that mean you have to enter it every time you want to SSH to the server (in order to unlock the key) or does it stay cached on most SSH clients (ssh on mac terminal, putty on windows, etc). Isn't watching failed logins kind of useless? I think it is more important to see what successful logins were made.
  • Lockyy 10y ago
    If you put a passphrase on your ssh key you only have to enter it when you initially add it to your ssh-agent.
    • vmarsy 10y ago
      yes, ssh-agent will let you enter the password once, and then it won't prompt you anymore. (see https://help.github.com/articles/working-with-ssh-key-passphrases/ https://help.github.com/articles/working-with-ssh-key-passph... )
  • Splines 10y ago
    > Isn't watching failed logins kind of useless? I think it is more important to see what successful logins were made. Are you talking about Fail2Ban? I'm not an expert but my guess is that it's defense in depth. Rate limiting failed logins is useful in the case where you messed up somewhere else and have a service/login that is vulnerable to password guessing. I would imagine that auditing successful logins is definitely useful but was left out of the guide for brevity.
  • onli 10y ago
    They do, don't they? Later down there is a section Enforce ssh key logins. However that makes activating the root account even stranger. The thing with the ssh key passphrase makes not much sense to me. I think this is just about "keep your private key save". On linux, passwords like this (like gpg) get cached by the usual password input clients. You are right about watching the successful logins first. He writes it is to raise awareness.
• mmgutz 10y ago
  Hmmm ... why does root need a password? `sudo su`
  • molecule 10y ago
    > You'll only need this root password if you lose your sudo password. Also, good to have in case sudoers file is corrupted, e.g. via bad edit, typo, etc..
    • merurr 10y ago
      cd /etc git init git add * git commit -am "before i screwed up"
• ec109685 10y ago
  It would be useful to discuss what prevents the server from being rooted without a trace during the 10 minutes it takes to execute these steps.
  • VLM 10y ago
    Production boxes are not allowed to be plugged into bare internet or DMZ unless they were seasoned and tested on the LAN first, and the LAN allows no external traffic in (stateful firewall). If people on your own LAN are trying to pown you, you got bigger problems to solve before installing another box. In the old days this was manually moving ethernet cables, now a days this is changing which VLAN the virtual image talks to or if you use something like openstack that implements its own firewall at the virtualization level you allow no external traffic in until the config and testing is done. Also you need to verify your install media is not powned, which means you need access to the md5sum of the media (and how do you know someone didn't MITM the correct md5sum?) and you need to verify your md5sum program isn't powned which means you need to verify your verification strategy isn't powned which means this gets recursive real quick.
  • listerOfSmeg 10y ago
    I build images locally in virtualbox/vmware player/kqemu/whatever and upload or copy the image over after they are configured minimally.
  • superuser2 10y ago
    In general a VPS should come up more or less up to date with your SSH key pre installed and no services running except SSH. It may permit password auth but no accounts would have passwords.
  • tikwidd 10y ago
    I was going to ask the same thing. How do you stop someone from owning a new Linode between the time you start the server and the time you disable password authentication?
• tobltobs 10y ago
  Can somebody help me out with this question: The default config for unattended-upgrades seems to not enable reboot even if a reboot would be required to activate the upgrades. Wouldn't that had made quite a few important upgrades in the last years effectless if they server did never get rebooted?
  • codelitt 10y ago
    Yes. You should still keep your eye out on patches. If a big vulnerability gets patched requiring a reboot, you'll want to attend to the upgrades yourself.
  • blakesterz 10y ago
    Exactly. I run this from time to time: ansible -i ./invenory/whateves all -e -K -u deploy_dude -s -m shell -a 'stat /var/run/reboot-required' At least on ubuntu, if that file is there, you need to reboot.
  • btgeekboy 10y ago
    Not sure if this is still true, but I've also seen cases where Ubuntu will happily continue to install kernel updates as they come down the pipe, right up until /boot is full of old kernels and ramdisks.
  • gog 10y ago
    If you turn on email reporting you get an email tagged with [reboot required].
• jldugger 10y ago
  > We don't even have a password for our root user. We'll want to select something random and complex. So you're taking something secure by default -- no password means no login allowed, and making it less secure. And if you have hundreds of these servers, you'll need to rotate them whenever someone on the team leaves. This is painful. Simple solution: leave root password blank, don't forget your sudo password. If you can't get in, use grub or a liveCD. Or tie auth to ldap or kerberos so you _can't_ forget. This is one area where Windows has a distinct advantage: AD more or less requires admins to think at the level of network of servers, and provides a baseline set of services always present.
  • shujito 10y ago
    What if it is a VPS?
    • throwanem 10y ago
      Boot from a rescue volume, or use the VPS provider's root password reset functionality.
    • optimuspaul 10y ago
      terminate and launch a new one? You have automation to provision your servers right?
  • gshulegaard 10y ago
    Agreed. Stopped reading when I got to that.
  • codelitt 10y ago
    It depends on your VPS, but many give a root password by default. I do make sure later in the article that `/etc/ssh/sshd_config` does not allow root login: PermitRootLogin no But you make a good point that a simple solution is just having no root password at all. If your VPS does have a root login by default, then I believe you can get rid of it with: sudo usermod -p '!' root The best part about sharing things like this is getting all sorts of great info and input on things.
• amelius 10y ago
  For protecting against brute-force login attempts, I use sshguard [1] I really think this should be installed by default on distros like Ubuntu. [1] http://www.sshguard.net/ http://www.sshguard.net/
  • codelitt 10y ago
    Out of curiosity, how does it compare to fail2ban?
    • stevekemp 10y ago
      Seems more flexible, and has IPv6 support by default.
      • takeda 10y ago
        It also was more robust for me and simpler to configure. Though I switched from sshguard to fail2ban, because after I started blocking attacks on ssh, script kiddies started to brute fore passwords using other services (SMTP/IMAP/POP3 etc.) Edit: looks like they added ability to monitor other services, I guess I'll re-evaluate it again.
  • MichaelGG 10y ago
    I can't see any benefit, what am I missing? Put SSH on a port that's not 22 and done, no more mass scanning. The only thing SSHGuard has ever done for me is to lock me out when I was accidentally using the wrong key.
    • oofabz 10y ago
      If SSH is on a non-standard port, it is still possible to brute-force access to the server. You will see fewer automated attempts but you are still vulnerable to a motivated attacker who port scans you and finds the SSH port. Such an attacker is less common than automated scans but is more of a threat. With Sshguard, you are no longer vulnerable to this type of attack at all, no matter which port you run SSH on.
• VLM 10y ago
  Technically you don't need the root password, you can always password recovery if you have access to the box. And how exactly did you lock yourself out of every account with sudo? Of course there's always "messed up my ldap or general network settings, can't log in to fix them". There's nothing wrong with setting your root password to a random string and throwing it away, after verifying your sudo works, I guess. I will admit to being lazy, and with full automation its faster to spawn a new virtual image and let ansible run its course than to do root password recovery where you boot and tell the bootloader to make the init system /bin/sh and hand edit /etc/shadow and /etc/passwd and then reboot again, etc etc. I mean I can set up a new image almost as fast as I can reboot an old image, and I set up images a lot more often than I do password recovery, so... Scrap the ssh commentary and set up ssh company wide as per stribika plus or minus local modifications: https://stribika.github.io/2015/01/04/secure-secure-shell.html https://stribika.github.io/2015/01/04/secure-secure-shell.ht... "On large scale, you'll be better off with a full automated setup using something like Ansible" At ANY scale you're better off, unless you're experimenting or time isn't money. It'll take longer to add the time to document and test what you're doing by hand than to convince ansible to do it for you. If you don't document or test you're just doomed, so its not like you can avoid that effort. With automation this is like "first two minutes on a server" not ten. Some people like to drop a .forward in root's homedir sending mail to your sysadmin mailing list or yourself. I THINK but might be wrong that if you do that you don't have to tell logwatch whom to email to, it'll go to root then forward to the right people. More than logwatch assumes root@something.whatever exists as an email address. You're missing setting up your centralized rsyslog or local equivalent, your munin/nagios/zabbix or local equivalent... I still configure zabbix by hand because I'm old fashioned but its possible to automate that. NTP is also missing. You can make Kerberos a very sad faced puppy if time isn't synced. And its easy to set up to point to local trusted servers. (Note, a post that's nothing but complaining still means the linked article is at least 99.9% correct, it is a nicely written wide ranging TODO list)
  • JoshTriplett 10y ago
    > And how exactly did you lock yourself out of every account with sudo? A single typo in /etc/sudoers or any /etc/sudoers.d file will lock you out of all sudo usage. visudo helps with that, but a single mistake (including in a sudoers.d file installed by a configuration management system or package) will lock you out.
• timroy 10y ago
  Thanks for this article - very clear, well-motivated, and concise. I'm saving this for myself and others.
• rodolphoarruda 10y ago
  I'd be more curious to see a "My first 10 minutes on an Ubuntu desktop" version of the article.
  • deleted 10y ago
    [deleted]
  • rcarmo 10y ago
    apt-get remove -y unity && apt-get install wmaker wmaker-data pcmanfm lxterminal Should take less than 10 minutes and be way faster to use :) (I actually use openbox and fbpanel, but WindowMaker is just too great to forget about)
  • Shorel 10y ago
    Or alternatively: "My first 10 minutes on an Ubuntu desktop (for users who don't hate Unity, which includes myself)". In my case: Change the Switch workspace keys from Ctrl+Alt+Arrow keys to Super+ Arrow keys. Remove LibreOffice, install WPS Office. Remove Transmission, install Deluge. Install indicator-multiload, indicator-sound-switcher. Install Kodi. Install Steam. A lot more stuff, but I have not written it down. =)
    • rodolphoarruda 10y ago
      Yes, I see a lot of reaction around Unity. To be honest, I had to google it to find out what it was. I have always used Ubuntu Gnome, so I had no clue about Unity nor its issues. Back to the topic here, I thought someone could outline a security checking for an ubuntu desktop to assess how secure the system is -- or maybe making it secure from a clean install. Edit: grammar
    • Scarbutt 10y ago
      First time I hear about wps office, why do you prefer it over libreoffice? mobile support?
      • Shorel 10y ago
        It's faster (C++ vs Java) and more compatible with the documents I have to open. It also pleasantly surprised me once: I received a Powerpoint presentation, and went to a customer meeting, where said presentation had to be presented. I plugged the HDMI to VGA adapter for the VideoBeam to the laptop, started the presentation, and the presentation was running in the external display, while the laptop display was still showing the normal 'Powerpoint' view. I could load the web browser in the laptop display to check some things while the presentation was still running undisturbed and the speaker and the audience was happily unaware of it. That level of professional use in software was something I did not come to expect in Ubuntu for any third party software. In fact I don't know if the other Office suites (including MS) have that feature. I guess they do, but still. It is wonderful when everything just works as intended.
• nanis 10y ago
  Sigh ... "principal of least privilege"
• tmaly 10y ago
  I have been meaning to write up a similar guide. I would like to recommend using just iptables instead of ufw, I had a case on my vps where an update to ufw failed and then the firewall was not working. With iptables, install iptables-persistent package so they are saved when you do restarts. Do not try to block entire country ip ranges as this slows the machine down substantially. fail2ban is great, I would recommend looking at some of your system logs to figure out new rules to add.
  • throwanem 10y ago
    Seconding the recommendation to avoid ufw. I haven't actually used it or had a problem with it, but if you understand iptables then you don't need it, and if you don't understand iptables, you're better off just learning to use iptables directly so you can handle those cases that ufw doesn't support or clean up after it when it breaks. I didn't know about iptables-persistent, but it's easy enough to just "iptables-save > /etc/network/iptables" once you're finished changing the config, and "iptables-restore < /etc/network/iptables" in /etc/rc.local. Pretty sure those binaries come with iptables itself, so they should always be available. (I've never dealt with a system that had iptables where they weren't.)
    • tmaly 10y ago
      I recommend iptables-persistent because fail2ban is adding rules dynamically, and it just makes it easier to handle with that aspect automated.
      • throwanem 10y ago
        I didn't notice that iptables-persistent actually saves the currently configured rules periodically. That's both kinda neat and a little scary, and I'm not entirely sure I see much value in persisting dynamic rules; it seems like it'd be easy to end up with a long chain of stale rules that way. Still worth knowing about the automatic persistence, though.
        • porker 10y ago
• nblr 10y ago
  Fail2ban? sshguard? unnecessary. Just disable ssh passwd auth (which generally is a good idea) -> done/done If you don't like lognoise from ssh scanners (even if you disable passwd auth), move your sshd port to some random high port and make note of it in your ~/.ssh/config Generally: if in doubt, take the more simple and elegant solution to a problem.
  • josho 10y ago
    I agree that fail2ban for ssh seems unnecessary. But, it also provides monitoring for other services like http and common exploits. I'd be interested in learning from the community if fail2ban adds much value. As I've looked into the service, it seems like simply running the latest security patches obviates the need for fail2ban.
    • treerunner 10y ago
      I run a web server with some 50+ Wordpress installs on it. You better believe Fail2Ban is necessary. Without it all resources would be consumed by brute force attacks. If someone knows of a better way I would like to hear about it.
      • falcolas 10y ago
        In my experience, better to let a webserver handle stopping bad traffic by whitelisting WordPress endpoints. Parsing a url and returning 404 from nginx is cheap and scalable, and allows through legitimate traffic that may be sharing an IP (such as TOR).
      • snowwrestler 10y ago
        We use the WordFence module to block brute force attacks, seems to work fine. I cannot believe that Wordpress still ships without basic rate limiting on its login form.
    • mwpmaybe 10y ago
      I too am curious to know what the consensus is. People seem to fall into one of two camps: 1. it's unnecessary if you disable password login or 2. it's an easy thing to add as yet another safety net on top of sane service configuration and firewall rules. I've taken some flak here for being in the #2 camp so I'm loathe to expose myself once again, but my thinking is that even with password authentication disabled, sshd is still vulnerable to DoS attacks. From the public internet or from other machines in my VPS provider's datacenter. The ssh and recidive fail2ban filters add some minimal defense against that.
• dawkins 10y ago
  I always worry that adding 2FA could make your machine inaccessible if anything happens to google-authenticator in this case. Maybe it's a little bit of paranoia but I don't like the idea of giving control over my ability to log into my server.
  • feld 10y ago
    google-authenticator is a fancy name for an algorithm described in RFC 6238. It does not talk to servers or require any network access. Google is not involved in any way; they just made the algorithm popular and branded it.
    • dave2000 10y ago
      I didn't read the question as "what if Google discontinues authenticator" but more as "what if I use Google authenticator on my phone and it gets stolen and i can't log in any more"
      • feld 10y ago
        it provides you with backup codes that you print and put somewhere safe
  • throwanem 10y ago
    It'd be perfectly reasonable if libpam-google-authenticator relied on Google's infrastructure, but despite the infelicitous name, it does not; it just implements the server side of TOTP. The authentication flow is identical to any other correct TOTP implementation, and you can use any compatible client; no integration with Google services or infrastructure is required. (In fact, I don't think it's even possible.) Speaking of TOTP without Google, if you use iOS and find the Google Authenticator app unsatisfactory, try Authy. It's good stuff, and well worth a few bucks.
    • stephenr 10y ago
      Hurricane Electric's network tools iOS app is free and has an OTP client built in, with iCloud Keychain sync. OTP Auth is another excellent, free OTP client. Edit: clarified both are free.
• jboynyc 10y ago
  I'm finding that another important step is this one: apt-get install etckeeper && cd /etc && etckeeper init Keeps your /etc under version control so you know what kinds of configuration changes you've perpetrated.
  • agumonkey 10y ago
    Ha yeah. I wonder how it's not installed by default, it's such a bliss (until nixos becames the defacto standard)
  • eropple 10y ago
    I used to use this, but I found that it's significantly less useful than a git repo with my server-specific Chef cookbook in it. Manually modifying servers was never a good idea; it's worse now, even with tools like this.
    • jboynyc 10y ago
      I suppose that's true, but in the field where I work (social science), servers are mostly spun up to scratch an immediate and idiosyncratic itch, so configuration tends to happen organically. I agree that's probably not a good idea, and learning Pupchefsible is well worth the effort. In the meantime, though, there's at least some degree of reproducibility with etckeeper.
      • eropple 10y ago
        I do this stuff professionally, and I've learned the hard way that you either have a reproducible environment or you don't. etckeeper isn't reproducible. Actually rolling back with something like etckeeper is much, much more likely to break something (by deleting a config file used by a newer service, say) than to save you. If it did something like separate branches for each service or component within /etc I might be more sympathetic...but at that point you have half of a CM system already and might as well just go the rest of the way. If you need reproducibility without a CM framework, keep backups of your machines.
• ryanmarsh 10y ago
  I don't mean to sound flippant but why can't these "lock down your new box" tutorials just be a bash script? Shouldn't they be?
  • komali2 10y ago
    I would say the same reason we teach web developers what a linked list is. They might never use it (watch me eat my own boot here) but it's a fundamental principle upon which a lot of the tools they use are built. It's good to know what's going on under the hood.
  • eonw 10y ago
    just cut and paste all of the commands into a single script if you want one?
  • rryan 10y ago
    My dotfiles repo includes an "initialize_debian_server.sh" that basically does all this. It's definitely doable.
  • kemayo 10y ago
    There's the argument that "just download and run this shell script as root" and "lock down your box" are fundamentally incompatible steps. It's one of those cases where a tutorial explaining every change you're making and why you're making it really is pretty important.
  • throwanem 10y ago
    Sure! That way it's just a matter of doing su - curl -sS https://some.random.host/trust-me.sh | bash - What could go wrong?
    • drdaeman 10y ago
      Not really. It's more like `./provision.sh your-new-host.example.org` (Because why in the world would anyone want to type in things by hand? Laziness for any repetetive manual labor is a greatest virtue of any good sysadmin.)
• chrisper 10y ago
  Instead of using unattended-upgrade, I prefer to subscribe to mailinglists and see when there are new securtiy updates. One could combine that with something like rundeck where you run apt-get upgrade.
• Someone1234 10y ago
  Why do people install fail2ban then disable password based authentication entirely? I legitimately don't understand the purpose. Also, they complain about log spam (from failed SSH attempts) this is one reason to move SSH to a different port. It does NOT increase security, but it DOES reduce log spam from bots trying for easy targets.
  • e1ven 10y ago
    Fail2ban can be used to block IPs based on any log file, not just SSH - I believe the author uses an Apache log in the example.
    • geerlingguy 10y ago
      It also has rules for mailers, and other utilities, which I often use just because they're nice and simple to work with.
  • treerunner 10y ago
    I always change the ssh port to something other than 22. It has always seemed to work well for me for most automated attacks. Perhaps this is not advisable for some reason?
    • carlisle_ 10y ago
      There is a slight security concern to binding it to a port above 1024 in that a non-privileged user can bind to that port and MITM. Here is a good summary of the different options you have with ssh and choosing a port: http://serverfault.com/questions/619898/should-i-change-the-ssh-port-to-1024 http://serverfault.com/questions/619898/should-i-change-the-...
    • supbpeerr 10y ago
      From experience, running ssh on non-default port have mostly reduced automated login attempts to 0, year after year.
  • vacri 10y ago
• usaphp 10y ago
  > "You should never be logging on to a server as root." Can someone explain me, let's say I disabled password logins and only allow login via a key, what are potential downsides of logging in as a root?
  • ghayes 10y ago
    Well for one, every command you run has root privileges (instead of requiring sudo). Every process you spawn has root privileges. You're safer keeping your privileges limited and sudoing when required. Also you lose your real-life audit log if multiple people log in as root.
    • codelitt 10y ago
      Also good reading is: https://en.wikipedia.org/wiki/Principle_of_least_privilege https://en.wikipedia.org/wiki/Principle_of_least_privilege
    • usaphp 10y ago
      But when I tried using a regular user to run anything on a server - it still requires me to do sudo, so the process will anyway have root privileges, no? So I end up typing sudo most of the times becauase most of processes do not work without root priveleges
• raimue 10y ago
  Be aware fail2ban does not handle IPv6 at all with its default configuration on Debian/Ubuntu. https://github.com/fail2ban/fail2ban/issues/1123 https://github.com/fail2ban/fail2ban/issues/1123
  • mwpmaybe 10y ago
    Indeed. I add this to sshd_config (with a note to remove it when this fail2ban issue is resolved): AddressFamily inet
  • codelitt 10y ago
    You're right. Took mwpmaybe's suggestion for the time being and only will accept IPv4 from ssh for now till that's fixed.
• chrisfosterelli 10y ago
  > sudo ufw allow from {your-ip} to any port 22 I'm surprised nobody mentioned this is a great way to shoot yourself in the foot if you don't have a static IP.
  • codelitt 10y ago
    Ooo. Fair point. I'll add that now and link to your comment.
    • chrisfosterelli 10y ago
      Great! Thanks for the article, it makes a good reference when setting up VPS's.
• seagreen 10y ago
  Great tactical advice, but what a sad situation to be in. "Run this command, then run this command, then run this command ..." There should be a single configuration file (or set of files) that declaratively describes the whole state of the machine. That way the exact situation of the server can be reviewed by just looking at files, instead of trying to poke and prod at the machine to see what commands have been run over the last X weeks.
  • benplumley 10y ago
    There is a set of files that describes the state of the machine, it's called the filesystem. Anything less doesn't describe the whole machine. The 'poking and prodding' is just a convenient way of querying the very small parts of the filesystem that are relevant to that query. That said, a script that pokes and prods the right places and reports a machine's 'security factor' and prompts improvements would be cool (and probably already exists).
    • rdrake 10y ago
      > That said, a script that pokes and prods the right places and reports a machine's 'security factor' and prompts improvements would be cool (and probably already exists). https://cisofy.com/lynis/ https://cisofy.com/lynis/
• adrianmsmith 10y ago
  What's the reason for using a firewall? Assuming that services which shouldn't be accessible to the outside only listen to localhost not the network (e.g. MySQL on a LAMP stack), isn't that sufficient? (Honest question, I don't have much experience with syadmin.)
  • raesene6 10y ago
    So there's a couple of reasons to add a firewall. 1. If an attacker gets unprivileged access it can slow them down (if properly configured) in getting new tools onto the system or adding a shell. 2. If a configuration error results in a service being started on a network accessible interface by accident the firewall gives you a bit of defence in depth protection against unauthorised connections to that server. 3. you can also use it for logging activity to feed into other systems.
    • MichaelGG 10y ago
      >1. If an attacker gets unprivileged access it can slow them down (if properly configured) in getting new tools onto the system or adding a shell. That only works if you have an outbound firewall. Which is very onerous - you'd either have to whitelist destinations (package repos, but what if you want to validate arbitrary certificate's CRLs?) or whitelist applications (but not wget etc.)
    • belorn 10y ago
      1) An attacker getting access will only be slowed down/detected if your firewall filters outgoing traffic, which practically no one does because of the inconvenience and maintenance costs. You also need to lock down outgoing traffic to port 80/443, which is how many intrusions download their payloads and calls home for instruction. If you however accept the cost and do use a outgoing filter, it's quite effecting in detecting and stopping attacks, and it is something I recommend for defending assets with high security demands or high risk. 2) As for configuration errors, it depends on what kind of practices you use as a sysadmin. Do you download and run random scripts found on blogs, use experimental versions, and do not spend time reading manuals? Or are you someone who will only run a Debian stable, has verbose settings in aptitude and reads patch notes? It's been a long time (i.e., almost 20 years) since the last time I saw a program that allowed vulnerable interfaces to be accessible on the network without significant warnings in the manual, comments in the config file and readme. Projects and package maintainers have significantly stepped up their security practices, that by the time something reaches stable it should be matured enough that shooting yourself by accident is difficult.
• tjohns 10y ago
  > I check our logwatch email every morning and thoroughly enjoy watching several hundreds (sometimes 1000s) of attempts at gaining access with little prevail. This is something that actually bugs me a bit. These attacks are so common, getting emails like this every day contributes to alarm fatigue. (https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue) I'd love to see the Linux nightly security scripts replaced with something that only sends out emails when there's an specific actionable event I need to pay attention to. Ideally in a way that can easily be aggregated over all the machines I manage.
  • dredmorbius 10y ago
    Among the things I'll do early in system configuration is to reduce such notifications. Things which should simply be activity logs are moved there. Conditions triggerring notifications are tuned so they don't (failtoban, rate limiting, firewall rules, ...). Makes life much more tractable.
  • MichaelGG 10y ago
    Yep and this doesn't demonstrate anything about security. Showing brute force scanners trying out "root/letmein123" doesn't teach anyone the importance of good security, just the importance of not using super-common user/passes. I cannot figure out why anyone would care or find anything useful in these logs. Change the port, call it a day. Getting worked up about random SSH attempts (or random HTTP "exploit" attempts) seems to be for admins with too much free time.
  • driverdan 10y ago
    I used to read logwatch daily when I was at a small shop and only had two servers. It was really interesting to see the attack trends and IP blocks they came from. It never gave me alarm fatigue because it's not an alarm. It's a log of something that has already passed. Most script kiddies are automatically banned by tools like fail2ban anyway. Seeing the data is pretty interesting.
• javajosh 10y ago
  It may be useful, at step 0, to check out the server and see basic server orientation. Which Linux is it (cat /etc/*-release)? How much ram and disk (htop, df)? How is the filesystem setup (mount)? What packages are already installed (dpkg -l)? What processes are running (ps aux, htop)? What did the last root, including me, do (history)? I also like to know where is the box physically, roughly (tracert, run locally).
• Theodores 10y ago
  I would be annoyed with a cryptic Audi password. I would prefer 'BatteryHorseStaple' passwords. Anything I can't remember gets written on a post it note and put next to my screen with what it is for. This is my behaviour and the problem with cryptic passwords is that there are others like me, willing to keep a good password secret and not willing to be so secret about a clumsy, easy to crack by machine but impossible to remember password.
  • codelitt 10y ago
    Get a PW manager instead. No passwords should be stored in your head (because every one should be different) and they should be stored behind encryption -- definitely not plain text nor sticky notes. http://keepass.info/download.html http://keepass.info/download.html
• feross 10y ago
  This is very similar to my "How To Set Up Your Linode For Maximum Awesomeness" guide: http://feross.org/how-to-setup-your-linode/ http://feross.org/how-to-setup-your-linode/
• babuskov 10y ago
  > First we'll want to make sure that we are supporting IPv6 How does that help security?
  • stqism 10y ago
    That comment was in regards to ufw as it doesn't support adding IPv6 rules by default in commands in older Ubuntu. If you don't enable IPv6 and the server supports it (odds are it does) all of the benefits of using ufw at all are totally ignored on IPv6.
    • deleted 10y ago
      [deleted]
• drzaiusapelord 10y ago
  Its a tradition to nitpick these kinds of lists. Here's my take. >I generally agree with Bryan that you'll want to disable normal updates and only enable security updates. Hmm, fairly certain the Ubuntu (and others) don't do major product updates or API breaking updates via apt-get. You shouldnt have to worry about breaking anything if you use normal updates. This seems a bit too conservative for me and leads to problems down the line of being on an ancient or bugged library and then having to do the update manually later, usually after wasting a couple hours googling why $sexy_new_application isn't working right on that server. He setup an email alert, but not an smtp to actually send it. Also, OSSEC takes a few seconds to install and is much nicer than emailing full logs. Lastly, fail2ban is becoming a sysadmin snake-oil/fix-all. Its use is questionable in many circumstances. There's a real chance of being locked out of your own server with this. If people are recommending it, they should be giving noob-friendly instruction to whitelist their IP at the very least.
• kikimeter 10y ago
  I created a script that does almost everything automatically using Ansible and Ansible Vault : https://github.com/guillaumevincent/Ansible-My-First-5-Minutes-On-A-Server https://github.com/guillaumevincent/Ansible-My-First-5-Minut...
• taf2 10y ago
  Not sure if others feel this way but adding this line to sudo never felt right to me... deploy ALL=(ALL) ALL I usually instead limit the deploy user to a smaller subset of commands e.g. the init.d script to control a service. obviously if someone gained access to deploy user we're probably sol anyway... but it just makes it seem safer... we have a to login as an ops user to install or update things on the boxes.
  • codelitt 10y ago
    Someone on /r/netsec rightly pointed out that you shouldn't ever add a user directly to sudoers anyways. You should add them to the sudo or wheel group. I've since updated the article. What I've described is a more of a base, but according the Principle of Least Privilege you could go even one step further and do what you're suggesting. You'd probably want to have a couple of users though. An admin user, a deploy user, and a maintain user all with different privileges.
    • vacri 10y ago
      > you shouldn't ever add a user directly to sudoers anyways What was the reason for that? I have the deploy user able to run a couple of individual commands without a sudo password (scripts that run canned updates, to be initiated from a buildserver), but I don't see how it would improve things to use a group instead that only holds that user.
  • mwpmaybe 10y ago
    I prefer to grant my person-user access to run any command as deploy, so I can kick off deployments from a remote machine using Ansible: - include: deploy.yml become: yes become_user: deploy And grant separate access for my person-user to restart services as root. And use sudo groups.
• windsurfer 10y ago
  I guess I'm a pretty big noob, but why do people recommend so strongly on password protecting your private key? Losing it pretty much dooms you whether or not it's password protected. It might get you a few hours or so to react and invalidate the public key, I guess...
  • ryanlol 10y ago
    If you've got a good password on your key, then nobody will be able to use it in years. It most certainly helps.
    • windsurfer 10y ago
      Years? How long a password would you need to make GPU cracking take years?
      • ryanlol 10y ago
        A very short one. Reasonable 10+ character passwords should remain out of reach for years. Also, AFAIK there doesn't currently exist any very effective GPU cracking software for SSH passphrases.
        • chrisseaton 10y ago
          I went to lookup the algorithm GPG uses to encrypt private keys, to help answer the original question, but couldn't seem to find that information anywhere. Do you know what it is?
          • windsurfer 10y ago
            According to a quick stack exchange search, OpenSSL uses 3DES for encryption of private keys.
      • chrisseaton 10y ago
        You're already given your own concrete time estimate - 'hours' - how did you work that out?
• ck2 10y ago
  Don't just change SSH key requirements, also change SSH port. Port 22 is possibly the most heavily scanned port around.
• z3t4 10y ago
  If you open up access from/to port 80 or 443, you also open up access to all trojans/spyware/telemetry/auto-update created in the last ten years. You'll want to limit access per user and process.
• stonogo 10y ago
  No production server should ever be manually configured.
  • jeremyt 10y ago
    How should this be done by, say, a small team of three with no SysAdmin?
    • stonogo 10y ago
      Some subset of the team should learn the use of tools like cobbler and puppet.
    • merurr 10y ago
      Take a day off and get familiar with ansible, it will save you tons of time in the long run and simplify your dev/staging/prod environment for years to come
• thatusertwo 10y ago
  I have a VPS, when I first got it, it had an additional user setup for some unknown reason. I didn't know it was there until my server was hacked by a bot. I'd suggest adding one step of checking the /home directory or other places to make sure no 'unknown' accounts have been set up.
• teddyh 10y ago
  I prefer the “Securing Debian Manual” – it’s an official manual from the Debian project. https://www.debian.org/doc/manuals/securing-debian-howto/ https://www.debian.org/doc/manuals/securing-debian-howto/
• archon810 10y ago
  My biggest concern with being on a VPS like Linode, once you're all done securing yourself and binding services to the local LAN IP, is an attack from within the network. The VPS you own is also accessible by others on the same subnet, contrary to what you might assume. I'd love to see a ufw guide for whitelisting only your own internal IPs to be allowed access to any services for ultimate security.
  • nisa 10y ago
    Not sure what you mean but ufw by default blocks everything on your interface so other machines in the local subnet shouldn't have access. If you want to have more security and no (or just a single) outgoing service configure OpenVPN with TLS and put all your local services in a local subnet for your machine. So not even a portscan can find something.
  • mwpmaybe 10y ago
    Something like Ansible can help with this a ton. My playbooks are littered with tasks such as: - ufw: rule=allow direction=in port={{ redis_port }} src={{ hostvars[item]['ansible_ssh_host'] }} with_items: "{{ groups['jobservers'] }}"
• PerfectElement 10y ago
  Is there a similar guide for Windows servers out there?
  • garthk 10y ago
    http://decentsecurity.com/ http://decentsecurity.com/?
• a_imho 10y ago
  I think 2FA is generally bad practice and quite sad it is ubiquitous in e.g. banking and people try to shove it everywhere. It is analogous to password rules, 8-14 characters, numbers, capital letters and other signs. Yet it is very rare you can use a 40+ character passphrase. It gives a false sense of added security, while being annoying at the same time imo. It is very common, for me at least, not to have access to my phone all the time, because I left it at home, in the car etc. Not to mention if you lose it (or someone steals it) you have a huge pita to deal with.
  • tjohns 10y ago
    2FA doesn't have to be annoying. Take a look at Yubikey devices as an example of how to do this right. The reality is that it is actually really useful at preventing some common attack vectors: password reuse, keyloggers, etc. It's even better if you're using a hardware dongle that supports U2F (or can be used as a smartcard for SSH), because that can even prevent active MITM attacks.
    • a_imho 10y ago
      no, my point is exactly that the 2 in 2FA is inherently annoying, because you need to have physical access to 2 different devices at the same time. How does it prevent password reuse? You can use the same (weak) password to lock your phone and login to your banking account (which is again, a false security). However it could be easily circumvented by random generating secure passwords for users (which needs clever advertising like 2FA, because they prefer convenience otherwise). In this case your phone is a single point of failure. You could even argue it increases the attack surface.
      • pfg 10y ago
        > How does it prevent password reuse? It does not prevent password reuse, it mitigates the risks of password reuse in that it adds the requirement of having physical access to a device, which is a show-stopper for most attackers. If you're using a password manager with sufficiently complex passphrases, the biggest remaining risk factor are targeted malware attacks (something like a keylogger), which is something that typical SMS- or TOTP-App-based 2FA implementions won't help you with, fair enough. Implementations where certain security-sensitive activities require separate confirmation and where the details are transmitted through a separate channel would mitigate this attack to a certain degree as well. As an example, some banks in Europe provide their customers with card readers with a PIN pad that shows transaction details on a separate display. Banks routinely include transaction details in SMS-based TAN mechanisms, which works as well, but is obviously not quite as good. > You could even argue it increases the attack surface. How?
• SadWebDeveloper 10y ago
  Forgot to check if the server isn't backdoored. You will be surprised how many providers add many backdoors and monitoring systems you don't need (m looking at you AWS guys).
• brndnmg 10y ago
  May I suggest Ansible or whatever other provisioning tool, you can subtract 9+ minutes from the title...
  • cfieber 10y ago
    please don't.. provision once and snapshot, and deploy the snapshot
• elbear 10y ago
  Here's an Ansible role (I made it) that automates the steps described in the article: https://github.com/LucianU/ansible-secure https://github.com/LucianU/ansible-secure.
  • locusm 10y ago
    I havent checked out your repo but I found myself chanting "Ansible FFS" while reading this post :) Well done.
  • edtechdev 10y ago
    See also https://github.com/geerlingguy/ansible-role-security https://github.com/geerlingguy/ansible-role-security mentioned above
• deleted 10y ago
  [deleted]
• Hello71 10y ago
  1. useradd -m deploy 2. "PasswordAuthentication no" probably won't work as you expect if UsePAM is on.
• agentgt 10y ago
  It might nice if there were some cloud vendor specific addendums. For example on rackspace you almost always want to install the monitoring daemon (it's actually fairly decent and small foot print).
• walrus01 10y ago
  For those saying "why fail2ban?", fail2ban can be used for a great deal more than just watching the sshd log. You can activate fail2ban rules for apache and nginx which help significantly with small DDoS, turning spurious traffic/login attempts into iptables DROP rules. And a lot of other daemons.
  • KB1JWQ 10y ago
    At least one log parsing tool I've seen in years past was vulnerable to log injection attacks. Hilarious proof of concept to own a box by way of PTR record. I haven't checked to see whether fail2ban suffers from this model or not.
• catmanjan 10y ago
  One of the suggestions is to make sure your public key has the .pub extension, and they imply that if someone didn't include the extension they would be reprimanded - any reason for this in particular?
  • AlexCoventry 10y ago
    Having a reliable convention like that reduces the risk of someone accidentally copying their private key to a server.
• cfieber 10y ago
  sure makes me glad all that (and so much more) happens in the first negative 10 minutes on any server I deploy. If you are doing this after your server has launched you are doing it wrong.
• dewarrn1 10y ago
  Nice guide, better comments, leaving this here for later reference.
• plusbryan 10y ago
  What was wrong with 5 minutes? :-)
  • codelitt 10y ago
    Nothing! (Except I doubt I can manually get it all done in 5 minutes =) ) Thanks for your great article. We just ended up adapting your approach with a couple modifications (like 2FA) and extending it to be more of a primer and explain the steps a bit more so that the younger engineers understood what each step performed was doing. I found myself pointing them to your article, but then having to explain what was being performed and it's purpose (not a bad thing - just different audience). As has been mentioned, in the real world, an Ansible Playbook should be performing these, but teach a man to fish, etc...
    • plusbryan 10y ago
      I love the article, and thanks for the credit.
• dmourati 10y ago
  Anyone remember Bastille Linux? https://help.ubuntu.com/community/BastilleLinux https://help.ubuntu.com/community/BastilleLinux
• p8donald 10y ago
  Since I changed the default SSH port of 22 to something else (like 4422), I no longer get any of these drive-by attacks and don't need fail2ban anymore. I also like to set up a simple Monit configuration to alert me about high cpu usage or when the disk space is about to run out. Instead of emailing me these alerts (and also weekly reports) I've configured Monit to post them to my Slack team of 1. https://peteris.rocks/blog/monit-configuration-with-slack/ https://peteris.rocks/blog/monit-configuration-with-slack/
  • kingosticks 10y ago
    You should still use fail2ban. https://news.ycombinator.com/item?id=11854576 https://news.ycombinator.com/item?id=11854576
• bikamonki 10y ago
  Why not make a certified secured best practice 99% covered snapshot and share it as part of the one-click installs that most VPS providers offer nowadays?
• solutions16 10y ago
  Basically (consultanthackers@outlook.com)he just helps you out with whatever hacking or spying activity,Fix your credit and clean your debts , Change your Uni grades and transcripts, Investigate a cheating spouse anything ! Stay classified stay certified, call (302) 365-0294 Thank me later