4 ms·

Sorry to disagree with a lot of posts here. >At the most basic level, when firing up a new server I follow the how to harden your server guides and install fai

by crypt1d 11y ago

Sorry to disagree with a lot of posts here. >At the most basic level, when firing up a new server I follow the how to harden your server guides and install fail2ban, disable root login, enable ssh only login etc. This is simply wrong. There are two things wrong with this approach: 1. If I understood correctly, you are essentially following random guides on the internet about setting up your security rig. Not too far from downloading untrusted binaries from the Internet. 2. It shows that you do not understand the core issue and what the main purpose of such tools is, how they function , etc. This is a bad approach because it does not scale well. Security should be built from ground up, not as icing on top. You have to interact with developers during the build cycle and be the paranoid one. So, for example, when somebody mentions FTP transfers you yell at them, instead of finding a workaround (eg, setting up a box with FTP but filtering IP addresses with iptables or whatever). Security is an architectural issue, not the issue of which tools to install. That being said about security, being a good sysadmin for me also means striving for simplicity, documenting and standardizing everything and being meticulous. How do you learn it? From experience. Listen to your senior colleagues and learn by doing. Also, RTFM (but seriously though, those tech notes are important). Disclaimer: Used to be a team leader of a large UNIX/Linux team at the big blue. Now I do DevOps.

blumkvist 11y ago
Learning from blogposts is a widespread issue it seems. I see content marketers and growth hackers who have never done a linear regression in their life. How they are "marketing" and what they do day-to-day is beyond me. If you want to learn something, go read (text)books. Not blog posts. You need core understanding. Nobody can give you that in a blog post. Even if the author actually knows what he is talking about, he is translating his knowledge into practical tips for a particular situation. He is applying his knowledge. You get applied knowledge, not real knowledge. When it comes time for you to do your job, the only things you know how to do are bits and pieces from situations which might or might not be applicable to your situation. Think of it like raw vs. aggregate data. You can easily memorize by heart what the aggregations of the data are. You can learn mean, quartiles, max, min, sum. But what do you do when it comes time to filter? The job of knowledge workers is to filter data, not to memorize aggregations.
- nstart 11y ago
  Would this list of books be a good starting point? http://everythingsysadmin.com/books.html http://everythingsysadmin.com/books.html
  - lamontcg 11y ago
    That is insufficent, try going through TCP/IP Illustrated Vol 1 at least and really learning TCP/IP and use tcpdump to rip apart packets. You should learn in detail the tcp state diagram and how to read netstat output. Know what the TCP 3-way handshake is and PTMU discovery and 4-way connection teardown and be able to apply that knowledge to debugging. Learn how to program in C at least well enough to write simple clients and servers, write a trivial threaded application using something like pthreads (just a couple ideas). Then start reading Advanced Programming in the Unix Environment. Really learn how to use Strace -- you should know what the difference is between a system call and a library call and be familiar with many of the common system calls you see in the output of that. You should understand the sockets API and file IO APIs. Should be able to understand socket and file descriptors and understand the output of lsof and be able to use it to debug problems. Pick up the latest version of the Daemon book and learn how a Unix system is architected and what the different kernel subsystems are. Start poking around in the linux kernel sources. Use your C knowledge to make some toy modifications to the kernel (a /proc entry that echos back 'hello world' when you cat it for example). Read Unix Systems for Modern Architecture by Schimmel as a start if you want to go further here. Take the same approach to the higher level aspects of systems. Your package manger is important so if you are on a RedHat system you should be able to take a trivial piece of C code that you wrote and properly package it so that you can build it, package it, and install it. If you've built by hand or seen C packages built by hand you should have encountered 'configure; make; make install'. You should write a toy C program with a portability issue (find a Linux/FreeBSD portability issue and write a toy C program with that problem in it), and write an autoconf script so that you can run your code on either O/S. Pick up either CFEngine, Puppet or Chef and learn how to use it (you can also add Salt and/or Ansible, but please don't consider yourself an expert if you don't understand the limitations of those two). Study Mark Burgesses Promise Theory. Learn at least one high level scripting language that is in common use: Ruby, Python or Perl. You should also learn bash (and ideally learn old school /bin/sh and its differences with bash as well), but you cannot stop at just shell scripts. Ideally you really learn how to program and write 10,000+ line object oriented programs, and write code that is tested. Learn what inheritance and composition are and why some software developers argue you should favor composition over inheritance. Learn the Law of Demeter and other principles. Install and use something other than Linux to broaden your horizons a bit. FreeBSD at least. Solaris or one of the Solaris derived distros would be even better. Having a Mac as a laptop is also useful but I'm not sure it replaces playing with different server O/Sen. Then, you definitely do also want Limoncelli's book and Nemeth's Unix and Linux System Administration Handbook and be able to configure a wide range of different systems apps (sshd, ntpd, SMTP mailers, apache, nginx, etc). If all you can do is install and configure apps, though, then you're no better than a scriptkiddie running scripts they download off the internet if you can't write them yourself. You won't understand what you're actually doing, and won't have a prayer of debugging some of the harder problems you can run into in architecting and debugging issue that come up in running those apps in a large and successful environment.
Mahn 11y ago
Following random guides on the internet doesn't necesarily have to be harmful if you don't simply copy-paste, but rather make an effort to understand what is the advice being given, why is it being given, and form an opinion about it. For instance, assuming you start from the very beginning, if a guide suggests disabling root login and you do your own research to understand what root accounts are, what can they do, and why would you want to disable it, and after doing your homework you happen to agree it is a good idea, then it's probably okay to follow that specific advice. That doesn't mean you are going to learn everything there is to server administration by following a random guide on the internet, but for a small startup it's not necesarily a bad way to get started since at the end of the day you are going to learn more from experience than by reading 50 books.
- kpcyrd 11y ago
  There are some surprises if you go down that hole. The best argument I could find for "disable root login" was "the attacker has to guess the username, too", which doesn't align with Kerckhoffs's principle and isn't the way security should be done, imo. Also, fail2ban is a protection against bruteforce. If bruteforce is an issue for you, you're doing something wrong. Please correct me if you know more.
  - thaumaturgy 11y ago
    Some of the brute force attempts against servers are so relentless now that they can consume a significant amount of server resources just causing the server to say, "no. no. no. no. no. no. no. no. no...." They also fill up your log files, needlessly consuming disk space and making it a pain to crawl through logs later on to troubleshoot legitimate issues. Plus, you can hook Fail2Ban so that other services can use it to buff up their filters. For instance, if someone's spamming your mail server, your mail server can trigger Fail2Ban and then Fail2Ban can tell your web server to also block the IP (or network) to help reduce common sources of WordPress spam. There are good reasons to use Fail2Ban, and the counterarguments that it doesn't actually improve security miss all the other benefits it brings. And, I've read all of Theo de Raadt's arguments against these approaches. I understand and mostly agree with them. I get that with ssh key only authentication and sane services configuration and so on that people can hammer away at your server all day and never accomplish anything. But that still doesn't mean I want to provide a test bed for every dumb script kiddie on the internet (and there are many).
nstart 11y ago
You are right with the following blog posts blindly is a bad idea. And I feel guilty that I didn't invest time to learn why I commented those lines in IP conf (it's bad enough that I don't remember which files I changed). So I've definitely got a place to start working on already. The mentorship part is a lot more difficult. Good sysadmins are really difficult to find in Sri Lanka. I've worked with a few companies so far. Some examples. 1) One company I worked for didn't even have a policy of hashed passwords and protection against SQL injection. They were developing major enterprise software. 2) I worked as an internal systems developer for a non IT team within another company. This was the one place I could have learnt the most at but the IT team was this very opaque "don't tell people what exactly we do" kind of team. 3) One last example. This other company that I worked at, the sysadmin was pretty good in keeping stuff up and running, but a lot of it was copy paste scripts. I got what I could out of the person but I couldn't pull out much. Where I'm from, the main cyber security body of the country gave blank looks when asked about heartbleed at a conference held recently after the whole thing exploded. All that to just sum up why I turned to HN to seek out advice as to what resources I should look at. A lot of threads on the net seem to veer more towards "be a good communicator" and "know your system". While necessary, it's a little too abstract for someone trying to find out what gaps exist and which ones need filling ASAP. Thanks a lot for the advice. I'll probably start reading up on all those files I had to edit when hardening the server. That should provide a good starting point.
- marcosdumay 11y ago
  > The mentorship part is a lot more difficult. Good sysadmins are really difficult to find in Sri Lanka. Congratulations, it looks like you are one of the top sysadmins at your country. Maybe you should try to look in other places, or maybe you really should really congratulate yourself, with no sarcasm at all, and start selling yourself like an expert.
- bsbechtel 11y ago
  You should reach out to @arunoda (https://twitter.com/arunoda https://twitter.com/arunoda), founder of Meteor Hacks. He's one of the leading voices in deployment architecture for the open source meteor.js project, and based in Sri Lanka.
drzaiusapelord 11y ago
> It shows that you do not understand the core issue and what the main purpose of such tools is, how they function , etc. The people who write those guides don't either. I almost never see mention of things like OSSEC or running a WAF or even a basic IDS/IPS. Just lots of naive "hackers get in through passwords, dont use passwords in ssh and enable fail2ban." That's on top of security via obscurity like changing port numbers. Uh, what? Security needs to have a layered onion approach: Network> IDS/IPS, Firewall Server> OSSEC, SELinux, smart permissions, WAF, AV Storage/Filesystem> Backups, tripwire DB> Hashed passwords, etc Software> Patching schedule, writing secure software from day 1, not giving into to team/client demands for insecurity Monitoring> OSSEC alerts, nagios/zabbix alerts, tripwire alerts, etc Once you get used to a security stack, this stuff comes easy. The problem is cheap VPS's have turned devs into sysadmins who just plant a naked linux box on the web, turn on fail2ban, and call it a day.

by crypt1d 11y ago

blumkvist 11y ago

Learning from blogposts is a widespread issue it seems. I see content marketers and growth hackers who have never done a linear regression in their life. How they are "marketing" and what they do day-to-day is beyond me. If you want to learn something, go read (text)books. Not blog posts. You need core understanding. Nobody can give you that in a blog post. Even if the author actually knows what he is talking about, he is translating his knowledge into practical tips for a particular situation. He is applying his knowledge. You get applied knowledge, not real knowledge. When it comes time for you to do your job, the only things you know how to do are bits and pieces from situations which might or might not be applicable to your situation. Think of it like raw vs. aggregate data. You can easily memorize by heart what the aggregations of the data are. You can learn mean, quartiles, max, min, sum. But what do you do when it comes time to filter? The job of knowledge workers is to filter data, not to memorize aggregations.

nstart 11y ago
Would this list of books be a good starting point? http://everythingsysadmin.com/books.html http://everythingsysadmin.com/books.html
- lamontcg 11y ago
  That is insufficent, try going through TCP/IP Illustrated Vol 1 at least and really learning TCP/IP and use tcpdump to rip apart packets. You should learn in detail the tcp state diagram and how to read netstat output. Know what the TCP 3-way handshake is and PTMU discovery and 4-way connection teardown and be able to apply that knowledge to debugging. Learn how to program in C at least well enough to write simple clients and servers, write a trivial threaded application using something like pthreads (just a couple ideas). Then start reading Advanced Programming in the Unix Environment. Really learn how to use Strace -- you should know what the difference is between a system call and a library call and be familiar with many of the common system calls you see in the output of that. You should understand the sockets API and file IO APIs. Should be able to understand socket and file descriptors and understand the output of lsof and be able to use it to debug problems. Pick up the latest version of the Daemon book and learn how a Unix system is architected and what the different kernel subsystems are. Start poking around in the linux kernel sources. Use your C knowledge to make some toy modifications to the kernel (a /proc entry that echos back 'hello world' when you cat it for example). Read Unix Systems for Modern Architecture by Schimmel as a start if you want to go further here. Take the same approach to the higher level aspects of systems. Your package manger is important so if you are on a RedHat system you should be able to take a trivial piece of C code that you wrote and properly package it so that you can build it, package it, and install it. If you've built by hand or seen C packages built by hand you should have encountered 'configure; make; make install'. You should write a toy C program with a portability issue (find a Linux/FreeBSD portability issue and write a toy C program with that problem in it), and write an autoconf script so that you can run your code on either O/S. Pick up either CFEngine, Puppet or Chef and learn how to use it (you can also add Salt and/or Ansible, but please don't consider yourself an expert if you don't understand the limitations of those two). Study Mark Burgesses Promise Theory. Learn at least one high level scripting language that is in common use: Ruby, Python or Perl. You should also learn bash (and ideally learn old school /bin/sh and its differences with bash as well), but you cannot stop at just shell scripts. Ideally you really learn how to program and write 10,000+ line object oriented programs, and write code that is tested. Learn what inheritance and composition are and why some software developers argue you should favor composition over inheritance. Learn the Law of Demeter and other principles. Install and use something other than Linux to broaden your horizons a bit. FreeBSD at least. Solaris or one of the Solaris derived distros would be even better. Having a Mac as a laptop is also useful but I'm not sure it replaces playing with different server O/Sen. Then, you definitely do also want Limoncelli's book and Nemeth's Unix and Linux System Administration Handbook and be able to configure a wide range of different systems apps (sshd, ntpd, SMTP mailers, apache, nginx, etc). If all you can do is install and configure apps, though, then you're no better than a scriptkiddie running scripts they download off the internet if you can't write them yourself. You won't understand what you're actually doing, and won't have a prayer of debugging some of the harder problems you can run into in architecting and debugging issue that come up in running those apps in a large and successful environment.

Mahn 11y ago

Following random guides on the internet doesn't necesarily have to be harmful if you don't simply copy-paste, but rather make an effort to understand what is the advice being given, why is it being given, and form an opinion about it. For instance, assuming you start from the very beginning, if a guide suggests disabling root login and you do your own research to understand what root accounts are, what can they do, and why would you want to disable it, and after doing your homework you happen to agree it is a good idea, then it's probably okay to follow that specific advice. That doesn't mean you are going to learn everything there is to server administration by following a random guide on the internet, but for a small startup it's not necesarily a bad way to get started since at the end of the day you are going to learn more from experience than by reading 50 books.

kpcyrd 11y ago
There are some surprises if you go down that hole. The best argument I could find for "disable root login" was "the attacker has to guess the username, too", which doesn't align with Kerckhoffs's principle and isn't the way security should be done, imo. Also, fail2ban is a protection against bruteforce. If bruteforce is an issue for you, you're doing something wrong. Please correct me if you know more.
- thaumaturgy 11y ago
  Some of the brute force attempts against servers are so relentless now that they can consume a significant amount of server resources just causing the server to say, "no. no. no. no. no. no. no. no. no...." They also fill up your log files, needlessly consuming disk space and making it a pain to crawl through logs later on to troubleshoot legitimate issues. Plus, you can hook Fail2Ban so that other services can use it to buff up their filters. For instance, if someone's spamming your mail server, your mail server can trigger Fail2Ban and then Fail2Ban can tell your web server to also block the IP (or network) to help reduce common sources of WordPress spam. There are good reasons to use Fail2Ban, and the counterarguments that it doesn't actually improve security miss all the other benefits it brings. And, I've read all of Theo de Raadt's arguments against these approaches. I understand and mostly agree with them. I get that with ssh key only authentication and sane services configuration and so on that people can hammer away at your server all day and never accomplish anything. But that still doesn't mean I want to provide a test bed for every dumb script kiddie on the internet (and there are many).

nstart 11y ago

You are right with the following blog posts blindly is a bad idea. And I feel guilty that I didn't invest time to learn why I commented those lines in IP conf (it's bad enough that I don't remember which files I changed). So I've definitely got a place to start working on already. The mentorship part is a lot more difficult. Good sysadmins are really difficult to find in Sri Lanka. I've worked with a few companies so far. Some examples. 1) One company I worked for didn't even have a policy of hashed passwords and protection against SQL injection. They were developing major enterprise software. 2) I worked as an internal systems developer for a non IT team within another company. This was the one place I could have learnt the most at but the IT team was this very opaque "don't tell people what exactly we do" kind of team. 3) One last example. This other company that I worked at, the sysadmin was pretty good in keeping stuff up and running, but a lot of it was copy paste scripts. I got what I could out of the person but I couldn't pull out much. Where I'm from, the main cyber security body of the country gave blank looks when asked about heartbleed at a conference held recently after the whole thing exploded. All that to just sum up why I turned to HN to seek out advice as to what resources I should look at. A lot of threads on the net seem to veer more towards "be a good communicator" and "know your system". While necessary, it's a little too abstract for someone trying to find out what gaps exist and which ones need filling ASAP. Thanks a lot for the advice. I'll probably start reading up on all those files I had to edit when hardening the server. That should provide a good starting point.

marcosdumay 11y ago
> The mentorship part is a lot more difficult. Good sysadmins are really difficult to find in Sri Lanka. Congratulations, it looks like you are one of the top sysadmins at your country. Maybe you should try to look in other places, or maybe you really should really congratulate yourself, with no sarcasm at all, and start selling yourself like an expert.
bsbechtel 11y ago
You should reach out to @arunoda (https://twitter.com/arunoda https://twitter.com/arunoda), founder of Meteor Hacks. He's one of the leading voices in deployment architecture for the open source meteor.js project, and based in Sri Lanka.

drzaiusapelord 11y ago

> It shows that you do not understand the core issue and what the main purpose of such tools is, how they function , etc. The people who write those guides don't either. I almost never see mention of things like OSSEC or running a WAF or even a basic IDS/IPS. Just lots of naive "hackers get in through passwords, dont use passwords in ssh and enable fail2ban." That's on top of security via obscurity like changing port numbers. Uh, what? Security needs to have a layered onion approach: Network> IDS/IPS, Firewall Server> OSSEC, SELinux, smart permissions, WAF, AV Storage/Filesystem> Backups, tripwire DB> Hashed passwords, etc Software> Patching schedule, writing secure software from day 1, not giving into to team/client demands for insecurity Monitoring> OSSEC alerts, nagios/zabbix alerts, tripwire alerts, etc Once you get used to a security stack, this stuff comes easy. The problem is cheap VPS's have turned devs into sysadmins who just plant a naked linux box on the web, turn on fail2ban, and call it a day.