5 ms·

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), am

by yan 13y ago

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and kept very thorough personal logs and documentation on the entire network. One of my first tasks was to change backup frequency (or other singular change) and going by how I usually did things at the time, just sudid a vi session, changed the frequency and restarted the service. She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows and gave me a look. She quickly reverted what I did, updated our config management tool, tested it, then deployed it, while explaining why this was the right way to do things. I slowly came around to doing things the right way and haven't thought much about the initial incident until we found her personal logs that she archived and left on our public network share for future reference. In the entries for the day that I started, we saw the following two lines: [*] 2007/09/09 09:58 - yan started. gave sudo privs and initial hire forms. [*] 2007/09/09 10:45 - revoked yan's sudo privs.

cunac 13y ago
not so bad 47 minutes in first day :-)
sillysaurus2 13y ago
She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or what topics should I study? I want to become an expert at modern sysops. Modern deployment, hardening, backup, managing dozens of boxen, etc. I've been thinking of going through any MIT OCW on the subject, but it seems like hard-earned experience might not necessarily translate well to an academic setting. What would you recommend I do?
- ixmatus 13y ago
  Papertrail is great for this...you can of course setup syslog to route to a central server and just be logged in tmux / screen on that particular machine to read off the stream of logs (I prefer papertrail though + saved searches and hipchat "pings" when saved searches are matched on incoming events). General devops / sysops/ sysadmin knowledge can be had through a variety of means - I got most of my knowledge from simply reading the FreeBSD manual and making a lot of mistakes with my own servers.
- sentenza 13y ago
  What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works. This way, a complete admin newbie would learn about digging through the systems by working out the kinks of practical everyday problems. Remember, this is only the most basic instruction, nowhere near enterprise-grade. If there was a "prospective admin" who had never before run Linux, I'd tell them to install and use Ubuntu/Mint. (Those guys whould usually only be trained to be a helping hand for a "senior" admin.) If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely. If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server. If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days. If you are already an advanced Linux or BSD user, my approach is of course not appropriate. Instead I'd recommend to pick skills that you want to learn (iptables? Exim?) and set that up. Read Manuals! Read RFCs! Best of luck.
  - sillysaurus2 13y ago
    What position are you starting from? Accurately assessing one's own competence is difficult and makes for boring reading, but since it's probably necessary here, I'll give some background. If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely. If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server. If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days. I'd say my current skill level is a mixture of those three. For example, I don't know how to deploy a web service which can send out email for users to e.g. reset passwords. So I don't know anything about email. On the other hand, I've been trying to hone my skills by hardening a Debian server using iptables. On my third hand, while I could set up a box at home that can be SSH'd remotely, I'm not yet confident I know all the best practices. I think the best SSH practices are: change the default SSH port, disable root login, and disable password-based login (use a password-protected keyfile instead). Beyond that, what is interesting to me is being able to set up dozens or hundreds of systems. Doing this by hand is fraught with error, so it seems like I should learn about virtualization + deployment systems. I've heard good things about Ansible and Salt, but I've also heard Salt considered security an afterthought, which didn't sound good. It's sounding like my best bet is just to try things, but I want to set things up correctly from a security perspective. I should also enhance my knowledge of networking... perhaps by spending a few weeks on OCW material regarding the networking stack. How packets are routed, the details of TCP, that sort of thing. Thanks so much for your insight!

by yan 13y ago

cunac 13y ago

not so bad 47 minutes in first day :-)

sillysaurus2 13y ago

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or what topics should I study? I want to become an expert at modern sysops. Modern deployment, hardening, backup, managing dozens of boxen, etc. I've been thinking of going through any MIT OCW on the subject, but it seems like hard-earned experience might not necessarily translate well to an academic setting. What would you recommend I do?

ixmatus 13y ago
Papertrail is great for this...you can of course setup syslog to route to a central server and just be logged in tmux / screen on that particular machine to read off the stream of logs (I prefer papertrail though + saved searches and hipchat "pings" when saved searches are matched on incoming events). General devops / sysops/ sysadmin knowledge can be had through a variety of means - I got most of my knowledge from simply reading the FreeBSD manual and making a lot of mistakes with my own servers.
sentenza 13y ago
What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works. This way, a complete admin newbie would learn about digging through the systems by working out the kinks of practical everyday problems. Remember, this is only the most basic instruction, nowhere near enterprise-grade. If there was a "prospective admin" who had never before run Linux, I'd tell them to install and use Ubuntu/Mint. (Those guys whould usually only be trained to be a helping hand for a "senior" admin.) If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely. If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server. If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days. If you are already an advanced Linux or BSD user, my approach is of course not appropriate. Instead I'd recommend to pick skills that you want to learn (iptables? Exim?) and set that up. Read Manuals! Read RFCs! Best of luck.
- sillysaurus2 13y ago
  What position are you starting from? Accurately assessing one's own competence is difficult and makes for boring reading, but since it's probably necessary here, I'll give some background. If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely. If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server. If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days. I'd say my current skill level is a mixture of those three. For example, I don't know how to deploy a web service which can send out email for users to e.g. reset passwords. So I don't know anything about email. On the other hand, I've been trying to hone my skills by hardening a Debian server using iptables. On my third hand, while I could set up a box at home that can be SSH'd remotely, I'm not yet confident I know all the best practices. I think the best SSH practices are: change the default SSH port, disable root login, and disable password-based login (use a password-protected keyfile instead). Beyond that, what is interesting to me is being able to set up dozens or hundreds of systems. Doing this by hand is fraught with error, so it seems like I should learn about virtualization + deployment systems. I've heard good things about Ansible and Salt, but I've also heard Salt considered security an afterthought, which didn't sound good. It's sounding like my best bet is just to try things, but I want to set things up correctly from a security perspective. I should also enhance my knowledge of networking... perhaps by spending a few weeks on OCW material regarding the networking stack. How packets are routed, the details of TCP, that sort of thing. Thanks so much for your insight!