4 ms·

A sysadmin team leader walks into the room and says, "Everyone, you are now DevOps engineers, carry on", and walks out to collect a bigger paycheck...

by hcrean 5y ago

ltbarcly3 5y ago
So this, but instead of starting with Sysadmins with tens of years of unix experience, start with junior developers with 1 year of experience using Docker as a developer and have them read a blog post on how-to set up Kubernetes.
- geodel 5y ago
  Now I feel this is how it is exactly happening at work even though I am not part of "devops" team. Any working solution at my place is now legacy ready to be dumped and replaced with next generation stack. Other day "devops" engineer couldn't figure out wrong directory permissions and whole deployment failed.
  - kazen44 5y ago
    we once had this issue at an old job. People wanted to replace a monitoring solution for something new, because snmp was "bad and outdated". SNMP is definitely not the holy grail of solutions (to be honest, its kind of a mess). But atleast SNMP is widly supported, works reasonably well, and gets you the data in a dashboard you need. Even for those old Cisco 7200's with an uptime of over a decade. Being new and shiny in operations is usually a sign of inmature technology. (with the bugs and issues associated with it).
    - aduitsis 5y ago
      Thank you for the post! At some point years ago I started realising that most problems in SNMP are stemming from badly implemented SNMP agents. Those problems can be: 1. Returning values different than what the MIB says. 2. Missing values that are not optional in the MIB. 3. Broken getnext code that returns an incorrect oid, which could even lead to loops. Net-snmp will detect it, but proper snmpwalk gets broken. 4. Broken getbulk repeater code that can cause crashes if the repeater is too large. 5. OIDs that will crash/stall/slow the agent. 6. Implementations that cannot handle simultaneous requests. Admittedly this is notoriously difficult to achieve: most MIB implementation code is not reentrant for various reasons. 7. SNMP SET is very difficult to implement, cannot be meaningfully used for almost anything beyond extremely simple stuff. 8. UDP encapsulation can cause MTU/fragmentation issues that are a nightmare to debug. Net-snmp actually implemented transports such as TCP and it was very nice, but it never caught on in general. Bonus, there is also the not related to implementation problem of: 9. The query semantics of SNMP are non-existent, and the indirection in the SMI is cumbersome, which means that as a client/manager you'll have to do all sorts of data wrangling to find what you need.
- Spivak 5y ago
  I really wish this wasn't the case but after sitting on the other side of the interview table I'll take the junior developer almost every time. All the best ops people start as devs with an interest in infra. The worst interviews I've ever been in have been sysadmins trying to pivot to infra work. You can't teach the tinkerer mentality or the culture of "I'll just look at the source and fix the problem."
  - trabant00 5y ago
    I am sorry but I just laughed at this. 20 years ago sysadmins used perl and bash to automate everything. Now devs that don't understand OS, networking, package managers, etc, think they will obsolete ops people by writing yaml configuration for a 0.12 version software, looking at its source code and fixing the limitations. May God have mercy on your soul as the hell you have developed for yourself has none.
    - Spivak 5y ago
      Look, I am that ops person sans the age. I desperately want to one of these mythical graybeards to show up for an interview but it just doesn't happen. I get "senior" UNIX sysadmins that can't write a basic Python script despite it being on their resumes [1], can't mock out a basic HA web architecture or a logging pipeline, and can't explain at all how Linux namespaces work. We tried to hire someone senior for 8 months before giving up and just grabbing a junior dev that wanted to do infra stuff and trained him up. I have worked at shops with people like you describe but it seems they're all happily employed. [1] Not whiteboard style, on their own time, on their own computer, not timed or watched. They can take a week for all we care.
      - trabant00 5y ago
        This is your personal recruiting problem and has nothing to do with "all the best ops start as devs" statement that you made. I have no idea how you could find true experienced sysadmins in your case. They exist though, that much I can tell you. I have 20 years of experience, I write bash, python, perl and lately mostly go. I am not a greybeard as I am just old enough to have been a junior for the greybeards that wrote linux kernel patches among other things to solve infra problems.

A sysadmin team leader walks into the room and says, "Everyone, you are now DevOps engineers, carry on", and walks out to collect a bigger paycheck...

by hcrean 5y ago

A sysadmin team leader walks into the room and says, "Everyone, you are now DevOps engineers, carry on", and walks out to collect a bigger paycheck...

ltbarcly3 5y ago

So this, but instead of starting with Sysadmins with tens of years of unix experience, start with junior developers with 1 year of experience using Docker as a developer and have them read a blog post on how-to set up Kubernetes.

geodel 5y ago
Now I feel this is how it is exactly happening at work even though I am not part of "devops" team. Any working solution at my place is now legacy ready to be dumped and replaced with next generation stack. Other day "devops" engineer couldn't figure out wrong directory permissions and whole deployment failed.
- kazen44 5y ago
  we once had this issue at an old job. People wanted to replace a monitoring solution for something new, because snmp was "bad and outdated". SNMP is definitely not the holy grail of solutions (to be honest, its kind of a mess). But atleast SNMP is widly supported, works reasonably well, and gets you the data in a dashboard you need. Even for those old Cisco 7200's with an uptime of over a decade. Being new and shiny in operations is usually a sign of inmature technology. (with the bugs and issues associated with it).
  - aduitsis 5y ago
    Thank you for the post! At some point years ago I started realising that most problems in SNMP are stemming from badly implemented SNMP agents. Those problems can be: 1. Returning values different than what the MIB says. 2. Missing values that are not optional in the MIB. 3. Broken getnext code that returns an incorrect oid, which could even lead to loops. Net-snmp will detect it, but proper snmpwalk gets broken. 4. Broken getbulk repeater code that can cause crashes if the repeater is too large. 5. OIDs that will crash/stall/slow the agent. 6. Implementations that cannot handle simultaneous requests. Admittedly this is notoriously difficult to achieve: most MIB implementation code is not reentrant for various reasons. 7. SNMP SET is very difficult to implement, cannot be meaningfully used for almost anything beyond extremely simple stuff. 8. UDP encapsulation can cause MTU/fragmentation issues that are a nightmare to debug. Net-snmp actually implemented transports such as TCP and it was very nice, but it never caught on in general. Bonus, there is also the not related to implementation problem of: 9. The query semantics of SNMP are non-existent, and the indirection in the SMI is cumbersome, which means that as a client/manager you'll have to do all sorts of data wrangling to find what you need.
Spivak 5y ago
I really wish this wasn't the case but after sitting on the other side of the interview table I'll take the junior developer almost every time. All the best ops people start as devs with an interest in infra. The worst interviews I've ever been in have been sysadmins trying to pivot to infra work. You can't teach the tinkerer mentality or the culture of "I'll just look at the source and fix the problem."
- trabant00 5y ago
  I am sorry but I just laughed at this. 20 years ago sysadmins used perl and bash to automate everything. Now devs that don't understand OS, networking, package managers, etc, think they will obsolete ops people by writing yaml configuration for a 0.12 version software, looking at its source code and fixing the limitations. May God have mercy on your soul as the hell you have developed for yourself has none.
  - Spivak 5y ago
    Look, I am that ops person sans the age. I desperately want to one of these mythical graybeards to show up for an interview but it just doesn't happen. I get "senior" UNIX sysadmins that can't write a basic Python script despite it being on their resumes [1], can't mock out a basic HA web architecture or a logging pipeline, and can't explain at all how Linux namespaces work. We tried to hire someone senior for 8 months before giving up and just grabbing a junior dev that wanted to do infra stuff and trained him up. I have worked at shops with people like you describe but it seems they're all happily employed. [1] Not whiteboard style, on their own time, on their own computer, not timed or watched. They can take a week for all we care.
    - trabant00 5y ago
      This is your personal recruiting problem and has nothing to do with "all the best ops start as devs" statement that you made. I have no idea how you could find true experienced sysadmins in your case. They exist though, that much I can tell you. I have 20 years of experience, I write bash, python, perl and lately mostly go. I am not a greybeard as I am just old enough to have been a junior for the greybeards that wrote linux kernel patches among other things to solve infra problems.