4 ms·

Ask HN: New job is frustrating – Handcrafted DevOps

6 points by kenbob 10y ago

I joined a midsize public company as a DevOps Engineer few months ago and unfortunately it's not working out well. I was given an impression that I will be doing automation work with internal virtualization environment, AWS, Jenkins, Puppet etc. In reality, the infrastructure is a mess and managers don't have any idea about implementing DevOps. Management thinks supporting developer collaboration/productivity tools like Gitlab/Atlassian suite/Slack is DevOps. Even these systems are handcrafted using common root password and no one seems to care. * It takes 1-2 days to provision a new VM because VMs are provisioned by another IT department * No systems are under configuration management and there doesn't seem any strong desire to do that in near future. "It's one time change, let's not automate" seems to be the approach * Everyone knows and uses standard 'root' password and custom passwords are written on wiki * Obviously no one knows how a system was configured and hence snapshots are preferred way of backing out any changes It's frustrating to me as previously I've worked with people having good DevOps mindset and we managed hundreds of production systems really well . Here it's all about emailing strangers and scheduling meetings to figure out stupid issues etc. I need to pay my bills and this job is offering better pay and willing to sponsor my green card. But it's frustrating..:( As my visa is expiring in a 15 months, I am not sure if it will be practical to switch my job now. How do I convince managers or seniors to focus on automation and monitoring tools? They are willing to throw money at commercial solutions like SumoLogic, Splunk, NewRelic, VMware, Cisco UCS etc., but don't understand the real problem that is within the organization. Senior managers and CEO doesn't seem to care about these things as revenues are going great. Sorry for the rant, but I wanted to vent out my frustration.

lsiebert 10y ago
Fundamentally, work for their problems, not yours, to solve your problem. Log patterns and avoid premature automation, but write down the time you and others take to fix issues and then you can say that technical debt because of blank cost the company X days this last Y period. Also work on projects that showcase you can get shit done to build up credibility. Everyone who comes in as a junior engineer with different ideas will come across as the hot shot who thinks he knows better. Reach out to the developers and ask them what are areas where automation would make their job easier. Reach out to management and ask them what they want more control over, or want done quicker through automation. Look for other costs, server costs, service costs, etc. Your frustration likely matters to nobody but yourself though. Find their frustrations and fix both simultaneously.
jlg23 10y ago
> How do I convince managers or seniors to focus on automation and monitoring tools? By providing solutions. You don't sound like you're busy doing stuff all the time so you can spend the time you are waiting for a meeting just doing things. Find a few low hanging fruits where you can make life for others easier, preferably things that save them time. Over time you build a reputation and people will start to actually listen to what you say. Until then, don't talk, just do.
- kenbob 10y ago
  Thanks for suggestion. I've tried that few times (configured staging hosts in minutes using Ansible which saved time for me at least), but other folks were not interested. Hopefully I can find things that other people do frequently by hand.
  - FlopV 10y ago
    Changing culture is not an easy task, but can be very rewarding. If you can work on those solutions that save X dollars or time, and can show the value that solution brings, people will start to buy into what you're selling. Goodluck.
saluki 10y ago
Sometimes you just have to sit back and enjoy the ride (and get your green card). I would work on automating what you can to save yourself time. Be on the look out for easy things that can save other time and present proposals to implement X and save Y time. I'm been in companies though where for whatever reason no one wants to change/improve anything. Some fear automating things saving time will cost them their job or they might not understand/not want to learn new things. So they are never going to be open to change. Be on guard for things going wrong and being blamed for trying new things. Learn that they are doing inside and out, then look at ways your team can do it better. If you just settle in and keep doing what you're doing take some time to learn new techniques and technologies that can help you later. Maybe start a blog, writing to build an audience to make you an 'expert' and open up possible product offerings in the future. (If that interests you). Good luck, some companies are never going to do things the 'right' way. Good luck passing along some best practices.
pjungwir 10y ago
There were sysadmins doing automation before people invented DevOps, and they had similar challenges: finding time for "projects" in between fixing the printer. A good book on this is Time Management for System Administrators by Thomas Limoncelli. It's short and humorous and might give you some ideas. I don't think you're going to change the organization, but you might be able to improve things here and there and get more personal satisfaction from your work. Maybe also find out who wrote that job description. Perhaps you can make a case that you were given a mandate to push the organization toward better practices. But in that case perhaps you should also ask for a promotion, because it doesn't sound like you have the authority to make the changes you'd like.
la6470 10y ago
Ha ha kiddo .. you think you know it all. You should have started 25 years back when OS install meant loading 25 different floppy disks and compiling your kernel. Each organization has a history and since you are getting paid to work there you should first respect the people there, understand why they are doing things the way they are doing before trying to act as the guy who knows all the new techie things and solution to all the problems.

6 points by kenbob 10y ago

lsiebert 10y ago

Fundamentally, work for their problems, not yours, to solve your problem. Log patterns and avoid premature automation, but write down the time you and others take to fix issues and then you can say that technical debt because of blank cost the company X days this last Y period. Also work on projects that showcase you can get shit done to build up credibility. Everyone who comes in as a junior engineer with different ideas will come across as the hot shot who thinks he knows better. Reach out to the developers and ask them what are areas where automation would make their job easier. Reach out to management and ask them what they want more control over, or want done quicker through automation. Look for other costs, server costs, service costs, etc. Your frustration likely matters to nobody but yourself though. Find their frustrations and fix both simultaneously.

jlg23 10y ago

> How do I convince managers or seniors to focus on automation and monitoring tools? By providing solutions. You don't sound like you're busy doing stuff all the time so you can spend the time you are waiting for a meeting just doing things. Find a few low hanging fruits where you can make life for others easier, preferably things that save them time. Over time you build a reputation and people will start to actually listen to what you say. Until then, don't talk, just do.

kenbob 10y ago
Thanks for suggestion. I've tried that few times (configured staging hosts in minutes using Ansible which saved time for me at least), but other folks were not interested. Hopefully I can find things that other people do frequently by hand.
- FlopV 10y ago
  Changing culture is not an easy task, but can be very rewarding. If you can work on those solutions that save X dollars or time, and can show the value that solution brings, people will start to buy into what you're selling. Goodluck.

saluki 10y ago

Sometimes you just have to sit back and enjoy the ride (and get your green card). I would work on automating what you can to save yourself time. Be on the look out for easy things that can save other time and present proposals to implement X and save Y time. I'm been in companies though where for whatever reason no one wants to change/improve anything. Some fear automating things saving time will cost them their job or they might not understand/not want to learn new things. So they are never going to be open to change. Be on guard for things going wrong and being blamed for trying new things. Learn that they are doing inside and out, then look at ways your team can do it better. If you just settle in and keep doing what you're doing take some time to learn new techniques and technologies that can help you later. Maybe start a blog, writing to build an audience to make you an 'expert' and open up possible product offerings in the future. (If that interests you). Good luck, some companies are never going to do things the 'right' way. Good luck passing along some best practices.

pjungwir 10y ago

There were sysadmins doing automation before people invented DevOps, and they had similar challenges: finding time for "projects" in between fixing the printer. A good book on this is Time Management for System Administrators by Thomas Limoncelli. It's short and humorous and might give you some ideas. I don't think you're going to change the organization, but you might be able to improve things here and there and get more personal satisfaction from your work. Maybe also find out who wrote that job description. Perhaps you can make a case that you were given a mandate to push the organization toward better practices. But in that case perhaps you should also ask for a promotion, because it doesn't sound like you have the authority to make the changes you'd like.

la6470 10y ago

Ha ha kiddo .. you think you know it all. You should have started 25 years back when OS install meant loading 25 different floppy disks and compiling your kernel. Each organization has a history and since you are getting paid to work there you should first respect the people there, understand why they are doing things the way they are doing before trying to act as the guy who knows all the new techie things and solution to all the problems.