9 ms·

Ask HN: Was hired to improve company's devops, founder won't listen to my ideas

79 points by milquetoastaf 8y ago

Three weeks ago I joined a fast growing startup on the West Coast to improve their dev operations and processes. The company is seeing crazy growth but they are still doing things like pushing directly to master and using prod deploys to test things out. I was hired to help improve this and implement things like staging environments, autoscaling of resources, and other DevOpsy stuff. Since joining I've received considerable pushback on my ideas. Obviously I am the new guy and don't expect to just dictate people around nor is that my intention. However what really gets me is that the founder just straight up does not seem to trust me. Every suggestion I give - even things as industry standard such as the use of a staging environment - is met with suspicion and doubt and the insistence that I hadn't done enough proper "research" on the idea. I find this highly offensive considering I've been working in DevOps for almost seven years and before that was a SysAdmin. So after two weeks, I got the approval to create a staging branch. I wrote up a huge document on the staging workflow, covering edge cases such as hotfixes and rollbacks. I put time on people's calendars and walked them through it including the founder. Yet he still continues to resist the change and pushes directly to master, ignores the new processes I put in place, and generally just does what he wants. He also keeps harping that I need to "earn his trust." I find this baffling - I mean, I was hired for a reason right? Why would they hire me if they didn't trust me? I am starting to get frustrated, especially since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give? I've worked with difficult clients before but I've never faced such a huge lack of trust.

invalidOrTaken 8y ago
Only thing I can think of: If you have freedom of action (you can choose what to do with your time), then do something small, perfect, and seamless. Present it as a fait accompli. That's your foot in the door. Repeat.
- invalidOrTaken 8y ago
  Later note: reading through all the responses, I'm impressed at what a great resource the minds and experience of HN are.
_skybird 8y ago
Don't make suggestions. Tell them, what they are doing wrong and why.. and then show them what to do. Trust seems like an excuse for being too lazy to adapt to changes.
itronitron 8y ago
If it were me, I would want to run a dev system off the master branch and then tag versions that get the OK and are deployed to the production system. That would give the founder some sense of immediate gratification while insulating the production system from goofs. disclaimer << I am not a devops person
lacker 8y ago
It sounds like you have a different idea of what you were hired to do than the founder does. Talk to the founder and figure out what they want you to be doing. If your manager is a different person than the founder, your manager should be able to help you resolve this. Often companies at some stage realize they need better devops procedures, but also realize they can't slow down their pace of progress, and there are tricky tradeoffs to be made. I encourage you to not be insulted when people have concerns - they may just be concerned that you do not yet have all the context about the company as they do, rather than critical of your experience.
cweagans 8y ago
Leave. There are better environments to work in.
- chrisbennet 8y ago
  This. You aren’t happy and you can’t make your boss happy. Accept that it was a mistake, “admit” that it was your fault for not understanding what they wanted, and go your separate ways. Don’t try to get a “dig” in on your way out, just leave on a good terms as you can.
- scorpioxy 8y ago
  I agree. Especially when the boss starts talking about "trust" and not listening to your proposals. Getting blamed for other people's decisions when you've already warned about the consequences is a flag that you've already lost the battle. Some people just can't be reasoned with. Granted, saying leave is easy to do but the OP might not be able to due to their financial situation or other circumstances. So maybe leaving is not the right option but I don't believe this relationship can be salvaged so checking out(either mentally or physically) is the only way forward, as I see it.
- timb07 8y ago
  Alternatively, get ready to leave. But before you do, push back against the founder. The next time he pushes directly to master, immediately revert the commit, but cherry-pick it over to a development or staging branch instead. Then notify him what you've done and why (including that you're trying to prevent breaking changes being made on master, like happened previously). How the founder responds to this will determine whether you leave, or whether you stay a bit longer and perhaps gain the founder's trust. But still expect you'll have to leave. :/
stephen82 8y ago
It's like reading the same story, but with different protagonist and type of job. I had a similar situation like yours, but I was in the traditional Sys Admin category. Basically I had to deal with toxic behavior for a number of years and in reality what happened was to stay where I first started for the whole time. When they had an issue, I would suggest solutions and useful alternatives, but I would get nothing but "No"s and "don't you dare do such thing!", so they could do their ways. Now, when they would mess up badly with something, they would come back running to me to fix their shit. I was never trusted nor had they believed in me and my skills to improve our entire infrastructure. I got tired arguing and fighting for nothing as I was getting nowhere. Eventually, I automated my tasks to the point of not working, like doing nothing at all for like...4 years or so(?) and right before leaving, I took notes based on my scripts' implementations and then deleted them for good. Lastly, the headquarters decided to close down our branch. I don't think I would like to work for such type of environment anymore, but I'm afraid the majority are like this more or less.
- jhot 8y ago
  I did the same thing of automating my own tasks. Only lasted about a year though because I got too bored.
- newman8r 8y ago
  Wouldn't the company own the scripts you wrote to automate their processes while you worked there? Sounds kind of spiteful not to hand them over, although I can empathize with your frustration (been in similar situations as well).
  - stephen82 8y ago
    I guess, but I am pretty sure if they knew, they would fire me therefore I could not risk it. So, the best thing to do was to delete them as a whole and move on with my life.
    - newman8r 8y ago
      Yeah I gotcha, sometimes that's all you can do. Sounds like they were terrible managers. I hadn't even considered the (obvious) fact that automating your own job can be dangerous.
justin_vanw 8y ago
What is the negative consequence of breaking the site for an hour? Is it "some people are a little annoyed", "nobody notices", or "customers start calling and are very upset" or even "customers file lawsuits"? The amount of friction you add to the development velocity has to be proportioned to the potential downsides. A system that shows doctors xray images in the ER cant go down or people might die. A site like twitter can go down and slightly improve the life of its users. It sounds like you are recommending big company super risk averse solutions at a startup, which is just silly. Besides, staging environments never work, for a long list of reasons. You should be recommending continuous integration and a focus on proper unit tests.
- dfee 8y ago
  > A site like twitter can go down and slightly improve the life of its users. Ha!
- whack 8y ago
  > Besides, staging environments never work, for a long list of reasons Can you go into more detail on this? I would have thought that having a staging environment where all changes get battle-tested by the wider team, before being deployed to prod, would be a good thing.
  - deleted 8y ago
    [deleted]
  - Xeago 8y ago
    Unless you can spin up and tear down a staging environment provisioned with data and instrumentation I too don’t believe in it. Without that you’ll be using something shared, where either one thing at a time is subjected to testing or you’re testing multiple things at once which could conflict (including falsely positive) with eachother. Hope that provides a new perspective. If the team is small, or the product is not susceptible to concurrent development you might not have this phenomenon.
    - Chyzwar 8y ago
lkrubner 8y ago
Some leaders are control freaks, and they resist anything that reduces their control of things. Some are so hungry for control that they behave in irrational and arbitrary ways, often undermining themselves and their companies. I wrote about this at length here: https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/0998997617/ https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09...
halis 8y ago
Maybe he just doesn't like what you're selling. Find out what direction the company wants to go and help them get closer to that. I would recommend temporary feature branches, that frequently merge into master and also release branches for the code that you want to push to production. You can always do hotfix branches off of the release branch. There's always that wet dream of no branches at all, every edge case automatically tested and pushing right to production if the tests pass. But this is really not practical for most places of business. But if you can promote some meaningful amount of: 1) Unit tests 2) Integration tests 3) Functional tests That automatically run when code is pushed, that's a good first step. Also, if you can get them on a regular release cadence of monthly or better, that would also be preferable, and you should be able to release code to pretty much any environment with the click of a button and with little to no manual intervention. Oh and use Vault. :) Stick with the basics and it will take them a long way.
- hartator 8y ago
  What’s Vault?
  - danielvf 8y ago
    https://www.vaultproject.io/ https://www.vaultproject.io/
  - barbecue_sauce 8y ago
    HashiCorp Vault. It's for storing secrets, keys, certificates, tokens, etc.
zwickerbne 8y ago
My suggestion is similar to the ones specified here. Part of the DevOps concepts is to try to take a scientific unbiased approach that is to experiment, measure and make more changes. You could always treat implementing these changes as experiments to see what works. Simplicity, trust, team rapport and support from top down really helps in order to implement change. If the team, founder and manager aren't really interested in changing you can either work on influencing and correcting their mindsets, fight them on it or move on.
danielvf 8y ago
As you said, the "earn trust" doesn't really make sense, so I'd guess there is actually some other reason. Is the way you've implemented the staging plan making life more difficult in some way? Does the founder have some other objective in mind? Once I worked with a company that had a super smart founder who hired a super smart sales manager. The sales manager quickly noticed that the company did not follow the usual industry practices around quoting projects, and on his own initiative started rolling out a new system, including training sales people, and changing the company's software systems. There was increasing friction with the founder over everything, and eventually the sales manager was let go/quit. Somehow the sales manager never understood that the core issue the founder had with his plan was that even though 95% of the industry operated with a secretive, price-what-the-customer-can-pay sales process, the whole company had been built over decades on transparent pricing for everyone. It was big part of the trust the customers had for the company and quite a competitive advantage. Which is all to say that to you, devops means "production doesn't go down", but to the founder, it could mean "Anyone can push to production even faster on more servers." :) Since the founder has basically said he doesn't trust you, it's likely there's something you are doing that he doesn't like. I hear that you are offended by the founders lack of trust, but being offended is a really good way of missing the actual meaning when he is talking. If things continue as they are, both of you are going to be very unhappy. Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants -- and when he says he wants something, push deeper to find out the why behind it! Sometimes people ask for a "universal plugin system for our app, so any can extend it" when all they really wanted was "we can translate the app to French". It's aways possible that the founder is a loon in this area, but I'd say that's only a 10% chance. 80% bet that there is a miscommunication.
- derekp7 8y ago
  The way I've had to explain this to coworkers when dealing with our former manager -- Take what he says, then figure out what he really means, and from there deliver what he needs. But dress what he actually needs into something that looks like what he meant to say, ignoring what he actually said.
  - 8y ago
lazyant 8y ago
So you've been given the responsibility but not the authority. You've been set up for failure. Demand clarification of what's expected from you and the backing to carry it out, because right now you are not able to do your job.
gnulinux 8y ago
Sounds like a very toxic company. You're there for a reason, if you have 7 years of experience you do not deserve this attitude. In the West Coast, you can probably find a new job in 3 minutes anyway. Leave and find a new job.
mysticlabs 8y ago
If the founder is who you report to then you need to schedule a 1-on-1 with them in private and ask them to clarify why they hired you. If the person you report to is someone else, you need to have this conversation with them first. Then have a follow up meeting with your boss and the founder and clarify your role and position to the founder with your manager. Just be upfront, and say hey I’m the new guy, and I’m getting pushback, and want to understand my role better so I can contribute my expertise and create value for the team. Not here to step on anyone’s toes, just trying to gather more information as I haven’t been on boarded properly and I’ve been here nearly 3 weeks. Then let them tell you what they want. If the response is not what you agreed to when hired and the role they want you to fill is not what you thought then look for another job. Better figure this out now then waste months or years working in a toxic environment. DevOps is one of the most sought after positions in the world right now, don’t settle for being bullied or working in a toxic environment. At the same time, you need to establish a clear line of communication first. Collect data, and then analyze what the situation is. Many startups don’t know why they’re hiring people and often aren’t sure why or what they’re hiring for. For example, I got recruited for business development from a company then in the interview realized they weren’t looking for business development. They were looking for an office manager to babysit the developers so they could not be in the office. When I asked about how I would take calls in an open office next to the dev team, and what kind of flexabity I would have to broker deals and deal structures they didn’t even know what I was talking about. The founders basically just wanted an exec assistant with a fancier title. It’s entirely possible you thought you were hired for devops but they don’t even know what devops is and think you’re just an IT guy or sys admin. That’s why you need to gather more info and figure out if this is something you even want to be involved in.
princetontiger 8y ago
Give it another month, and if behavior doesn't change... well, get out!!!
karmakaze 8y ago
Get aligned on values as others here have said. We had a similar situation with types of tests and coverage for a product that did not yet find fit. In retrospect everyone could see that it was overkill. After the pivot we now commit to master run fast unit tests in CI, fast acceptance tests in staging and most valuable journey tests for core user actions, like sign up/in do the main action, get notified. If it passes you can push to prod or do some more ad-hoc in staging. Pipeline is fast, have stability, and everyone's happy. Top pri after prod issue is not blocking CI. If not a quick fix, revert til green. We also have a separate dev integration env for testing changes with some uncertainty. Go fast and only break minor things.
edoceo 8y ago
Presumably you made it through a rigorous process to be hired? And maybe have some background in the space (it looks like you do). I'm a former coder turned CEO. What I see here is a leader who is afraid to let go. And at the same time blocking proven models. And after that you carry the fail-bag. One time, 15+ years ago I joined a small company as an IT Director - reports to CEO - had a pretty good time investment through the interview process. The second day I had similar feeling to what you've expressed. I quit. Sharp talent can find a new gig - quick. Like is too short for you to waste time explaining that 2>1 to folks who are paying you to tell them that - and then ignore you.
eximius 8y ago
If you're willing to relocate, the company I work for is in need of someone like you...
karmakaze 8y ago
Also add that 'flake tests' were a sore point until it became clear which ones were actually the system failing and which tests didn't reflect actual usage. By keeping stats on test faulures it was clear which were causing distraction time and fix them or system cause first. After that we got stability much more quickly than always treading seemingly random errors.
djangowithme 8y ago
"So after two weeks, I got the approval to create a staging branch." - lol
poulsbohemian 8y ago
>founder has pushed breaking changes directly to master Is founder also CEO? If founder is ceo and is coding rather than working on sales / marketing / traction, then you need to run briskly away
- shrthnd 8y ago
  Concurred.
- em-bee 8y ago
  why? if he is a tech founder, and there are co-founders that work on sales and marketing, why should he give up the CEO position? or do you mean to say that from his behavior he is obviously not qualified? greetings, eMBee.
  - poulsbohemian 8y ago
    Put simply: Lead Developer != CEO. Unless it is super early stage (pre-employees, pre-revenue, pre-everything), the CEO should not be hands-on-keyboard. If they are, there are several problems: 1) Delegation. They are demonstrating a management failure to delegate. 2) Even in a technology-oriented business, the job of the CEO is not to cut code. The function of the CEO is to manage and lead all the things related to the growth and success of the business (think: sale, marketing, hiring and recruiting, raising capital, interacting with investors, and a whole lot more). If they want to undertake the technical functions of the business, they should become CTO / VP of Engineering / etc and turn the reigns over to a CEO who can execute on the business. 3) Even if there are other team members engaged in sales and marketing, see point #2 - the CEO is fundamentally responsible for the growth and financial success of the business and it is highly unlikely that technical tasks should be their priority.
cbanek 8y ago
It sounds like you're in a rough spot. Luckily, it sounds like from what you're talking about that there's a lot you can offer and there's a lot of low hanging fruit that can be picked. Here's what I suggest: 1. Find the easiest things to do that will save people time, or fix problems that people have been talking about. Especially useful if founder is one of the people helped. 2. Once you've found some easy things to do (<1wk), figure out which ones you can do without any help, resources, or authority. 3. Do it, and show it to him, or talk it up with the team and save them some time. 4. Tell the founder you've saved people time, and him money. Invest in some of your next ideas to save more time. Don't ask for permission. Just say you're doing it. 5. Success builds on success. One thing I'd definitely point out is sometimes you have to go skunkworks. It doesn't sound like you've been there long and they are harping on you to ship something. You likely have plenty of time to just do some easy things, like automating paperwork or forms. In terms of your staging workflow, again I'd apply concepts that I don't need permission for. Instead of blocking commits to master or even needing a staging branch, you could write some automation that takes a git hash. There's lots of ways to work around needing permissions or official things. Never underestimate the use of a button to do something, rather than typing it into a terminal. It also sounds like you may have people problems. Is your founder stressed? It sounds like he's having a rough time with scaling growth. Find out what is stressing him out at work and use your talent to eliminate it. If all else fails, use FIRE. (and quit.) Good luck.
natbobc 8y ago
It doesn't sound like a great situation to be in. It sounds like the founder has strong opinions. I suspect he subscribes to Trunk Based Development[1] and Continuous Delivery[2]. He also likely wants to follow the practises that "leading organisations" follow in "Accelerate: State of DevOps Report"[3]. You might find it beneficial to research those subjects prior to discussions with him. 1 - https://trunkbaseddevelopment.com/ 2 - https://continuousdelivery.com/ 3 - https://cloudplatformonline.com/2018-state-of-devops.html
- teh_klev 8y ago
  Just some friendly advice. When you're referencing stuff elsewhere on the web do it like this: [1] https://trunkbaseddevelopment.com/ https://trunkbaseddevelopment.com/ [2] https://continuousdelivery.com/ https://continuousdelivery.com/ [3] https://cloudplatformonline.com/2018-state-of-devops.html https://cloudplatformonline.com/2018-state-of-devops.html This means the links are clickable. Using the code indentation thing you did there causes all sorts of grief for mobile device users.
  - natbobc 8y ago
    Thanks! Was faffing with it... the "help" isn't very helpful on how it formats. Sadly can't edit now. :/
- imron 8y ago
  Can't believe this comment is so low. I would guess this is exactly what is happening, an it sounds like OP is trying to push git flow or similar. I would push back against that as well. Trunk based development beats git flow every time.
anon31415926535 8y ago
What could help you stay sane is to recast how you view your job. Here is one way to think about it: your job is to help establish a culture of improved reliability. You are accountable for improving the process but you don't have the authority to decide the process. A large part of being successful in this role is sales. There is a pattern that works fairly well at starting a new engagement. Initially, the main goal is not to fix things; instead you need to focus on building trust. As you've found out, without trust, you won't have success. The easiest way to build trust is to find a problem that the team acknowledges and is small. Tell them how you will solve the problem and have them agree. Solve the problem. Then repeat with another problem until you have some trust. A good indication that you have enough credibility is when they stay asking for your input. Now, you need to find a problem that they are blind to or have given up on solving. Convince them it's a problem worth solving. Metrics are your friend here. Then get buy-in for your proposed solution. Keep visibility add the work is solved. Sacrifice overall speed for immediate partial wins. At this point, hopefully you've reached a credibility tipping point and established a good working relationship. I also had advice for dealing with change resistance, but that is not your problem. Your problem is building your personal credibility by delivering something the team cares about and wants solved. Try not feel insulted. It's not possible. They're only human. :)
epynonymous 8y ago
i think a little conflict is fine, working through conflict can often times build trust, use this as a learning experience because you're always going to have conflict, even if this conflict feels like a core belief mismatch. i find the founder is a bit too super direct and lacks tact, i'm a senior director, and have never told anyone to earn my trust, it's a given, i gain nothing by explicitly stating this, but that's besides the point, different strokes, different folks. i also see a bit of a power struggle here, because your role, perhaps at other companies, has been more control and process oriented, this is a startup with few processes and control seems to be completely with the founder. here's what i suggest, change is inevitably hard to deal with, it takes time and it's very natural for people to resist, so the first thing for you is to decide whether you're ok with this taking a long time, if not, there are a ton of other opportunities out there, don't waste time. if you're willing to stick it out and see value in the business, in the role, then figure out how to do things in incremental steps aka non-earth shattering, it shouldn't be, adhere to my god-like, every company does it this way devops pattern or else (even if it is god-like). baby steps where it's easier for the team/founder to accept, less friction/inertia. another thing is through statistics, founders change caused this which impacted that, we had 3h downtime because founder's change hosed the system, this caused roughly $20,000 hit to revenue, or we lost x users as a results of this. if he's ok with this after repeatedly pointing this out, maybe it's because his goal is different, he's looking for speed of execution as opposed to short term revenue gain, in which case you need to try to align and figure out ways to break things even faster, but with simple ways of rolling back. maybe he/she wants a/b type testing with quick rollbacks in case stuff hits the fan. obviously aligning with the founder is going to be critical. i also suggest a book titled "what got you here won't get your there". you should treat every new role with an open outlook, there are always new challenges to sharpen your skills and experience, if everything's similar to what you did before, you're essentially not learning much.
arountheworld 8y ago
I worked with a start-up where owner confessed he is just looking to get out as quickly as possible. We were not supposed to build processes that would slow down the development at the cost of huge technical debt - but that wouldn't be for him to worry. He did have a number of hires so that on paper it would look nice to investors. We had an experienced and old CEO whose job was to smile and drink with investors, number of developers to fiddle with code, two senior devs that were doing actual coding and a devops guy who spent time on Reddit and baby sitting juniors so they don't break the app too badly when deploying to production and marketing team that was making up ideas what they could shoe next to investors. Company was paying "customers" to use the product. I left early when I realised the equity package is made up. I'd say run! Not worth your reputation.
segmondy 8y ago
You're trying to do too much but moving too slow. You're asking for permission instead of just doing things. You wrote up a huge document? This is a startup not an enterprise! What's the worst thing that can happen? prod breaks right? Make sure it's being backed up! Teach them to tag the branches before merging into master, at least they can have a reference to rollback. Make sure rollback on master can happen with one command. Build the stage without permission, make it a CI/CD. Show them, taadaah! When you push, you can go test on stage, see? huge documents, calendars? meetings? this is a fast growing startup, all your ideas are great but you are executing it too slow and slowing everyone else down that's why the push back is happening.
trelliscoded 8y ago
> especially since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app > use of a staging environment - is met with suspicion and doubt If he doesn't want to give you authority to design the deployment pipeline, then he doesn't get to make you responsible for it. Responsibility/authority mismatch is a classic management mistake, they cover it in lower division business supervision and management classes. It's generally a result of one or more personality defects and/or something interfering with proper reasoning. If you can't get it fixed by figuring out how to work with the founder, then you need to find another job because it's only going to get worse as things ramp up. Based on what you're saying about wanting more research and suspicion when presented with change, it sounds like they have Groucho Marx syndrome due to either a self-esteem issue or impostor syndrome. The usual line of reasoning is: 1. I need expert help. I will look outside my organization for it. 2. I hired expert help, and they are now working for my company. 3. It is self-evident that this person does not know what they are doing because they are working for my organization, and I have a low opinion of myself and/or my organization. 4. I need expert help. Rinse and repeat. I'm currently working with a founder who constantly pushes to master, and I didn't push back because he earned my trust by being a really decent programmer. He's broken things in the past, but in general he fixes things faster than I can detect them so I'm ok with it. The engineering organization doesn't have many complaints about it because the company needs him free to work on the hiring pipeline, which is crucial because our staff keeps getting poached. If he was mediocre, I'd have privilege separated his accounts a long time ago and enabled annoying 3FA on his account with master commit privileges.
xrd 8y ago
I'm doubtful any of these "technical" solutions proposed here will work. What I mean is this: it sounds like the CEO is completely scattered. Nothing you put in front of him will be enough to shake him out of it. He is only getting 10% of what you are telling him. No matter how well written the document you put out there, no matter what consensus you build, it won't change his relationship to what he is dealing with. It's terrifying to be the CEO. He's failing constantly and fearful of being exposed as a fraud. What everyone else deals with, yes, but this time it seems more significant. My suggestion: find other people who have traveled the same path as he has, specifically the issues you were brought into solve. Find a way to connect them with him. Storytelling from a powerful communicator is the only thing that will break him free of this state. Once you have that in place, the solutions you put in front of him will be easy.
deleted 8y ago
[deleted]
tastyham 8y ago
Sounds like there is a miscommunication about what your role is and possibly what each of you mean by "devops".
watwut 8y ago
What about making production branch which goes live and is tested before and let them push into master if they like it so much? Practically, it does not matter which branch and how it is called. Important is that you don't push untested changes to live. Also, writing large document is not how you convince people. If they dont think your ideas are good, they perceive it just as an attempt to force them to do something that does not make sense to them. > since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give? This is large red flag. Toxic politics warning. Document what you do, document discussions decisions etc. Diplomatic is partly way to go, especially since it is founder, but if situation call for it be ready to defend yourself. Also, if that sort of founder behavior becomes usual, run away. No matter what your tech skills are, this will only get worst as other employees will naturally take clues from founder and imitate behavior. This is where production branch I suggested up there can really help. > He also keeps harping that I need to "earn his trust." Another red flag warning. Sometimes it is not about who you are and what you do, sometimes it is about who they are. And who they are suggests they will not build productive effective workplace. It will generate ugly office politics. If that happens, leave before you are so frustrated that you lost confidence or burned out.
m_ransing 8y ago
Put yourself in the founder shoes and think. He/She already has some way of doing things, which is working till now, and you are trying to break it. There will be some resistance to this inertia. When you talked about the standard, people tend to make themselves as exception saying that "our case is different". Check if this is the case. It seems that you are starting big. Trying to push the changes, which people are not used to. Start with small changes, something like statistical analysis tools or some small automated test case tool. Something which will be simple and not much time consuming for developers. Once they see its value, may be then they can "trust" you with big changes. Then there is psychological approach for making changes. You can present your ideas in such a way that the person buying (in your case founder), feels that he/she has a choice or a major role in decision making. But then you present choices, such that the one you prefer comes out as obviously best. Check if this approach can work in your case. Finally, it's your life and your job. Mental stability is more important that the company you are working in. Don't hesitate to fly away on first opportunity.
Blackstone4 8y ago
How does the founder behave with other employees? Especially the new ones. Focus on listening and validating their thoughts. i.e. "What are your concerns?" "Sounds like you're worried about X. I can see where you're coming from". You don't have to agree with them just make them feel like they are being heard. Rinse and repeat. If this doesn't work..... Find a new job.
thecolorblue 8y ago
I find myself in a similar situation. I took a job as a software developer at a hardware shop. They have different processes to what I am used to. I keep pushing for more robust project management but I am getting a lot of push back ("don't tell me how to do my job" level push back). There are two parts to this: 1. Build a better relationship with the founder (carefully) 2. Focus on adding value. New processes are not going well, so scrap them and do the best you can with the processes they have. Your arguments for change will go over better if you are adding a lot of value to the company. That is were credibility comes from in small groups like this.
EnderMB 8y ago
I've had similar issues in the past, and here's what I did to resolve them: 1. Get the others on board. I assume there are other developers in your team. Pitch your ideas to them, and not management, and get whoever is in charge of the developers to agree that your plan is the way to go. If there isn't a lead dev or dev team manager, just get them all to agree on a date where they switch to your way of doing things. 2. Do, rather than ask. I had the same issues with a director that fancied themselves as a coder. To get around this, I implemented a plan, set up a continuous delivery lifecycle, and put it into place. Their FTP access was revoked, and when they went through the (stupid) process to get FTP access again to tweak some CSS the next deployment wiped their changes out. 3. Let them fuck up, and be accountable for their fuck-ups. Ultimately, you can't stop someone in power from abusing it, so when he/she does and it inevitably fucks up, tell them they fucked up and the team will fix it. Going back to my FTPing director, he tried to fuck with some HTML on a Razor template and ended up breaking a page. We had a basic script that checked the sitemap periodically for anything other than 200 responses, and alarm bells soon rang. I asked around on Slack to see who had made live changes without going through the process, and after he (eventually) owned up, we re-deployed the last release and fixed the error. That public log of their fuckup was enough to ensure they stuck to the process after. 4. Own it, but take your time to come up with a solution. I think your biggest mistake is to immediately jump in and start making changes. I know they're a startup, but if I were you I would've committed to a week or two of simply watching and helping the team go about their business as it is. Get everything to tell you what the problems with their current setup is, and lead them towards the answer you want to provide. After that time expires, deliver your findings, backed up with all the feedback you've received from the entire team, and move to step 1. 5. Be prepared to walk if things don't improve. Sometimes, there's only so much you can do. You can have literally everyone on board, except one person in charge, and they'll be more than happy to derail everything in order to do what they want. If you reach a point where you feel that there isn't a way around a blocker, and you're absolutely sure that it's not your (or your teams) fault, then walk. Line up an interview, and simply tell them that you've accepted a job elsewhere. Don't rub their noses in it, or say the obvious reason why you're leaving. The team will probably already know why. Just leave, and thank them for hiring you.
squozzer 8y ago
You are probably a scapegoat. Pull the ripcord.
JSeymourATL 8y ago
> He also keeps harping that I need to "earn his trust." If he doesn't trust you, he won't listen to your advice. Use a collaborative model. Try to figure out the 'pictures in his head'. Good food for thought on this subject from Stuart Diamond > https://www.microsoft.com/en-us/research/video/getting-more-how-to-negotiate-to-achieve-your-goals-in-the-real-world/ https://www.microsoft.com/en-us/research/video/getting-more-...
vfulco2 8y ago
Get out while you can. You were brought on as a checkbox on someone's list, either a BoD, external investors, someone's due diligence. You'll have little to no impact with your domain expertise sadly. I have seen a few times where the boss would not listen to any standard reasoning or best practices.
lackbeard 8y ago
My guess as to what's going on here is that what the founder actually wants is for you to be the expert on the deployment environment & related tools to help ship things faster (in the short-run) and fix things when they break. On the other hand, what you actually want is to spend a bunch of time up-front building tools and processes to keep things stable. Somehow what each of you expects and wants from the other was miscommunicated.
cmcginty 8y ago
"If you can’t measure it, you can’t improve it." I would suggest starting by gathering data to show you CEO where you think you can make improvements. Sometimes just applying a standard processes is not enough to convince management of the benefits if they are not yet bought in.
akulbe 8y ago
Your profile says you're in WA. How close are you to Portland? We should have a beer and swap stories. I'm in a similar situation.
aprdm 8y ago
I find it amazing how people can make so many assumptions from this Ask HN post and already assume how the CEO is, what the start up needs, if OP is right or wrong and whatnot. DevOps is much about culture and it's not a one solution fits-it-all that you can grab from your past 7 years and readily apply to a company you just joined. I was responsible for a very big devops transformation in a medium company (3k employees, 80 devs), took around a year and a half to go from ssh to server to deploy to everyone being on Ansible and automated deploys with CI/dev/staging/prod. I would say to chill :), you just joined the company, 3w is not a lot of time to understand the culture, the devs, what they should optimize for, what is causing them pain, what is taking them time, what is making them money and etc. Also, have some real 1:1s with the CEO (or whoever you report to) and be very direct about how you are feeling and what the expectations are. Case in point, I really recommend the two books in this link: - https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html I don't believe any of that can be applied blindly to a company that isn't google size.
weliketocode 8y ago
You're new to the organization and were hired to help the company go from rough, ship-first, relatively low quality practices to high quality and scalable best practices. In order to make this a reality, you're going to need to first be very understanding and cognizant of the existing team and the reason they do what they do. This is the only way to inform your decisions on: 1. Which changes are most high priority ie business impact 2. Which changes will require significant team buy-in 3. Which changes will make the most sense for your team I very strongly recommend you start with some easy wins: - Bad key management? Use env variables - Autoscaling (as you mentioned) - Clean up no longer in-use aws services to save $$$ Once you've made an improvement or two, then start tackling some of the problems that require more buy-in. This is anything that affects others' workflows or pace of work. If the founder is cowboy-coding and pushing things directly to prod, you're going to need to get his buy-in to change his workflow. Come up with a document, plan, or RFC for anything that requires buy-in. Get comments from relevant parties for your plan. Then, and only then, implement the plan. 3 weeks really isn't that long. You still have plenty of time to right this ship.

79 points by milquetoastaf 8y ago

invalidOrTaken 8y ago

Only thing I can think of: If you have freedom of action (you can choose what to do with your time), then do something small, perfect, and seamless. Present it as a fait accompli. That's your foot in the door. Repeat.

invalidOrTaken 8y ago
Later note: reading through all the responses, I'm impressed at what a great resource the minds and experience of HN are.

_skybird 8y ago

Don't make suggestions. Tell them, what they are doing wrong and why.. and then show them what to do. Trust seems like an excuse for being too lazy to adapt to changes.

itronitron 8y ago

If it were me, I would want to run a dev system off the master branch and then tag versions that get the OK and are deployed to the production system. That would give the founder some sense of immediate gratification while insulating the production system from goofs. disclaimer << I am not a devops person

lacker 8y ago

It sounds like you have a different idea of what you were hired to do than the founder does. Talk to the founder and figure out what they want you to be doing. If your manager is a different person than the founder, your manager should be able to help you resolve this. Often companies at some stage realize they need better devops procedures, but also realize they can't slow down their pace of progress, and there are tricky tradeoffs to be made. I encourage you to not be insulted when people have concerns - they may just be concerned that you do not yet have all the context about the company as they do, rather than critical of your experience.

cweagans 8y ago

Leave. There are better environments to work in.

chrisbennet 8y ago
This. You aren’t happy and you can’t make your boss happy. Accept that it was a mistake, “admit” that it was your fault for not understanding what they wanted, and go your separate ways. Don’t try to get a “dig” in on your way out, just leave on a good terms as you can.
scorpioxy 8y ago
I agree. Especially when the boss starts talking about "trust" and not listening to your proposals. Getting blamed for other people's decisions when you've already warned about the consequences is a flag that you've already lost the battle. Some people just can't be reasoned with. Granted, saying leave is easy to do but the OP might not be able to due to their financial situation or other circumstances. So maybe leaving is not the right option but I don't believe this relationship can be salvaged so checking out(either mentally or physically) is the only way forward, as I see it.
timb07 8y ago
Alternatively, get ready to leave. But before you do, push back against the founder. The next time he pushes directly to master, immediately revert the commit, but cherry-pick it over to a development or staging branch instead. Then notify him what you've done and why (including that you're trying to prevent breaking changes being made on master, like happened previously). How the founder responds to this will determine whether you leave, or whether you stay a bit longer and perhaps gain the founder's trust. But still expect you'll have to leave. :/

stephen82 8y ago

It's like reading the same story, but with different protagonist and type of job. I had a similar situation like yours, but I was in the traditional Sys Admin category. Basically I had to deal with toxic behavior for a number of years and in reality what happened was to stay where I first started for the whole time. When they had an issue, I would suggest solutions and useful alternatives, but I would get nothing but "No"s and "don't you dare do such thing!", so they could do their ways. Now, when they would mess up badly with something, they would come back running to me to fix their shit. I was never trusted nor had they believed in me and my skills to improve our entire infrastructure. I got tired arguing and fighting for nothing as I was getting nowhere. Eventually, I automated my tasks to the point of not working, like doing nothing at all for like...4 years or so(?) and right before leaving, I took notes based on my scripts' implementations and then deleted them for good. Lastly, the headquarters decided to close down our branch. I don't think I would like to work for such type of environment anymore, but I'm afraid the majority are like this more or less.

jhot 8y ago
I did the same thing of automating my own tasks. Only lasted about a year though because I got too bored.
newman8r 8y ago
Wouldn't the company own the scripts you wrote to automate their processes while you worked there? Sounds kind of spiteful not to hand them over, although I can empathize with your frustration (been in similar situations as well).
- stephen82 8y ago
  I guess, but I am pretty sure if they knew, they would fire me therefore I could not risk it. So, the best thing to do was to delete them as a whole and move on with my life.
  - newman8r 8y ago
    Yeah I gotcha, sometimes that's all you can do. Sounds like they were terrible managers. I hadn't even considered the (obvious) fact that automating your own job can be dangerous.

justin_vanw 8y ago

What is the negative consequence of breaking the site for an hour? Is it "some people are a little annoyed", "nobody notices", or "customers start calling and are very upset" or even "customers file lawsuits"? The amount of friction you add to the development velocity has to be proportioned to the potential downsides. A system that shows doctors xray images in the ER cant go down or people might die. A site like twitter can go down and slightly improve the life of its users. It sounds like you are recommending big company super risk averse solutions at a startup, which is just silly. Besides, staging environments never work, for a long list of reasons. You should be recommending continuous integration and a focus on proper unit tests.

dfee 8y ago
> A site like twitter can go down and slightly improve the life of its users. Ha!
whack 8y ago
> Besides, staging environments never work, for a long list of reasons Can you go into more detail on this? I would have thought that having a staging environment where all changes get battle-tested by the wider team, before being deployed to prod, would be a good thing.
- deleted 8y ago
  [deleted]
- Xeago 8y ago
  Unless you can spin up and tear down a staging environment provisioned with data and instrumentation I too don’t believe in it. Without that you’ll be using something shared, where either one thing at a time is subjected to testing or you’re testing multiple things at once which could conflict (including falsely positive) with eachother. Hope that provides a new perspective. If the team is small, or the product is not susceptible to concurrent development you might not have this phenomenon.
  - Chyzwar 8y ago

lkrubner 8y ago

Some leaders are control freaks, and they resist anything that reduces their control of things. Some are so hungry for control that they behave in irrational and arbitrary ways, often undermining themselves and their companies. I wrote about this at length here: https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/0998997617/ https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09...

halis 8y ago

Maybe he just doesn't like what you're selling. Find out what direction the company wants to go and help them get closer to that. I would recommend temporary feature branches, that frequently merge into master and also release branches for the code that you want to push to production. You can always do hotfix branches off of the release branch. There's always that wet dream of no branches at all, every edge case automatically tested and pushing right to production if the tests pass. But this is really not practical for most places of business. But if you can promote some meaningful amount of: 1) Unit tests 2) Integration tests 3) Functional tests That automatically run when code is pushed, that's a good first step. Also, if you can get them on a regular release cadence of monthly or better, that would also be preferable, and you should be able to release code to pretty much any environment with the click of a button and with little to no manual intervention. Oh and use Vault. :) Stick with the basics and it will take them a long way.

hartator 8y ago
What’s Vault?
- danielvf 8y ago
  https://www.vaultproject.io/ https://www.vaultproject.io/
- barbecue_sauce 8y ago
  HashiCorp Vault. It's for storing secrets, keys, certificates, tokens, etc.

zwickerbne 8y ago

My suggestion is similar to the ones specified here. Part of the DevOps concepts is to try to take a scientific unbiased approach that is to experiment, measure and make more changes. You could always treat implementing these changes as experiments to see what works. Simplicity, trust, team rapport and support from top down really helps in order to implement change. If the team, founder and manager aren't really interested in changing you can either work on influencing and correcting their mindsets, fight them on it or move on.

danielvf 8y ago

As you said, the "earn trust" doesn't really make sense, so I'd guess there is actually some other reason. Is the way you've implemented the staging plan making life more difficult in some way? Does the founder have some other objective in mind? Once I worked with a company that had a super smart founder who hired a super smart sales manager. The sales manager quickly noticed that the company did not follow the usual industry practices around quoting projects, and on his own initiative started rolling out a new system, including training sales people, and changing the company's software systems. There was increasing friction with the founder over everything, and eventually the sales manager was let go/quit. Somehow the sales manager never understood that the core issue the founder had with his plan was that even though 95% of the industry operated with a secretive, price-what-the-customer-can-pay sales process, the whole company had been built over decades on transparent pricing for everyone. It was big part of the trust the customers had for the company and quite a competitive advantage. Which is all to say that to you, devops means "production doesn't go down", but to the founder, it could mean "Anyone can push to production even faster on more servers." :) Since the founder has basically said he doesn't trust you, it's likely there's something you are doing that he doesn't like. I hear that you are offended by the founders lack of trust, but being offended is a really good way of missing the actual meaning when he is talking. If things continue as they are, both of you are going to be very unhappy. Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants -- and when he says he wants something, push deeper to find out the why behind it! Sometimes people ask for a "universal plugin system for our app, so any can extend it" when all they really wanted was "we can translate the app to French". It's aways possible that the founder is a loon in this area, but I'd say that's only a 10% chance. 80% bet that there is a miscommunication.

derekp7 8y ago
The way I've had to explain this to coworkers when dealing with our former manager -- Take what he says, then figure out what he really means, and from there deliver what he needs. But dress what he actually needs into something that looks like what he meant to say, ignoring what he actually said.
- 8y ago

lazyant 8y ago

So you've been given the responsibility but not the authority. You've been set up for failure. Demand clarification of what's expected from you and the backing to carry it out, because right now you are not able to do your job.

gnulinux 8y ago

Sounds like a very toxic company. You're there for a reason, if you have 7 years of experience you do not deserve this attitude. In the West Coast, you can probably find a new job in 3 minutes anyway. Leave and find a new job.

mysticlabs 8y ago

If the founder is who you report to then you need to schedule a 1-on-1 with them in private and ask them to clarify why they hired you. If the person you report to is someone else, you need to have this conversation with them first. Then have a follow up meeting with your boss and the founder and clarify your role and position to the founder with your manager. Just be upfront, and say hey I’m the new guy, and I’m getting pushback, and want to understand my role better so I can contribute my expertise and create value for the team. Not here to step on anyone’s toes, just trying to gather more information as I haven’t been on boarded properly and I’ve been here nearly 3 weeks. Then let them tell you what they want. If the response is not what you agreed to when hired and the role they want you to fill is not what you thought then look for another job. Better figure this out now then waste months or years working in a toxic environment. DevOps is one of the most sought after positions in the world right now, don’t settle for being bullied or working in a toxic environment. At the same time, you need to establish a clear line of communication first. Collect data, and then analyze what the situation is. Many startups don’t know why they’re hiring people and often aren’t sure why or what they’re hiring for. For example, I got recruited for business development from a company then in the interview realized they weren’t looking for business development. They were looking for an office manager to babysit the developers so they could not be in the office. When I asked about how I would take calls in an open office next to the dev team, and what kind of flexabity I would have to broker deals and deal structures they didn’t even know what I was talking about. The founders basically just wanted an exec assistant with a fancier title. It’s entirely possible you thought you were hired for devops but they don’t even know what devops is and think you’re just an IT guy or sys admin. That’s why you need to gather more info and figure out if this is something you even want to be involved in.

princetontiger 8y ago

Give it another month, and if behavior doesn't change... well, get out!!!

karmakaze 8y ago

Get aligned on values as others here have said. We had a similar situation with types of tests and coverage for a product that did not yet find fit. In retrospect everyone could see that it was overkill. After the pivot we now commit to master run fast unit tests in CI, fast acceptance tests in staging and most valuable journey tests for core user actions, like sign up/in do the main action, get notified. If it passes you can push to prod or do some more ad-hoc in staging. Pipeline is fast, have stability, and everyone's happy. Top pri after prod issue is not blocking CI. If not a quick fix, revert til green. We also have a separate dev integration env for testing changes with some uncertainty. Go fast and only break minor things.

edoceo 8y ago

Presumably you made it through a rigorous process to be hired? And maybe have some background in the space (it looks like you do). I'm a former coder turned CEO. What I see here is a leader who is afraid to let go. And at the same time blocking proven models. And after that you carry the fail-bag. One time, 15+ years ago I joined a small company as an IT Director - reports to CEO - had a pretty good time investment through the interview process. The second day I had similar feeling to what you've expressed. I quit. Sharp talent can find a new gig - quick. Like is too short for you to waste time explaining that 2>1 to folks who are paying you to tell them that - and then ignore you.

eximius 8y ago

If you're willing to relocate, the company I work for is in need of someone like you...

karmakaze 8y ago

Also add that 'flake tests' were a sore point until it became clear which ones were actually the system failing and which tests didn't reflect actual usage. By keeping stats on test faulures it was clear which were causing distraction time and fix them or system cause first. After that we got stability much more quickly than always treading seemingly random errors.

djangowithme 8y ago

"So after two weeks, I got the approval to create a staging branch." - lol

poulsbohemian 8y ago

>founder has pushed breaking changes directly to master Is founder also CEO? If founder is ceo and is coding rather than working on sales / marketing / traction, then you need to run briskly away

shrthnd 8y ago
Concurred.
em-bee 8y ago
why? if he is a tech founder, and there are co-founders that work on sales and marketing, why should he give up the CEO position? or do you mean to say that from his behavior he is obviously not qualified? greetings, eMBee.
- poulsbohemian 8y ago
  Put simply: Lead Developer != CEO. Unless it is super early stage (pre-employees, pre-revenue, pre-everything), the CEO should not be hands-on-keyboard. If they are, there are several problems: 1) Delegation. They are demonstrating a management failure to delegate. 2) Even in a technology-oriented business, the job of the CEO is not to cut code. The function of the CEO is to manage and lead all the things related to the growth and success of the business (think: sale, marketing, hiring and recruiting, raising capital, interacting with investors, and a whole lot more). If they want to undertake the technical functions of the business, they should become CTO / VP of Engineering / etc and turn the reigns over to a CEO who can execute on the business. 3) Even if there are other team members engaged in sales and marketing, see point #2 - the CEO is fundamentally responsible for the growth and financial success of the business and it is highly unlikely that technical tasks should be their priority.

cbanek 8y ago

It sounds like you're in a rough spot. Luckily, it sounds like from what you're talking about that there's a lot you can offer and there's a lot of low hanging fruit that can be picked. Here's what I suggest: 1. Find the easiest things to do that will save people time, or fix problems that people have been talking about. Especially useful if founder is one of the people helped. 2. Once you've found some easy things to do (<1wk), figure out which ones you can do without any help, resources, or authority. 3. Do it, and show it to him, or talk it up with the team and save them some time. 4. Tell the founder you've saved people time, and him money. Invest in some of your next ideas to save more time. Don't ask for permission. Just say you're doing it. 5. Success builds on success. One thing I'd definitely point out is sometimes you have to go skunkworks. It doesn't sound like you've been there long and they are harping on you to ship something. You likely have plenty of time to just do some easy things, like automating paperwork or forms. In terms of your staging workflow, again I'd apply concepts that I don't need permission for. Instead of blocking commits to master or even needing a staging branch, you could write some automation that takes a git hash. There's lots of ways to work around needing permissions or official things. Never underestimate the use of a button to do something, rather than typing it into a terminal. It also sounds like you may have people problems. Is your founder stressed? It sounds like he's having a rough time with scaling growth. Find out what is stressing him out at work and use your talent to eliminate it. If all else fails, use FIRE. (and quit.) Good luck.

natbobc 8y ago

It doesn't sound like a great situation to be in. It sounds like the founder has strong opinions. I suspect he subscribes to Trunk Based Development[1] and Continuous Delivery[2]. He also likely wants to follow the practises that "leading organisations" follow in "Accelerate: State of DevOps Report"[3]. You might find it beneficial to research those subjects prior to discussions with him. 1 - https://trunkbaseddevelopment.com/ 2 - https://continuousdelivery.com/ 3 - https://cloudplatformonline.com/2018-state-of-devops.html

teh_klev 8y ago
Just some friendly advice. When you're referencing stuff elsewhere on the web do it like this: [1] https://trunkbaseddevelopment.com/ https://trunkbaseddevelopment.com/ [2] https://continuousdelivery.com/ https://continuousdelivery.com/ [3] https://cloudplatformonline.com/2018-state-of-devops.html https://cloudplatformonline.com/2018-state-of-devops.html This means the links are clickable. Using the code indentation thing you did there causes all sorts of grief for mobile device users.
- natbobc 8y ago
  Thanks! Was faffing with it... the "help" isn't very helpful on how it formats. Sadly can't edit now. :/
imron 8y ago
Can't believe this comment is so low. I would guess this is exactly what is happening, an it sounds like OP is trying to push git flow or similar. I would push back against that as well. Trunk based development beats git flow every time.

anon31415926535 8y ago

What could help you stay sane is to recast how you view your job. Here is one way to think about it: your job is to help establish a culture of improved reliability. You are accountable for improving the process but you don't have the authority to decide the process. A large part of being successful in this role is sales. There is a pattern that works fairly well at starting a new engagement. Initially, the main goal is not to fix things; instead you need to focus on building trust. As you've found out, without trust, you won't have success. The easiest way to build trust is to find a problem that the team acknowledges and is small. Tell them how you will solve the problem and have them agree. Solve the problem. Then repeat with another problem until you have some trust. A good indication that you have enough credibility is when they stay asking for your input. Now, you need to find a problem that they are blind to or have given up on solving. Convince them it's a problem worth solving. Metrics are your friend here. Then get buy-in for your proposed solution. Keep visibility add the work is solved. Sacrifice overall speed for immediate partial wins. At this point, hopefully you've reached a credibility tipping point and established a good working relationship. I also had advice for dealing with change resistance, but that is not your problem. Your problem is building your personal credibility by delivering something the team cares about and wants solved. Try not feel insulted. It's not possible. They're only human. :)

epynonymous 8y ago

i think a little conflict is fine, working through conflict can often times build trust, use this as a learning experience because you're always going to have conflict, even if this conflict feels like a core belief mismatch. i find the founder is a bit too super direct and lacks tact, i'm a senior director, and have never told anyone to earn my trust, it's a given, i gain nothing by explicitly stating this, but that's besides the point, different strokes, different folks. i also see a bit of a power struggle here, because your role, perhaps at other companies, has been more control and process oriented, this is a startup with few processes and control seems to be completely with the founder. here's what i suggest, change is inevitably hard to deal with, it takes time and it's very natural for people to resist, so the first thing for you is to decide whether you're ok with this taking a long time, if not, there are a ton of other opportunities out there, don't waste time. if you're willing to stick it out and see value in the business, in the role, then figure out how to do things in incremental steps aka non-earth shattering, it shouldn't be, adhere to my god-like, every company does it this way devops pattern or else (even if it is god-like). baby steps where it's easier for the team/founder to accept, less friction/inertia. another thing is through statistics, founders change caused this which impacted that, we had 3h downtime because founder's change hosed the system, this caused roughly $20,000 hit to revenue, or we lost x users as a results of this. if he's ok with this after repeatedly pointing this out, maybe it's because his goal is different, he's looking for speed of execution as opposed to short term revenue gain, in which case you need to try to align and figure out ways to break things even faster, but with simple ways of rolling back. maybe he/she wants a/b type testing with quick rollbacks in case stuff hits the fan. obviously aligning with the founder is going to be critical. i also suggest a book titled "what got you here won't get your there". you should treat every new role with an open outlook, there are always new challenges to sharpen your skills and experience, if everything's similar to what you did before, you're essentially not learning much.

arountheworld 8y ago

I worked with a start-up where owner confessed he is just looking to get out as quickly as possible. We were not supposed to build processes that would slow down the development at the cost of huge technical debt - but that wouldn't be for him to worry. He did have a number of hires so that on paper it would look nice to investors. We had an experienced and old CEO whose job was to smile and drink with investors, number of developers to fiddle with code, two senior devs that were doing actual coding and a devops guy who spent time on Reddit and baby sitting juniors so they don't break the app too badly when deploying to production and marketing team that was making up ideas what they could shoe next to investors. Company was paying "customers" to use the product. I left early when I realised the equity package is made up. I'd say run! Not worth your reputation.

segmondy 8y ago

You're trying to do too much but moving too slow. You're asking for permission instead of just doing things. You wrote up a huge document? This is a startup not an enterprise! What's the worst thing that can happen? prod breaks right? Make sure it's being backed up! Teach them to tag the branches before merging into master, at least they can have a reference to rollback. Make sure rollback on master can happen with one command. Build the stage without permission, make it a CI/CD. Show them, taadaah! When you push, you can go test on stage, see? huge documents, calendars? meetings? this is a fast growing startup, all your ideas are great but you are executing it too slow and slowing everyone else down that's why the push back is happening.

trelliscoded 8y ago

> especially since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app > use of a staging environment - is met with suspicion and doubt If he doesn't want to give you authority to design the deployment pipeline, then he doesn't get to make you responsible for it. Responsibility/authority mismatch is a classic management mistake, they cover it in lower division business supervision and management classes. It's generally a result of one or more personality defects and/or something interfering with proper reasoning. If you can't get it fixed by figuring out how to work with the founder, then you need to find another job because it's only going to get worse as things ramp up. Based on what you're saying about wanting more research and suspicion when presented with change, it sounds like they have Groucho Marx syndrome due to either a self-esteem issue or impostor syndrome. The usual line of reasoning is: 1. I need expert help. I will look outside my organization for it. 2. I hired expert help, and they are now working for my company. 3. It is self-evident that this person does not know what they are doing because they are working for my organization, and I have a low opinion of myself and/or my organization. 4. I need expert help. Rinse and repeat. I'm currently working with a founder who constantly pushes to master, and I didn't push back because he earned my trust by being a really decent programmer. He's broken things in the past, but in general he fixes things faster than I can detect them so I'm ok with it. The engineering organization doesn't have many complaints about it because the company needs him free to work on the hiring pipeline, which is crucial because our staff keeps getting poached. If he was mediocre, I'd have privilege separated his accounts a long time ago and enabled annoying 3FA on his account with master commit privileges.

xrd 8y ago

I'm doubtful any of these "technical" solutions proposed here will work. What I mean is this: it sounds like the CEO is completely scattered. Nothing you put in front of him will be enough to shake him out of it. He is only getting 10% of what you are telling him. No matter how well written the document you put out there, no matter what consensus you build, it won't change his relationship to what he is dealing with. It's terrifying to be the CEO. He's failing constantly and fearful of being exposed as a fraud. What everyone else deals with, yes, but this time it seems more significant. My suggestion: find other people who have traveled the same path as he has, specifically the issues you were brought into solve. Find a way to connect them with him. Storytelling from a powerful communicator is the only thing that will break him free of this state. Once you have that in place, the solutions you put in front of him will be easy.

deleted 8y ago

[deleted]

tastyham 8y ago

Sounds like there is a miscommunication about what your role is and possibly what each of you mean by "devops".

watwut 8y ago

What about making production branch which goes live and is tested before and let them push into master if they like it so much? Practically, it does not matter which branch and how it is called. Important is that you don't push untested changes to live. Also, writing large document is not how you convince people. If they dont think your ideas are good, they perceive it just as an attempt to force them to do something that does not make sense to them. > since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give? This is large red flag. Toxic politics warning. Document what you do, document discussions decisions etc. Diplomatic is partly way to go, especially since it is founder, but if situation call for it be ready to defend yourself. Also, if that sort of founder behavior becomes usual, run away. No matter what your tech skills are, this will only get worst as other employees will naturally take clues from founder and imitate behavior. This is where production branch I suggested up there can really help. > He also keeps harping that I need to "earn his trust." Another red flag warning. Sometimes it is not about who you are and what you do, sometimes it is about who they are. And who they are suggests they will not build productive effective workplace. It will generate ugly office politics. If that happens, leave before you are so frustrated that you lost confidence or burned out.

m_ransing 8y ago

Put yourself in the founder shoes and think. He/She already has some way of doing things, which is working till now, and you are trying to break it. There will be some resistance to this inertia. When you talked about the standard, people tend to make themselves as exception saying that "our case is different". Check if this is the case. It seems that you are starting big. Trying to push the changes, which people are not used to. Start with small changes, something like statistical analysis tools or some small automated test case tool. Something which will be simple and not much time consuming for developers. Once they see its value, may be then they can "trust" you with big changes. Then there is psychological approach for making changes. You can present your ideas in such a way that the person buying (in your case founder), feels that he/she has a choice or a major role in decision making. But then you present choices, such that the one you prefer comes out as obviously best. Check if this approach can work in your case. Finally, it's your life and your job. Mental stability is more important that the company you are working in. Don't hesitate to fly away on first opportunity.

Blackstone4 8y ago

How does the founder behave with other employees? Especially the new ones. Focus on listening and validating their thoughts. i.e. "What are your concerns?" "Sounds like you're worried about X. I can see where you're coming from". You don't have to agree with them just make them feel like they are being heard. Rinse and repeat. If this doesn't work..... Find a new job.

thecolorblue 8y ago

I find myself in a similar situation. I took a job as a software developer at a hardware shop. They have different processes to what I am used to. I keep pushing for more robust project management but I am getting a lot of push back ("don't tell me how to do my job" level push back). There are two parts to this: 1. Build a better relationship with the founder (carefully) 2. Focus on adding value. New processes are not going well, so scrap them and do the best you can with the processes they have. Your arguments for change will go over better if you are adding a lot of value to the company. That is were credibility comes from in small groups like this.

EnderMB 8y ago

I've had similar issues in the past, and here's what I did to resolve them: 1. Get the others on board. I assume there are other developers in your team. Pitch your ideas to them, and not management, and get whoever is in charge of the developers to agree that your plan is the way to go. If there isn't a lead dev or dev team manager, just get them all to agree on a date where they switch to your way of doing things. 2. Do, rather than ask. I had the same issues with a director that fancied themselves as a coder. To get around this, I implemented a plan, set up a continuous delivery lifecycle, and put it into place. Their FTP access was revoked, and when they went through the (stupid) process to get FTP access again to tweak some CSS the next deployment wiped their changes out. 3. Let them fuck up, and be accountable for their fuck-ups. Ultimately, you can't stop someone in power from abusing it, so when he/she does and it inevitably fucks up, tell them they fucked up and the team will fix it. Going back to my FTPing director, he tried to fuck with some HTML on a Razor template and ended up breaking a page. We had a basic script that checked the sitemap periodically for anything other than 200 responses, and alarm bells soon rang. I asked around on Slack to see who had made live changes without going through the process, and after he (eventually) owned up, we re-deployed the last release and fixed the error. That public log of their fuckup was enough to ensure they stuck to the process after. 4. Own it, but take your time to come up with a solution. I think your biggest mistake is to immediately jump in and start making changes. I know they're a startup, but if I were you I would've committed to a week or two of simply watching and helping the team go about their business as it is. Get everything to tell you what the problems with their current setup is, and lead them towards the answer you want to provide. After that time expires, deliver your findings, backed up with all the feedback you've received from the entire team, and move to step 1. 5. Be prepared to walk if things don't improve. Sometimes, there's only so much you can do. You can have literally everyone on board, except one person in charge, and they'll be more than happy to derail everything in order to do what they want. If you reach a point where you feel that there isn't a way around a blocker, and you're absolutely sure that it's not your (or your teams) fault, then walk. Line up an interview, and simply tell them that you've accepted a job elsewhere. Don't rub their noses in it, or say the obvious reason why you're leaving. The team will probably already know why. Just leave, and thank them for hiring you.

squozzer 8y ago

You are probably a scapegoat. Pull the ripcord.

JSeymourATL 8y ago

> He also keeps harping that I need to "earn his trust." If he doesn't trust you, he won't listen to your advice. Use a collaborative model. Try to figure out the 'pictures in his head'. Good food for thought on this subject from Stuart Diamond > https://www.microsoft.com/en-us/research/video/getting-more-how-to-negotiate-to-achieve-your-goals-in-the-real-world/ https://www.microsoft.com/en-us/research/video/getting-more-...

vfulco2 8y ago

Get out while you can. You were brought on as a checkbox on someone's list, either a BoD, external investors, someone's due diligence. You'll have little to no impact with your domain expertise sadly. I have seen a few times where the boss would not listen to any standard reasoning or best practices.

lackbeard 8y ago

My guess as to what's going on here is that what the founder actually wants is for you to be the expert on the deployment environment & related tools to help ship things faster (in the short-run) and fix things when they break. On the other hand, what you actually want is to spend a bunch of time up-front building tools and processes to keep things stable. Somehow what each of you expects and wants from the other was miscommunicated.

cmcginty 8y ago

"If you can’t measure it, you can’t improve it." I would suggest starting by gathering data to show you CEO where you think you can make improvements. Sometimes just applying a standard processes is not enough to convince management of the benefits if they are not yet bought in.

akulbe 8y ago

Your profile says you're in WA. How close are you to Portland? We should have a beer and swap stories. I'm in a similar situation.

aprdm 8y ago

I find it amazing how people can make so many assumptions from this Ask HN post and already assume how the CEO is, what the start up needs, if OP is right or wrong and whatnot. DevOps is much about culture and it's not a one solution fits-it-all that you can grab from your past 7 years and readily apply to a company you just joined. I was responsible for a very big devops transformation in a medium company (3k employees, 80 devs), took around a year and a half to go from ssh to server to deploy to everyone being on Ansible and automated deploys with CI/dev/staging/prod. I would say to chill :), you just joined the company, 3w is not a lot of time to understand the culture, the devs, what they should optimize for, what is causing them pain, what is taking them time, what is making them money and etc. Also, have some real 1:1s with the CEO (or whoever you report to) and be very direct about how you are feeling and what the expectations are. Case in point, I really recommend the two books in this link: - https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html I don't believe any of that can be applied blindly to a company that isn't google size.

weliketocode 8y ago

You're new to the organization and were hired to help the company go from rough, ship-first, relatively low quality practices to high quality and scalable best practices. In order to make this a reality, you're going to need to first be very understanding and cognizant of the existing team and the reason they do what they do. This is the only way to inform your decisions on: 1. Which changes are most high priority ie business impact 2. Which changes will require significant team buy-in 3. Which changes will make the most sense for your team I very strongly recommend you start with some easy wins: - Bad key management? Use env variables - Autoscaling (as you mentioned) - Clean up no longer in-use aws services to save $$$ Once you've made an improvement or two, then start tackling some of the problems that require more buy-in. This is anything that affects others' workflows or pace of work. If the founder is cowboy-coding and pushing things directly to prod, you're going to need to get his buy-in to change his workflow. Come up with a document, plan, or RFC for anything that requires buy-in. Get comments from relevant parties for your plan. Then, and only then, implement the plan. 3 weeks really isn't that long. You still have plenty of time to right this ship.