4 ms·
Cloud migrating production first in 12 hours or less
Cloud migrating production without a plan, and how it made me a better programmer ;)
We have a b2b webapp product in cloud_provider_a which started having trouble serving requests, timing out.
My team (supporting another product in cloud_provider_b) was tasked with setting up a duplicate of prod in cloud_provider_b. I can't even remember why mgmt thought that would help.
By the end of the day, by hook or by crook the call was made to switch PRODUCTION traffic into a newly made stack in a different cloud than where that application was currently hosted. Despite protest from the engineering and QA teams.
Its all a blur now. Things went horribly wrong for two days. On the third day my team (with no prior knowledge of how this product works) found that the docker containers running the app had an nginx config with a low worker_connections setting.
We changed that one line and the app is back up. But the cost has been high, our customers had a far higher level of disruption these 3 days. Our engineers feel burnt out and disrespected, and our QA teams are appalled.
AMA I guess
- wtf_did_we_do 7y agoI read here a while back that as Engineers we have an ethical obligation to only present good options to management. That when pressed for more options we need to be careful on what we present. I think I have gained a lot of understanding around this now, and have pushed back time and time again to prevent management demanding dangerous and risky changes without even quantifying the impact of those, positive or negative. If only I had that before this mess began.
- K0SM0S 7y agoWhy aren't you interviewing for another job right now instead of sharing on HN? ;-) haha Joke aside. "No pain no gain" as they say. So what's one thing that was painful today but eventually is learning / growth / wisdom for you?
- wtf_did_we_do 7y agoI am. I think others in the company are now too. To be told that our knowledge and expertise isn't respected was a real blow. I did also learn a lot about cloud architecture though.
- K0SM0S 7y agoAh yeah nothing like the most barebones PoC to learn a lot about something. Bird's eye view and all that. Good that you're already thinking 'next'! The blow you speak of is common unfortunately between different types, and there is always a sorta 'dominant' type in every business. Sometimes it's the engineers, sometimes it's the marketing, sometimes it's sales, sometimes it's c-execs. That's one piece of info worth having to make more informed career choices, depending on your goal — you usually thrive as part of the dominant type, i.e. "ethos" of the company. Look at the CEO and background of the managing backbone. It usually says a lot about a company's culture, who matters and who's recycled perpetually.
- wtf_did_we_do 7y agoIt's definitely engineering that get recycled, and our C level is made of people focused on sales. Now I know what to look for in a company, this event paid out in wisdom coins for me.
- tuldia 7y agoThat is great! You are doing system administration now, enjoy! I believe people should be doing that more often, you know, just moving stuff around.
- wtf_did_we_do 7y agoI wonder for how much longer? If my team was asked to spend a day investigating instead of migrating we would have found it and solved it without any overtime and midnight call outs. At least I've grown a pair over the last few days.
- imhoguy 7y agoI see same too often recently. There is too much printed money on tables. Quality assurance degrades, business in scoping histeria, management micromanaging everything ad-hoc aka flat but same old structures, engineering forced to deliver over weekends and cut every corner. This bubble has to crash or people will go mental en masse.