6 ms·
My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting O
by 0xbadcafebee 13y ago
My Devops horror stories, one sentence each:
- Somebody deployed new features on a Friday at 5pm.
- Fifteen hundred machines running mod_perl.
- Supporting Oracle - TWICE.
- It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is.
- Troubleshooting a bug in a site, view source.... and see SQL in the JS.
- stevenklein 13y agofun stuff
- danudey 13y agoI built a system where our developers can do instant deployments of any of our software packages (and instant point-in-time rollbacks), and then do zero-downtime restarts of services. Now we deploy dozens of times a day and I never get called on a Friday night because someone did something stupid. Edit: I do get called when I did something stupid and it broke the deployment system. But that's gotten much rarer lately.
- rdoherty 13y agoWould love to hear more about this system, sounds really cool!
- mikko-apo 13y agoI've built a similar system, around 5 year ago. Users were able to deploy any version to any cluster from a nice UI. Basically you could select any version, click "Install" button and follow the logs in real time. Behind the scenes it was a decentralized continuous delivery system. Very cool stuff, highly automated. Reduced a lot of work and sped up development cycles from months to minutes. Served quite a large software development organization (1000+). I think we had 1000 servers in 5 datacenters around the globe. Nowdays I'm working on an open source version of that system, it's still missing few critical features but hopefully I'll get the first release out next spring. btw, I'm looking for projects/contacts that would be interested in trying out how the system would fit their needs.
- existencebox 13y agoHey Mikko; I'm a sysadmin at a research university, and I'd be very curious in at least "picking your brain" about your tool. I can't make any promises about actual usage, but I always love to see a novel approaches to relevant problems. Do you have any sort of github/project page?
- mikko-apo 13y agoHi Cool, sysadmin at a research university sounds like a nice position to be at. Yes, it's already on github. Unfortunately, since it's missing those critical features it's not easy to see how the whole system is going to work. If you to talk just drop me an email at gmail. mikko.apo is the account.
- mcfunley 13y agoAutomated tests trigger automated security, all admins are banned, user groups are automatically notified that certain admins are no longer admins.
- ascendantlogic 13y agoI read the last two lines, threw up under my desk and blacked out.
- ecopoesis 13y agoI really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.
- randall 13y agoYeah, with "real" CI which some people seem to hate, checking in bad code becomes the problem, which seems way better than just waiting to deploy.
- justinsteele 13y agoIt's not always the person who pushed it on a friday that ends up fixing it, though. They can be unreachable, without a computer, etc etc. It's just easier to change less during hours you have less people on hand, is all.
- jmccree 13y agoI'd agree that in a perfect scenario, you should be able to push code at any time confidently. But for many companies and projects this is not really available. As well, in many organizations the person who has to fix broken stuff is not the same as who develops and pushes code. I'm not saying that's a good thing, but it is reality for many people. Even if it only happens once in your career, once you've had a dev push out code at 5pm friday night, jet out the door and hit the bar, meanwhile you (the sysadmin/ops on call) get woken up at 1am by site down alert, and have to debug/rollback the changes while the dev who pushed them is unreachable, you learn to really avoid friday evening pushes. Fool me once...
- walshemj 13y agoAmen Brother Its sad that most places dont have a proper technical copy (with a full copy of live data) to do full tests on TDD is all very well but you need to test the entire system.
- deleted 13y ago[deleted]
- mkramlich 13y ago> - Troubleshooting a bug in a site, view source.... and see SQL in the JS. this is why I refuse to do "View Source" on the HealthCare.gov website. I'm afraid of what I might see.
- aaronem 13y agoI'm just gonna leave this right here: if ('en' === 'en') { $('#desktop-nav .middle').append('<a class="liteac-login topnav myprofile" href="/marketplace/auth/global/en_US/myProfile#landingPage">My Profile</a>'); $('.mobile-nav-right').append('<a class="mobile-right-bottom liteac-login" href="/marketplace/auth/global/en_US/myProfile#landingPage">My Profile</a>'); } else { $('#desktop-nav .middle').append('<a class="liteac-login topnav myprofile" href="/marketplace/auth/global/es_MX/myProfile#landingPage">My Profile</a>'); $('.mobile-nav-right').append('<a class="mobile-right-bottom liteac-login" href="/marketplace/auth/global/es_MX/myProfile#landingPage">My Profile</a>'); }
- illicium 13y agoAh, now I see why it has 500 million lines of source code.
- mkramlich 13y agoand costs 200M$+ because, you know, static HTML and some CRUD/lookup logic behind-the-scenes is just that hard
- DCoder 13y agoSeen this pattern many times before. One of those 'en' strings is the current user's language being written into the source, the other is hardcoded. If your server-side templating engine is impotent and only supports variable interpolation without conditionals, this approach is easier than pulling the right JS snippet from somewhere else.
- pygy_ 13y ago- It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. How did you locate it? Measuring ping latency from other machines?
- linc01n 13y agoWe won't deploy any code even after Wed afternoon. Don't get ourselves in any troubles.