3 ms·
It can sometimes be very hard to cold start in such circumstances, because the demand from all the services trying to access their secrets at once can create a
by ashtonkem 5y ago
It can sometimes be very hard to cold start in such circumstances, because the demand from all the services trying to access their secrets at once can create a “thundering herd” that knocks the recently restored service over again.
My guess is that they’ve had to shut most services down, restore the secret store, and then bring up internal services in a careful sequence to avoid a thundering herd. At $JOB even understanding the correct sequence to cold start our organization would probably take more than 24hrs.
- noir_lord 5y agoAt $JOB, I've often pointed out that no-one actually knows if given a sufficiently bad issue and a reasonable timeline we even could bring it back up. In power grids its called a Black Start, I think that's something we could nick as a concept.
- sidlls 5y agoJust one more reason I don't take the "engineering" part of "software engineering" seriously. Hardly surprising given the admiration of tinkering and hacking that runs through the industry.
- jatone 5y agomodern software spans the globe, often communicating with 3rd party systems, operates 24/7/365, and is often at the whims of deadlines. its some of the most technically challenging endeavors in existence. its not really surprising how you'd end up in these situations. that being said. rate limits and error handling people. jesus. =)
- jiveturkey 5y agoreal engineering is full of the same examples. i’m thinking of civil infrastructure but there is no shortage in any domain
- ashtonkem 5y agoThe 19th century is full of “real engineering” that is frankly clown shoes stuff. Stories about how early truss bridges were designed makes modern software engineering seem quite orderly by comparison.
- sidlls 5y agoI've always been amused at this comparison. You're not wrong. We wouldn't tolerate or excuse a group of people attempting engineering based on 19th century practices. Why should we excuse software engineers (if, indeed, their goal is to be "engineers" in any meaningful sense) for not building on the last two centuries of engineering knowledge?
- ashtonkem 5y agoBecause software engineering is not the same as mechanical engineering. We’ve tried to apply mechanical engineering principles to software projects; it didn’t work.
- dunham 5y agoBack in college (90’s), we had a Solaris shop. One time after a long power outage we discovered that we couldn’t bring up the system because of a circular dependency. It’s been decades, something along the lines of a NIS server depending on a file server and vice-versa. We managed to manually get the services running and then do a clean reboot after the rest of the systems came up.