6 ms·
re: Chaos Monkey. I love this, and I'm 48. 99.999% uptime meant that once a year your program did something completely disastrous because the server flunked. A
by marcus_holmes 10y ago
re: Chaos Monkey. I love this, and I'm 48. 99.999% uptime meant that once a year your program did something completely disastrous because the server flunked.
Assuming our hardware was reliable was a mistake. Writing code so that it's chaos-monkey-proof is a great step forward.
As for the generational thing... meh, we're ugly. People like hiring the beautiful folks. Startup folks think they're immortal, having some greying old fart in the corner wheezing away reminds them that they're not.
- informatimago 10y agoThey should hire us for remote work! :-)
- lsc 10y agoChaos Monkey is actually a reference to a tool that I think netflix released? huh. yeah, I see it on github: https://github.com/Netflix/SimianArmy https://github.com/Netflix/SimianArmy I remember someone giving a talk on it in the mid aughts, and I think it's a wonderful way to describe what happens in real-world networks and systems. >Assuming our hardware was reliable was a mistake. Writing code so that it's chaos-monkey-proof is a great step forward. Hah. Part of it is that I am the guy who is responsible for keeping that multi-year uptime (and on most of my own hardware, this is mostly a matter of vetting updates, security and otherwise, and implementing workarounds) - but I like to point out that building a system that can deal with the chaos monkey is... well, it's way harder than throwing up a reliable SQL server with a reliable backup and someone like me on pager. Especially if you want to beat the "the system fucks up once a year" record that even mediocre server-grade hardware can achieve. That said, if you can write such software, I agree that the end result is better, I mean, like half my skillset is in Linux/BSD and environments where reboots cost big bucks, but I'm not going to cry about turning in my pager. I'm just saying, it's hard and most people screw it up and end up building something less reliable than well-managed hardware with a competent oncall. >As for the generational thing... meh, we're ugly. People like hiring the beautiful folks. Startup folks think they're immortal, having some greying old fart in the corner wheezing away reminds them that they're not. Eh, I think that most of the startups that only hire pretty people also tend to pay such that they can't afford someone who is good who has a few decades of experience, so it might be more of a budgetary/age thing; young people, generally, will work for less. I think it might also be a San Francisco thing. The fitness of the people interviewing me goes up and the wages they offer me go down when I head up that direction.
- throwanem 10y agoThe strong impression I get is that it's a seller's market for programming work most places in the United States, but a buyer's market in and around the Bay Area. I've never worked there myself, only visited, but I have known a lot of folks from there with hair-raising tales to tell. Things seem a lot more equitable elsewhere.
- lsc 10y agoEh, I'm contrasting San Francisco, above, with the South Bay; both are "bay area" (conventionally, the south bay is "silicon valley" - historically, San Francisco is not "silicon valley" though that definition seems to move every bubble.) In the South Bay, I largely don't have cultural fit problems, and I can pretty easily score pretty good jobs at second and third-tier companies (getting jobs at top-tier companies is a real challenge, still, but considering the compensation and prestige at the end of that particular tunnel, I can see why.) In San Francisco, though, I quite often have 'cultural fit' issues, and often can't even get into startups that pay really poorly.
- flukus 10y agoOn the other hand, most places don't need that level of reliablity. If a company email server went down for a few hours it's not that big a deal. Even for most large corps, only a certain few things need that level of reliability. I often see it get written into contracts with very little thought as to what it means. You get some great looks from managment when you break the figure down into minutes though, especially when you don't have 24/7 staffing. A poorly timed cosmic ray could ruin your SLA. * It would where I'm at now, because someone thought it would be a good idea to handle a critical business process by emails csv's.
- bandrami 10y agoI've also seen people make the other mistake, though. I've had people screaming about 2 hours of downtime violating our 99.999% SLA, apparently not realizing the five nines means the thing can be down eight hours a year (an entire business day) and still be in spec.
- entropy_ 10y agoThat's not true. 5 nines, ie 99.999% uptime means that the maximum downtime in a year is about 315 seconds: 60x60x24x365 x (1 - (99.999/100)) = 315.36 Wikipedia confirms: https://en.wikipedia.org/wiki/High_availability#Percentage_calculation https://en.wikipedia.org/wiki/High_availability#Percentage_c... So yes, 2hrs of downtime would most definitely violate the SLA. The 8 hours you refer to is the figure for 3 nines.
- bandrami 10y agoD'oh. It was three nines, thanks