4 ms·

I consider 'toomuchtodo a voice of authority on operations based on much reading and discussion (particularly on moving to physical, which we've discussed befor

I consider 'toomuchtodo a voice of authority on operations based on much reading and discussion (particularly on moving to physical, which we've discussed before), have performed the very exercise being discussed four times in my own career ranging from a couple cabinets to a couple hundred million in capital, completely agree with the entire comment to which you are replying, and feel that your jab about "serious experience in the real world" was totally unnecessary. If moving operations around is insurmountably difficult, you built operations incorrectly. Put another, even broader, way with a few more implications: if you are totally reliant on one vendor for continuity of any part of your operations, you built operations incorrectly and are introducing unnecessary risk. If us-east-1 goes down and you cease generating revenue as a result, you have built operations incorrectly. That's really all there is to it. And yes, I realize this means 80%, maybe more, of the operations in in the world is built incorrectly. We just learned Snap's is[0]. Maybe even yours! And that's fine as long as you're working on it. Good news: said exercise is a good chance to fix it! Now that half the crowd is inhaling to bombastically retort that undoubtedly controversial, yet completely true, paragraph, allow me to quickly redirect: What's "the real world," anyway? Most of HN forgets a Windows/.NET ecosystem exists, not to mention extra-valley gigs in, say, Nebraska. Would you say the lone sysadmin holding together a hospital in Des Moines is gaining "real world" experience and able to meet you in discussion? Seriously, I hate "the real world" and the people who fire it as a volley during an argument. Even your career is not indicative of "the real world." (Nor is mine.) [0]: Flagrantly so. I've spent the better part of an hour trying to concoct a scenario where that deal is even remotely in the win column for Snap. Still trying. You enshrined a business disincentive (nay, prohibition!) toward optimizing your opex into a five year contract and $400mm a year operating Snap was some kind of win? ... How on Earth? Even with a quarter billion DAUs...

toomuchtodo 10y ago
I really appreciate the kind words, and wish I had more than one upvote for your comment. I'm not here for profit or ego, just to share what I have learned and experienced for the benefit of others. "I am old, here are my mistakes, do not make them" sort of thing. Whether anyone believes it, there's a serious Bay Area echo chamber (which HN overlaps with); there is a whole tech world out there that isn't startup culture/tools/methodologies. If you're ever in Tampa or Chicago, let's grab a beer or dinner my treat. Would love to share war stories.
- eitally 10y ago
  While I agree with your position in principle, I also come from a "legacy IT" world, the kind where systems are moved to public cloud because developers are several steps ahead of the DC admins, or because the CIO wants all net new dev to be on infrastructure he or she doesn't own. These are not "digital natives" (who have no excuse, imho). These are the thick, long tail or large businesses with aging infrastructure and a lack of willingness -- or ability -- to pay for top talent. An additional point is that even the second tier PaaS/IaaS providers (like GE/Predix, for example) are trying to get out of the DC ownership business. There's no compelling reason for them to keep their own DCs when a) it's expensive to run & keep fresh, b) it's CapEx, and c) it costs them expensive heads to organize and manage everything.
  - late2part 10y ago
    Snap shouldn't fit the model of legacy IT. To some degree, myspace did and that's part of why they failed.
res0nat0r 10y ago
I also hate when people say "you should just be able to flip a button and switch cloud providers or regions or you're doing it wrong." That shows about as much of a lack of real world experience as you are bemoaning. It doesn't work that way in real life.
- jsmthrowaway 10y ago
  It's an ideal to strive toward. "Real world" and "real life" dismissals of said ideal are stupid, lame excuses for people who want to make a stupid, lame argument to cover up stupid, lame technical debt in operations by either (a) assailing the credentials of the speaker, as this entire thread has spent much time doing, and/or (b) providing a "well, everybody else isn't investing in this, neither should we" cop-out. And yes, you are doing it wrong. Rather than getting defensive, pulling the knives out, downvoting on sight, and trying to wipe away or justify doing it wrong, why don't you instead realize that it's motivation to do it better? We should all do things better, and every time a single AWS region goes down and takes out half the Internet I sigh because it's this exact, stupid, lame justification fest that results in that situation. Work toward correct operations. You will never reach it. The cognitive dissonance of these two statements is totally acceptable. If you build a new service today and don't account for DR and security up front it just goes to show me that you haven't learned from the very public failures of those who have come before you, and that to me shows a lack of real world experience. And yes, I'm aware of several shops that can literally flip a switch, and we're talking cage and ASN, not hobby-scale iOS backends. It's not like finding the holy grail and requires developers who "get" operations. It does exist. And it exists in real life! \u1F632!
010a 10y ago
The "real world" argument is a classic No True Scotsman fallacy.

toomuchtodo 10y ago

I really appreciate the kind words, and wish I had more than one upvote for your comment. I'm not here for profit or ego, just to share what I have learned and experienced for the benefit of others. "I am old, here are my mistakes, do not make them" sort of thing. Whether anyone believes it, there's a serious Bay Area echo chamber (which HN overlaps with); there is a whole tech world out there that isn't startup culture/tools/methodologies. If you're ever in Tampa or Chicago, let's grab a beer or dinner my treat. Would love to share war stories.

eitally 10y ago
While I agree with your position in principle, I also come from a "legacy IT" world, the kind where systems are moved to public cloud because developers are several steps ahead of the DC admins, or because the CIO wants all net new dev to be on infrastructure he or she doesn't own. These are not "digital natives" (who have no excuse, imho). These are the thick, long tail or large businesses with aging infrastructure and a lack of willingness -- or ability -- to pay for top talent. An additional point is that even the second tier PaaS/IaaS providers (like GE/Predix, for example) are trying to get out of the DC ownership business. There's no compelling reason for them to keep their own DCs when a) it's expensive to run & keep fresh, b) it's CapEx, and c) it costs them expensive heads to organize and manage everything.
- late2part 10y ago
  Snap shouldn't fit the model of legacy IT. To some degree, myspace did and that's part of why they failed.

res0nat0r 10y ago

I also hate when people say "you should just be able to flip a button and switch cloud providers or regions or you're doing it wrong." That shows about as much of a lack of real world experience as you are bemoaning. It doesn't work that way in real life.

jsmthrowaway 10y ago
It's an ideal to strive toward. "Real world" and "real life" dismissals of said ideal are stupid, lame excuses for people who want to make a stupid, lame argument to cover up stupid, lame technical debt in operations by either (a) assailing the credentials of the speaker, as this entire thread has spent much time doing, and/or (b) providing a "well, everybody else isn't investing in this, neither should we" cop-out. And yes, you are doing it wrong. Rather than getting defensive, pulling the knives out, downvoting on sight, and trying to wipe away or justify doing it wrong, why don't you instead realize that it's motivation to do it better? We should all do things better, and every time a single AWS region goes down and takes out half the Internet I sigh because it's this exact, stupid, lame justification fest that results in that situation. Work toward correct operations. You will never reach it. The cognitive dissonance of these two statements is totally acceptable. If you build a new service today and don't account for DR and security up front it just goes to show me that you haven't learned from the very public failures of those who have come before you, and that to me shows a lack of real world experience. And yes, I'm aware of several shops that can literally flip a switch, and we're talking cage and ASN, not hobby-scale iOS backends. It's not like finding the holy grail and requires developers who "get" operations. It does exist. And it exists in real life! \u1F632!

010a 10y ago

The "real world" argument is a classic No True Scotsman fallacy.