7 ms·

Joyent's messaging about "we're cloud, but with perfect uptime" was always broken. It's mildly gross that the current messaging sounds like they're throwing a

by alrs 12y ago

Joyent's messaging about "we're cloud, but with perfect uptime" was always broken. It's mildly gross that the current messaging sounds like they're throwing a sysadmin under the bus. If fat fingers can down a data center, that's an engineering problem. I care about an object store that never loses data and an API that always has an answer for me, even if it's saying things that I don't want to hear. 99.999 sounds stuck-in-the-90s.

evan_ 12y ago
> sounds like they're throwing a sysadmin under the bus at least they didn't name the operator in question...
- elijahwright 12y ago
  Our internal culture is such that everyone on the team would rather be blamed for something than accuse someone else of doing it. That's shitty, and not something you do to someone. You fix the problem and then you move on. If it makes you happy, blame me - I don't mind.
  - hack_edu 12y ago
    BTW, this is the right way to do it. :)
    - niels_olson 12y ago
      "elijahwright" shall henceforth be used in place of "scapegoat"
      - elijahwright 12y ago
        Awesome! It's what I've always wanted!!!
      - arakkisu 12y ago
        it was that way at tech, no reason for it to change now
        elijahwright 12y ago
        Now I have to figure out who you are. :-)
  - sokoloff 12y ago
    At my $DAYJOB, we are always careful to figure out exactly what happened, including by whom. It's not to assign personal blame, but I believe it's critical that everyone agrees on the facts (who, what, when, where, and [if possible] why). Response and conversation is always focused on "how do we prevent this in the future?", not on punishing whoever was involved in the past. IOW, I agree with I believe is your intent, but differ on the implementation. Blameless transparency is the term we use (and we probably stole that from somewhere else). It's a very powerful signal to the whole team when you first see individuals "admitting" to exactly what they did, how it caused or contributed to the outage, and to hear them thanked for their contribution of understanding in the post-mortem. Senior leadership (including myself, who originally instituted the entire process a decade ago) is very clear that we want to know the facts and that in seeking and using those facts, we're only focused on the future, no matter how boneheaded the individual actions appear with the benefit of hindsight and knowledge that they'd lead (in)directly to an outage. I run operations and also participate in the promotion discussions for all technologists, and in 11 years, I've never heard a negative shadow cast onto a sysadmin/sysengineer from their actions during or leading to a production outage. And we've (collectively) made our fair share of mistakes over the years. That doesn't stop good employees from feeling bad about it, but that's a personal feeling they have, not from the fear of it being a professional black mark.
    - M2Ys4U 12y ago
      I think there's a difference in how you approach this with an internal-facing view and an external one. Internally, You're right. But externally the company fucked up, not the individual.
      - sokoloff 12y ago
        100% agree, and it is my oversight to not draw that distinction more clearly. We have the luxury (so far) of only reporting internally.
  - deleted 12y ago
    [deleted]
knodi 12y ago
Sure blame on the engineers. You give power, people use it badly blame the engineer for giving too much power. You don't give enough power sysadmins/users bitch and yell why don't we have enough power, we're not children. Its always the engineer fault. :(
- CHY872 12y ago
  It's a combined fault. Clearly the operator made a mistake, but the system shouldn't have let such a calamitous operation take place without at least three levels of "Are you sure" (or something smarter like "Confirm how many servers would you like to reboot:") before it lets you take down thousands of servers.
- alrs 12y ago
  Systems engineers, software engineers, architects, whatever. We're all in the same gang. My point is that the problem in this case is likely the system's design, not one engineer's typing abilities.
  - jsmthrowaway 12y ago
    This comes down to operational philosophy, in the end. The point you're dancing around is whether the system should permit grave actions that don't make any sense when you're designing the system. By the rules, every single system on a commercial aircraft has a circuit breaker. Pilots make the "what if X catches on fire?" case, which is actually pretty compelling. However, that also means there are several switches overhead that will ostensibly crash the airplane if pulled. Pilots lobby very strongly for the aircraft not to fight them in any way because they are the only ones with the data, in the moment, now. They have final command over the aircraft in every way. I use this to point out that as you're designing systems for operations people -- something we're increasingly doing ourselves as devops/SRE takes hold -- you might think you can anticipate every scenario and design suitable safeguards into the system. However, sometimes, when Halley's Comet refracts some moonlight into swamp gas and takes your fleet down, you as an operator have to do some really crazy shit. It's in that moment, when all hell has broken loose, I'm at the helm, and based on the data available to me I have made a decision to shoot the system in the head: if the system fights me and prolongs an outage because we argued about whether we'd ever need to reboot a fleet all at once, I'm replacing the system as the first item in my postmortem. If you make me walk row to row flipping PDUs, we're going to have words. That's just my philosophy. Give the operators the knives and let them cut themselves, trusting that you've hired smart people and understanding mistakes will happen. Your philosophy may vary. By all means, ask me to confirm. Ask me for a physical key, even. But if you ever prevent me from doing what I know must be done, you are in my way. I have yet to meet a system that is smarter than an operator when the shit hits the fan (especially when the shit hits the fan). There's probably a broader term for operational philosophy like this.
berns 12y ago
Joyent's marketing is not the most transparent. They haven't updated AWS prices in their pricing page since AWS lowered their prices two months ago.
- 4ad 12y ago
  What? Joyent doesn't use AWS.

by alrs 12y ago

evan_ 12y ago

> sounds like they're throwing a sysadmin under the bus at least they didn't name the operator in question...

elijahwright 12y ago
Our internal culture is such that everyone on the team would rather be blamed for something than accuse someone else of doing it. That's shitty, and not something you do to someone. You fix the problem and then you move on. If it makes you happy, blame me - I don't mind.
- hack_edu 12y ago
  BTW, this is the right way to do it. :)
  - niels_olson 12y ago
    "elijahwright" shall henceforth be used in place of "scapegoat"
    - elijahwright 12y ago
      Awesome! It's what I've always wanted!!!
    - arakkisu 12y ago
      it was that way at tech, no reason for it to change now
      - elijahwright 12y ago
        Now I have to figure out who you are. :-)
- sokoloff 12y ago
  At my $DAYJOB, we are always careful to figure out exactly what happened, including by whom. It's not to assign personal blame, but I believe it's critical that everyone agrees on the facts (who, what, when, where, and [if possible] why). Response and conversation is always focused on "how do we prevent this in the future?", not on punishing whoever was involved in the past. IOW, I agree with I believe is your intent, but differ on the implementation. Blameless transparency is the term we use (and we probably stole that from somewhere else). It's a very powerful signal to the whole team when you first see individuals "admitting" to exactly what they did, how it caused or contributed to the outage, and to hear them thanked for their contribution of understanding in the post-mortem. Senior leadership (including myself, who originally instituted the entire process a decade ago) is very clear that we want to know the facts and that in seeking and using those facts, we're only focused on the future, no matter how boneheaded the individual actions appear with the benefit of hindsight and knowledge that they'd lead (in)directly to an outage. I run operations and also participate in the promotion discussions for all technologists, and in 11 years, I've never heard a negative shadow cast onto a sysadmin/sysengineer from their actions during or leading to a production outage. And we've (collectively) made our fair share of mistakes over the years. That doesn't stop good employees from feeling bad about it, but that's a personal feeling they have, not from the fear of it being a professional black mark.
  - M2Ys4U 12y ago
    I think there's a difference in how you approach this with an internal-facing view and an external one. Internally, You're right. But externally the company fucked up, not the individual.
    - sokoloff 12y ago
      100% agree, and it is my oversight to not draw that distinction more clearly. We have the luxury (so far) of only reporting internally.
- deleted 12y ago
  [deleted]

knodi 12y ago

Sure blame on the engineers. You give power, people use it badly blame the engineer for giving too much power. You don't give enough power sysadmins/users bitch and yell why don't we have enough power, we're not children. Its always the engineer fault. :(

CHY872 12y ago
It's a combined fault. Clearly the operator made a mistake, but the system shouldn't have let such a calamitous operation take place without at least three levels of "Are you sure" (or something smarter like "Confirm how many servers would you like to reboot:") before it lets you take down thousands of servers.
alrs 12y ago
Systems engineers, software engineers, architects, whatever. We're all in the same gang. My point is that the problem in this case is likely the system's design, not one engineer's typing abilities.
- jsmthrowaway 12y ago
  This comes down to operational philosophy, in the end. The point you're dancing around is whether the system should permit grave actions that don't make any sense when you're designing the system. By the rules, every single system on a commercial aircraft has a circuit breaker. Pilots make the "what if X catches on fire?" case, which is actually pretty compelling. However, that also means there are several switches overhead that will ostensibly crash the airplane if pulled. Pilots lobby very strongly for the aircraft not to fight them in any way because they are the only ones with the data, in the moment, now. They have final command over the aircraft in every way. I use this to point out that as you're designing systems for operations people -- something we're increasingly doing ourselves as devops/SRE takes hold -- you might think you can anticipate every scenario and design suitable safeguards into the system. However, sometimes, when Halley's Comet refracts some moonlight into swamp gas and takes your fleet down, you as an operator have to do some really crazy shit. It's in that moment, when all hell has broken loose, I'm at the helm, and based on the data available to me I have made a decision to shoot the system in the head: if the system fights me and prolongs an outage because we argued about whether we'd ever need to reboot a fleet all at once, I'm replacing the system as the first item in my postmortem. If you make me walk row to row flipping PDUs, we're going to have words. That's just my philosophy. Give the operators the knives and let them cut themselves, trusting that you've hired smart people and understanding mistakes will happen. Your philosophy may vary. By all means, ask me to confirm. Ask me for a physical key, even. But if you ever prevent me from doing what I know must be done, you are in my way. I have yet to meet a system that is smarter than an operator when the shit hits the fan (especially when the shit hits the fan). There's probably a broader term for operational philosophy like this.

berns 12y ago

Joyent's marketing is not the most transparent. They haven't updated AWS prices in their pricing page since AWS lowered their prices two months ago.

4ad 12y ago
What? Joyent doesn't use AWS.