5 ms·

No matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)

pdelgallego 7y ago
"You can't legislate against failure, but you can focus on fast detection and response" -- Chris Pinkham
- PenguinCoder 7y ago
  "Prevention is ideal, but detection is a must"
- iso947 7y ago
  It’s amazing how this is accepted in the software world. Move fast and break things, such a different philosophy to other areas.
  - dasil003 7y ago
    It’s evolutionary pressure; software is malleable and potential functionality is limitless. Software companies that didn’t ascribe to this philosophy were repeatedly killed by ones that did until it became the status quo.
  - dspillett 7y ago
    I think the quote from Pinkham is more about dealing with genuinely unavoidable failures, not those due to moving without appropriate due diligence.
  - mattigames 7y ago
    I mean, that's rather disputable: The Apollo 1 exploded, medical mistakes have a toll of 250.000 deaths per year in the US alone; among many other serious mistakes on vast different areas, I think unreliability is unfortunately a constant on the human race.
- _salmon 7y ago
  "I will not be harassed in my own private domicile" -- Jesse Pinkman
  - edm0nd 7y ago
    "I AM THE ONE WHO KNOCKS" -- Walter White
alexis_fr 7y ago
Let’s reflect on Amazon’s 99.999999999% (literally their number) durability on S3.
- dragonwriter 7y ago
  Durability isn't uptime.
  - brainwipe 7y ago
    Correct. Availability is 99.99. https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/
    - victords 7y ago
      Regardless, the SLA that AWS (& other cloud providers) meet is quite impressive.
      - Cthulhu_ 7y ago
        But what do they offer if they don't meet it? I mean a discount on your monthly bill somehow doesn't sound like it would cover a potential loss.
amelius 7y ago
The interesting thing is that Git is entirely non-centralized, so in theory they could simply redirect to servers onto which the data has been mirrored.
- deleted 7y ago
  [deleted]
- tribaal 7y ago
  Yes, git is. The issues, comments, PRs, wikis etc... that we all came to depend on aren't.
  - jsmith45 7y ago
    The wiki is stored as a git repository, although to my knowledge the others are not.
- Cthulhu_ 7y ago
  Git is, but the APIs and all the services they provide around it aren't. That said, I think it's a bit weird that they don't store the data of the services around the code itself in git, like they do with e.g. sites. That way you'd have an `issues` branch that you could still access if github is down. But that would probably pave the way for easy migrations away from Github.
  - chrisweekly 7y ago
    >"But that would probably pave the way for easy migrations away from Github." bingo
    - amelius 7y ago
      At least they could have used the concepts in Git's design. But it seems they didn't learn much from the tool they based their service on.
      - 7y ago
legohead 7y ago
could learn a few things from Netflix and chaos engineering...

pdelgallego 7y ago

"You can't legislate against failure, but you can focus on fast detection and response" -- Chris Pinkham

PenguinCoder 7y ago
"Prevention is ideal, but detection is a must"
iso947 7y ago
It’s amazing how this is accepted in the software world. Move fast and break things, such a different philosophy to other areas.
- dasil003 7y ago
  It’s evolutionary pressure; software is malleable and potential functionality is limitless. Software companies that didn’t ascribe to this philosophy were repeatedly killed by ones that did until it became the status quo.
- dspillett 7y ago
  I think the quote from Pinkham is more about dealing with genuinely unavoidable failures, not those due to moving without appropriate due diligence.
- mattigames 7y ago
  I mean, that's rather disputable: The Apollo 1 exploded, medical mistakes have a toll of 250.000 deaths per year in the US alone; among many other serious mistakes on vast different areas, I think unreliability is unfortunately a constant on the human race.
_salmon 7y ago
"I will not be harassed in my own private domicile" -- Jesse Pinkman
- edm0nd 7y ago
  "I AM THE ONE WHO KNOCKS" -- Walter White

alexis_fr 7y ago

Let’s reflect on Amazon’s 99.999999999% (literally their number) durability on S3.

dragonwriter 7y ago
Durability isn't uptime.
- brainwipe 7y ago
  Correct. Availability is 99.99. https://aws.amazon.com/s3/storage-classes/ https://aws.amazon.com/s3/storage-classes/
  - victords 7y ago
    Regardless, the SLA that AWS (& other cloud providers) meet is quite impressive.
    - Cthulhu_ 7y ago
      But what do they offer if they don't meet it? I mean a discount on your monthly bill somehow doesn't sound like it would cover a potential loss.

amelius 7y ago

The interesting thing is that Git is entirely non-centralized, so in theory they could simply redirect to servers onto which the data has been mirrored.

deleted 7y ago
[deleted]
tribaal 7y ago
Yes, git is. The issues, comments, PRs, wikis etc... that we all came to depend on aren't.
- jsmith45 7y ago
  The wiki is stored as a git repository, although to my knowledge the others are not.
Cthulhu_ 7y ago
Git is, but the APIs and all the services they provide around it aren't. That said, I think it's a bit weird that they don't store the data of the services around the code itself in git, like they do with e.g. sites. That way you'd have an `issues` branch that you could still access if github is down. But that would probably pave the way for easy migrations away from Github.
- chrisweekly 7y ago
  >"But that would probably pave the way for easy migrations away from Github." bingo
  - amelius 7y ago
    At least they could have used the concepts in Git's design. But it seems they didn't learn much from the tool they based their service on.
    - 7y ago

legohead 7y ago

could learn a few things from Netflix and chaos engineering...