8 ms·

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

toomuchtodo 12y ago
Devops Borat: This not mistake. This how we test recovery operations in production.
- ibisum 12y ago
  This one goes on the first page of my playbook as of today.
  - deleted 12y ago
    [deleted]
timClicks 12y ago
Propogation of a software mistake is what appears to have caused the Gmail outage of 2011 http://youtu.be/eNliOm9NtCM?t=28m49s http://youtu.be/eNliOm9NtCM?t=28m49s
mey 12y ago
I wish that account was still updating.
- reitanqild 12y ago
  For anyone who hasn't read it yet the last tweet on that account is gold.
- misnome 12y ago
  I found an alternative in https://twitter.com/BigDataBorat https://twitter.com/BigDataBorat, not quite as insightful but still occasionally funny
ar7hur 12y ago
"In startup we are practice Outage Driven Infrastructure."
- wiml 12y ago
  I'm not sure that's so unlike Netflix's Chaos Monkey.
laumars 12y ago
I think it does happen often but isn't as well reported. I certainly know of more than one place that's suffered from this kind of accident (thankfully not places I personally work so I've not had to deal with the fallout. These are places I have friends or family who work there)
wiredfool 12y ago
The most frightening movie ever for sysadmins -- The Sorcerer's Apprentice.
IgorPartola 12y ago
@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month rollout of a large product. It itched like crazy, but I did work much faster to get rid of it.
- Havoc 12y ago
  >best minds of generation are deal with flaky VPN client So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only solution). So much for Cisco...
  - Nexxxeh 12y ago
    At the risk of asking a potentially dumb question, why can they only reimage the machine at head office?
    - Havoc 12y ago
      Not dumb at all. This is a professional service firm, so there is no real head office per se, but rather your "home office" - I just simplified it a bit for hn purposes. Couple of reasons. Each country rolls their own custom image. Plus I need an office that has the encryption keys for the full disk encryption. Plus only 3 offices globally carry copies of my data (used when they can't pull the data off the hdd). If I'm flying anyway I might as well go to home office - I know they have all the required stuff for my laptop.
  - philtar 12y ago
    Are you calling yourself one of the best minds of our generation?
jerf 12y ago
My favorite personal variant: "To err is human. To screw up a million times per second, you need a computer."
shill 12y ago
Worst mistake is always happen in batch. --Devops Borat https://twitter.com/DEVOPS_BORAT/status/251885078794366976 https://twitter.com/DEVOPS_BORAT/status/251885078794366976
antmldr 12y ago
>Frankly, I'm surprised things like this don't happen more often. They do. This happened to the largest bank in Australia mid 2012[1]. Very similar circumstances. I've been told that SCCM's UI doesn't help here- something about the default action when nothing is selected to apply it to all devices managed by SCCM. Someone more familiar with SCCM may want to correct me here. [1] http://delimiter.com.au/2012/07/30/disastrous-patch-cripples-commbank/ http://delimiter.com.au/2012/07/30/disastrous-patch-cripples...

toomuchtodo 12y ago

Devops Borat: This not mistake. This how we test recovery operations in production.

ibisum 12y ago
This one goes on the first page of my playbook as of today.
- deleted 12y ago
  [deleted]

timClicks 12y ago

Propogation of a software mistake is what appears to have caused the Gmail outage of 2011 http://youtu.be/eNliOm9NtCM?t=28m49s http://youtu.be/eNliOm9NtCM?t=28m49s

mey 12y ago

I wish that account was still updating.

reitanqild 12y ago
For anyone who hasn't read it yet the last tweet on that account is gold.
misnome 12y ago
I found an alternative in https://twitter.com/BigDataBorat https://twitter.com/BigDataBorat, not quite as insightful but still occasionally funny

ar7hur 12y ago

"In startup we are practice Outage Driven Infrastructure."

wiml 12y ago
I'm not sure that's so unlike Netflix's Chaos Monkey.

laumars 12y ago

I think it does happen often but isn't as well reported. I certainly know of more than one place that's suffered from this kind of accident (thankfully not places I personally work so I've not had to deal with the fallout. These are places I have friends or family who work there)

wiredfool 12y ago

The most frightening movie ever for sysadmins -- The Sorcerer's Apprentice.

IgorPartola 12y ago

@DEVOPS_BORAT is actually very insightful in about 1/5 tweets. Snide for sure, but there are quite a few good points in there if you read carefully: "In devops we have best minds of generation are deal with flaky VPN client." "Single point of failure in private cloud is of usually Unix guy with neckbeard." These are gold. Edit: based on the above advice I once grew out a neckbeard while going through a multi-month rollout of a large product. It itched like crazy, but I did work much faster to get rid of it.

Havoc 12y ago
>best minds of generation are deal with flaky VPN client So true. I'm on the receiving side of this..."No you can't work on that multi million deadline project of yours...the only way to fix the VPN is to re-image the machine back at head office [an international flight away]". Me..."Could you repeat that?" And thats a Cisco Enterprise VPN...(turns out IT was right...re-image & avoid conflicting software is the only solution). So much for Cisco...
- Nexxxeh 12y ago
  At the risk of asking a potentially dumb question, why can they only reimage the machine at head office?
  - Havoc 12y ago
    Not dumb at all. This is a professional service firm, so there is no real head office per se, but rather your "home office" - I just simplified it a bit for hn purposes. Couple of reasons. Each country rolls their own custom image. Plus I need an office that has the encryption keys for the full disk encryption. Plus only 3 offices globally carry copies of my data (used when they can't pull the data off the hdd). If I'm flying anyway I might as well go to home office - I know they have all the required stuff for my laptop.
- philtar 12y ago
  Are you calling yourself one of the best minds of our generation?

jerf 12y ago

My favorite personal variant: "To err is human. To screw up a million times per second, you need a computer."

shill 12y ago

Worst mistake is always happen in batch. --Devops Borat https://twitter.com/DEVOPS_BORAT/status/251885078794366976 https://twitter.com/DEVOPS_BORAT/status/251885078794366976

antmldr 12y ago

>Frankly, I'm surprised things like this don't happen more often. They do. This happened to the largest bank in Australia mid 2012[1]. Very similar circumstances. I've been told that SCCM's UI doesn't help here- something about the default action when nothing is selected to apply it to all devices managed by SCCM. Someone more familiar with SCCM may want to correct me here. [1] http://delimiter.com.au/2012/07/30/disastrous-patch-cripples-commbank/ http://delimiter.com.au/2012/07/30/disastrous-patch-cripples...