13 ms·

“A Windows 7 deployment image was accidently sent to all Windows machines”

337 points by slyall 12y ago

k_sze 12y ago
Funny how they mention iTunes as one of the "key components" that are restored first, whereas Visio, Project, Adobe application are relegated to a second round.
- krexit 12y ago
  It's possible they're delivering academic materials through iTunes, I suppose.
- lstamour 12y ago
  Not hard to imagine. The others likely have specialty licenses and so aren't as easily distributed to everyone. In addition, Adobe software itself wasn't working earlier ;-)
- zachlipton 12y ago
  Presumably iTunes is part of their base system image for all workstations, along with Office, Firefox, Adobe Reader, and the like. In other words, a basic set of software to handle standard officework tasks. iTunes is free and IT would probably rather distribute it everywhere than have people trying to install it themselves (or calling the helpdesk to get someone with administrator rights to do it). They then offer additional applications on an as-needed basis to individuals and departments with specific tasks. So the designers who do print publications and the faculty who teach digital art might get the Adobe suite, while people in Facilities who plan construction will get Project. This keeps licensing costs down and simplifies systems according to their uses.
  - commandar 12y ago
    Generally you're going to keep your base images in SCCM limited to software that's only infrequently updated. Otherwise somebody has to update the entire image every time an update gets pushed out. Instead, you package them up and deploy the apps on top of the base image at install time. It takes a little longer to deploy, but it takes less admin time to manage since the actual installs are automated anyway.
facorreia 12y ago
> A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks. Wow. That is very unfortunate, to say the least...
- spiantino 12y ago
  > As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.
  - malkia 12y ago
    I feel bad, but I laughed at this for few seconds...
    - __david__ 12y ago
      I wouldn't feel bad. I guffawed at most of this story. Not in a "haha, what a bunch of morons. Serves those jerks right!" kind of way, but more in a "oh dear, that's the worst thing that can possibly happen! Oh no it gets worse??". I've been through IT catastrophes (and caused a couple myself) and I could easily see this happening to me. Still, it's funny as anything.
  - rasz_pl 12y ago
    We apologize again for the fault in the subtitles. Those responsible for sacking the people who have just been sacked have been sacked
sorennielsen 12y ago
This happened at one a former workplace too. Only the Solaris and Linux servers was untouched. It "mildly" amused the *nix operations guys to see all the "point and click" colleagues panic.
filmgirlcw 12y ago
I've never been prouder of my alma mater. /s
- gt1337h 12y ago
  You should be. Emory is the 2nd best school in Georgia! :)
miles 12y ago
Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.
- song 12y ago
  Yes, shit happens. It's how you solve it and make sure that it doesn't happen again that's important. With the way they're communicating, they seem to be on the top of their game
- harrystone 12y ago
  It is impressive. I can't imagine dealing with that kind of nightmare. I once worked as an admin on Solaris boxes at a big pharma company. There were ~77,000 users in their LDAP directory. I was very careful.
  - cfreeman 12y ago
    That sounds terrifying.
- keithpeter 12y ago
  At one place I worked, in the days of XP, the 'index server'(?) had a problem and un-installed all the application programs. The basic OS was there but MS Office, MSIE, all the doodads just got removed as each machine logged in. This was a small college, so the IT guys just went round explaining and told us to log out and log in again. Applications re-installed. No data loss and so no shouting. Stuff happens. We did 'assignment action planning' that morning: mind maps, essay plans and research ideas. Results better than normal anyway
- filmgirlcw 12y ago
  100% agree. My initial response was to laugh, but dude, I so feel for the team that has to clean the mess up.
- swasheck 12y ago
  It's not a standard across the board, but I've found academic institutions to be more honest about their technological mistakes (outside of large-scale breaches) than the private sector.
smegel 12y ago
Automation can also mean automated disaster.
- batmansbelt 12y ago
  ...Or automated the-opposite-of-disaster.
- deleted 12y ago
  [deleted]
- Already__Taken 12y ago
  Bigger tools have bigger sharper edges. It's important the message is stressed as the developers to operators worlds merge in the middle.
- ameoba 12y ago
  Why did the Roman Empire collapse? What is the Latin for office automation?
deleted 12y ago
[deleted]
perlgeek 12y ago
https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.
- toomuchtodo 12y ago
  Devops Borat: This not mistake. This how we test recovery operations in production.
  - ibisum 12y ago
    This one goes on the first page of my playbook as of today.
    - deleted 12y ago
      [deleted]
- timClicks 12y ago
  Propogation of a software mistake is what appears to have caused the Gmail outage of 2011 http://youtu.be/eNliOm9NtCM?t=28m49s http://youtu.be/eNliOm9NtCM?t=28m49s
- mey 12y ago
  I wish that account was still updating.
  - reitanqild 12y ago
    For anyone who hasn't read it yet the last tweet on that account is gold.
  - misnome 12y ago
    I found an alternative in https://twitter.com/BigDataBorat https://twitter.com/BigDataBorat, not quite as insightful but still occasionally funny
beloch 12y ago
This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work they'd done over the weekend (This was a weekend near the end of the semester when everyone was in crunch mode). Long story short, the sysadmin was hired back and paid more than most of the profs. Academia may tend to skimp on salaries for certain positions, but sysadmins probably shouldn't be one of them.
- deleted 12y ago
  [deleted]
- gertef 12y ago
  The BOFH sabotaged his systems before departing? Didn't build a backup plan? And they hired him back?
  - SDGT 12y ago
    You know what? Fuck them. Fuck higher education completely. Undervalued and underpaid is the name of the game for any important IT roles in that shit hole of an industry. Disclaimer: dev at a university.
    - angersock 12y ago
      Preach it. That said, if you get a cushy job, it can stay cushy for a long time.
    - beloch 12y ago
      To be fair, a lot of the people working in support roles in academia are pretty much unemployable in the real world. They show up at 10 am, take constant smoke breaks all day, and leave at 3 pm. When I was in physics (which used the main university servers for most things) we had a sysadmin who was in charge of some printers and a couple of server boxes. He had inherited those boxes from a former student who set them up, but he was functionally illiterate in managing them. At one point I needed a package installed. Not only could he not figure out how to install a package on a ubuntu server on his own, he couldn't do it with emailed instructions either. I had to go up and physically stand over him telling him what to click on and what to type. To make matters worse, he was so hard to actually catch "in the office", that I had to have the department secretary (whose office he was next to) alert me when he showed up. Not surprisingly, the functions of those servers were soon moved to desktop machines in various offices. As far as I know he's still working there though. He's a union employee and it would be a ride through deepest, hottest, hell to get rid of him. Note: I am not saying all university support staff are like this. Some definitely are though, and they're probably the reason why good people sometimes find it hard to be properly remunerated in academia.
deleted 12y ago
[deleted]
Fuzzwah 12y ago
I've just been hired to run a project using SCCM to upgrade ~5000 PCs from XP to Win7. This was amazing reading. Reading such a detailed wrap up of an IT team going through my worst possible nightmare was enlightening.
- sswaner 12y ago
  It must be reassuring that your new employer is so proactive in getting that XP upgrade project running.
  - Sanddancer 12y ago
    Sometimes these products get bogged down because of three important problems outside of the control of IT. First off, you need to get in touch with all the upstream vendors to get updates to any sort of custom software that has compatibility problems with newer versions of windows; Vista/7 got a lot more strict about giving admin access, for example, which may cause problems in the updates. Second, you've got to keep in mind the training costs. There are a lot of users who may be brilliant financial minds that can make numbers dance and bow to their whims, but get terribly locked up if an icon changes. Doesn't make them horrible people by any means, but you've got to keep it in consideration when planning a rollout. Finally, you have to keep in mind the petty turf wars. If Joe in Accounting gets the upgrade to 7 before Bob in Legal, Bob in Legal may feel slighted and start raising a holy shitstorm, even if he's scheduled to be upgraded a week later. Upgrades are ugly, no matter when they happen. Sometimes that proactive upgrade project takes many years just because of all the moving parts involved.
    - lugg 12y ago
      I think you missed the joke.
- asdfaoeu 12y ago
  Looks like it will be pretty easy these guys did it by accident.
rfolstad 12y ago
On the bright side they are no longer running XP!
jonmrodriguez 12y ago
Forgive my beginner question: Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?
- InclinedPlane 12y ago
  Deletion and formatting doesn't necessarily destroy data, it just destroys the pointers to the data. If they're lucky the data can be recovered via software utilities (undelete) with backfill from backups. If they're unlucky then important and un-backedup data has been written over, and those people are going to be SoL.
  - johnchristopher 12y ago
    Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.
    - fyrabanks 12y ago
      Yes, there are a lot of options, both commercial and non-commercial, for full drive backups (then you restore the using those and the incrementals). Do that for your provisioning servers and you can redeploy a lot of the infrastructure based on that.
      - johnchristopher 12y ago
        I see, Thank you. Somehow I was under the impression that automating and scaling the tedious work of analyzing and restoring boot sectors and the like couldn't be done. I suppose that it's easier to plan for though if you restore a whole drive from a backup image rather than restoring a random set of files by going on a sector hunt ?
    - deleted 12y ago
      [deleted]
sergiotapia 12y ago
Isn't this more the fault of the system architect than the guy who accidentally fired the bad deploys? It's similar to a database firehose: If you accidentally start deleting all data you should have a quick working backup ready to quickly bring the dead box up to production.
- fred_durst 12y ago
  I don't know. This could very well be a case of not much more than a bad drag and drop in SCCM. Its not quite that simple, but I'm not sure this is some custom process they setup.
  - not_kurt_godel 12y ago
    Any tool that allows you to easily perform the antithesis of its function without making it abundantly clear what clicking the OK button will do is fundamentally broken.
deleted 12y ago
[deleted]
Fuzzwah 12y ago
I was just watching the "What’s New with OS Deployment in Configuration Manager and the Microsoft Deployment Toolkit" session from TechED and hit the section on "check readiness" option which MS have added to SCCM 2012 in R2. It sounds like having this in part of the task sequence at Emory would have (at the very least) stopped this OS push from at least hosing all the servers. http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCIT-B340 http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCI...
stark3 12y ago
There was a similar catastrophe at Jewel osco stores many years ago. Nightly, items added to the store pos were merged back with the main item file at each store location. The format of the merged data was exactly the same as loading a new file, except the first statement would be /EDIT instead of /LOAD. One of the programmers decided to eliminate some code by combining the two functions, with a switch to control whether /LOAD or /EDIT was used for the first statement. There was a bug in the program, and the edits were sent down as loads. A guy I knew, Barry, was the main operator that night. He started getting calls from the stores after around 10 of them had been reloaded with 5 or 6 items. Barry said it was the first time he got to meet the president of the company that day.
- pierlux 12y ago
  Failure should never be the only time you get to meet upper management :(
  - ceejayoz 12y ago
    A night IT operator for an organization with 176 locations is pretty unlikely to ever meet the company president.
stark3 12y ago
There was a similar catastrophe at Jewel osco stores many years ago. Nightly, items added to the store pos were merged back with the main item file at each store location. The format of the merged data was exactly the same as loading a new file, except the first statement would be /EDIT instead of /LOAD. One of the programmers decided to eliminate some code by combining the two functions, with a switch to control whether /LOAD or /EDIT was used for the first statement. There was a bug in the program, and the edits were sent down as loads. A guy I knew, Barry, was the main operator that night. He started getting calls from the stores after around 10 of them had been reloaded with 5 or 6 items. Barry said it was the first time he got to meet the president of the company that day. This was back when phone lines were used to transmit data, and it took several hours just to transmit an item file. So, the store locations were out of business until the item file could be restored.
pling 12y ago
Not quite as disastrous but when I was at university the resident administrators configured the entire site's tftp server (everything was netbooted Suns) to boot from the network. This was fine until there was a site-wide power blip and it was shut down. When it came back it couldn't tftp to itself to boot because it wasn't booted yet (feel the paradox!). Cue 300 angry workstation users descend on the computer centre with pitchforks and torches because their workstations couldn't boot either... Bad stuff doesn't just happen to Windows networks.
grumblepeet 12y ago
I _very_ nearly did this whilst working for a University back in the early noughties. Luckily I managed to get to the server before the "advert" activated and wiped out everything. It was so easy to do I am surprised that it is stil possible. I feel for their pain, but it does sound like they are doing a good job of mopping up. I did allow myself a snort of laughter when I read the bit about the server being re imaged as well. That is pretty darn impressive carpet bombing the entire campus.
leccine 12y ago
We accidentally re-imaged all of the Windows servers with Linux the other day. Nobody noticed though...
- jacalata 12y ago
  That probably means nobody is using the servers you maintain and your department should be eliminated completely.
  - leccine 12y ago
    Or somebody was just joking... Who knows...
    - jacalata 12y ago
      maybe I was joking too!
      - negativity 12y ago
        Oh, you!
      - zyx321 12y ago
        I've found that jokes tend to get downvoted on HN. Your post seems to be still in the black.
        leccine 12y ago
        No sense of humor is bad for you. But on HN it is normal.
    - marcosdumay 12y ago
      Looks like two people can joke at the same time.
zacharycohn 12y ago
I thought this "accident" may have been on purpose... until they mentioned the servers. In my days of university tech support.
Fomite 12y ago
Reminds me of some emails that went out at my old university during a cluster outage, and got progressively more informal as the night went on, detailing people leaving dinners with extended families, a growing sense of desperation, etc. The last email might as well have ended with "Tell my wife I love her." It was both direct and funny enough that I was only mildly annoyed that the cluster was down.
tbyehl 12y ago
I've built a few systems for deploying Windows... and the last thing that every one of them did before writing a new partition table and laying down an image was to check for existing partitions and require manual intervention if any were found.
mantrax5 12y ago
You know how in movies you need at least two people to bring their special secret keys, plug them in, and turn them at once to enable a self-destruct sequence? That is a real principle in interface design - if something would be really, really bad to activate unintentionally, make it really, really hard to activate. If you design a nuclear missile facility, you don't put the "launch nukes" button right next to "check email" and "open facebook". Same way it shouldn't be easy for users to delete or corrupt their data by accident due to some omnipotent action innocently shoved right in between other trivial actions. I wouldn't blame the person who triggered this re-imaging process. I'd blame those who designed the re-imaging interface, to allow it to happen so easily by accident.
- kalleboo 12y ago
  In my experience, the key is that the UI makes it clear exactly what you're doing. What I mean is, instead of a button that says "Start Imaging", it should be "Start Imaging of the 12,600 computers this rule applies to". Of course, that's a lot more work for the programmer, so it's never done.
  - lstamour 12y ago
    But then that leads to people automatically clicking "Yes" to the "Are you sure..." dialog. Though even I would pause at "Are you sure you want to reformat 12,500 machines including this one?" ;-)
    - rectangletangle 12y ago
      Having people type in some sort of auto-generated confirmation key might give them pause, seeing as the technique is seldom used.
      - kalleboo 12y ago
        Or in my original example, have them enter in the # of computers it's going to apply to.
  - 12y ago
randlet 12y ago
Reading that just made me feel sick to my stomach and my heart goes out to the poor gal/guy that pushed "Go" on that one. Shit happens, but a screw up that big can be devastating to ones psyche.
CamperBob2 12y ago
stark3, you seem to be hellbanned.
- taspeotis 12y ago
  What gives you that idea? https://news.ycombinator.com/item?id=7758203 https://news.ycombinator.com/item?id=7758203 is a dupe of https://news.ycombinator.com/item?id=7758193 https://news.ycombinator.com/item?id=7758193
  - CamperBob2 12y ago
    (Sorry, didn't see the dupe.)
svec 12y ago
With great power comes great responsibility.
8ig8 12y ago
Mistakes are made. In related news... Lawn care error kills most of Ohio college's grass http://www.wral.com/lawn-care-error-kills-most-of-ohio-college-s-grass/13650157/ http://www.wral.com/lawn-care-error-kills-most-of-ohio-colle...
- unclesaamm 12y ago
  Haha this is great. How did you find this? Just to clarify for the passersby, it isn't the University of Ohio, rather, Findlay University
  - 8ig8 12y ago
    Coincidently on the local news website. > Findlay University did not release the name of the company that made the mistake but is working with the business' insurance company to pay for it to be fixed. Findlay apparently doesn't value transparency as much as Emory. > The university says grass was killed on as many as 54 of the campus' 72 acres.
- dredmorbius 12y ago
  Makes you wonder what chemicals are going into lawns generally. One of the lessons from my college days (informally acquired, take with appropriate quantities of salt) was that walking barefoot was as much a risk for chemical exposure as puncture wounds.
- frogpelt 12y ago
  I can understand the Embry problem a lot more simply because the distance between the decision and implementation of the decision is not that far. How on earth do you "accidentally" load up enough weed killer to treat 54 acres of grass and never realize its the wrong stuff?
  - dfc 12y ago
    How do you click/not-click the button in SCCM...? How are these two events any different?
rfrey 12y ago
My nomination of the top bullet point of 2014: * As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.
mehrdada 12y ago
As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted. I guess that's how robot apocalypse is gonna look like.
ww520 12y ago
Disasters as well as mistakes are unavoidable, such is life. A hallmark of a competent organization is how they handle the situation and recover from disasters or mistakes. So far all the signs have indicated they are doing great in recovering. I just hope there won't be onerous processes and restriction afterward due to desire on "make sure it won't happen again" stance.
lucio 12y ago
reads like a short dystopian novel
keehun 12y ago
I asked my friend attending Emory right now, and he didn't even realize anything was going on. He says that the Emory IT department has a notorious distinction on campus as being regularly terrible, mostly with an unreliable internet connection. However, it looks like they handled this accident the best they could! Perhaps this accident would not have happened at a more reliable IT department.
- deleted 12y ago
  [deleted]
gojomo 12y ago
"... to the cloud!" http://www.youtube.com/watch?v=jR6xbulUmsg http://www.youtube.com/watch?v=jR6xbulUmsg "Yay, cloud!"
chromaton 12y ago
Reminds me of The Website Is Down, episode 4: https://www.youtube.com/watch?v=v0mwT3DkG4w https://www.youtube.com/watch?v=v0mwT3DkG4w
imgur 12y ago
> As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted. That made me laugh. Poor SCCM server :)
deckar01 12y ago
"As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted." Unicast fail.
durkie 12y ago
hah! delighted to see this here. my roommate works at the emory library and has had a fun slow week there of coming home early many days because no one could do work. they were apparently also given laptops as an interim solution, but those somehow also wiped themselves eventually (?). poor IT people...just as they're starting to get a handle on the actual sitation it starts blowing up on the internet.
nissehulth 12y ago
Next time I'm about to complain about a bad day at the office, I will read this story again.

337 points by slyall 12y ago

k_sze 12y ago

Funny how they mention iTunes as one of the "key components" that are restored first, whereas Visio, Project, Adobe application are relegated to a second round.

krexit 12y ago
It's possible they're delivering academic materials through iTunes, I suppose.
lstamour 12y ago
Not hard to imagine. The others likely have specialty licenses and so aren't as easily distributed to everyone. In addition, Adobe software itself wasn't working earlier ;-)
zachlipton 12y ago
Presumably iTunes is part of their base system image for all workstations, along with Office, Firefox, Adobe Reader, and the like. In other words, a basic set of software to handle standard officework tasks. iTunes is free and IT would probably rather distribute it everywhere than have people trying to install it themselves (or calling the helpdesk to get someone with administrator rights to do it). They then offer additional applications on an as-needed basis to individuals and departments with specific tasks. So the designers who do print publications and the faculty who teach digital art might get the Adobe suite, while people in Facilities who plan construction will get Project. This keeps licensing costs down and simplifies systems according to their uses.
- commandar 12y ago
  Generally you're going to keep your base images in SCCM limited to software that's only infrequently updated. Otherwise somebody has to update the entire image every time an update gets pushed out. Instead, you package them up and deploy the apps on top of the base image at install time. It takes a little longer to deploy, but it takes less admin time to manage since the actual installs are automated anyway.

facorreia 12y ago

> A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks. Wow. That is very unfortunate, to say the least...

spiantino 12y ago
> As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.
- malkia 12y ago
  I feel bad, but I laughed at this for few seconds...
  - __david__ 12y ago
    I wouldn't feel bad. I guffawed at most of this story. Not in a "haha, what a bunch of morons. Serves those jerks right!" kind of way, but more in a "oh dear, that's the worst thing that can possibly happen! Oh no it gets worse??". I've been through IT catastrophes (and caused a couple myself) and I could easily see this happening to me. Still, it's funny as anything.
- rasz_pl 12y ago
  We apologize again for the fault in the subtitles. Those responsible for sacking the people who have just been sacked have been sacked

sorennielsen 12y ago

This happened at one a former workplace too. Only the Solaris and Linux servers was untouched. It "mildly" amused the *nix operations guys to see all the "point and click" colleagues panic.

filmgirlcw 12y ago

I've never been prouder of my alma mater. /s

gt1337h 12y ago
You should be. Emory is the 2nd best school in Georgia! :)

miles 12y ago

Snark and sarcasm aside, I am impressed with the level of detail that the IT department is sharing; it is refreshing to see such a disaster being discussed so openly and honestly, while at the same time treating customers like adults.

song 12y ago
Yes, shit happens. It's how you solve it and make sure that it doesn't happen again that's important. With the way they're communicating, they seem to be on the top of their game
harrystone 12y ago
It is impressive. I can't imagine dealing with that kind of nightmare. I once worked as an admin on Solaris boxes at a big pharma company. There were ~77,000 users in their LDAP directory. I was very careful.
- cfreeman 12y ago
  That sounds terrifying.
keithpeter 12y ago
At one place I worked, in the days of XP, the 'index server'(?) had a problem and un-installed all the application programs. The basic OS was there but MS Office, MSIE, all the doodads just got removed as each machine logged in. This was a small college, so the IT guys just went round explaining and told us to log out and log in again. Applications re-installed. No data loss and so no shouting. Stuff happens. We did 'assignment action planning' that morning: mind maps, essay plans and research ideas. Results better than normal anyway
filmgirlcw 12y ago
100% agree. My initial response was to laugh, but dude, I so feel for the team that has to clean the mess up.
swasheck 12y ago
It's not a standard across the board, but I've found academic institutions to be more honest about their technological mistakes (outside of large-scale breaches) than the private sector.

smegel 12y ago

Automation can also mean automated disaster.

batmansbelt 12y ago
...Or automated the-opposite-of-disaster.
deleted 12y ago
[deleted]
Already__Taken 12y ago
Bigger tools have bigger sharper edges. It's important the message is stressed as the developers to operators worlds merge in the middle.
ameoba 12y ago
Why did the Roman Empire collapse? What is the Latin for office automation?

deleted 12y ago

[deleted]

perlgeek 12y ago

https://twitter.com/DEVOPS_BORAT/status/41587168870797312 https://twitter.com/DEVOPS_BORAT/status/41587168870797312 "To make error is human. To propagate error to all server in automatic way is #devops." Frankly, I'm surprised things like this don't happen more often. Kudos for the incident management. Also a big plus for having working backups, it seems.

toomuchtodo 12y ago
Devops Borat: This not mistake. This how we test recovery operations in production.
- ibisum 12y ago
  This one goes on the first page of my playbook as of today.
  - deleted 12y ago
    [deleted]
timClicks 12y ago
Propogation of a software mistake is what appears to have caused the Gmail outage of 2011 http://youtu.be/eNliOm9NtCM?t=28m49s http://youtu.be/eNliOm9NtCM?t=28m49s
mey 12y ago
I wish that account was still updating.
- reitanqild 12y ago
  For anyone who hasn't read it yet the last tweet on that account is gold.
- misnome 12y ago
  I found an alternative in https://twitter.com/BigDataBorat https://twitter.com/BigDataBorat, not quite as insightful but still occasionally funny

beloch 12y ago

This reminds me of my undergrad CPSC days. The CPSC department had their own *nix-based mainframe system that was separate from the rest of the University. The sysadmin was a pretty smart guy who was making less than a third of what he could get in industry. Eventually he got fed up and left. About a week or two later the servers had a whole cascade of failures that resulted in everyone losing every last bit of work they'd done over the weekend (This was a weekend near the end of the semester when everyone was in crunch mode). Long story short, the sysadmin was hired back and paid more than most of the profs. Academia may tend to skimp on salaries for certain positions, but sysadmins probably shouldn't be one of them.

deleted 12y ago
[deleted]
gertef 12y ago
The BOFH sabotaged his systems before departing? Didn't build a backup plan? And they hired him back?
- SDGT 12y ago
  You know what? Fuck them. Fuck higher education completely. Undervalued and underpaid is the name of the game for any important IT roles in that shit hole of an industry. Disclaimer: dev at a university.
  - angersock 12y ago
    Preach it. That said, if you get a cushy job, it can stay cushy for a long time.
  - beloch 12y ago
    To be fair, a lot of the people working in support roles in academia are pretty much unemployable in the real world. They show up at 10 am, take constant smoke breaks all day, and leave at 3 pm. When I was in physics (which used the main university servers for most things) we had a sysadmin who was in charge of some printers and a couple of server boxes. He had inherited those boxes from a former student who set them up, but he was functionally illiterate in managing them. At one point I needed a package installed. Not only could he not figure out how to install a package on a ubuntu server on his own, he couldn't do it with emailed instructions either. I had to go up and physically stand over him telling him what to click on and what to type. To make matters worse, he was so hard to actually catch "in the office", that I had to have the department secretary (whose office he was next to) alert me when he showed up. Not surprisingly, the functions of those servers were soon moved to desktop machines in various offices. As far as I know he's still working there though. He's a union employee and it would be a ride through deepest, hottest, hell to get rid of him. Note: I am not saying all university support staff are like this. Some definitely are though, and they're probably the reason why good people sometimes find it hard to be properly remunerated in academia.

deleted 12y ago

[deleted]

Fuzzwah 12y ago

I've just been hired to run a project using SCCM to upgrade ~5000 PCs from XP to Win7. This was amazing reading. Reading such a detailed wrap up of an IT team going through my worst possible nightmare was enlightening.

sswaner 12y ago
It must be reassuring that your new employer is so proactive in getting that XP upgrade project running.
- Sanddancer 12y ago
  Sometimes these products get bogged down because of three important problems outside of the control of IT. First off, you need to get in touch with all the upstream vendors to get updates to any sort of custom software that has compatibility problems with newer versions of windows; Vista/7 got a lot more strict about giving admin access, for example, which may cause problems in the updates. Second, you've got to keep in mind the training costs. There are a lot of users who may be brilliant financial minds that can make numbers dance and bow to their whims, but get terribly locked up if an icon changes. Doesn't make them horrible people by any means, but you've got to keep it in consideration when planning a rollout. Finally, you have to keep in mind the petty turf wars. If Joe in Accounting gets the upgrade to 7 before Bob in Legal, Bob in Legal may feel slighted and start raising a holy shitstorm, even if he's scheduled to be upgraded a week later. Upgrades are ugly, no matter when they happen. Sometimes that proactive upgrade project takes many years just because of all the moving parts involved.
  - lugg 12y ago
    I think you missed the joke.
asdfaoeu 12y ago
Looks like it will be pretty easy these guys did it by accident.

rfolstad 12y ago

On the bright side they are no longer running XP!

jonmrodriguez 12y ago

Forgive my beginner question: Since a reformat was done to the affected machines, does this mean that researchers' datasets, drafts of papers, and other IP were lost? Or were researchers' machines not affected?

InclinedPlane 12y ago
Deletion and formatting doesn't necessarily destroy data, it just destroys the pointers to the data. If they're lucky the data can be recovered via software utilities (undelete) with backfill from backups. If they're unlucky then important and un-backedup data has been written over, and those people are going to be SoL.
- johnchristopher 12y ago
  Are there actual backup procedures out there that foresee and automate the restoration of wiped drives and partitions ? I might be wrong bu I doubt it should even be considered though.
  - fyrabanks 12y ago
    Yes, there are a lot of options, both commercial and non-commercial, for full drive backups (then you restore the using those and the incrementals). Do that for your provisioning servers and you can redeploy a lot of the infrastructure based on that.
    - johnchristopher 12y ago
      I see, Thank you. Somehow I was under the impression that automating and scaling the tedious work of analyzing and restoring boot sectors and the like couldn't be done. I suppose that it's easier to plan for though if you restore a whole drive from a backup image rather than restoring a random set of files by going on a sector hunt ?
  - deleted 12y ago
    [deleted]

sergiotapia 12y ago

Isn't this more the fault of the system architect than the guy who accidentally fired the bad deploys? It's similar to a database firehose: If you accidentally start deleting all data you should have a quick working backup ready to quickly bring the dead box up to production.

fred_durst 12y ago
I don't know. This could very well be a case of not much more than a bad drag and drop in SCCM. Its not quite that simple, but I'm not sure this is some custom process they setup.
- not_kurt_godel 12y ago
  Any tool that allows you to easily perform the antithesis of its function without making it abundantly clear what clicking the OK button will do is fundamentally broken.

deleted 12y ago

[deleted]

Fuzzwah 12y ago

I was just watching the "What’s New with OS Deployment in Configuration Manager and the Microsoft Deployment Toolkit" session from TechED and hit the section on "check readiness" option which MS have added to SCCM 2012 in R2. It sounds like having this in part of the task sequence at Emory would have (at the very least) stopped this OS push from at least hosing all the servers. http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCIT-B340 http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/PCI...

stark3 12y ago

There was a similar catastrophe at Jewel osco stores many years ago. Nightly, items added to the store pos were merged back with the main item file at each store location. The format of the merged data was exactly the same as loading a new file, except the first statement would be /EDIT instead of /LOAD. One of the programmers decided to eliminate some code by combining the two functions, with a switch to control whether /LOAD or /EDIT was used for the first statement. There was a bug in the program, and the edits were sent down as loads. A guy I knew, Barry, was the main operator that night. He started getting calls from the stores after around 10 of them had been reloaded with 5 or 6 items. Barry said it was the first time he got to meet the president of the company that day.

pierlux 12y ago
Failure should never be the only time you get to meet upper management :(
- ceejayoz 12y ago
  A night IT operator for an organization with 176 locations is pretty unlikely to ever meet the company president.

stark3 12y ago

pling 12y ago

Not quite as disastrous but when I was at university the resident administrators configured the entire site's tftp server (everything was netbooted Suns) to boot from the network. This was fine until there was a site-wide power blip and it was shut down. When it came back it couldn't tftp to itself to boot because it wasn't booted yet (feel the paradox!). Cue 300 angry workstation users descend on the computer centre with pitchforks and torches because their workstations couldn't boot either... Bad stuff doesn't just happen to Windows networks.

grumblepeet 12y ago

I _very_ nearly did this whilst working for a University back in the early noughties. Luckily I managed to get to the server before the "advert" activated and wiped out everything. It was so easy to do I am surprised that it is stil possible. I feel for their pain, but it does sound like they are doing a good job of mopping up. I did allow myself a snort of laughter when I read the bit about the server being re imaged as well. That is pretty darn impressive carpet bombing the entire campus.

leccine 12y ago

We accidentally re-imaged all of the Windows servers with Linux the other day. Nobody noticed though...

jacalata 12y ago
That probably means nobody is using the servers you maintain and your department should be eliminated completely.
- leccine 12y ago
  Or somebody was just joking... Who knows...
  - jacalata 12y ago
    maybe I was joking too!
    - negativity 12y ago
      Oh, you!
    - zyx321 12y ago
      I've found that jokes tend to get downvoted on HN. Your post seems to be still in the black.
      - leccine 12y ago
        No sense of humor is bad for you. But on HN it is normal.
  - marcosdumay 12y ago
    Looks like two people can joke at the same time.

zacharycohn 12y ago

I thought this "accident" may have been on purpose... until they mentioned the servers. In my days of university tech support.

Fomite 12y ago

Reminds me of some emails that went out at my old university during a cluster outage, and got progressively more informal as the night went on, detailing people leaving dinners with extended families, a growing sense of desperation, etc. The last email might as well have ended with "Tell my wife I love her." It was both direct and funny enough that I was only mildly annoyed that the cluster was down.

tbyehl 12y ago

I've built a few systems for deploying Windows... and the last thing that every one of them did before writing a new partition table and laying down an image was to check for existing partitions and require manual intervention if any were found.

mantrax5 12y ago

You know how in movies you need at least two people to bring their special secret keys, plug them in, and turn them at once to enable a self-destruct sequence? That is a real principle in interface design - if something would be really, really bad to activate unintentionally, make it really, really hard to activate. If you design a nuclear missile facility, you don't put the "launch nukes" button right next to "check email" and "open facebook". Same way it shouldn't be easy for users to delete or corrupt their data by accident due to some omnipotent action innocently shoved right in between other trivial actions. I wouldn't blame the person who triggered this re-imaging process. I'd blame those who designed the re-imaging interface, to allow it to happen so easily by accident.

kalleboo 12y ago
In my experience, the key is that the UI makes it clear exactly what you're doing. What I mean is, instead of a button that says "Start Imaging", it should be "Start Imaging of the 12,600 computers this rule applies to". Of course, that's a lot more work for the programmer, so it's never done.
- lstamour 12y ago
  But then that leads to people automatically clicking "Yes" to the "Are you sure..." dialog. Though even I would pause at "Are you sure you want to reformat 12,500 machines including this one?" ;-)
  - rectangletangle 12y ago
    Having people type in some sort of auto-generated confirmation key might give them pause, seeing as the technique is seldom used.
    - kalleboo 12y ago
      Or in my original example, have them enter in the # of computers it's going to apply to.
- 12y ago

randlet 12y ago

Reading that just made me feel sick to my stomach and my heart goes out to the poor gal/guy that pushed "Go" on that one. Shit happens, but a screw up that big can be devastating to ones psyche.

CamperBob2 12y ago

stark3, you seem to be hellbanned.

taspeotis 12y ago
What gives you that idea? https://news.ycombinator.com/item?id=7758203 https://news.ycombinator.com/item?id=7758203 is a dupe of https://news.ycombinator.com/item?id=7758193 https://news.ycombinator.com/item?id=7758193
- CamperBob2 12y ago
  (Sorry, didn't see the dupe.)

svec 12y ago

With great power comes great responsibility.

8ig8 12y ago

Mistakes are made. In related news... Lawn care error kills most of Ohio college's grass http://www.wral.com/lawn-care-error-kills-most-of-ohio-college-s-grass/13650157/ http://www.wral.com/lawn-care-error-kills-most-of-ohio-colle...

unclesaamm 12y ago
Haha this is great. How did you find this? Just to clarify for the passersby, it isn't the University of Ohio, rather, Findlay University
- 8ig8 12y ago
  Coincidently on the local news website. > Findlay University did not release the name of the company that made the mistake but is working with the business' insurance company to pay for it to be fixed. Findlay apparently doesn't value transparency as much as Emory. > The university says grass was killed on as many as 54 of the campus' 72 acres.
dredmorbius 12y ago
Makes you wonder what chemicals are going into lawns generally. One of the lessons from my college days (informally acquired, take with appropriate quantities of salt) was that walking barefoot was as much a risk for chemical exposure as puncture wounds.
frogpelt 12y ago
I can understand the Embry problem a lot more simply because the distance between the decision and implementation of the decision is not that far. How on earth do you "accidentally" load up enough weed killer to treat 54 acres of grass and never realize its the wrong stuff?
- dfc 12y ago
  How do you click/not-click the button in SCCM...? How are these two events any different?

rfrey 12y ago

My nomination of the top bullet point of 2014: * As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

mehrdada 12y ago

As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted. I guess that's how robot apocalypse is gonna look like.

ww520 12y ago

Disasters as well as mistakes are unavoidable, such is life. A hallmark of a competent organization is how they handle the situation and recover from disasters or mistakes. So far all the signs have indicated they are doing great in recovering. I just hope there won't be onerous processes and restriction afterward due to desire on "make sure it won't happen again" stance.

lucio 12y ago

reads like a short dystopian novel

keehun 12y ago

I asked my friend attending Emory right now, and he didn't even realize anything was going on. He says that the Emory IT department has a notorious distinction on campus as being regularly terrible, mostly with an unreliable internet connection. However, it looks like they handled this accident the best they could! Perhaps this accident would not have happened at a more reliable IT department.

deleted 12y ago
[deleted]

gojomo 12y ago

"... to the cloud!" http://www.youtube.com/watch?v=jR6xbulUmsg http://www.youtube.com/watch?v=jR6xbulUmsg "Yay, cloud!"

chromaton 12y ago

Reminds me of The Website Is Down, episode 4: https://www.youtube.com/watch?v=v0mwT3DkG4w https://www.youtube.com/watch?v=v0mwT3DkG4w

imgur 12y ago

> As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted. That made me laugh. Poor SCCM server :)

deckar01 12y ago

"As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted." Unicast fail.

durkie 12y ago

hah! delighted to see this here. my roommate works at the emory library and has had a fun slow week there of coming home early many days because no one could do work. they were apparently also given laptops as an interim solution, but those somehow also wiped themselves eventually (?). poor IT people...just as they're starting to get a handle on the actual sitation it starts blowing up on the internet.

nissehulth 12y ago

Next time I'm about to complain about a bad day at the office, I will read this story again.