12 ms·
Computer Crash Wipes Out Years of Air Force Investigation Records
- jtchang 10y agoReally? Like there are no backups? Is this amateur hour?
- stonogo 10y agoThis is a Lockheed snafu, so yes, it is amateur hour. Much of the data is backed up, and more of it will be retrievable from original sources. Maybe not all of it; I'll be very interested to see what gaps remain when they declare the issue resolved.
- jdavis703 10y agoThe cynic in me says the gaps will be for whoever was being investigated and triggered this corruption.
- 762236 10y agoLockheed is one of those companies that fires senior people to save money. Imagine how this plays out with turbine blades.
- tamana 10y agoWhy do companies do this, instead of "pay cut, take it or leave it?"
- stonogo 10y agoIt's never phrased that boldly. People aren't "fired", their contracts are not renewed, or they are declared redundant because of the new kid with an advanced degree... usually just after they've finished training the new kid. The most common move is to cut the highest paid section of the workforce after they acquire the service contract from a competitor. All major defense firms do this, which leads to problems maintaining institutional continuity.
- WDCDev 10y agoI would withhold judgement on Lockheed. It wouldn't surprise me if someone there raised the lack of a backup strategy as an problem with the govt. ... only to be ignored or told there wasn't enough budget for it.
- mnarayan01 10y agoEdit: Agreed.
- koolba 10y agoThat's a pretty stupid reason not to have backups. If anything, a lack of backups makes me feel like any safeguards for protecting PII would be sub par.
- niels_olson 10y agoThis is a very sound leap of logic you're making.
- _quick_q 10y agoFrom the article: >> It’s possible that some data is backed up at local bases where investigations originated.
- molecule 10y ago> He also criticized the Air Force for notifying Congress on Friday afternoon, five days after senior service leaders was told about the problem. Prime time for burying bad news.
- toyg 10y agoThe joys of IT. Back in the day, to wipe years of records you had to burn or shred tons of paper in multiple locations. Now you just crash one little system... Despite 20 years of Microsoft-induced crashes, clear best practices and yadda yadda, most organizations will still have a single backup system for any given data repository -- if they have one at all. And the answer when shit happens is "outsource! Go to the cloud!" -- so that local managers won't be responsible when the snafu happens and data is lost (if it happens to Google, it can and will happen to anyone). Progress, eh.
- Shivetya 10y agoIt never ceases to amaze me how many teams have the idea that DR is simply buy some new hardware. sadly its not just a government failing.
- pslam 10y ago> The joys of IT. Back in the day, to wipe years of records you had to burn or shred tons of paper in multiple locations. Now you just crash one little system. [...] most organizations will still have a single backup system for any given data repository -- if they have one at all. No, back in the day, you could still wipe years of records because most people only used a single filing cabinet with no duplication or off-site copies. "Destroyed in a fire" was the canonical way to destroy all records, easily. This story has nothing to do with the introduction of IT. Incompetence is still incompetence.
- droopyEyelids 10y agoFire? Why be so bombastic. All it takes is one mis-applied label to permanently destroy records in a large enough filing system.
- icebraining 10y agoBernard: Shall I file it? Hacker: Shall you file it? Shred it! Bernard: Shred it? Hacker: No one must ever be able to find it again! Bernard: In that case, Minister, I think it's best I file it. Yes, Minister - The Death List
- taneq 10y agoAh, the great flood of '76. Thousands of records were lost. Very convenient, that flood.
- emp_zealoth 10y agoGenuine question - can you post some more information?
- throwawayornot 10y agomaybe the other poster meant this one? Oct 1976 Frederick, MD: http://www.fredericknewspost.com/archive/flood-flashback/article_a8907abd-8ead-5abb-b180-84fc1ed61684.html http://www.fredericknewspost.com/archive/flood-flashback/art...
- taneq 10y agoIt was a quote from the TV show 'Yes, Minister': James Hacker: How am I going to explain the missing documents to "The Mail"? Sir Humphrey Appleby: Well, this is what we normally do in circumstances like these. James Hacker: [reads memo] This file contains the complete set of papers, except for a number of secret documents, a few others which are part of still active files, some correspondence lost in the floods of 1967... James Hacker: Was 1967 a particularly bad winter? Sir Humphrey Appleby: No, a marvellous winter. We lost no end of embarrassing files. James Hacker: [reads] Some records which went astray in the move to London and others when the War Office was incorporated in the Ministry of Defence, and the normal withdrawal of papers whose publication could give grounds for an action for libel or breach of confidence or cause embarrassment to friendly governments. James Hacker: That's pretty comprehensive. How many does that normally leave for them to look at? James Hacker: How many does it actually leave? About a hundred?... Fifty?... Ten?... Five?... Four?... Three?... Two?... One?... Zero? Sir Humphrey Appleby: Yes, Minister. (Source: http://www.imdb.com/character/ch0030014/quotes http://www.imdb.com/character/ch0030014/quotes) Edit: Although apparently a similar real-life incident occurred when Hurricane Sandy wiped out a significant portion of an FBI record archive: https://nsarchive.wordpress.com/2014/09/16/archival-neglect-flooding-of-fbi-archives-destroyed-hundreds-of-thousands-of-pages-of-files-related-to-civil-rights-movement-history/ https://nsarchive.wordpress.com/2014/09/16/archival-neglect-...
- dmd 10y agoThe records were not wiped out by a computer crash. They were wiped out by gross incompetence.
- deleted 10y ago[deleted]
- paulgerhardt 10y agoIf you're going to say something about paper, see also the 1973 Personnel Records Archive Fire: https://en.wikipedia.org/wiki/National_Personnel_Records_Center_fire https://en.wikipedia.org/wiki/National_Personnel_Records_Cen...
- deleted 10y ago[deleted]
- oval_atom 10y agoYeah this is believable. Where I worked, in the secured network there were maybe 15 workstations with a maximum of maybe 5 people working at any one time, unless there was a "TOUR" going on. We had 2 contractor and 1 government IT person supporting this network. I asked the one of the contractor ITs when was the last backup. Response: "Do you want me to do a backup of your workstation tonight?" WTF!!! I am no IT slob but the minimum is Mon-Thur backup of new and changed files that day, and Friday night was full network backup. Why wasn't this done?!!!! The tape backup makes too much noise. I guess the bitch couldn't read her endless Romance Novels because of the noise. I find an intern, not a full employee, working on a design file in the account of another engineer. WTF!!! We believe that since the intern was leaving soon, it would appear that the design was the work of the engineer. Security violation!!!!! The contractor IT people swore they did not know this was going on...... BS!!! With so few on the network at any one time, and in the mornings only the intern was on the network, these contractor IT people were unaware!!!! The government IT person knew, but she did not want to report it because she was well aware of the contractor's history of retaliation. BTW..... during the investigation, the government engineer asked where was it written that they could not share passwords!!!!! Can you hear Snowden laughing at this. All swept under the lumpy rug....... the violations would make the organization look bad. As for me..... I got the IR treatment.... that's Isolation and Retaliation... then after 2 and a half days, supervisor sent me an email as to my where abouts.... my reply: Retired as of 4:00 PM PST, 3 days previous.
- icebraining 10y agohttp://wiki.lspace.org/mediawiki/Multiple_exclamation_marks http://wiki.lspace.org/mediawiki/Multiple_exclamation_marks
- metaphor 10y agoDear 18F [1]: Help, please. [1] https://cloud.gov/ https://cloud.gov/
- niels_olson 10y ago18F is severely outgunned. They need your help. I can't say that enough. The entire Western US is on fire, there's a drought, they have one fire hose and their next boss may turn it off.
- lbhnact 10y agoNice to see an encouraging response for 18F here. They are fighting the good fight, and most people don't even know what they are up against.
- RcouF1uZ4gsC 10y agoThere has been too much recently of government agencies losing data due to "crashes" or "inadvertent" deletions. For example the IRS, state department, and now the Air Force. A really easy way to fix this is to make a law so that if government data such as this gets lost, then the cabinet level person responsible for overseeing the agency immediately loses their cabinet level position and is barred from further work with the federal government. This would do wonders to align incentives all through the federal agencies when it comes to data safeguards.
- sverige 10y agoYeah, forgive me if I'm too cynical, but Hanlon's razor fails when this sort of thing involves criminal investigations of government employees. Edit: it also inspired a variation on an old joke about how the same word means different things to different people. For example, suppose the order goes out to "secure the data center." Marines report back, "We have destroyed the data center." Army reports, "We have killed everyone in the data center and are holding the position." Navy: "We locked the doors when we left for the day." Air Force: "We signed a three-year contract with an outsourcing company, with an option to extend at the same price for ten more years." Which of course means that just because I don't think Hanlon's razor applies doesn't mean I think they're competent.
- deleted 10y ago[deleted]
- jMyles 10y ago> Yeah, forgive me if I'm too cynical, but Hanlon's razor fails when this sort of thing involves criminal investigations of government employees. I've had this thought before. Can you think of a general, defensible clause to add to Hanlon's razor to account for this?
- thaumasiotes 10y ago"There is no point in trying to distinguish incompetence from malice when it directly benefits the party committing the error." You need to treat it all as malice.
- PakG1 10y agoSigh. When I was in university, I worked at a computer store one summer that provided IT services to local offices. One of the clients was the Mexican consulate. We were hired to do various things, including installing a tape backup system. I was installing some service packs at about 5pm. Windows NT4. 5pm, everyone's gone home, safe, right? Server asks if I want to restart, I say yes. Then a guy walks into the server room, asks if I had restarted the server. I say yes. Whoops, OK, I guess someone was working. Next day, the database was dead. Seemingly because I had restarted the server while someone was using it. It was a custom database job, and the contractor who had made it was on vacation in some other country. Here's the kicker. Usually tape backups have rolling tapes, right? But we hadn't started rolling tapes yet. So the good backup from 2 days prior was overwritten the night before with the crashed database. Ugh. And I was due to go back to school in 2 weeks in another city. Ouch. I still feel bad about that to this day. No idea how my boss ended up. Never ever let someone who doesn't know what they're doing mess around with your mission critical systems where the consequences are not recoverable. Always guide them. Although in my case, it seems I may have known more than my boss at the time, so maybe wasn't possible for me....
- tamana 10y agoIf they were running a Windows server, they didn't care about their data.
- Falkon1313 10y ago>Data about current investigations has also been lost, which is delaying them. [...] >"We've opened an investigation to try to find out what’s going on, but right now, we just don’t know," Stefanek said. I wonder where they're putting the files for that investigation. I wonder if their backups were corrupted too, or if they just overwrite the old ones with newer ones as a cost-saving measure. Or whether no one ever tried restoring from them and for whatever reason they never could have been restored from. Since people rarely prioritize it until it's too late, maybe 'restore from backup' drills should be a common thing.
- crystaln 10y agoAny information on the contract with Lockheed or the consequences of catastrophic failure?
- MikeNomad 10y agoGood Grief. For a lot less than an F-35 program, they could install a TSM rig, with off sight backup. Ours (at a state university) deals with petabytes of data. We've had to escalate to IBM's Tier 3 support a couple of times over the past decade, but we have never lost a file.
- King-Aaron 10y agoI recall an episode of the X-Files where they talked about a 'basement fire', this reminds me of that. When you hear about a high-level government department "losing" files, it's usually because they've had an office party around the paper shredder. Or something along those lines.
- chipperyman573 10y ago>office party around the paper shredder This phrase made me laugh, but I don't really understand what it means, can you explain please?
- King-Aaron 10y ago"frantically destroying documents" I suppose would be another way to frame it... But I seem to remember that was the wording on the show, or pretty close to it. It made it sound like the staff were running around and being very energetic in their operation of the paper shredder.
- Bartweiss 10y agoIt's the 'big government agency' version of that movie scene where the hacker tosses all their drives in the microwave as the feds kick down the door.
- fit2rule 10y agoJust ask the Russians for a backup. And if that fails, the next-higher authority: the NSA.
- sklogic 10y agoAny sufficiently advanced incompetence is indistinguishable from a malice.
- ww520 10y agoMay the additional security requirement deter sensible backup procedure? Like you can't have copies of backup somewhere? Or backups have to be encrypted and the keys are in a USB drive somewhere, which had gone missing/bad and the encrypted data can't be recovered?
- aminorex 10y agoDo please keep in mind that on September 10th, 2001, the secretary of defense reported to Congress that the department could not account for 2.3 trillion dollars of spending.
- HackerPschorr 10y agoOh, come on. Nobody ever made any backups?
- jacquesm 10y agoAlways hard to tell the difference between gross incompetence and malice.
- FatAmerican 10y agoIt's becoming clear that all government data should be part of blockchain(s).
- cadou191 10y agoFor University Grades changing +Erase Criminal Records hack +Driver License +Facebook hack, Twitter hack, Instagram hack, Gmail, Yahoo hack +Already hacked Paypal, MasterCard, Credit Card untraceable balance +Email interception get a hold of consultanthackers@outlook.com , certified , professional and highly time conscious... call (302) 365-0294 and its done !