8 ms·
Fired employee wipes out whole season of TV show
- Splines 16y agoMan, that sucks. Hopefully it's a lesson to everyone that you don't leave the keys to the kingdom to one guy, especially if they've just been fired.
- r00fus 16y agoNo backups? 300GB doesn't sound like much. I just processed and shuttled 2.3TB today (sneakernet backup).
- famousactress 16y agoI was just thinking the same thing.. 300gb, mostly of presumably high-res art and video.. and that's 2 years work? Sounds really small.
- robin_reala 16y agoTwo man-years maybe?
- dagw 16y ago2D Animation doesn't take up much space. Beyond a few painted high res background, most files are small and a lot of things are procedurally generated and take up almost no space at all. I worked on a 2D animated movie 10 years ago, and the data needed to render the final movie weighed in at less than 100 GB (and that was 2-3 years of work by 30-40 people). Also I understood from the story that they didn't loose everything, just enough to make it impossible to recreate the episodes without basically starting from scratch.
- MichaelApproved 16y agothe lost dada is said to include fragments of shows It wasn't all their data but enough to make what was left unusable.
- robryan 16y agoThis is surprising, it would be like sitting my web apps code/ database on one live server without a local copy, and possibly multiple backups elsewhere. How can a TV production company justify this? I mean it would cost just about nothing relative to production costs to put 300gig on S3.
- mbreese 16y agoYou're assuming that the production company can afford to have someone around who knows how to put data on S3. My guess is that they outsourced this to the company who had the fired employee. Still, they should still have some kind of original files on some other medium somewhere... but who knows, maybe their production workflow revolved around this data-host. In which case they are really borked.
- detst 16y agoWe have tons of decades-old video alive and well today. They may not have expertise with S3 but if they don't have the expertise to ensure robust backups, they're doing it wrong.
- robryan 16y agoYeah they may not but you have the weight up the costs of getting a contractor in to setup an almost fail safe system against losing probably millions of dollars of production. I would assume it's more about people assuming something like this would never happen as to not justify the outlay of protecting against it.
- alexqgb 16y agoIn other news, soon-to-be-fired producer learns the true meaning of 'backup'. Also, 'redundancy'.
- maushu 16y agoIsn't the data hosting company job to do backups and redundancy? Unless they clearly state that they don't do that.
- gm 16y agoI'd say the opposite: Unless they clearly state they _do_ backups, they are not (legally) responsible for doing so.
- isleyaardvark 16y agoThat's what data hosting is. If they were doing webhosting, sure, but the article repeatedly states they were doing "data hosting". Here's the company's web page detailing their data backup plan: http://www.cyberlynk.net/services_corporate/dsp_backup.cfm http://www.cyberlynk.net/services_corporate/dsp_backup.cfm
- gm 16y agoWhat I'm saying is that whatever is in the contract is what matters. Data hosting, web hosting, data storage, whatever terms are used matter little. EDIT: Case in point: We probably all agree that dropbox could be called "data storage", but they specifically say that backups and backup costs are to be done by the client (http://www.dropbox.com/dmca#terms http://www.dropbox.com/dmca#terms), not by them.
- alexqgb 16y agoRedundancy - by definition - cannot be supplied by a single company. Yes, your vendor may have their own redundant systems in place so that their internal problems don't become your problems. And that's nice. They should. But you always have a Plan B. And you always keep a local copy.
- mbreese 16y agoThe thing that really sucks for the TV show is that it wasn't even their own employee. It was an employee for their data hosting company. You know... the guys who you're supposed to outsource to to make sure that this stuff doesn't happen.
- d2 16y agoYou mean The Cloud has disgruntled employees?
- jrockway 16y agoYou know... the guys who you're supposed to outsource to to make sure that this stuff doesn't happen. Actually, outsourcing is designed to be cheap, not to lower risk. Trusting the lowest bidder to an important task usually results in disappointment.
- pero 16y agoActually, big corporations outsource to transfer risk and not because it's cheap. To compliance/legal departments, paying exorbitant fees to qualified vendors is saving money in their risk analyses.
- mbreese 16y agoExactly. If you don't have expertise in house, then it can be cheaper to hire that task out to an outside company rather than hire someone to manage it in house. This sure sucks for the TV show, but at least they have someone else to hold accountable. If this had happened and they handled their own IT, there wouldn't be anyone but themselves to hold accountable. And you can bet that means a lawsuit with damages.
- dhughes 16y agoCan you hire him to wipe Jersey Shore?
- dschobel 16y agoDon't watch the show then. Or do you object to other people watching content you don't personally approve of? Because there are many terms for that particular personality type, none of them flattering.
- michaelcampbell 16y agoLiteralNet is in full force today, I see.
- daemin 16y agoShouldn't the company have off-site backups of the data its hosting, like, say, on tape or something?
- jrockway 16y agoThat employee may be going to jail, but it's not his fault that this happened. If your fired employees have access to your servers, you fucked up the firing horribly. Similarly, if one data center going offline can take out your entire season of work, perhaps you need to learn what redundancy is.
- uptown 16y agoWhat you said about redundancy is true, but it's still his fault this happened.
- jemfinch 16y agoI would not be surprised if their "data hosting service" claimed redundancy as a feature.
- ericd 16y agoHow is this not his fault?
- jrockway 16y agoWhat I'm saying is that relying on crazy people to not go crazy is not a good data loss prevention strategy. (In the end, putting someone in a situation where they'll do something bad is worse than being the person that does something bad. A good example is the Colgan Air crash from a few years ago. Anybody knows that when your plane is stalling you're supposed to push the control column forward and apply full engine power. The plane does the pushing the control column forward part automatically! But for some reason, the trained airline captain did the exact opposite, stalling the plane and killing everyone on board. Why? Because he got paid so little that he couldn't afford a hotel room to sleep in before his work that day, and had to commute across the country and then work a full shift. This is the airline's fault for impairing someone to the extent that he killed 50+ people by doing the exact opposite of what any trained pilot would do. $300 for a hotel room and everyone on that flight would still be alive today. Similarly, when you fire someone that works with critical data, you need to make sure the guy's access is revoked. It's a precaution that needs to be taken, as this case shows. When necessary precautions are omitted, bad things happen. Planes crash, and important data goes away forever.)
- hartror 16y agoI assume this is them: http://www.cyberlynk.net/ http://www.cyberlynk.net/ with the late 90s "new" graphic against off-site data backup. I hope the show producers sue someone, either Cyberlynk if they claimed they did redundant backup or their IT guy/company for setting them up with such a bum deal.
- trafficlight 16y agoI just don't understand how there could have only been a single copy of those videos.
- Retric 16y agoWho says he only deleted a single file? It's not hard to delete stuff from online backup systems.
- Udo 16y agoNo, I think what trafficlight meant was: how can there be only a single copy of this material? What about the media they actually shot the episodes on? Surely there must be some kind of copy on the cutting room computers? It looks as if they're mainly out to collect damages from the company.
- Retric 16y agoThe data breach allegedly knocked out 6,480 WER1 electronic files, or 300 gigabytes of data, comprising two years of work from hundreds of contributors globally, including animation artwork and live action video production. Sounds like they hosted everything in the "cloud". Consider what happens when you edit a Google doc, there is no offline backup just whatever Google provides. So sure they are not at zero, but plenty of stuff can and was lost.
- Udo 16y agoThat's what they claim, sure. But does it really work like that? It seems to me that there would have to be local copies of those videos, if for no other reason than synching production quality videos to the cloud in realtime is probably unrealistic. Maybe some HNer working in the TV industry can elaborate?
- trafficlight 16y agoFrom what little video editing I've been involved in, we had source material in 2 or 3 different places, plus multiple edits of said material, plus rough edited timelines, plus the final product. And keeping everything in the cloud would be insane. In our shop every video editing workstation had it's own RAID array just for realtime editing. In a large shop you'd have a SAN of some kind.
- dzorz 16y agoApril 1st?
- ctdonath 16y agoMust be. I thought the link would be about Charlie Sheen.
- sabalaba 16y ago(Warning anecdotal evidence) Having worked in China on a number of startups and worked alongside people who have had to fire employees, it's often common practice to just lock fired employees out of their systems. I've had this happen to people I know and had to do it once when letting a programmer go (which is nearly impossible to do after a probationary period due to the labor laws in the country, it's very very difficult to fire somebody, the company has to provide reason, pay social benefits, or face the employee in a court--forget it if you're a foreign company). I have heard stories of letting people go to find out that all of the work that employee did for the last month or two deleted, source missing, machines just wiped clean. I read that title and kind of laughed because, well, it is a seemingly commonplace response in some parts of the world.
- yardie 16y agoIn my case I gave a months notice I was resigning and was locked out immediately. Imagine how much it sucks to come into work everyday with nothing to do except go to lunch. When I wasn't offering the other developers advice on the code I had worked on or APIs I was familiar with I was just shooting the breeze. This was the hole in the policy. It was designed to keep disgruntled employees from damaging the system but the ones leaving on good terms were treated exactly the same.
- michaelcampbell 16y agoThis is more/less common in certain industries, and perhaps partially based on regulatory issues too, although I'm guessing there. I have worked on financial software the majority of my career; at one point in a Wall St. investment banking firm. The very explicit protocol there was if you were going to leave the 2 week notice was customary, but you would probably be locked out of everything instantly, and would be around for hands-off consulting at the whim of your manager for the duration. It was all handled very professionally and friendly, at least for me, and they let me keep working for the 2 weeks since anything I touched had a LOT of layers to go through before it ever saw production data.
- 16y ago
- monological 16y agoIs the guys name really Micheal Scott? April fools!
- tlrobinson 16y agoFor once I agree with the editorializing of a headline on Hacker News.
- Gorm-Casper 16y agoAm I the only one who found the idea with tiny bubbles at the scrollbar more interesting than the story itself? Front-end developers' curse :P
- nvictor 16y agoapril fools?
- nopassrecover 16y agoSurely the data company has redundancy or backups or data recovery software available? Surely they considered the risk of a staff member deleting copies / a hacker / a fire. Surely the producers have at least working copies somewhere (I can't see people live-editing on a cloud server). Surely they considered the risk of the company going bust / data loss / their own staff deleting the copies. Would assume this was an April Fools thing it's that bad (except for the fact it quite obviously isn't due to the quite obvious reputation damage that will effectively destroy this company).
- deleted 16y ago[deleted]
- scythe 16y agoKind of ironic that this would happen on World Backup Day, perhaps? http://worldbackupday.com/ http://worldbackupday.com/
- pinwale 16y agoThe real site is http://worldbackupday.net http://worldbackupday.net
- shadowpwner 16y agoInteresting how this happens on the same day of World Backup day. (http://blog.reddit.com/2011/03/better-have-backup-plan-world-backup.html http://blog.reddit.com/2011/03/better-have-backup-plan-world...)
- ThomPete 16y agoAprils fool?
- motters 16y agoThis is one thing which can happen when you put all your data into "the cloud" and hope for the best. One risk which any business has to take into account is the possibility of a rogue employee or ex-employee running amok. Putting all your eggs (or data) into one basket, especially when that basket is administrated by people who don't particularly care either way what happens to your business, may not be the wisest strategy.
- pbhjpbhj 16y ago>"According to a lawsuit that was filed last week in Hawaii District Court, a man named Michael Scott Jewson was terminated from CyberLynk. From his parents' residence, he allegedly accessed CyberLynk's data and intentionally wiped it out. Jewson is alleged to have been charged in February with a federal computer crime violation and admitted his guilt in a plea agreement. The data breach allegedly knocked out 6,480 WER1 electronic files, or 300 gigabytes of data, comprising two years of work from hundreds of contributors globally, including animation artwork and live action video production." These sort of reports often over do it with the use of "allegedly" as an attempt to mitigate any libel claim. "Jewson is alleged to have been charged" Surely it's a matter of record whether he was charged. If the reporter can't even confirm this then they don't have a story. "The data breach allegedly knocked out 6,480 WER1 electronic files" Again they don't need that "allegedly", it's presumably a reported fact, the allegation they have to be wary off is the one that says the person responsible. This really annoys me, allegedly.
- ubuntuftw 16y agoFormer TV news person here. While I agree with you on your first point (that the reporter shouldn't have to use the word "allegedly" if he/she can confirm the man was charged), personally, I think using allegedly in the sentence about the files being knocked out is warranted. "The data breach" is referring to the data breach the man *allegedly carried out. TV news reporters != attorneys. So when in doubt, attribution is your best friend.
- pbhjpbhj 16y agoThe data breach he allegedly carried out is reported as a known quantity. The only allegation that needs mitigating is that it was this person that carried it out. If one is unsure of the details then it would be "reportedly" or an "unconfirmed source stated" or similar, no? Thanks for your viewpoint though.
- ubuntuftw 16y agoThe problem is that the reporter doesn't know that the details of the breach mentioned in this article are facts. The figures, that there were 6,480 files totaling 300 GB of data likely came from the prosecution. So the reporter needs to get himself off the hook and either attribute it ("the prosecution says") or use words like "allegedly," "reportedly," etc.