4 ms·

I've dealt with e-discovery sets. No one really has answers to what to do when you have a litigation hold on data. Legislation commonly requires "retention of a

I've dealt with e-discovery sets. No one really has answers to what to do when you have a litigation hold on data. Legislation commonly requires "retention of anything related to X case", but how do you know what's relevant and what isn't? When you are a third party the ambiguity increases. So you end up with an everything and kitchen sink data dump. Even with everything the data is commonly useless without context. You have files without access logs and logs referrencing local namespaces etc... With a 25 petabyte discovery, I'm not surprised that everyone's scratching their heads on what to do next. This isn't just an MPAA/Megaupload problem. Even a smaller dataset like a 10-20TB discovery has numerous problems. Hosting/indexing/classifying/reviewing millions of documents is an open issue for the legal field. What do you do when there are multiple parties who all need to see "everything"? If everyone does their own thing how do you reference materials in a consistent manner across the interested parties? If you all agree to host the data in a neutral place who pays for it? What if the technology of that host benefits one party at the expense of another? For years the legal field has had a "print it all out and have a team of paralegals go over it" viewpoint. Clients don't pay for computers, but they do pay for paralegal hours. Only recently has that become untenable. Discovery sizes are growing exponentially per year. It's common to have a new discovery set come in larger that every previous set combined, and the legal industry doesn't really know what to do about it.

astrodust 15y ago
If it's anything like the usual proceedings, the lawyers involved would probably prefer if they had several copies of the data, too. I'm sure most of these drives were arrayed in such a fashion where they're unreadable unless in the proper equipment. It's not like you can just buy a pile of off-the-shelf external drives and start copying, either, as the contents might be unreadable unless the proper software is installed and configured correctly.
- phillmv 15y ago
  Not to mention that it would cost (1Tb * $109 @ newegg) * ( 25,000 Tb) $2,725,000 just to replicate it.
  - papercrane 15y ago
    I'm sure you could get a bulk discount though.
- CoryMathews 15y ago
  Wouldn't making more copies of the files be exactly why they are taking the site down in the first place?
deleted 15y ago
[deleted]
katane 15y ago
This isn't a problem, its the 'perfect moment' for a frighteningly ambitious, disruptive startup, per HN speak. I think its clear that this would be a problem in an industry where the most used tool is a bookshelf of various common interpretations of law. It just doesn't scale.
- Tyrannosaurs 15y ago
  No, it's a problem. The idea that law firms don't do IT is born of ignorance. Even the idea that the most used tool is a bookshelf of common interpretations of law is wrong, unsurprisingly all that stuff is online these days. Why on earth would anyone with two brain cells to rub together search for case histories on paper when there's Lexus Nexus which will do it in fractions of a second and is updated on a regular basis? Yes these people are pretty conservative they're not stupid and they have money (a lot of money). Large law firms will typically have IT departments of hundreds and there's a significant market for companies specialising in supporting the legal sector. But this is a tough problem because not only are there huge amounts of data involved but because people are actively trying to hinder your search. The law says that you have to to disclose everything relevant but it's pretty much common practice to also disclose a bunch of things that might be and a shed load of stuff you know full well isn't. This is a double win for the defending firm - not only can no-one accuse you of not providing everything they might want to see, but you get to do so in a way that makes very very difficult for them to find the stuff that matters. Think of it this way - the problem is basically the same one Google have (already a pretty tough problem) only instead of the people providing the pages being keen that their information be discovered, they're actively trying to hide it.

astrodust 15y ago

If it's anything like the usual proceedings, the lawyers involved would probably prefer if they had several copies of the data, too. I'm sure most of these drives were arrayed in such a fashion where they're unreadable unless in the proper equipment. It's not like you can just buy a pile of off-the-shelf external drives and start copying, either, as the contents might be unreadable unless the proper software is installed and configured correctly.

phillmv 15y ago
Not to mention that it would cost (1Tb * $109 @ newegg) * ( 25,000 Tb) $2,725,000 just to replicate it.
- papercrane 15y ago
  I'm sure you could get a bulk discount though.
CoryMathews 15y ago
Wouldn't making more copies of the files be exactly why they are taking the site down in the first place?

deleted 15y ago

[deleted]

katane 15y ago

This isn't a problem, its the 'perfect moment' for a frighteningly ambitious, disruptive startup, per HN speak. I think its clear that this would be a problem in an industry where the most used tool is a bookshelf of various common interpretations of law. It just doesn't scale.

Tyrannosaurs 15y ago
No, it's a problem. The idea that law firms don't do IT is born of ignorance. Even the idea that the most used tool is a bookshelf of common interpretations of law is wrong, unsurprisingly all that stuff is online these days. Why on earth would anyone with two brain cells to rub together search for case histories on paper when there's Lexus Nexus which will do it in fractions of a second and is updated on a regular basis? Yes these people are pretty conservative they're not stupid and they have money (a lot of money). Large law firms will typically have IT departments of hundreds and there's a significant market for companies specialising in supporting the legal sector. But this is a tough problem because not only are there huge amounts of data involved but because people are actively trying to hinder your search. The law says that you have to to disclose everything relevant but it's pretty much common practice to also disclose a bunch of things that might be and a shed load of stuff you know full well isn't. This is a double win for the defending firm - not only can no-one accuse you of not providing everything they might want to see, but you get to do so in a way that makes very very difficult for them to find the stuff that matters. Think of it this way - the problem is basically the same one Google have (already a pretty tough problem) only instead of the people providing the pages being keen that their information be discovered, they're actively trying to hide it.