3 ms·

He crawled a bunch (780 Gigs / ~20 million pages) of 'public' court records that were behind some kind of auth (pay) wall via a free trial and opened them up. L

by ajju 17y ago

He crawled a bunch (780 Gigs / ~20 million pages) of 'public' court records that were behind some kind of auth (pay) wall via a free trial and opened them up. Likely as a result of this crawling, the free trial was shut down. The pages he crawled constitute 20% of the total records. The feds then started investigating him. http://thelede.blogs.nytimes.com/2009/02/13/steal-these-federal-records-okay-not-literally/ http://thelede.blogs.nytimes.com/2009/02/13/steal-these-fede...

rguzman 17y ago
Court records are, in general, public. Not 'public'. Just go to the library. However, PACER costs money and so do the favored online vendors (westlaw and lexis-nexis). Even if they aren't public. They should be. This stuff can be used as law -- precedents. The only valid objection I see is that there needs to be some process to protect the privacy of the people involved in the case. But that's second-order concern.
- ajju 17y ago
  They should be public but are behind a wall. Thus, the quotes.
- igurari 17y ago
  PACER records are public, even that they cost money to access. So are most of the records on Lexis and Westlaw. Westlaw has tried to claim a copyright in certain annotations on the court documents, but they have no copyright in the text of the court records themselves.[1] [1] Matthew Bender & Co. v. West Publishing Co., 158 F.3d 693 (2d Cir. 1998) - http://bulk.resource.org/courts.gov/c/F3/158/158.F3d.693.97-7430.html http://bulk.resource.org/courts.gov/c/F3/158/158.F3d.693.97-...
- bbgm 17y ago
  I don't know much about what PACER includes and what exactly is covered by Westlaw and Lexis-Nexis, but Altlaw (http://altlaw.org http://altlaw.org) is a "free, full-text, searchable database of U.S. case law", and I've heard it being mentioned in the context of Lexis-Nexis. Also has the extra cool aspect of having been done by Stuart Sierra, whom all the Clojure folks around here probably know well.
- dpifke 17y ago
  "The only valid objection I see is that there needs to be some process to protect the privacy of the people involved in the case." There already is: you can file documents under seal or in redacted form. Most courts require you to do so if you're including information such as social security numbers. See, i.e. http://www.courtinfo.ca.gov/invitationstocomment/documents/spr07-19.pdf http://www.courtinfo.ca.gov/invitationstocomment/documents/s... Nothing that's available via PACER couldn't be read by walking down to the courthouse and requesting the physical file.
jpwagner 17y ago
But when lawyers told Mr. Malamud and Mr. Swartz that they appeared to have broken no laws, Mr. Malamud sent Mr. Swartz a message saying, “You should just lay low for a while.” It's not illegal, so what's going on? Trying to collect $1.5MM?
- ajju 17y ago
  Either they think it is illegal or what is more likely is that anytime a sysadmin at a federal facility sees "atypical" behavior (as defined by them) they contact the feds. It is very unlikely that a sysadmin in the federal court system would think about whether the crawling session overloading his server is legal, forget moral and I am sure if he/she complains, the FBI is obligated to investigate. I am not saying this is what should happen but this is what likely happened.
  - eli 17y ago
    Well, yeah, and those fees they're missing out on indirectly fund that sysadmin's department.
- gojomo 17y ago
  Interpretation of 'unauthorized access' is murky and there's a lot -- too much -- discretion in how access legitimacy and damages are evaluated. (See also: the submission last week about an author's estimation most people commit 'three felonies a day'.) In this matter, I'd bet the NYTimes coverage was more valuable in deterring prosecution than any legal counsel or even influential friends -- it meant any trumped-up narrative of nefarious intent would be fighting an uphill battle against the story already started in 'the paper of record'.
falsestprophet 17y ago
PACER data is now being collected and made available to the public by the RECAP project run by the Center for Information Technology Policy at Princeton. https://www.recapthelaw.org/how-it-works/ https://www.recapthelaw.org/how-it-works/

by ajju 17y ago

rguzman 17y ago

Court records are, in general, public. Not 'public'. Just go to the library. However, PACER costs money and so do the favored online vendors (westlaw and lexis-nexis). Even if they aren't public. They should be. This stuff can be used as law -- precedents. The only valid objection I see is that there needs to be some process to protect the privacy of the people involved in the case. But that's second-order concern.

ajju 17y ago
They should be public but are behind a wall. Thus, the quotes.
igurari 17y ago
PACER records are public, even that they cost money to access. So are most of the records on Lexis and Westlaw. Westlaw has tried to claim a copyright in certain annotations on the court documents, but they have no copyright in the text of the court records themselves.[1] [1] Matthew Bender & Co. v. West Publishing Co., 158 F.3d 693 (2d Cir. 1998) - http://bulk.resource.org/courts.gov/c/F3/158/158.F3d.693.97-7430.html http://bulk.resource.org/courts.gov/c/F3/158/158.F3d.693.97-...
bbgm 17y ago
I don't know much about what PACER includes and what exactly is covered by Westlaw and Lexis-Nexis, but Altlaw (http://altlaw.org http://altlaw.org) is a "free, full-text, searchable database of U.S. case law", and I've heard it being mentioned in the context of Lexis-Nexis. Also has the extra cool aspect of having been done by Stuart Sierra, whom all the Clojure folks around here probably know well.
dpifke 17y ago
"The only valid objection I see is that there needs to be some process to protect the privacy of the people involved in the case." There already is: you can file documents under seal or in redacted form. Most courts require you to do so if you're including information such as social security numbers. See, i.e. http://www.courtinfo.ca.gov/invitationstocomment/documents/spr07-19.pdf http://www.courtinfo.ca.gov/invitationstocomment/documents/s... Nothing that's available via PACER couldn't be read by walking down to the courthouse and requesting the physical file.

jpwagner 17y ago

But when lawyers told Mr. Malamud and Mr. Swartz that they appeared to have broken no laws, Mr. Malamud sent Mr. Swartz a message saying, “You should just lay low for a while.” It's not illegal, so what's going on? Trying to collect $1.5MM?

ajju 17y ago
Either they think it is illegal or what is more likely is that anytime a sysadmin at a federal facility sees "atypical" behavior (as defined by them) they contact the feds. It is very unlikely that a sysadmin in the federal court system would think about whether the crawling session overloading his server is legal, forget moral and I am sure if he/she complains, the FBI is obligated to investigate. I am not saying this is what should happen but this is what likely happened.
- eli 17y ago
  Well, yeah, and those fees they're missing out on indirectly fund that sysadmin's department.
gojomo 17y ago
Interpretation of 'unauthorized access' is murky and there's a lot -- too much -- discretion in how access legitimacy and damages are evaluated. (See also: the submission last week about an author's estimation most people commit 'three felonies a day'.) In this matter, I'd bet the NYTimes coverage was more valuable in deterring prosecution than any legal counsel or even influential friends -- it meant any trumped-up narrative of nefarious intent would be fighting an uphill battle against the story already started in 'the paper of record'.

falsestprophet 17y ago

PACER data is now being collected and made available to the public by the RECAP project run by the Center for Information Technology Policy at Princeton. https://www.recapthelaw.org/how-it-works/ https://www.recapthelaw.org/how-it-works/