4 ms·

I was hoping the data would be in a text format so that I could build a tree of blame -- so basically you could see exactly how 'information' from someone who m

I was hoping the data would be in a text format so that I could build a tree of blame -- so basically you could see exactly how 'information' from someone who may not be giving accurate information impacted future prisoners. Sadly, it's all just embedded PDF files.

burgerbrain 16y ago
OCR it. It sounds like an interesting thing to look at.
- mrcharles 16y ago
  I thought about that, but given that I'd only need a tiny bit of info, and OCR being (potentially) prone to error, at that point it would probably be easier to just manually enter the information I'd need. But I'm lazy.
conesus 16y ago
It's all part of DocumentCloud, which means API access to the text of all that Guantanamo data. See how easy this can be: curl http://www.documentcloud.org/api/search.json?q=group:nytimes "total": 2821, "page": 1, "per_page": 10, "q": "group:nytimes", "documents": [{ "id": "86275-isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo", "title": "ISN 10015 - Abd al Rahim al Nashiri - JTF-GTMO Detainee Assessment", "access": "public", "pages": 15, "description": null, "source": null, "created_at": "Sun, 24 Apr 2011 15:50:19 +0000", "updated_at": "Mon, 25 Apr 2011 15:12:53 +0000", "canonical_url": "http://www.documentcloud.org/documents/86275-isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.html", "resources": { "pdf": "http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.pdf", "text": "http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.txt", "thumbnail": "http://s3.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p1-thumbnail.gif", "search": "http://www.documentcloud.org/documents/86275/search.json?q={query}", "page": { "text": "http://www.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p{page}.txt", "image": "http://s3.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p{page}-{size}.gif" } } }, ... ]} Look at documents[0].resources.text: curl http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.txt There you go. Now go build that tree and report back. :-)
- mrcharles 16y ago
  Oh, very cool. Thanks for the heads up-- If I get some free time I'll do this.
- jashkenas 16y ago
  I'm afraid that only a handful (< 20, so far) of the newly released documents have been redacted and published thus far. You won't get too far trying to build that tree.
  - mrcharles 16y ago
    Sure but I can start writing code now, in anticipation of the full set.
- mrcharles 16y ago
  So I was fiddling a bit. Sadly, despite being able to skip the OCR step, there's obviously a lot of OCR errors in the text as is, which would make reliably stripping out data somewhat more difficult. The reality is that I don't really have the time to commit to this unless everything worked perfect. Ah well, maybe someone else will do it.

burgerbrain 16y ago

OCR it. It sounds like an interesting thing to look at.

mrcharles 16y ago
I thought about that, but given that I'd only need a tiny bit of info, and OCR being (potentially) prone to error, at that point it would probably be easier to just manually enter the information I'd need. But I'm lazy.

conesus 16y ago

It's all part of DocumentCloud, which means API access to the text of all that Guantanamo data. See how easy this can be: curl http://www.documentcloud.org/api/search.json?q=group:nytimes "total": 2821, "page": 1, "per_page": 10, "q": "group:nytimes", "documents": [{ "id": "86275-isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo", "title": "ISN 10015 - Abd al Rahim al Nashiri - JTF-GTMO Detainee Assessment", "access": "public", "pages": 15, "description": null, "source": null, "created_at": "Sun, 24 Apr 2011 15:50:19 +0000", "updated_at": "Mon, 25 Apr 2011 15:12:53 +0000", "canonical_url": "http://www.documentcloud.org/documents/86275-isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.html", "resources": { "pdf": "http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.pdf", "text": "http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.txt", "thumbnail": "http://s3.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p1-thumbnail.gif", "search": "http://www.documentcloud.org/documents/86275/search.json?q={query}", "page": { "text": "http://www.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p{page}.txt", "image": "http://s3.documentcloud.org/documents/86275/pages/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo-p{page}-{size}.gif" } } }, ... ]} Look at documents[0].resources.text: curl http://s3.documentcloud.org/documents/86275/isn-10015-abd-al-rahim-al-nashiri-jtf-gtmo.txt There you go. Now go build that tree and report back. :-)

mrcharles 16y ago
Oh, very cool. Thanks for the heads up-- If I get some free time I'll do this.
jashkenas 16y ago
I'm afraid that only a handful (< 20, so far) of the newly released documents have been redacted and published thus far. You won't get too far trying to build that tree.
- mrcharles 16y ago
  Sure but I can start writing code now, in anticipation of the full set.
mrcharles 16y ago
So I was fiddling a bit. Sadly, despite being able to skip the OCR step, there's obviously a lot of OCR errors in the text as is, which would make reliably stripping out data somewhat more difficult. The reality is that I don't really have the time to commit to this unless everything worked perfect. Ah well, maybe someone else will do it.