5 ms·

> The government initially released the document in a PDF format which renders it like an image, impossible to search. When PDF files are uploaded to the Archiv

> The government initially released the document in a PDF format which renders it like an image, impossible to search. When PDF files are uploaded to the Archive, we automatically run them through an Optical Character Recognition (OCR) process. This turns those images into text, making it much easier to move between sections and search for specific words or phrases. This allows journalists and the public to more easily parse through volumes of information contained within these massive documents. Shouldn't an accessible, searchable and digital version of government documents be the default form in which they are released? Why do citizens have to resort to the Internet Archive to see such a basic right fulfilled? In my country this happens frequently, as well. Too often, documents that should be accessible and widely publicized are hidden somewhere on the internet with practically no incoming links and in a form which is the digital equivalent of a grainy fax. Sometimes the intention of superficially fulfilling a transparency obligation, without actually communicating anything is palpable.

sohkamyung 7y ago
One possibly is because they want to make sure the redacted sections are really redacted. You can do a web search and discover that others have tried to redact sections of PDF documents, only to later discover that people can uncover the redacted sections via software tools. Scanning a redacted document is the simplest way of making sure redacted sections stay redacted as there is no underlying information to reveal.
- rossenberg79 7y ago
  You nailed it. Aside from that anyone who is serious about search is going to run it through an OCR and get the raw text anyway. Why bother doing more work?
  - lostlogin 7y ago
    Offering a plain text file with the odd “[redacted text]” alongside a formatted PDF is hardly onerous. That said, there are numerous instances of governments releasing a .doc or similar and someone opening it and doing a few presses of ‘undo’.
    - rossenberg79 7y ago
      There’s just no reason to. Someone will fuck it up, and it will not have been worth the effort anyway.
- ALittleLight 7y ago
  Releasing the plain text is all that's needed. I fail to see how redacting by overwriting the relevant text is at all reversible - e.g. replace every redacted word with five X's. Replace multiple redactions with ten X's. That actually seems more secure than what's currently done, where you can guess at the words by their length.
  - WrtCdEvrydy 7y ago
    Trust me, people have failed... and we've fucked it up in the past. The easiest way is to use a PDF redactor, print it and rescan it.
  - wahern 7y ago
    > Releasing the plain text is all that's needed. I doubt Mueller's team wrote the report in plain text. They probably used Microsoft Word and exported a PDF document. When I'm president, all government agencies will be forced to use LaTeX. Then we can have both plaintext source, PDFs with actual text, and PDFs that don't leak redacted material.
    - caseysoftware 7y ago
      LaTeX? That will effectively prevent most government agencies from writing reports.. oh, that was your goal, wasn't it? Well played.
    - i_grow_beans 7y ago
      And I suppose they'll be published in GovtHub, so that anyone can fork them and add their annotations. I'd vote for you, but I'm not a citizen.
    - antt 7y ago
      Context is much better for documents: http://www.berenddeboer.net/tex/LaTeX2ConTeXt.pdf http://www.berenddeboer.net/tex/LaTeX2ConTeXt.pdf
maxlybbert 7y ago
The government is legally required to create an accessible version of the report for disabled people, and the Justice Department already promised to do so. While I can understand not wanting to delay the report’s release because that version wasn’t available, (1) I don’t know what the law says about eventually fulfilling the requirement, and (2) the Justice Department actually has the original documents and doesn’t need OCR to create the accessible PDF, so I’m surprised they haven’t done so yet.
gizmo686 7y ago
Much of the policy around classified materials is designed to be idiot proof. I imagine the process for releasing the document was something along the lines of: go into secure room; print out redacted document; review redacted document; take redacted document to non-secure scanner; release scanned version. Sending digital files across a security boundary is just a much more dangerous operation in general than sending paper. Given the prevelence and quality of OCR (particuarly when run on a good quality print, as was in this case), there is just not enough of a reason to take the risks of sending files across the boundary. EDIT: Idiot proof is probably not a fair characterization either. The other part of security policies is to be genius proof. What if someone involved is trying to exfiltrate data. There is a lot more places for them to hide it if you give them a digital pathway out then if you make them take it out on pieces of paper that will be reviewed.
mbrumlow 7y ago
> Shouldn't an accessible, searchable and digital version of government documents be the default form in which they are released? The way it was done was the most sane way from a security standpoint. In my opinion it is the first time I have thought that somebody who knows what they are doing is actually on the case. I say this because it is just too easy for you to leak out the redacted parts. People have tried, and failed. We don't know if what is under those black blots are really important or not, but if they were, and the leaking of such information could cause somebody harm -- then this is the way to do it.
blihp 7y ago
I suspect the reason they released a scanned version of the redacted document is probably a policy decision to avoid serious screw-ups.[1] Also, organizations of all sizes have known how to photocopy/scan documents for decades so it's a path of least resistance thing. The other thing is that those in charge of releasing this report had no interest in doing anything, including a simple OCR pass[2], on that scanned document that would help disseminate the information in this report faster... it's against their interests to do so.[3] So you're right, a more useful document should have been released. But for the previously mentioned reasons, it wasn't. As long as no critical mass of citizens demands digital competence from their government, their government will not offer it. [1] There was a story in the last month or two about how someone screwed up redaction in an electronic document in a court case where they effectively only painted black bars over the text but the underlying text was in the release document. [2] It wouldn't surprise me one bit if lawyers viewed an image-only PDF document dump as a digital form of the 'paper blizzard'. It really isn't (not even close) as there was already an OCR'd version of the document available within the hour of its release and some NLP analysis within the day. [3] In the event that objections are raised, they can always throw up the 'we were moving as fast as we could to get it released and this was the best we could do given time constraints' defense. Most politicians and judges would likely buy that story since they don't tend to be technologically literate themselves.
- rayiner 7y ago
  > [2] It wouldn't surprise me one bit if lawyers viewed an image-only PDF document dump as a digital form of the 'paper blizzard'. I highly doubt any federal litigator would think of having to OCR a document as being some sort of road block. Acrobat Pro (which has excellent OCR just a couple of clicks away) is universal in the field. Many federal courts will reject electronic filings if they’re not fully text searchable. It’s almost certain that someone just forgot to run the OCR after redaction.
rayiner 7y ago
The government releases documents as PDFs, which is the gold standard for document exchange. Here, all that happened is that the OCR text was stripped when the underlying Word document was redacted, and someone forgot to re-run OCR on the post-redaction version. But it’s trivial to fix. (The article’a belly-aching about the fact someone scanned in the document from paper is misplaced. I have definitely done the exact same thing, because a purely digital conversion keeps all sorts of metadata. Yeah there are ways to strip various layers, but printing to paper and scanning it back in is a sure fire way to strip the metadata and make sure that redactions “stick.”)
plebio 7y ago
The report on the Department of Justice website is already searchable. No one's been relying on the Archive to search it.

sohkamyung 7y ago

One possibly is because they want to make sure the redacted sections are really redacted. You can do a web search and discover that others have tried to redact sections of PDF documents, only to later discover that people can uncover the redacted sections via software tools. Scanning a redacted document is the simplest way of making sure redacted sections stay redacted as there is no underlying information to reveal.

rossenberg79 7y ago
You nailed it. Aside from that anyone who is serious about search is going to run it through an OCR and get the raw text anyway. Why bother doing more work?
- lostlogin 7y ago
  Offering a plain text file with the odd “[redacted text]” alongside a formatted PDF is hardly onerous. That said, there are numerous instances of governments releasing a .doc or similar and someone opening it and doing a few presses of ‘undo’.
  - rossenberg79 7y ago
    There’s just no reason to. Someone will fuck it up, and it will not have been worth the effort anyway.
ALittleLight 7y ago
Releasing the plain text is all that's needed. I fail to see how redacting by overwriting the relevant text is at all reversible - e.g. replace every redacted word with five X's. Replace multiple redactions with ten X's. That actually seems more secure than what's currently done, where you can guess at the words by their length.
- WrtCdEvrydy 7y ago
  Trust me, people have failed... and we've fucked it up in the past. The easiest way is to use a PDF redactor, print it and rescan it.
- wahern 7y ago
  > Releasing the plain text is all that's needed. I doubt Mueller's team wrote the report in plain text. They probably used Microsoft Word and exported a PDF document. When I'm president, all government agencies will be forced to use LaTeX. Then we can have both plaintext source, PDFs with actual text, and PDFs that don't leak redacted material.
  - caseysoftware 7y ago
    LaTeX? That will effectively prevent most government agencies from writing reports.. oh, that was your goal, wasn't it? Well played.
  - i_grow_beans 7y ago
    And I suppose they'll be published in GovtHub, so that anyone can fork them and add their annotations. I'd vote for you, but I'm not a citizen.
  - antt 7y ago
    Context is much better for documents: http://www.berenddeboer.net/tex/LaTeX2ConTeXt.pdf http://www.berenddeboer.net/tex/LaTeX2ConTeXt.pdf

maxlybbert 7y ago

The government is legally required to create an accessible version of the report for disabled people, and the Justice Department already promised to do so. While I can understand not wanting to delay the report’s release because that version wasn’t available, (1) I don’t know what the law says about eventually fulfilling the requirement, and (2) the Justice Department actually has the original documents and doesn’t need OCR to create the accessible PDF, so I’m surprised they haven’t done so yet.

gizmo686 7y ago

Much of the policy around classified materials is designed to be idiot proof. I imagine the process for releasing the document was something along the lines of: go into secure room; print out redacted document; review redacted document; take redacted document to non-secure scanner; release scanned version. Sending digital files across a security boundary is just a much more dangerous operation in general than sending paper. Given the prevelence and quality of OCR (particuarly when run on a good quality print, as was in this case), there is just not enough of a reason to take the risks of sending files across the boundary. EDIT: Idiot proof is probably not a fair characterization either. The other part of security policies is to be genius proof. What if someone involved is trying to exfiltrate data. There is a lot more places for them to hide it if you give them a digital pathway out then if you make them take it out on pieces of paper that will be reviewed.

mbrumlow 7y ago

> Shouldn't an accessible, searchable and digital version of government documents be the default form in which they are released? The way it was done was the most sane way from a security standpoint. In my opinion it is the first time I have thought that somebody who knows what they are doing is actually on the case. I say this because it is just too easy for you to leak out the redacted parts. People have tried, and failed. We don't know if what is under those black blots are really important or not, but if they were, and the leaking of such information could cause somebody harm -- then this is the way to do it.

blihp 7y ago

I suspect the reason they released a scanned version of the redacted document is probably a policy decision to avoid serious screw-ups.[1] Also, organizations of all sizes have known how to photocopy/scan documents for decades so it's a path of least resistance thing. The other thing is that those in charge of releasing this report had no interest in doing anything, including a simple OCR pass[2], on that scanned document that would help disseminate the information in this report faster... it's against their interests to do so.[3] So you're right, a more useful document should have been released. But for the previously mentioned reasons, it wasn't. As long as no critical mass of citizens demands digital competence from their government, their government will not offer it. [1] There was a story in the last month or two about how someone screwed up redaction in an electronic document in a court case where they effectively only painted black bars over the text but the underlying text was in the release document. [2] It wouldn't surprise me one bit if lawyers viewed an image-only PDF document dump as a digital form of the 'paper blizzard'. It really isn't (not even close) as there was already an OCR'd version of the document available within the hour of its release and some NLP analysis within the day. [3] In the event that objections are raised, they can always throw up the 'we were moving as fast as we could to get it released and this was the best we could do given time constraints' defense. Most politicians and judges would likely buy that story since they don't tend to be technologically literate themselves.

rayiner 7y ago
> [2] It wouldn't surprise me one bit if lawyers viewed an image-only PDF document dump as a digital form of the 'paper blizzard'. I highly doubt any federal litigator would think of having to OCR a document as being some sort of road block. Acrobat Pro (which has excellent OCR just a couple of clicks away) is universal in the field. Many federal courts will reject electronic filings if they’re not fully text searchable. It’s almost certain that someone just forgot to run the OCR after redaction.

rayiner 7y ago

The government releases documents as PDFs, which is the gold standard for document exchange. Here, all that happened is that the OCR text was stripped when the underlying Word document was redacted, and someone forgot to re-run OCR on the post-redaction version. But it’s trivial to fix. (The article’a belly-aching about the fact someone scanned in the document from paper is misplaced. I have definitely done the exact same thing, because a purely digital conversion keeps all sorts of metadata. Yeah there are ways to strip various layers, but printing to paper and scanning it back in is a sure fire way to strip the metadata and make sure that redactions “stick.”)

plebio 7y ago

The report on the Department of Justice website is already searchable. No one's been relying on the Archive to search it.