4 ms·

Because Hillary wants to make it hard to analyze them. Drug companies used the same technique to hide their relationships with doctors. https://www.propublica.

by spikels 11y ago

Because Hillary wants to make it hard to analyze them. Drug companies used the same technique to hide their relationships with doctors. https://www.propublica.org/nerds/item/doc-dollars-guides-collecting-the-data https://www.propublica.org/nerds/item/doc-dollars-guides-col...

danso 11y ago
I was the original programmer for that project...while I didn't enjoy spending part of my late 20s learning about PDF to text conversions...I wouldn't necessarily attribute their use of PDF to devious motives. It's easy to forget sometimes that much of the world still operates on paper...and electronic documents that translate directly to paper have some value. These financial records were disclosed as part of legal settlements, and they were also used by hospital compliance officers...While obviously just sending out CSVs or Excel sheets would be great, it adds a possible breakage point between the company and its audience...i.e. hospital officers complaining that the spreadsheet looks funky on their version of Excel or whatever the hell they're using. The thing that makes PDFs so frustrating to parse is what makes them so "secure" and stable to end-users. That's why I advise my students to not just send URLs and web-clips to potential employers, but to have PDF versions of their resumes and work because inevitably, the "boss" wants a printout on his desk...and you don't want his office's shitty printer (and/or whatever IE browser rendered the HTML) to leave a bad first impression. And let's face it, if you are an end-user who is not a programmer, a PDF is very convenient. You can even do Ctrl-F to look for things...and if you aren't a programmer, that's all the search-power you need to do your compliance work. After the investigation was published, I fielded a lot of questions from auditors who were convinced I had built some amazing proprietary search algorithm...such people just don't know the gap between doing Ctrl-F versus "SELECT * FROM doctors where LAST_NAME like ..." :) There were a couple of companies who arguably went out of their way to obfuscate their records. They had built websites that consisted of Flash widgets designed to look and act like a (crappy) HTML table...except with Flash, you can prevent the user from select/copy/etc. Unfortunately, the developers of those widgets never told the companies that their Flash containers operated by making XHR requests to external XML/JSON files...so these companies ended up being the easiest to gather the data from.
- brc 11y ago
  Handling PDFs and OCR is always troublesome. But in this case, if I understand you correctly, the original source documents were PDF? That they were distributed as PDF for all the reasons you state? That doesn't apply in this case - the original documents were not PDF, and had no reason to be PDF except to make it more difficult.
  - halter73 11y ago
    Fortunately, the PDFs already contain the plain text as metadata. I believe they are what are known as a searchable image PDFs. The code posted here isn't doing any OCR, but whatever generated the PDFs (Acrobat?) might have.
    - brc 11y ago
      How did they get the plain text as metadata? Was the scanning equipment doing OCR and setting that?
      - danso 11y ago
        I believe the U.S. agencies use ABBYY FineReader, which does a pretty good job with OCR and text resolution. The U.S. Senate used it when releasing the CIA torture docs awhile ago: http://www.nytimes.com/interactive/2014/12/09/world/cia-torture-report-document.html?_r=0 http://www.nytimes.com/interactive/2014/12/09/world/cia-tort...
  - spikels 11y ago
    According to news reports Hillary handed over the emails to the Dept. of State as hard copies. These were then processed by them into PDFs including doing OCR. http://money.cnn.com/2015/03/11/technology/security/hillary-email-paper/index.html http://money.cnn.com/2015/03/11/technology/security/hillary-... Edit: Hilarious part was where State says this is not a problem because "it would have to print Clinton's emails in the normal review process."
    - 11y ago

by spikels 11y ago

danso 11y ago

I was the original programmer for that project...while I didn't enjoy spending part of my late 20s learning about PDF to text conversions...I wouldn't necessarily attribute their use of PDF to devious motives. It's easy to forget sometimes that much of the world still operates on paper...and electronic documents that translate directly to paper have some value. These financial records were disclosed as part of legal settlements, and they were also used by hospital compliance officers...While obviously just sending out CSVs or Excel sheets would be great, it adds a possible breakage point between the company and its audience...i.e. hospital officers complaining that the spreadsheet looks funky on their version of Excel or whatever the hell they're using. The thing that makes PDFs so frustrating to parse is what makes them so "secure" and stable to end-users. That's why I advise my students to not just send URLs and web-clips to potential employers, but to have PDF versions of their resumes and work because inevitably, the "boss" wants a printout on his desk...and you don't want his office's shitty printer (and/or whatever IE browser rendered the HTML) to leave a bad first impression. And let's face it, if you are an end-user who is not a programmer, a PDF is very convenient. You can even do Ctrl-F to look for things...and if you aren't a programmer, that's all the search-power you need to do your compliance work. After the investigation was published, I fielded a lot of questions from auditors who were convinced I had built some amazing proprietary search algorithm...such people just don't know the gap between doing Ctrl-F versus "SELECT * FROM doctors where LAST_NAME like ..." :) There were a couple of companies who arguably went out of their way to obfuscate their records. They had built websites that consisted of Flash widgets designed to look and act like a (crappy) HTML table...except with Flash, you can prevent the user from select/copy/etc. Unfortunately, the developers of those widgets never told the companies that their Flash containers operated by making XHR requests to external XML/JSON files...so these companies ended up being the easiest to gather the data from.

brc 11y ago
Handling PDFs and OCR is always troublesome. But in this case, if I understand you correctly, the original source documents were PDF? That they were distributed as PDF for all the reasons you state? That doesn't apply in this case - the original documents were not PDF, and had no reason to be PDF except to make it more difficult.
- halter73 11y ago
  Fortunately, the PDFs already contain the plain text as metadata. I believe they are what are known as a searchable image PDFs. The code posted here isn't doing any OCR, but whatever generated the PDFs (Acrobat?) might have.
  - brc 11y ago
    How did they get the plain text as metadata? Was the scanning equipment doing OCR and setting that?
    - danso 11y ago
      I believe the U.S. agencies use ABBYY FineReader, which does a pretty good job with OCR and text resolution. The U.S. Senate used it when releasing the CIA torture docs awhile ago: http://www.nytimes.com/interactive/2014/12/09/world/cia-torture-report-document.html?_r=0 http://www.nytimes.com/interactive/2014/12/09/world/cia-tort...
- spikels 11y ago
  According to news reports Hillary handed over the emails to the Dept. of State as hard copies. These were then processed by them into PDFs including doing OCR. http://money.cnn.com/2015/03/11/technology/security/hillary-email-paper/index.html http://money.cnn.com/2015/03/11/technology/security/hillary-... Edit: Hilarious part was where State says this is not a problem because "it would have to print Clinton's emails in the normal review process."
  - 11y ago