7 ms·

PDF processing and analysis with open-source tools (2021)

186 points by miles 4y ago

devy 4y ago
Amazing share on the tooling side of thing for PDF processing. More on PDF and why it's hard to do analysis [1]. TL;DR "PDF was never really designed as a data input format, but rather, it was designed as an output format giving fine grained control over the resulting document." By the way, the PDF 1.4 (most widely used version) specification is over 1000 pages![2] To make it even harder, not everybody follow the spec :) [1] https://news.ycombinator.com/item?id=33146364 https://news.ycombinator.com/item?id=33146364 [2] https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/pdfreference1.4.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...
- bee_rider 4y ago
  Yeah I think 95% of the time, the correct way to fix a problem with a PDF is to go back to the tool that generated it, and fix the issue in the input. But, sometimes non-ideal circumstances intervene.
  - perelin 4y ago
    Make that 99%
- jimjimjim 4y ago
  yes, until very recently, if what adobe acrobat/reader did was different to the spec then the spec was wrong. Very hard to tell a client that their document is corrupted when they can read it just fine in adobe.
tyingq 4y ago
When it comes up for me, I've usually been able to use the mentioned pdftotext from poppler-utils. With just one tweak that isn't mentioned in the article...for some documents, adding the "-layout" option gets the text in a state where I can parse it more easily.
chasil 4y ago
This is missing the "pdfsizeopt" suite, that bundles statically compiled utilities to reduce size. Static compilation means that it will run on most Linux platforms without extra required software. I believe one aspect of it will remove characters from included fonts that are not used. It really is quite impressive. https://github.com/pts/pdfsizeopt https://github.com/pts/pdfsizeopt
pfarrell 4y ago
I would add Tabula [0] to this list. I’ve used it to extract tabular data from pdfs, especially in acquiring Covid data at the height of the pandemic. It’s got an MIT license and does extraction of table data from pdfs really well. 0: https://tabula.technology/ https://tabula.technology/
- MilStdJunkie 4y ago
  Yeah, seconding this. I am surprised Tabula didn't make this list. Invaluable.
MAGZine 4y ago
One interesting thing to note about pdf libraries is there seems to be a general trend towards A/GPL licenses, presumably because working with PDFs are tedious but still the standard for conveying data intended for human consumption/archival. The commercial licensing for some of these is a little hairy, too. I've seen rev share clauses which is, uh, hilarious. Depending on what you're doing, check your licenses!
woojoo666 4y ago
I still haven't found a FOSS tool capable of signing pdfs (like docusign)
- POPOSYS 4y ago
  Also a simple tool to check signatures, if they are valid etc.
  - POPOSYS 4y ago
    BTW pdfsig from the poppler tools is mentioned in the list of tools.
  - oever 4y ago
    Okular has recently gotten improved support for verifying cryptographic signatures in PDF files. https://tsdgeos.blogspot.com/2022/03/okular-signature-support-now-works-on.html https://tsdgeos.blogspot.com/2022/03/okular-signature-suppor...
  - gettalong 4y ago
    Verifying digital signatures is quite hard. Yeah, checking whether the stored hash is valid is simple but all the other things are hard, for example, whether the additions done to a signed PDF are actually allowed or not. Or whether the signature is correct in all the small details, e.g. used algorithm, the included information. And then it sometimes depends on the environment. For example, sometimes a digital signature is only considered valid if all the revocation information and all the certificates are included in the PDF.
- jahewson 4y ago
  PDFBox can do this but to have to build the CLI tool yourself from this example: https://github.com/apache/pdfbox/blob/5b00807463279f1002e24531aa0f07c12ee03b0d/examples/src/main/java/org/apache/pdfbox/examples/signature/CreateSignature.java https://github.com/apache/pdfbox/blob/5b00807463279f1002e245... Everything about PDF signatures is maddeningly complex.
- tpoacher 4y ago
ademup 4y ago
For my use case, I ended up converting the PDF to a single png image and then use Amazon Textract on it. This allows me to easily convert pdf tables into csv files all from within php. Would love to find a cheaper (local) option vs AWS, but this works.
- lhuser123 4y ago
  > Would love to find a cheaper (local) option vs AWS How about tesseract (https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract) There’s even a library for php (https://github.com/thiagoalessio/tesseract-ocr-for-php https://github.com/thiagoalessio/tesseract-ocr-for-php). Haven’t used it. I did used python Pytesseract & works fairly well.
  - ademup 4y ago
    Haven't seen this one, thanks!
POPOSYS 4y ago
I am still looking for a good open source templating system that does use PDF files as input, so you are not bound to a special tool / format to generate the templates but can use whatever produces PDF files itself. Does anybody know one? All printing workflows use proprietary formats as input and bind you to one tool or producer. Could Scribus help with that?
- mrazomor 4y ago
  One thing that worked well in practice (years ago): - design svg template (eg. in inkscape) - set placeholders (as svg is just xml -- {%address%} etc) - replace the placeholders with the actual value & use headless inkscape to produce PDF
  - POPOSYS 4y ago
    This is interesting. Does exist some kind of extraction of the PDF generating part of inkscape, so you do not have to start inkscape all over again for generating PDF files? Or is there a good and fast command line tool, that can produce PDF from Inkscape PDF files?
    - tkp 4y ago
      Inkscape can be used from the commandline to export pdfs : https://wiki.inkscape.org/wiki/Using_the_Command_Line https://wiki.inkscape.org/wiki/Using_the_Command_Line
    - franga2000 4y ago
      Inkscape's format is just SVG, so any tool that can do SVG-PDF will do. I've seen cairosvg, imagemagick and rsvg used for this, but if your deployment allows, it does make the most sense to use inkscape from the CLI to avoid any inconsistencies: $ inkscape rendered.svg --export-pdf=output.pdf I think there's also a way to do batch processing where you don't need to spin up a new inkscape process for each file (which takes time), but I don't remember how it works anymore.
- Archelaos 4y ago
crecker 4y ago
This tool might be helpful for comparing pdfs: https://github.com/serhack/pdf-diff https://github.com/serhack/pdf-diff
tpoacher 4y ago
I'm still lamenting the demise of the "Impress!ve" pdf presentation tool. It was super cool. Currently unnmaintained and spitting errors, unfortunately.
janci 4y ago
I'd like to add this tool to the list: https://pdfsam.org/ https://pdfsam.org/ It does exactly what it says - splits and merges PDFs
gyulai 4y ago
Don't forget https://weasyprint.org https://weasyprint.org
matt7340 4y ago
PDF processing seems like a security minefield. What are folks doing to mitigate that problem prior to (or as part of) processing? Or as part of any system that accepts PDFs with the intent that they’re shared with other systems and users.
triggercut 4y ago
I've been looking for a tool that lets you markup pdf documents with bounding boxes around text and lets you add metadata (like a section number, paragraph number, page number) you can use for downstream processing and extracting the text from particular areas to process. Kind of like a reverse LaTeX Document Structure The closest I've come in the past is manually setting xy coordinates to do crops in pdftotext which was pretty time consuming. I'm sure it wouldn't be too difficult (famous last words) to use annotation objects with structure codes or some such but I'm surprised even after all these years that there isn't something that lets you do this more simply.

186 points by miles 4y ago

devy 4y ago

Amazing share on the tooling side of thing for PDF processing. More on PDF and why it's hard to do analysis [1]. TL;DR "PDF was never really designed as a data input format, but rather, it was designed as an output format giving fine grained control over the resulting document." By the way, the PDF 1.4 (most widely used version) specification is over 1000 pages![2] To make it even harder, not everybody follow the spec :) [1] https://news.ycombinator.com/item?id=33146364 https://news.ycombinator.com/item?id=33146364 [2] https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/pdfreference1.4.pdf https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...

bee_rider 4y ago
Yeah I think 95% of the time, the correct way to fix a problem with a PDF is to go back to the tool that generated it, and fix the issue in the input. But, sometimes non-ideal circumstances intervene.
- perelin 4y ago
  Make that 99%
jimjimjim 4y ago
yes, until very recently, if what adobe acrobat/reader did was different to the spec then the spec was wrong. Very hard to tell a client that their document is corrupted when they can read it just fine in adobe.

tyingq 4y ago

When it comes up for me, I've usually been able to use the mentioned pdftotext from poppler-utils. With just one tweak that isn't mentioned in the article...for some documents, adding the "-layout" option gets the text in a state where I can parse it more easily.

chasil 4y ago

This is missing the "pdfsizeopt" suite, that bundles statically compiled utilities to reduce size. Static compilation means that it will run on most Linux platforms without extra required software. I believe one aspect of it will remove characters from included fonts that are not used. It really is quite impressive. https://github.com/pts/pdfsizeopt https://github.com/pts/pdfsizeopt

pfarrell 4y ago

I would add Tabula [0] to this list. I’ve used it to extract tabular data from pdfs, especially in acquiring Covid data at the height of the pandemic. It’s got an MIT license and does extraction of table data from pdfs really well. 0: https://tabula.technology/ https://tabula.technology/

MilStdJunkie 4y ago
Yeah, seconding this. I am surprised Tabula didn't make this list. Invaluable.

MAGZine 4y ago

One interesting thing to note about pdf libraries is there seems to be a general trend towards A/GPL licenses, presumably because working with PDFs are tedious but still the standard for conveying data intended for human consumption/archival. The commercial licensing for some of these is a little hairy, too. I've seen rev share clauses which is, uh, hilarious. Depending on what you're doing, check your licenses!

woojoo666 4y ago

I still haven't found a FOSS tool capable of signing pdfs (like docusign)

POPOSYS 4y ago
Also a simple tool to check signatures, if they are valid etc.
- POPOSYS 4y ago
  BTW pdfsig from the poppler tools is mentioned in the list of tools.
- oever 4y ago
  Okular has recently gotten improved support for verifying cryptographic signatures in PDF files. https://tsdgeos.blogspot.com/2022/03/okular-signature-support-now-works-on.html https://tsdgeos.blogspot.com/2022/03/okular-signature-suppor...
- gettalong 4y ago
  Verifying digital signatures is quite hard. Yeah, checking whether the stored hash is valid is simple but all the other things are hard, for example, whether the additions done to a signed PDF are actually allowed or not. Or whether the signature is correct in all the small details, e.g. used algorithm, the included information. And then it sometimes depends on the environment. For example, sometimes a digital signature is only considered valid if all the revocation information and all the certificates are included in the PDF.
jahewson 4y ago
PDFBox can do this but to have to build the CLI tool yourself from this example: https://github.com/apache/pdfbox/blob/5b00807463279f1002e24531aa0f07c12ee03b0d/examples/src/main/java/org/apache/pdfbox/examples/signature/CreateSignature.java https://github.com/apache/pdfbox/blob/5b00807463279f1002e245... Everything about PDF signatures is maddeningly complex.
tpoacher 4y ago

ademup 4y ago

For my use case, I ended up converting the PDF to a single png image and then use Amazon Textract on it. This allows me to easily convert pdf tables into csv files all from within php. Would love to find a cheaper (local) option vs AWS, but this works.

lhuser123 4y ago
> Would love to find a cheaper (local) option vs AWS How about tesseract (https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract) There’s even a library for php (https://github.com/thiagoalessio/tesseract-ocr-for-php https://github.com/thiagoalessio/tesseract-ocr-for-php). Haven’t used it. I did used python Pytesseract & works fairly well.
- ademup 4y ago
  Haven't seen this one, thanks!

POPOSYS 4y ago

I am still looking for a good open source templating system that does use PDF files as input, so you are not bound to a special tool / format to generate the templates but can use whatever produces PDF files itself. Does anybody know one? All printing workflows use proprietary formats as input and bind you to one tool or producer. Could Scribus help with that?

mrazomor 4y ago
One thing that worked well in practice (years ago): - design svg template (eg. in inkscape) - set placeholders (as svg is just xml -- {%address%} etc) - replace the placeholders with the actual value & use headless inkscape to produce PDF
- POPOSYS 4y ago
  This is interesting. Does exist some kind of extraction of the PDF generating part of inkscape, so you do not have to start inkscape all over again for generating PDF files? Or is there a good and fast command line tool, that can produce PDF from Inkscape PDF files?
  - tkp 4y ago
    Inkscape can be used from the commandline to export pdfs : https://wiki.inkscape.org/wiki/Using_the_Command_Line https://wiki.inkscape.org/wiki/Using_the_Command_Line
  - franga2000 4y ago
    Inkscape's format is just SVG, so any tool that can do SVG-PDF will do. I've seen cairosvg, imagemagick and rsvg used for this, but if your deployment allows, it does make the most sense to use inkscape from the CLI to avoid any inconsistencies: $ inkscape rendered.svg --export-pdf=output.pdf I think there's also a way to do batch processing where you don't need to spin up a new inkscape process for each file (which takes time), but I don't remember how it works anymore.
Archelaos 4y ago

crecker 4y ago

This tool might be helpful for comparing pdfs: https://github.com/serhack/pdf-diff https://github.com/serhack/pdf-diff

tpoacher 4y ago

I'm still lamenting the demise of the "Impress!ve" pdf presentation tool. It was super cool. Currently unnmaintained and spitting errors, unfortunately.

janci 4y ago

I'd like to add this tool to the list: https://pdfsam.org/ https://pdfsam.org/ It does exactly what it says - splits and merges PDFs

gyulai 4y ago

Don't forget https://weasyprint.org https://weasyprint.org

matt7340 4y ago

PDF processing seems like a security minefield. What are folks doing to mitigate that problem prior to (or as part of) processing? Or as part of any system that accepts PDFs with the intent that they’re shared with other systems and users.

triggercut 4y ago

I've been looking for a tool that lets you markup pdf documents with bounding boxes around text and lets you add metadata (like a section number, paragraph number, page number) you can use for downstream processing and extracting the text from particular areas to process. Kind of like a reverse LaTeX Document Structure The closest I've come in the past is manually setting xy coordinates to do crops in pdftotext which was pretty time consuming. I'm sure it wouldn't be too difficult (famous last words) to use annotation objects with structure codes or some such but I'm surprised even after all these years that there isn't something that lets you do this more simply.