4 ms·

I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned ho

by _bohm 8y ago

I've spent a great deal of time parsing data out of government PDFs that isn't attainable by any other means as a part of my job. In the process I've learned how difficult this information can be to access even for people who don't require it to be in a machine readable format. It certainly has been an interesting exercise in how far simple web scraping tools can be pushed, though.

ams6110 8y ago
Amazon Textract was recently announced, sounds like it might be good for that. Haven't tried it myself. https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-amazon-textract-now-in-preview-easily-extract-text-and-data-from-virtually-any-document/ https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...
- _bohm 8y ago
  I applied to the beta, but they never got back to me :\
  - bpchaps 8y ago
    Have you tried Apache's tika? It's pretty decent.
    - _bohm 8y ago
      Nope, I'll have to give it a spin. Thanks for the recommendation!
TheAceOfHearts 8y ago
Do you have any tool suggestions or general advice for someone trying to do this? A while back I was trying to extract text from some government PDFs in order to make the information more accessible for others, but I became a bit overwhelmed when I started reading up on PDFs.
- _bohm 8y ago
  Sure! In terms of raw text extraction (for documents that don't require OCR), the most useful tools I've worked with have been pdftotext [0] and PyMuPDF [1]. For extracting useful details, really, my best advice is to make sure that your regex skills are sharp. I've been meaning to explore the possibility of using NLP tools for named entity recognition, but unfortunately I don't have much of a background there. The rest kind of it kind of just comes down to using good software engineering practices to help keep yourself sane. Find useful abstractions for common tasks you need to perform and build a library around them, make sure that your data processing pipeline is designed with enough flexibility to handle inputs in different formats so that adding or modifying parsing logic becomes trivial, etc. [0] https://www.xpdfreader.com/pdftotext-man.html https://www.xpdfreader.com/pdftotext-man.html [1] https://pymupdf.readthedocs.io/en/latest/ https://pymupdf.readthedocs.io/en/latest/
  - ocrcustomserver 8y ago
    pdfminer is another good library (Python).

by _bohm 8y ago

ams6110 8y ago

Amazon Textract was recently announced, sounds like it might be good for that. Haven't tried it myself. https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-amazon-textract-now-in-preview-easily-extract-text-and-data-from-virtually-any-document/ https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...

_bohm 8y ago
I applied to the beta, but they never got back to me :\
- bpchaps 8y ago
  Have you tried Apache's tika? It's pretty decent.
  - _bohm 8y ago
    Nope, I'll have to give it a spin. Thanks for the recommendation!

TheAceOfHearts 8y ago

Do you have any tool suggestions or general advice for someone trying to do this? A while back I was trying to extract text from some government PDFs in order to make the information more accessible for others, but I became a bit overwhelmed when I started reading up on PDFs.

_bohm 8y ago
Sure! In terms of raw text extraction (for documents that don't require OCR), the most useful tools I've worked with have been pdftotext [0] and PyMuPDF [1]. For extracting useful details, really, my best advice is to make sure that your regex skills are sharp. I've been meaning to explore the possibility of using NLP tools for named entity recognition, but unfortunately I don't have much of a background there. The rest kind of it kind of just comes down to using good software engineering practices to help keep yourself sane. Find useful abstractions for common tasks you need to perform and build a library around them, make sure that your data processing pipeline is designed with enough flexibility to handle inputs in different formats so that adding or modifying parsing logic becomes trivial, etc. [0] https://www.xpdfreader.com/pdftotext-man.html https://www.xpdfreader.com/pdftotext-man.html [1] https://pymupdf.readthedocs.io/en/latest/ https://pymupdf.readthedocs.io/en/latest/
- ocrcustomserver 8y ago
  pdfminer is another good library (Python).