6 ms·

Extracting campaign finance data from gnarly PDFs using deep learning

95 points by danso 7y ago

manojlds 7y ago
Looks like a very shallow article that hasn't actually solved the meat of the problems. Am I reading it wrong?
- jonathanstray 7y ago
  Heh. It’s my work, so maybe I can clarify the goals. This is meant to be a proof of concept. I took a week and was able to show that relatively simple deep learning techniques are capable of generalizing over unseen form types with high accuracy. I also showed that tokens-plus-geometry is a viable format, and that hand-crafted feature engineering is still necessary (and still used in SOTA approaches). I also believe that preparing and cleaning this data set, and bringing a challenging investigative journalism problem to the attention of other researchers, would be valuable even if I hadn’t done any work on this baseline solution. This is a problem that journalists currently expend a huge amount of time and money on, which reduces the effectiveness of transparency around political ad spending information.
  - coffeecat 7y ago
    I'm curious if you considered or attempted to use pdfplumber's table extraction methods to separate tabular from non-tabular text. That would be my starting point on a problem like this, as picking the relevant rows of a table is far easier than picking from the set of all tokens. By the way, when you say tokens, you're referring to non-whitespace characters separated by whitespace? How reliable have you found pdfplumber to be in picking out words/tokens?
    - jonathanstray 7y ago
      I didn’t try separating out tables because the total field isn’t actually “inside” the table in many cases. Certainly the other fields I want are not. pdfplumber seems mostly ok at extracting tokens. Sometimes it seems to combine tokens that should be separate. I suspect a few percent of the error is actually problems earlier in the data pipeline, as opposed to the model proper.
  - steve19 7y ago
    Really enjoyed the video, thank you for sharing. I think the parents critism was more that the article was a little light compared to thr video. For example, you didn't have screenshots of the scanned pdfs in the article.
jonathanyc 7y ago
> This project aimed to find out, and successfully extracted the easiest of the fields (total amount) at 90% accuracy using a relatively simple network. Why does this page not even say what this field was? For all we know, OpenCV would have worked better.
- oaeide 7y ago
  As it states, the field in question is "Total amount"
ma2rten 7y ago
Accuracy may not be the best metric, if your dataset is imbalanced.
- jonathanstray 7y ago
  It’s not a binary classifier. Every invoice in the dataset has a total amount written somewhere on it. Accuracy here is whether the network chooses the correct token from each PDF.
lalaland1125 7y ago
My personal bet is that you would have probably gotten better results by simply feeding your data into xgboost or lightgbm. This current fad of focusing on neural networks just seems sorta silly when we have simpler and (often) better performing models. Just look at what sorts of models win kaggle competitions.
- steve19 7y ago
  Maybe but I think 90% of the work is wrangling the data and understanding the problem, it wouldn't be that hard to substitute gradient boosting for the NN
dlphn___xyz 7y ago
how does the performance compare to a simple ocr reader?
- steve19 7y ago
  It operates on ocr output. It's job is to identify the total expenditure by looking at ocr output.
ChuckMcM 7y ago
This is a problem that fascinates me as well. How humans effortlessly look at a document with tables and columns Etc. and effortlessly extract all of the 'factoid' bits from the document and often their relationships. Back at IBM there had been some work on extracting tables from PDFs but it turns out to be a pretty challenging problem to generalize.
sweeneyrod 7y ago
Deep learning (and indeed any kind of machine learning) is really not necessary here. I wrote a very simple baseline that estimates the total gross amount as the biggest numerical value present (favouring ones that begin with a dollar sign and/or occur more than once). This achieves 93% accuracy (code is here https://gist.github.com/rlmacsween/166b7c1c1b0c5f466a0fe9b467d61537 https://gist.github.com/rlmacsween/166b7c1c1b0c5f466a0fe9b46...) in comparison to the 90% for the deep learning model.
- jonathanstray 7y ago
  Nice! An excellent baseline, something to beat.

95 points by danso 7y ago

manojlds 7y ago

Looks like a very shallow article that hasn't actually solved the meat of the problems. Am I reading it wrong?

jonathanstray 7y ago
Heh. It’s my work, so maybe I can clarify the goals. This is meant to be a proof of concept. I took a week and was able to show that relatively simple deep learning techniques are capable of generalizing over unseen form types with high accuracy. I also showed that tokens-plus-geometry is a viable format, and that hand-crafted feature engineering is still necessary (and still used in SOTA approaches). I also believe that preparing and cleaning this data set, and bringing a challenging investigative journalism problem to the attention of other researchers, would be valuable even if I hadn’t done any work on this baseline solution. This is a problem that journalists currently expend a huge amount of time and money on, which reduces the effectiveness of transparency around political ad spending information.
- coffeecat 7y ago
  I'm curious if you considered or attempted to use pdfplumber's table extraction methods to separate tabular from non-tabular text. That would be my starting point on a problem like this, as picking the relevant rows of a table is far easier than picking from the set of all tokens. By the way, when you say tokens, you're referring to non-whitespace characters separated by whitespace? How reliable have you found pdfplumber to be in picking out words/tokens?
  - jonathanstray 7y ago
    I didn’t try separating out tables because the total field isn’t actually “inside” the table in many cases. Certainly the other fields I want are not. pdfplumber seems mostly ok at extracting tokens. Sometimes it seems to combine tokens that should be separate. I suspect a few percent of the error is actually problems earlier in the data pipeline, as opposed to the model proper.
- steve19 7y ago
  Really enjoyed the video, thank you for sharing. I think the parents critism was more that the article was a little light compared to thr video. For example, you didn't have screenshots of the scanned pdfs in the article.

jonathanyc 7y ago

> This project aimed to find out, and successfully extracted the easiest of the fields (total amount) at 90% accuracy using a relatively simple network. Why does this page not even say what this field was? For all we know, OpenCV would have worked better.

oaeide 7y ago
As it states, the field in question is "Total amount"

ma2rten 7y ago

Accuracy may not be the best metric, if your dataset is imbalanced.

jonathanstray 7y ago
It’s not a binary classifier. Every invoice in the dataset has a total amount written somewhere on it. Accuracy here is whether the network chooses the correct token from each PDF.

lalaland1125 7y ago

My personal bet is that you would have probably gotten better results by simply feeding your data into xgboost or lightgbm. This current fad of focusing on neural networks just seems sorta silly when we have simpler and (often) better performing models. Just look at what sorts of models win kaggle competitions.

steve19 7y ago
Maybe but I think 90% of the work is wrangling the data and understanding the problem, it wouldn't be that hard to substitute gradient boosting for the NN

dlphn___xyz 7y ago

how does the performance compare to a simple ocr reader?

steve19 7y ago
It operates on ocr output. It's job is to identify the total expenditure by looking at ocr output.

ChuckMcM 7y ago

This is a problem that fascinates me as well. How humans effortlessly look at a document with tables and columns Etc. and effortlessly extract all of the 'factoid' bits from the document and often their relationships. Back at IBM there had been some work on extracting tables from PDFs but it turns out to be a pretty challenging problem to generalize.

sweeneyrod 7y ago

Deep learning (and indeed any kind of machine learning) is really not necessary here. I wrote a very simple baseline that estimates the total gross amount as the biggest numerical value present (favouring ones that begin with a dollar sign and/or occur more than once). This achieves 93% accuracy (code is here https://gist.github.com/rlmacsween/166b7c1c1b0c5f466a0fe9b467d61537 https://gist.github.com/rlmacsween/166b7c1c1b0c5f466a0fe9b46...) in comparison to the 90% for the deep learning model.

jonathanstray 7y ago
Nice! An excellent baseline, something to beat.