17 ms·

Parsing my monthly credit card bill (PDF), categorizing, and displaying statistics.

mehrdadn 8y ago
Ooh, I've been looking for something like this. Could I ask how you parse your bills? Did you just hack together your own script or did you find a parser online? I tried doing this several times but it seems rather painful to get right since the formats can differ and since copying text from PDFs can sometimes ignore the table layouts and whatnot.
- geerlingguy 8y ago
  The poor person’s solution would be mint.com... though the security model there is dubious.
  - mehrdadn 8y ago
    I've tried Mint... aside from the security I neither trust its privacy model nor did I see it working very well either.
    - burkestar 8y ago
      It actually gets a lot better at categorization the more you use it. It has at least some primitive machine learning (possibly rules engine?). Categorize a transaction once and it usually gets it right subsequently.
      - mehrdadn 8y ago
        Yeah I'm aware of that, but I still didn't find it that great. Unfortunately it's been a few years so I don't remember exactly why.
- mkl 8y ago
  There are no general solutions, unfortunately. Every PDF-generating system will do things differently, so parsing them must be custom for each type. Generated PDFs are usually very systematic, though, and PDF commands are text (compressed). You can uncompress with PDFTK: pdftk input.pdf output output.pdf uncompress Then try grepping or using whatever tools you like (there will be binary parts of the file still, like embedded fonts and bitmaps).
  - mehrdadn 8y ago
    I just tried it on a bank statement and the only text I see is the PDF commands (stuff like "/MediaBox [0 0 612 792]", "/F1 125 Tf", and whatnot). There's no actual PDF content text that I can find, whether it's my name, the bank name, or anything else.
    - mkl 8y ago
      There are a few possibilities. The main PDF commands for text are "Tj" and "TJ", so you can try searching for them. Text commands must also be between "BT" and "ET". Maybe a bit harder to search for are the other two text commands: ' and ". TJ is "Show text, allowing individual glyph positioning" which means strings are not necessarily in the PDF as the strings themselves, but have glyph positioning info mixed into them. This is the most common issue I've come across. Another possibility is that the fonts have been combined/compressed/butchered, so that they're not using ASCII/Unicode (this can sometimes save space by combining bits of multiple fonts. Searching for Tj/TJ/'/" should tell you if that's the case. If it is, technically there should be a translation table somewhere in the PDF file (so that clipboard operations work in PDF viewers). Look for CustomEncoding, I think. Particularly bad PDF creation libraries can put characters in one by one. Characters (alone or in strings) can also be in there as octal or hex. Finally, particularly obnoxious libraries can do it without text commands at all, but convert everything to vector drawing commands, or even worse, bitmaps. I couldn't remember CustomEncoding, and this came up when I searched. Seems helpful: https://stackoverflow.com/questions/29467539/encoding-of-pdf-text-string https://stackoverflow.com/questions/29467539/encoding-of-pdf...
      - 8y ago
- petters 8y ago
  I have just written my own parser. Took some work when my bank switched from txt to PDF. Then I match keywords in the strings to categorize the expenses. These days, this would be called AI :) Edit: I use pdftotext, which has a mode that keeps the spatial structure of tables. Works for my bank.

Ooh, I've been looking for something like this. Could I ask how you parse your bills? Did you just hack together your own script or did you find a parser online? I tried doing this several times but it seems rather painful to get right since the formats can differ and since copying text from PDFs can sometimes ignore the table layouts and whatnot.

geerlingguy 8y ago
The poor person’s solution would be mint.com... though the security model there is dubious.
- mehrdadn 8y ago
  I've tried Mint... aside from the security I neither trust its privacy model nor did I see it working very well either.
  - burkestar 8y ago
    It actually gets a lot better at categorization the more you use it. It has at least some primitive machine learning (possibly rules engine?). Categorize a transaction once and it usually gets it right subsequently.
    - mehrdadn 8y ago
      Yeah I'm aware of that, but I still didn't find it that great. Unfortunately it's been a few years so I don't remember exactly why.
mkl 8y ago
There are no general solutions, unfortunately. Every PDF-generating system will do things differently, so parsing them must be custom for each type. Generated PDFs are usually very systematic, though, and PDF commands are text (compressed). You can uncompress with PDFTK: pdftk input.pdf output output.pdf uncompress Then try grepping or using whatever tools you like (there will be binary parts of the file still, like embedded fonts and bitmaps).
- mehrdadn 8y ago
  I just tried it on a bank statement and the only text I see is the PDF commands (stuff like "/MediaBox [0 0 612 792]", "/F1 125 Tf", and whatnot). There's no actual PDF content text that I can find, whether it's my name, the bank name, or anything else.
  - mkl 8y ago
    There are a few possibilities. The main PDF commands for text are "Tj" and "TJ", so you can try searching for them. Text commands must also be between "BT" and "ET". Maybe a bit harder to search for are the other two text commands: ' and ". TJ is "Show text, allowing individual glyph positioning" which means strings are not necessarily in the PDF as the strings themselves, but have glyph positioning info mixed into them. This is the most common issue I've come across. Another possibility is that the fonts have been combined/compressed/butchered, so that they're not using ASCII/Unicode (this can sometimes save space by combining bits of multiple fonts. Searching for Tj/TJ/'/" should tell you if that's the case. If it is, technically there should be a translation table somewhere in the PDF file (so that clipboard operations work in PDF viewers). Look for CustomEncoding, I think. Particularly bad PDF creation libraries can put characters in one by one. Characters (alone or in strings) can also be in there as octal or hex. Finally, particularly obnoxious libraries can do it without text commands at all, but convert everything to vector drawing commands, or even worse, bitmaps. I couldn't remember CustomEncoding, and this came up when I searched. Seems helpful: https://stackoverflow.com/questions/29467539/encoding-of-pdf-text-string https://stackoverflow.com/questions/29467539/encoding-of-pdf...
    - 8y ago
petters 8y ago
I have just written my own parser. Took some work when my bank switched from txt to PDF. Then I match keywords in the strings to categorize the expenses. These days, this would be called AI :) Edit: I use pdftotext, which has a mode that keeps the spatial structure of tables. Works for my bank.