4 ms·

Wow, your setup sounds very similar to what I've converged on. Except I wrote a fairly substantial pipeline that preprocesses the data before giving it to beanc

Wow, your setup sounds very similar to what I've converged on. Except I wrote a fairly substantial pipeline that preprocesses the data before giving it to beancount and Fava. Maybe I missed something, but it seemed like beancount wants everything to live in One Giant Journal file. I really wanted a pipeline where each bank statement PDF would output one file with a corresponding list of transactions (this stage can run completely in parallel and I use "ninja" to make it very fast). Then another process can run over these files looking for matches (+$X, -$X), and spit out "transaction groups", where each transaction group is a set of transaction ids that sum to zero. And then a different interactive tool lets you categorize transactions and spits out transaction groups with embedded "expense" transactions. It's all non-destructive; each tool only adds data, and nothing ever modifies existing files. Then a final step can combine all these files and spit out a beancount file for Fava. How does this compare with the way beancount's importer does it? How does its importer handle transfers and categorization? Is it destructive or non-destructive?

mattbillenstein 6y ago
You're parsing PDF? I'd imagine most places allow some sort of data export like csv or something? That's how I get the data out of Chase and BofA.
- mehrdadn 6y ago
  There are a few problems with exporting data... here's what comes to my mind at the moment: - It's an additional step and unnecessary when you're already downloading PDF statements, which you probably should be doing regardless - Data is often unavailable for export after a few months or a year or two (depending on the bank), far earlier than PDFs - If you have PDF scans (especially from earlier days) then you need to parse the OCR anyway - If you're using official APIs (instead of writing a scraper or or downloading by hand) then you may need to pay extra
  - mattbillenstein 6y ago
    To each their own, csv for me has been the easiest thing moving forward every month or three. I did do a historical net worth calculation and the only data I had going back far enough was PDFs, so through a mix of bash scripts and pdftotext, I was able to get a number for each month back about 10 years. But I ended up just putting that monthly balance for each account in a google sheet so I could sum and plot it there. Now I just stick the month-end balance for each account in a sheet to keep this updated.
- haberman 6y ago
  Yes, in my experience CSV/XML/etc export is spotty. Some institutions don't have it at all, and even when they do the time range can be limited or hard to time-window reliably. For example, I can't get CSV about my pay stub and all the places money flows (taxes, insurance, etc). So I use PDF as the best way to get all the data. Parsing PDF is a huge PITA, since PDF is really designed only for layout and not to encode semantic document structure. But if you want the greatest amount of visibility, nothing is as authoritative as the actual statements.
  - mattbillenstein 6y ago
    Yeah, I guess I don't really look at paystubs often and my banks support good csv export for like 2+ years, so I'll always have it when I need when I refresh those sheets. I do year-to-date in those sheets to make them not huge. Regarding taxes, I do a tax year calculation using my W2's and returns to compute an effective tax rate and figure out ways I can improve my situation re taxes -- but my budgeting basically only starts once the net money hits a bank account which has worked pretty well.
cobby 6y ago
> beancount wants everything to live in One Giant Journal file Actually, beancount supports include directive. See includes section [1] of the language syntax document [1]: https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmYZSygUaR0uOMW1GV3YE0/edit https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmY...
- haberman 6y ago
  Thanks for the reference. But a single transaction can span multiple financial institutions. Say you pay off a credit card. One half of the transaction comes from the credit card statement, the other half comes from your bank. The two halves sum to zero. My understanding is that the whole transaction has to live in one file with Beancount. This doesn't seem amenable to having each bank statement generate a single Beancount output file. But I could be wrong. I don't know how Beancount's importer works.
  - freddie_mercury 6y ago
    If you really want to do that, there is a beancount plugin called ZeroSum that does it https://github.com/redstreet/beancount_reds_plugins/tree/master/beancount_reds_plugins/zerosum https://github.com/redstreet/beancount_reds_plugins/tree/mas...
  - jonahbenton 6y ago
    Conceptually this is one transaction; from a book keeping (and beancount) perspective it is, or can be, two. Often (in the US) the dates are different- debit from the bank on day X, post to credit card on day X + 1 or X + 2. For my stuff I want the dates in beancount to reflect the dates on the respective statements, so that the balances, which are date-specific, match. So I capture these as two transactions. I do it as- debit the bank, credit the "ether", then debit the "ether" credit the credit card. Beancount just wants the postings within a literal transaction to sum to zero. If this conceptual transaction is split into two literal transactions, with some fictional bridge account in between, they can live in different beancount files.
  - arnarbi 6y ago
mehrdadn 6y ago
Can I ask how you parse PDFs? I'm curious both in terms of reading the PDF data (Python library?) and parsing it (regex?)... and do you have to deal with OCR as well?
- haberman 6y ago
  I use "pdftotext -layout" and then parse that. Here is some more info from people who have tried this approach: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=pdftotext%20layout&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
  - mehrdadn 6y ago
    Thanks!

mattbillenstein 6y ago

You're parsing PDF? I'd imagine most places allow some sort of data export like csv or something? That's how I get the data out of Chase and BofA.

mehrdadn 6y ago
There are a few problems with exporting data... here's what comes to my mind at the moment: - It's an additional step and unnecessary when you're already downloading PDF statements, which you probably should be doing regardless - Data is often unavailable for export after a few months or a year or two (depending on the bank), far earlier than PDFs - If you have PDF scans (especially from earlier days) then you need to parse the OCR anyway - If you're using official APIs (instead of writing a scraper or or downloading by hand) then you may need to pay extra
- mattbillenstein 6y ago
  To each their own, csv for me has been the easiest thing moving forward every month or three. I did do a historical net worth calculation and the only data I had going back far enough was PDFs, so through a mix of bash scripts and pdftotext, I was able to get a number for each month back about 10 years. But I ended up just putting that monthly balance for each account in a google sheet so I could sum and plot it there. Now I just stick the month-end balance for each account in a sheet to keep this updated.
haberman 6y ago
Yes, in my experience CSV/XML/etc export is spotty. Some institutions don't have it at all, and even when they do the time range can be limited or hard to time-window reliably. For example, I can't get CSV about my pay stub and all the places money flows (taxes, insurance, etc). So I use PDF as the best way to get all the data. Parsing PDF is a huge PITA, since PDF is really designed only for layout and not to encode semantic document structure. But if you want the greatest amount of visibility, nothing is as authoritative as the actual statements.
- mattbillenstein 6y ago
  Yeah, I guess I don't really look at paystubs often and my banks support good csv export for like 2+ years, so I'll always have it when I need when I refresh those sheets. I do year-to-date in those sheets to make them not huge. Regarding taxes, I do a tax year calculation using my W2's and returns to compute an effective tax rate and figure out ways I can improve my situation re taxes -- but my budgeting basically only starts once the net money hits a bank account which has worked pretty well.

cobby 6y ago

> beancount wants everything to live in One Giant Journal file Actually, beancount supports include directive. See includes section [1] of the language syntax document [1]: https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmYZSygUaR0uOMW1GV3YE0/edit https://docs.google.com/document/d/1wAMVrKIA2qtRGmoVDSUBJGmY...

haberman 6y ago
Thanks for the reference. But a single transaction can span multiple financial institutions. Say you pay off a credit card. One half of the transaction comes from the credit card statement, the other half comes from your bank. The two halves sum to zero. My understanding is that the whole transaction has to live in one file with Beancount. This doesn't seem amenable to having each bank statement generate a single Beancount output file. But I could be wrong. I don't know how Beancount's importer works.
- freddie_mercury 6y ago
  If you really want to do that, there is a beancount plugin called ZeroSum that does it https://github.com/redstreet/beancount_reds_plugins/tree/master/beancount_reds_plugins/zerosum https://github.com/redstreet/beancount_reds_plugins/tree/mas...
- jonahbenton 6y ago
  Conceptually this is one transaction; from a book keeping (and beancount) perspective it is, or can be, two. Often (in the US) the dates are different- debit from the bank on day X, post to credit card on day X + 1 or X + 2. For my stuff I want the dates in beancount to reflect the dates on the respective statements, so that the balances, which are date-specific, match. So I capture these as two transactions. I do it as- debit the bank, credit the "ether", then debit the "ether" credit the credit card. Beancount just wants the postings within a literal transaction to sum to zero. If this conceptual transaction is split into two literal transactions, with some fictional bridge account in between, they can live in different beancount files.
- arnarbi 6y ago

mehrdadn 6y ago

Can I ask how you parse PDFs? I'm curious both in terms of reading the PDF data (Python library?) and parsing it (regex?)... and do you have to deal with OCR as well?

haberman 6y ago
I use "pdftotext -layout" and then parse that. Here is some more info from people who have tried this approach: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=pdftotext%20layout&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- mehrdadn 6y ago
  Thanks!