7 ms·

I automated downloading and reconciling my financials from all my bank accounts. https://disjoint.ca/projects/ledger-reconciler https://disjoint.ca/projects/le

I automated downloading and reconciling my financials from all my bank accounts. https://disjoint.ca/projects/ledger-reconciler https://disjoint.ca/projects/ledger-reconciler

bberenberg 9y ago
This is great, would love to see support for more data sources.
- marvinpinto 9y ago
  Thanks! Unfortunately I can only add support for things I have access to (bank accounts and such) and I do plan on adding a few more. To work around this I tried to make it as developer-friendly as I could so we'll see!
- burkemw3 9y ago
  ledger-autosync [0] does similar things with more data sources available [0]: https://github.com/egh/ledger-autosync https://github.com/egh/ledger-autosync
throwaway613834 9y ago
Holy cow, I've been looking for this but for the US. Anybody know of any?
- marvinpinto 9y ago
  If you're up for it, you can add your own bank/creditcard company/whatever as a plugin to it - I think the examples there are decent. I could also help if need be!
  - throwaway613834 9y ago
    Haha thanks! The problem is in the US most banks would need screen-scraping; they don't have APIs, so it's a lot more work to do such a thing.
    - marvinpinto 9y ago
      ledger-reconciler uses headless chrome underneath to screen-scrape all its information from banking/creditcard websites. That's how all the plugins hook into it. It does not use any public or private banking APIs. Hope that helps!
      - throwaway613834 9y ago
        Oh dang I didn't know! I'll check it out at some point then, thanks! :)
    - jldugger 9y ago
      Many actually do have APIs, they're just poorly documented in public.
    - noughth 9y ago
      Many US banks support OFX (though they may call it QFX, DirectConnect, or simply advertise Quicken support). The OFX spec is a bit of a pain, but there are a number of open-source libraries to query/parse it. I wrote one in Go: https://github.com/aclindsa/ofxgo https://github.com/aclindsa/ofxgo
- jldugger 9y ago
  Transaction pull can be done in GNUCash with aqbanking. It's not fully automated, as it's a GUI and there's a manual review step. And also the aqbanking interface is terribad.
- jpeeler 9y ago
  This might be helpful, though I haven't looked at it in a very long time: https://github.com/madhat2r/plaid2text https://github.com/madhat2r/plaid2text Essentially it is some python to download transactions using the plaid API (and requires an account to do so).
jldugger 9y ago
On a related note, I have gnucash set up to autocreate a ton of transactions 90 days in advance. Rent, salary, typical bills, student loan repayment, etc. At some point my plan is to automate the following tasks: 1. Sweep checking into savings based on forecasted transactions. Useful since interest is like 10x more in savings. 2. Update asset values daily. 3. Update autocreated transactions as the invoices are emailed to me. 4. Calculate a rebalancing strategy that factors in allowed rollovers and expense ratios.
- mschuster91 9y ago
  > On a related note, I have gnucash set up to autocreate a ton of transactions 90 days in advance. Rent, salary, typical bills, student loan repayment, etc. Why do you need to do the transactions? In Germany, I can simply create a recurring transaction for rent and loan repayment, and utility/phone bills are directly debited from my bank account.
  - jldugger 9y ago
    You misunderstand. They're already scheduled largely. This is about writing down in the books in January that I'm going to spend in March. GNUCash's summary view has a column 'future minimum' for each account that uses this data; looking at it informs you when you might go negative without corrective action. Or conversely, tells you how much money you can safely shift out of checking.
    - mschuster91 9y ago
      Ah, that explains it. Nice idea!
- greggyb 9y ago
  Perhaps something you've already considered, but if you're going to spend time on automation of sweeping extra out of checking, you might want to put that into some index funds rather than a savings account to optimize on return.
cup-of-tea 9y ago
I've been working on my own classifier for updating my ledger because I struggled too much trying to get other people's stuff to work. I'm getting about ~90% accuracy using a SVM from scikit-learn, but it can only ever be semi-automated. My plan is that every month I'll go through my various accounts and help to update my ledger. One thing that I don't think other software does is automatically remove the duplication when transferring between bank accounts/credit cards. How does yours deal with transfers between accounts? I really want a nice auto-completing thing to type in the account names when the classifier gets them wrong (or to add a new account if necessary). The best UI for this is ido or ivy in emacs. But it would be nice to have a tool that runs outside emacs. The best I've found is fzf but it doesn't support tab completion which is a shame.
- marvinpinto 9y ago
  I handle transfers between accounts manually - as in when I see duplicates show up, I remove one of them. A neat thing that ledger-reconciler does is to print the balance (as listed on the banking website) for each account. That information combined with Ledger's balance report tells me immediately whether something is off and if investigation is needed. This in itself has saved me so much time that I can deal with manually editing the occassional duplicate entry (internal account transfers). I also keep all my ledger data in git so it's very easy to see what has changed, etc. Have you looked at the reckon gem[0] for auto-classification? That is basically what this program uses. [0]: https://github.com/cantino/reckon https://github.com/cantino/reckon
  - cup-of-tea 9y ago
    I tried reckon amongst other things but couldn't get it to read my CSV files properly. I got frustrated enough that I thought writing my own would be easier. Putting the account balances in the ledger file is certainly a good idea. At least one of my bank CSVs provides the balances with each transaction. It doesn't seem too hard to automatically merge the transfers between accounts, though. But maybe there is a difficult I haven't thought of yet. Where there will definitely need some manual tweaking is on the those transactions where ledger really shines: many to many transactions. Have you built in some functional to flag a transaction as TODO when you know it will need manual tweaking later?

I automated downloading and reconciling my financials from all my bank accounts. https://disjoint.ca/projects/ledger-reconciler https://disjoint.ca/projects/ledger-reconciler

bberenberg 9y ago

This is great, would love to see support for more data sources.

marvinpinto 9y ago
Thanks! Unfortunately I can only add support for things I have access to (bank accounts and such) and I do plan on adding a few more. To work around this I tried to make it as developer-friendly as I could so we'll see!
burkemw3 9y ago
ledger-autosync [0] does similar things with more data sources available [0]: https://github.com/egh/ledger-autosync https://github.com/egh/ledger-autosync

throwaway613834 9y ago

Holy cow, I've been looking for this but for the US. Anybody know of any?

marvinpinto 9y ago
If you're up for it, you can add your own bank/creditcard company/whatever as a plugin to it - I think the examples there are decent. I could also help if need be!
- throwaway613834 9y ago
  Haha thanks! The problem is in the US most banks would need screen-scraping; they don't have APIs, so it's a lot more work to do such a thing.
  - marvinpinto 9y ago
    ledger-reconciler uses headless chrome underneath to screen-scrape all its information from banking/creditcard websites. That's how all the plugins hook into it. It does not use any public or private banking APIs. Hope that helps!
    - throwaway613834 9y ago
      Oh dang I didn't know! I'll check it out at some point then, thanks! :)
  - jldugger 9y ago
    Many actually do have APIs, they're just poorly documented in public.
  - noughth 9y ago
    Many US banks support OFX (though they may call it QFX, DirectConnect, or simply advertise Quicken support). The OFX spec is a bit of a pain, but there are a number of open-source libraries to query/parse it. I wrote one in Go: https://github.com/aclindsa/ofxgo https://github.com/aclindsa/ofxgo
jldugger 9y ago
Transaction pull can be done in GNUCash with aqbanking. It's not fully automated, as it's a GUI and there's a manual review step. And also the aqbanking interface is terribad.
jpeeler 9y ago
This might be helpful, though I haven't looked at it in a very long time: https://github.com/madhat2r/plaid2text https://github.com/madhat2r/plaid2text Essentially it is some python to download transactions using the plaid API (and requires an account to do so).

jldugger 9y ago

On a related note, I have gnucash set up to autocreate a ton of transactions 90 days in advance. Rent, salary, typical bills, student loan repayment, etc. At some point my plan is to automate the following tasks: 1. Sweep checking into savings based on forecasted transactions. Useful since interest is like 10x more in savings. 2. Update asset values daily. 3. Update autocreated transactions as the invoices are emailed to me. 4. Calculate a rebalancing strategy that factors in allowed rollovers and expense ratios.

mschuster91 9y ago
> On a related note, I have gnucash set up to autocreate a ton of transactions 90 days in advance. Rent, salary, typical bills, student loan repayment, etc. Why do you need to do the transactions? In Germany, I can simply create a recurring transaction for rent and loan repayment, and utility/phone bills are directly debited from my bank account.
- jldugger 9y ago
  You misunderstand. They're already scheduled largely. This is about writing down in the books in January that I'm going to spend in March. GNUCash's summary view has a column 'future minimum' for each account that uses this data; looking at it informs you when you might go negative without corrective action. Or conversely, tells you how much money you can safely shift out of checking.
  - mschuster91 9y ago
    Ah, that explains it. Nice idea!
greggyb 9y ago
Perhaps something you've already considered, but if you're going to spend time on automation of sweeping extra out of checking, you might want to put that into some index funds rather than a savings account to optimize on return.

cup-of-tea 9y ago

I've been working on my own classifier for updating my ledger because I struggled too much trying to get other people's stuff to work. I'm getting about ~90% accuracy using a SVM from scikit-learn, but it can only ever be semi-automated. My plan is that every month I'll go through my various accounts and help to update my ledger. One thing that I don't think other software does is automatically remove the duplication when transferring between bank accounts/credit cards. How does yours deal with transfers between accounts? I really want a nice auto-completing thing to type in the account names when the classifier gets them wrong (or to add a new account if necessary). The best UI for this is ido or ivy in emacs. But it would be nice to have a tool that runs outside emacs. The best I've found is fzf but it doesn't support tab completion which is a shame.

marvinpinto 9y ago
I handle transfers between accounts manually - as in when I see duplicates show up, I remove one of them. A neat thing that ledger-reconciler does is to print the balance (as listed on the banking website) for each account. That information combined with Ledger's balance report tells me immediately whether something is off and if investigation is needed. This in itself has saved me so much time that I can deal with manually editing the occassional duplicate entry (internal account transfers). I also keep all my ledger data in git so it's very easy to see what has changed, etc. Have you looked at the reckon gem[0] for auto-classification? That is basically what this program uses. [0]: https://github.com/cantino/reckon https://github.com/cantino/reckon
- cup-of-tea 9y ago
  I tried reckon amongst other things but couldn't get it to read my CSV files properly. I got frustrated enough that I thought writing my own would be easier. Putting the account balances in the ledger file is certainly a good idea. At least one of my bank CSVs provides the balances with each transaction. It doesn't seem too hard to automatically merge the transfers between accounts, though. But maybe there is a difficult I haven't thought of yet. Where there will definitely need some manual tweaking is on the those transactions where ledger really shines: many to many transactions. Have you built in some functional to flag a transaction as TODO when you know it will need manual tweaking later?