5 ms·

As for automatically pulling transactions, it's still shocking to me that anyone handed over their banking username and password to a third party, quicken et al

As for automatically pulling transactions, it's still shocking to me that anyone handed over their banking username and password to a third party, quicken et al at least use a system now where the bank grants read only permission as an app integration, but still I'm not a fan of every bank transaction becoming someone else's anonymized data harvest. I'm evaluating whether self hosting accounting software is at all feasible to meet CMMC requirements and so far have my sights set on ERPNext. I configured my bank to send me an email alert for every transaction and intend to parse that and appene it to my ledger, but the API to do so is fairly annoying, so hearing that beancount is meant to be automatable is intriguing indeed BTW the other thing that shocks me about quicken classic is lack of version control - the database can become corrupt and your only recourse is restoring from backup or sending the file to support and having them manually fix it!

evrimoztamur 2y ago
The space was absolutely insane prior to PSD2 in Europe. Luckily most banks now have API endpoints, but there's a new class of middlemen wrangling the APIS: the aggregators who pipe all your data from the banks to apps. It's better than handing out the keys to the old class of scraping aggregators, I suppose, but the outcome is still not that much better in my opinion.
FredPret 2y ago
I automated the flow of transaction -> text message -> MacOS -> Beancount once, but it never added up. - some transactions go through without a text - some transactions generate two identical texts - for Forex transactions, the amount in the text and the amount on your statement will not match - some texts are ambiguous in that they could have been generated by two different kinds of transaction; especially true of deposits & credit card payments In the end I gave up, accepted that accounting will never be smooth & simple, and now just generate a CSV every month by hand.
Wronnay 2y ago
Do you plan to open source or sell your Email to ledger / ERPNext solution? I actually plan the exact same thing for my use cases...
Helmut10001 2y ago
If you are in Europe, there is a very nice OSS software called Hibiscus [1] with APIs-connectivity for most banks, and (local) web scrapers for those banks that have no APIs. I am working on a plugin to pull categories and transactions from the Hibiscus DB (H2 or PG) to Beancount [2]. It is not there yet, but the process overall looks promising. [1]: https://www.willuhn.de/ https://www.willuhn.de/ [2]: https://github.com/Sieboldianus/beancount-hibiscus-importer https://github.com/Sieboldianus/beancount-hibiscus-importer
- amaccuish 2y ago
  Another Hibiscus fan. I'm pretty sure FinTS/HBCI is a mostly only used here in Germany sadly. Which is a shame because all this "OpenBanking" stuff requires registering/paying/certificates etc.
- woolion 2y ago
  https://github.com/willuhn/hibiscus https://github.com/willuhn/hibiscus >Developer Information (in german) Which sends to a Javadoc which is actually a 404. To be honest, I wouldn't work on/with a project that uses not-English as primary language (even if it was my own). Maybe a bit of cleanup and documentation would help make it more accessible.
  - Helmut10001 2y ago
    I agree. However, in those niche areas where the software is the only choice available, I can make compromises. This software has been developed over 10+ years, and many conventions did not exist that long ago. It's still great that the software exists (because there is no alternative).
    - woolion 2y ago
      Sure, I did not want to be overly dismissive; I've worked in the space and most options suck and/or are fairly expensive for what they are, so any decent to good OSS option would be really appreciated. However, these points would make for a hard sell. Actually, a client is migrating their system and needs to redo their bank synchronization. Shall I try with Google Translate, or is there someone who would also work on updating some of the resources to English?
human_llm 2y ago
ERPNext is fairly easy to automate. We have a number of Python scripts that use the ERPNext API to create sales invoices, add and reconcile transactions from PayPal, Stripe, etc. We originally used Quickbooks Online and I'm glad we decided to switch to ERPNext a few years ago.
fangpenlin 2y ago
Hi, the author here. If you are okay with Plaid[1], many of their bank connections are now using OAuth-style authentication instead of password sharing. I actually added a new feature called Direct Connect[2] a while back to allow any plaintext accounting book users to pull CSV directly via Plaid API through BeanHub. We don't train AI models with our customers' transactions, and if we want to, we will ask for explicit consent (not just ToS) and anonymize customers' data. If you're okay with the above, the key to achieving a high automation level is the ability to pull CSV transaction files directly from the bank in a standard format. Maybe you can give it a try. We have 30 days free trial period. I am not so familiar with the CMMC requirements, as you mentioned, but for us to access transactions from some banks, such as Chase, Plaid requires us to pass an auditing process about our security measurements. Is the CMMC compliance your company needs to meet to take a third-party software vendor into considerations? [1]: https://plaid.com https://plaid.com [2]: https://beanhub.io/blog/2025/01/16/direct-connect-repository/ https://beanhub.io/blog/2025/01/16/direct-connect-repository...
ashish01 2y ago
With a one-time setup, a low-fi solution is to receive an email from your bank for every transaction and extract the data into a ledger entry.
walterbell 2y ago
> not a fan of every bank transaction becoming someone else's anonymized data harvest In 2024, CFPB had mandated all U.S. banks to open up data history to SaaS/fintech vendors, if the customer gives permission.
- dspillett 2y ago
  And that customer (me too) is not a fan of the idea of giving permission for every bank transaction to become part of some vendor's data harvest. The banks having to do it if we want them to, doesn't not mean we have to want them to.
  - robertlagrant 2y ago
    > if the customer gives permission
    - dspillett 2y ago
      I took from you replying “not a fan of every bank transaction becoming” with the fact that the banks have to give access at the customer's request, and the requirement somehow impacted the “not a fan” because it is not optional. > if the customer gives permission Of course once permission has been given, it is part of that dataset (whether or not that is made clear by the app) and numerous other datasets (unless the law requires companies not share that data, and companies all fully obey that law, and they don't have any little data accidents).
patmcc 2y ago
>>>quicken et al at least use a system now where the bank grants read only permission as an app integration ...not always, they don't. Some financial institutes provide this, some do not. I know very specifically that some FIs do not have any way to grant read-only access, and that aggregators like Plaid, Flinks, Quicken, etc. will simply fallback on good-old-fashioned "give us your username/password and we'll screen scrape your account info". I can personally attest to this being true in 2025 in Canada, at least.

evrimoztamur 2y ago

The space was absolutely insane prior to PSD2 in Europe. Luckily most banks now have API endpoints, but there's a new class of middlemen wrangling the APIS: the aggregators who pipe all your data from the banks to apps. It's better than handing out the keys to the old class of scraping aggregators, I suppose, but the outcome is still not that much better in my opinion.

FredPret 2y ago

I automated the flow of transaction -> text message -> MacOS -> Beancount once, but it never added up. - some transactions go through without a text - some transactions generate two identical texts - for Forex transactions, the amount in the text and the amount on your statement will not match - some texts are ambiguous in that they could have been generated by two different kinds of transaction; especially true of deposits & credit card payments In the end I gave up, accepted that accounting will never be smooth & simple, and now just generate a CSV every month by hand.

Wronnay 2y ago

Do you plan to open source or sell your Email to ledger / ERPNext solution? I actually plan the exact same thing for my use cases...

Helmut10001 2y ago

If you are in Europe, there is a very nice OSS software called Hibiscus [1] with APIs-connectivity for most banks, and (local) web scrapers for those banks that have no APIs. I am working on a plugin to pull categories and transactions from the Hibiscus DB (H2 or PG) to Beancount [2]. It is not there yet, but the process overall looks promising. [1]: https://www.willuhn.de/ https://www.willuhn.de/ [2]: https://github.com/Sieboldianus/beancount-hibiscus-importer https://github.com/Sieboldianus/beancount-hibiscus-importer

amaccuish 2y ago
Another Hibiscus fan. I'm pretty sure FinTS/HBCI is a mostly only used here in Germany sadly. Which is a shame because all this "OpenBanking" stuff requires registering/paying/certificates etc.
woolion 2y ago
https://github.com/willuhn/hibiscus https://github.com/willuhn/hibiscus >Developer Information (in german) Which sends to a Javadoc which is actually a 404. To be honest, I wouldn't work on/with a project that uses not-English as primary language (even if it was my own). Maybe a bit of cleanup and documentation would help make it more accessible.
- Helmut10001 2y ago
  I agree. However, in those niche areas where the software is the only choice available, I can make compromises. This software has been developed over 10+ years, and many conventions did not exist that long ago. It's still great that the software exists (because there is no alternative).
  - woolion 2y ago
    Sure, I did not want to be overly dismissive; I've worked in the space and most options suck and/or are fairly expensive for what they are, so any decent to good OSS option would be really appreciated. However, these points would make for a hard sell. Actually, a client is migrating their system and needs to redo their bank synchronization. Shall I try with Google Translate, or is there someone who would also work on updating some of the resources to English?

human_llm 2y ago

ERPNext is fairly easy to automate. We have a number of Python scripts that use the ERPNext API to create sales invoices, add and reconcile transactions from PayPal, Stripe, etc. We originally used Quickbooks Online and I'm glad we decided to switch to ERPNext a few years ago.

fangpenlin 2y ago

Hi, the author here. If you are okay with Plaid[1], many of their bank connections are now using OAuth-style authentication instead of password sharing. I actually added a new feature called Direct Connect[2] a while back to allow any plaintext accounting book users to pull CSV directly via Plaid API through BeanHub. We don't train AI models with our customers' transactions, and if we want to, we will ask for explicit consent (not just ToS) and anonymize customers' data. If you're okay with the above, the key to achieving a high automation level is the ability to pull CSV transaction files directly from the bank in a standard format. Maybe you can give it a try. We have 30 days free trial period. I am not so familiar with the CMMC requirements, as you mentioned, but for us to access transactions from some banks, such as Chase, Plaid requires us to pass an auditing process about our security measurements. Is the CMMC compliance your company needs to meet to take a third-party software vendor into considerations? [1]: https://plaid.com https://plaid.com [2]: https://beanhub.io/blog/2025/01/16/direct-connect-repository/ https://beanhub.io/blog/2025/01/16/direct-connect-repository...

ashish01 2y ago

With a one-time setup, a low-fi solution is to receive an email from your bank for every transaction and extract the data into a ledger entry.

walterbell 2y ago

> not a fan of every bank transaction becoming someone else's anonymized data harvest In 2024, CFPB had mandated all U.S. banks to open up data history to SaaS/fintech vendors, if the customer gives permission.

dspillett 2y ago
And that customer (me too) is not a fan of the idea of giving permission for every bank transaction to become part of some vendor's data harvest. The banks having to do it if we want them to, doesn't not mean we have to want them to.
- robertlagrant 2y ago
  > if the customer gives permission
  - dspillett 2y ago
    I took from you replying “not a fan of every bank transaction becoming” with the fact that the banks have to give access at the customer's request, and the requirement somehow impacted the “not a fan” because it is not optional. > if the customer gives permission Of course once permission has been given, it is part of that dataset (whether or not that is made clear by the app) and numerous other datasets (unless the law requires companies not share that data, and companies all fully obey that law, and they don't have any little data accidents).

patmcc 2y ago

>>>quicken et al at least use a system now where the bank grants read only permission as an app integration ...not always, they don't. Some financial institutes provide this, some do not. I know very specifically that some FIs do not have any way to grant read-only access, and that aggregators like Plaid, Flinks, Quicken, etc. will simply fallback on good-old-fashioned "give us your username/password and we'll screen scrape your account info". I can personally attest to this being true in 2025 in Canada, at least.