4 ms·

While we're on the topic, does anyone know where one might find scripts to scrape bank statements so you don't have to download them manually every month? (This

While we're on the topic, does anyone know where one might find scripts to scrape bank statements so you don't have to download them manually every month? (This is one thing I would find headless browsers useful for...)

pricechild 9y ago
Every UK bank I've used allows some format of csv/qif/ofx export. Are you in the US?
- wfunction 9y ago
  The format isn't the issue. The problem is I want it to be API-friendly so I don't even have to think about it; my system should download it automatically. But yes, I'm talking about the US.
- vidarh 9y ago
  They generally do. But many UK banks make it hard to automate things. Either insisting on 2FA in all cases, or having a secondary login without 2FA that only gives very limited access. Some way of authorising API access to read-only access to things like statements would be fantastic, to the extent that I'd consider changing banks over it, if you know of any UK banks that offer it.
  - thibaut_barrere 9y ago
    As well, most bank OFX/CSV exports I've dealt with are truncated in some ways (e.g. truncated labels), which make it harder to really leverage sometimes.
  - kingosticks 9y ago
    Monzo will be offering current accounts soon. https://monzo.com/blog/2017/04/05/banking-licence/ https://monzo.com/blog/2017/04/05/banking-licence/
  - pricechild 9y ago
    Ahh of course. Apologies for missing that.
- lightbyte 9y ago
  My bank offers that, but it costs I believe ~$15 a month.
vidarh 9y ago
If your authentication is basic enough, it should be a problem to write one. The problem with a lot of banks is things like two-factor auth to do anything. If I didn't have a mortgage locked in at a crazily great rate, I'd consider changing banks just to get one that'll let me automate statement downloads. The alternative is to build a device to press buttons on their 2FA device... Come to thing of it that might be a fun hack.
- wfunction 9y ago
  2FA isn't my issue here. Actually downloading the statements is. Lots of banks use JS, some go through really weird hoops getting you the PDF that are difficult for non-experienced people to automate. Stuff like JS in embedded iframes that generate the link on the fly and open a new tab that you have to navigate. It's hard to accurately detect all the links and handle things like "Next Page" and so on, especially for more than one bank. It's quite nontrivial.
  - vidarh 9y ago
    If you don't have 2FA issues, then while I agree it's non-trivial, it's certainly doable with a headless browser. But yes, I'd love for there to be simpler ways to do this in general.
- sjtgraham 9y ago
  Who do you bank with? I might be able to help.
thibaut_barrere 9y ago
Boobank (http://weboob.org/applications/boobank http://weboob.org/applications/boobank) is such a collection of scripts (althought mostly centered on french banks).
RandomBookmarks 9y ago
>Scrape bank statements... There is Kantu: https://www.a9t9.com/kantu/web-automation https://www.a9t9.com/kantu/web-automation It uses screenshots and OCR to automate web browsing and scraping, to you do not even have to "touch" the DOM. What you do is you simply draw a frame around the areas that you need to have extracted and OCR'ed. It also works with PDFs.
sjtgraham 9y ago
For UK banks check out https://teller.io/ https://teller.io/ for an API to your bank account (Disclosure: my company)

pricechild 9y ago

Every UK bank I've used allows some format of csv/qif/ofx export. Are you in the US?

wfunction 9y ago
The format isn't the issue. The problem is I want it to be API-friendly so I don't even have to think about it; my system should download it automatically. But yes, I'm talking about the US.
vidarh 9y ago
They generally do. But many UK banks make it hard to automate things. Either insisting on 2FA in all cases, or having a secondary login without 2FA that only gives very limited access. Some way of authorising API access to read-only access to things like statements would be fantastic, to the extent that I'd consider changing banks over it, if you know of any UK banks that offer it.
- thibaut_barrere 9y ago
  As well, most bank OFX/CSV exports I've dealt with are truncated in some ways (e.g. truncated labels), which make it harder to really leverage sometimes.
- kingosticks 9y ago
  Monzo will be offering current accounts soon. https://monzo.com/blog/2017/04/05/banking-licence/ https://monzo.com/blog/2017/04/05/banking-licence/
- pricechild 9y ago
  Ahh of course. Apologies for missing that.
lightbyte 9y ago
My bank offers that, but it costs I believe ~$15 a month.

vidarh 9y ago

If your authentication is basic enough, it should be a problem to write one. The problem with a lot of banks is things like two-factor auth to do anything. If I didn't have a mortgage locked in at a crazily great rate, I'd consider changing banks just to get one that'll let me automate statement downloads. The alternative is to build a device to press buttons on their 2FA device... Come to thing of it that might be a fun hack.

wfunction 9y ago
2FA isn't my issue here. Actually downloading the statements is. Lots of banks use JS, some go through really weird hoops getting you the PDF that are difficult for non-experienced people to automate. Stuff like JS in embedded iframes that generate the link on the fly and open a new tab that you have to navigate. It's hard to accurately detect all the links and handle things like "Next Page" and so on, especially for more than one bank. It's quite nontrivial.
- vidarh 9y ago
  If you don't have 2FA issues, then while I agree it's non-trivial, it's certainly doable with a headless browser. But yes, I'd love for there to be simpler ways to do this in general.
sjtgraham 9y ago
Who do you bank with? I might be able to help.

thibaut_barrere 9y ago

Boobank (http://weboob.org/applications/boobank http://weboob.org/applications/boobank) is such a collection of scripts (althought mostly centered on french banks).

RandomBookmarks 9y ago

>Scrape bank statements... There is Kantu: https://www.a9t9.com/kantu/web-automation https://www.a9t9.com/kantu/web-automation It uses screenshots and OCR to automate web browsing and scraping, to you do not even have to "touch" the DOM. What you do is you simply draw a frame around the areas that you need to have extracted and OCR'ed. It also works with PDFs.

sjtgraham 9y ago

For UK banks check out https://teller.io/ https://teller.io/ for an API to your bank account (Disclosure: my company)