6 ms·
> It is also worth mentioning there has never been a single case of fraud or loss attributed to "screen-scraping" This response makes me angry. Every service w
by viraptor 9y ago
> It is also worth mentioning there has never been a single case of fraud or loss attributed to "screen-scraping"
This response makes me angry. Every service worth attacking will have security problems at some point. You're running a store of bank credentials, which you have to have access to (as opposed to password managers for example which can store user encrypted data). Given enough time, one of these services will get hacked and "this has never happened before" is not going to be a good answer. Someone will be the first one.
I'm happy your service will push more banks to provide APIs. But I'm already doing bank screen scraping for myself because I don't trust services which require my credentials. I hope people consider that risk seriously.
- djsumdog 9y agoThis is what always worried me about services like Mint.
- curun1r 9y agoMint is probably the one you don't have to worry about. The service that stores credentials and scrapes bank sites is the same one that powers TurboTax and QuickBooks. Intuit spends a lot of money keeping that service secure and they're well aware of how catastrophic a breech would be. In short, that service is run out of private datacenters, not publicly available. You'd have to compromise not just Mint/TurboTax/QuickBooks but then the FICDS service, which only has methods that take a token representing the credentials and passes back transactions, not the actual credentials. It's the competitors to Mint who can't afford their own datacenters and don't have the many millions of dollars to put into securing their service the way Intuit does that should be much more worrisome. Disclaimer: I used to work at Intuit, though not on any of the products I mentioned. But I have had many conversations with the people that run Mint and FICDS and I know the overall architecture used. I give my bank credentials to Mint without any worry. It would probably require a state-level entity to get in, and they'd likely need to get someone hired into that group to facilitate it. The service is very secure.
- tempestn 9y agoI don't know, if the general level of bug-ridden-ness of their products is any indication, I wouldn't put much stock in their security. There are glaring bugs in Quickbooks for example that persist year after year, having been reported repeatedly, even as they continue to release a new version every single year with various visual tweaks and seemingly not much else.
- mfrye0 9y agoAgreed. I used to work for a division of Intuit (not financial stuff) and a lot of the dev work I saw was pretty shoddy. Lots of weird bugs constantly. With that said, security is taken very seriously and most of the stuff I experienced was UI related. So hard to say...
- curun1r 9y agoQuickBooks, even the online version, are decades old and have major issues that prevent the teams from getting much done. They took a year to do clean-up and basically didn't introduce any new features. Not much got done just due to the complexity of changing anything without causing problems. FICDS, on the other hand, was spun up more recently and is in much better shape. I wouldn't look at an application like QBO and see much that would inform a behind the scenes service like FICDS. The more interesting product, to me, is TurboTax. The seasonality of it allows Intuit to basically rebuild it from the ground up every year. It's really a different mindset on the San Diego campus (where TT is developed) and in Mountain View (where QBO is developed). San Diego is much more willing to take (non-security) risks with the product because small changes can add up to big wins. I remember a talk by one of the guys in charge of A/B testing on TT who said that a tenth of a percentage point increase in conversion rate was good for tens of millions in added profit. Contrast that with QuickBooks, where the difficulty to use is actually a feature. Accountants spend years learning the app and learning the work arounds for those bugs you mentioned. That knowledge and experience becomes a barrier to entry into the field and a job skill that they're compensated for. The result is that QB/QBO is aimed at accountants with years of experience in the product. Their livelihood depends on it and they don't like change. So the teams there do as little as possible that can cause problems and know that they'll still get good reviews if the get almost nothing done so long as they don't cause problems. It's deeply disfunctional and yet explains why you can put a lot of trust in FICDS. They too have had bugs for years. For example, they don't save cookies between scraping sessions and so are always an untrusted browser to the banks. I have at least two accounts that constantly send me 2FA tokens whenever Mint tries (and fails) to sync. But no one gets fired for not fixing bugs. You get fired for compromising the security or stability of the service, and the easiest way to not do that is to push as little possibly vulnerable code to prod. It's the anti-Zuckerberg environment...move slow and don't break things.
- fweespeech 9y agoTbh, its only worth considering for pure "credit card" services where its essentially read-only already and any damage can be attributed to fraud you can perform a chargeback against. That being said, if you run all your spending through your credit cards except obvious notables you update manually you are golden. (i.e. Rent/Mortgage)
- d0while 9y agoMe too.
- gregable 9y agoIt would be interesting if there were a way to run a local client that holds your credentials and does the screen scraping to send back to this API (or others).
- viraptor 9y agoTechnically, they could provide a self-hosted version. But that may not be a very profitable business. Also, screen scraping needs continuous updating as the bank's interface changes.
- thwarted 9y agoI believe Wesabe had an implementation that did this. From http://blog.precipice.org/why-wesabe-lost-to-mint/ http://blog.precipice.org/why-wesabe-lost-to-mint/ Wesabe built our own data acquisition system, first using downloadable client programs (partially because that was easier and partially to preserve users’ privacy)
- echion 9y ago> I'm already doing bank screen scraping for myself Any tools you recommend, or is it all hand-rolled?
- viraptor 9y agoThe best option I found (for read-only access) was chromedriver > download CSV statement > process that into your own database. Using minimal standard screen scraping will be hard because most bank apps are a complicated mess of client and server side processing using ancient tech. After many tries, I just used python+chromedriver and that's doable in ~100 loc per bank. CSV statements because OFX and others are a complete mess in most cases. I haven't seen a single compliant and correct file so far. (This may be AU specific) Own database / spreadsheet so you have a local copy you can easily filter on. Obtaining specific date range from a bank may be an interesting challenge. I default to importing last 30 days every day with duplicates detected using a hash(date-description-amount-idx) where idx increases for every date-desc-amount duplicate. So far I'm happy. Actually I'm in the process of writing a post about it - will submit sometime next week most likely.
- mkurz 9y agoLooking forward to that blog post since we need to solve something similar.
- deejbee 9y agoComing from a slightly different angle but it may be of use, I wrote a chrome browser extension that automates logging into banks and other financial sites. It also scrapes the balances once logged in to provide a net worth total. It currently works on around 25 banks (mainly UK). You can also build your own logins and all passwords/private data are stored encrypted in the browser. I wrote it as I just spend too much time digging out passwords all the time and the usual crowd like LastPass are a) stored online and b) fail to login automatically to complex input sequences like banks.
- smichael 9y agoI hand-rolled a wells fargo scraper recently using the haskell webdriver package and Selenium. It was working fairly well until they started requiring a captcha - something I've never seen as a human user. I'm not quite sure what triggered that or how to get around it.
- BrandiATMuhkuh 9y agoI like how Jude solved the problem of the password. They do all the scraping on the phone. So no password is stored on their servers. https://blog.jude.io https://blog.jude.io