21 ms·
The Secret API of Banks
- dewey 8y agoFor those in the EU there's something interesting coming next year, banks need to provide open API to interact with each other: https://thenextweb.com/worldofbanking/2018/06/27/openbanking/ https://thenextweb.com/worldofbanking/2018/06/27/openbanking... Already right now in Germany there are a lot of banks that share a common API format which is why there are a lot of banking apps where you can just log into your bank and don't need bank specific apps. It's called HBCI / FinTS (https://en.wikipedia.org/wiki/FinTS https://en.wikipedia.org/wiki/FinTS) and it's great to have that possibility without doing some web-scraping to get your data out. One example of another german bank API would be: https://api-docs.fidor.de/v1/introduction/welcome-text https://api-docs.fidor.de/v1/introduction/welcome-text
- AhtiK 8y agoIs this new initiative strictly for an API between the official banks or there's a chance that a regular company or, let's say a private person, can use it to access its own data & submit transfers? In a related note, I do hope that the German "lastschriftmandat"/direct debit gets also replaced with something more transparent and with a better control. Similar to an e-invoice concept used by a few other countries like Estonia.
- TomMarius 8y agoIt's not completely public, but it's open to all companies and people. Your application needs to meet certain requirements and needs to be approved etc.
- kozziollek 8y agoSimilar thing in Poland: https://polishapi.org/en/ https://polishapi.org/en/
- Rjevski 8y agoPSD2 and “open” banking is bullshit. I wish this myth would die - it is anything but “open”. If you want to gain access to APIs, you need to become an “AISP” (as they are called in the UK), this requires a certification and a load of other nonsense akin to PCI-DSS. This is for read-only access - for “write” access including the ability to edit payees or make payments you need to become a “PISP” which I assume requires even more paperwork. I also said APIs because the regulation does not mandate any kind of API format, so every bank has their own with different capabilities as far as what data is returned and in which format. Some of them are truly awful. And finally, “open” banking still does not allow you to get a personal access token for your own account.
- solenya 8y ago"...the regulation does not mandate any kind of API format.." That isn't true. Anybody which wants to be in the UK Open Banking Group (mandatory for the 9 biggest banks) must follow these very strict specs: https://openbanking.atlassian.net/wiki/spaces/DZ/pages/16385802/Specifications https://openbanking.atlassian.net/wiki/spaces/DZ/pages/16385...
- sjtgraham 8y agoNot true. The spec is loose, many fields are not mandatory, and the "conformance suite" only tests the security profile, not anything related to the schema. Even with that minimal level of scrutiny as of last month none of the CMA9 were compliant with the CS.
- pbreit 8y agoI've always kind of wondered if XML/JSON delivered over SMTP was a viable webhook approach?
- Splendor 8y agoI'm not sure I'd feel better about having every single transaction in email. That doesn't seem any more secure to me.
- packet_nerd 8y agoConcerning, but still much better than giving a third party your username and password which could be used to initiate a transfer or something.
- dewey 8y agoThat really depends on your threat model. If someone already has access to your email account it's game over anyway most of the time. I don't see much of a problem there if you are not worried about your ISP / mail provider taking your data.
- driverdan 8y agoSo long as the bank's email server connects directly to your email server over an encrypted connection it's just as secure as anything else.
- eboyjr 8y agoThis is an awesome alternative to scraping your own bank transactions. I'm considering using this technique to automatically add rows to my personal GnuCash MySQL database.
- beamatronic 8y agoIs there a good existing framework that can turn emails into events? Some sort of IMAP client ?
- mgamble 8y agoSendgrid has an inbound parse feature (https://sendgrid.com/docs/User_Guide/Settings/parse.html https://sendgrid.com/docs/User_Guide/Settings/parse.html) which you can use to start building a framework - it takes any inbound email to a domain and forwards it as an HTTP request to an endpoint you specify. Used it for a few projects and its worked great.
- jsjohnst 8y agoName your language of choice and I’ll link you to a good one.
- unl 8y agoClojure
- jsjohnst 8y agohttps://github.com/owainlewis/clojure-mail/ https://github.com/owainlewis/clojure-mail/
- beamatronic 8y agoJava
- jsjohnst 8y ago
- lgregg 8y agoI'm definitely going to check this out for my credit union.
- bluetidepro 8y agoI was expecting this to outline how you could actually use various APIs from different banks, but was still pleasantly happy with the actual content of the article. This is a clever idea I had never thought about doing. Kudos to OP!
- fleetside72 8y ago"please enter a value greater than 0" lol oh well
- nbap 8y agoYeah, I've been doing this for a couple of months now. I just happen to use a combination of Gmail filters, IFTTT actions to post to a webhook of a custom app hosted for free at Heroku to parse and send to Google Spreadsheets. It's been serving me well, definitely better than my bank's app.
- win_ini 8y agoI'm sorry, I'm not sure I heard you right. Did you say you can build a custom app but still use IFTTT? Here's a link to http://Zapier.com http://Zapier.com
- nbap 8y agoThanks but I'm aware of Zapier. The custom app is fairly easy to maintain, I can keep track of changes and it provides total flexibility for my needs in terms of parsing and how to store the data.
- driverdan 8y agoI downvoted you because you seem to be implying that programmers shouldn't use IFTTT. There's nothing wrong with IFTTT if it meets your needs.
- netsharc 8y agoNot original commenter, I used to use and like them, but at one point they changed all URLs to use their shortener (I was using it to archive all links I post on FB into an Evernote note). Great, they want to do more analytics and they're making sure all URLs in content they're supposed to be just plumbing will die with them, very customer friendly. And then there's this: https://blog.pinboard.in/2016/03/my_heroic_and_lazy_stand_against_ifttt/ https://blog.pinboard.in/2016/03/my_heroic_and_lazy_stand_ag...
- ccarse 8y agoOP, could you share the code for this?
- nodesocket 8y agoAgree, open sourcing the e-mail parsing and pandas code would be amazing!
- mail2vks 8y agoAgree. Upvoting need for open sourcing.
- gduverger 8y agoI've been getting a good amount of requests for open sourcing it. It would require a few hours of work but I might take the time one of those weekends. Please feel free to send me an email (georges.duverger@gmail.com) if you'd like me to keep you updated.
- edgartaor 8y agoAgree. I'm interested in how he generate those reports.
- loisaidasam 8y agoI tried doing something like this, but it didn't work for Bank of America, as their minimum alert level is $100. When I lived in Slovenia, I could set SMS alerts from my bank there (UniCredit). I wrote an Android app [1] that intercepted those SMS messages and parsed out the relevant bits. Those were the days... [1] https://github.com/loisaidasam/poor-mans-money-counter https://github.com/loisaidasam/poor-mans-money-counter
- djrogers 8y ago> I wrote an Android app [1] that intercepted those SMS messages and parsed out the relevant bits. Sounds like a nice hack, but at a meta level, I find the idea that a mobile platform even allows an app to ‘intercept’ SMS messages to be a little troubling.
- TomMarius 8y agoWell, it doesn't. You have to do a lot of things in order to allow it that it's impossible to not know exactly what you're doing. I think it's positive that I am in control of my device when I want to.
- jsjohnst 8y agoActually there are quite a few popular Android apps that bulk harvested SMSes, so your point in defense of its security is rather weak.
- misterbwong 8y agoThis is very clever but makes me sad. It’s 2018 and the best, cleanest way of monitoring and storing my own transactions programmatically is by scraping an email.
- orf 8y ago*in America Banks provide an API in Europe. In fact it's a legal requirement that's coming into force in 2019, and there are a lot of 'mobile-first' banks like Monzo and Revolut which make this entirely un-needed in the first place (providing spending exports, decent analytics, push notifications, etc etc). Welcome to the future. Contact your local politician if you want to join us. Maybe also ask about chip and pin while you're at it!
- bhhaskin 8y agoSure, but as other users like Rjevski have pointed out the PSD2 API is going to be nearly impossible for the common person to use. Unless you want to become pretty much PCI-DSS certified.
- orf 8y agoSo you're surprised or concerned that applications that wish to extract financial data abour consumers programmatically from their bank accounts need to conform to regulations regarding the storage of said data? Oh no. What a travesty.
- jsjohnst 8y agoFrom GP: > API is going to be nearly impossible for the common person to use Reading what you are replying to and making an argument against that rather than a hypothetical straw man that GP didn’t imply or mean will get you further.
- orf 8y agoApologies, I misread it as him complaining that small, one man startups will not be able to compete due to the necessary regulatory burden.
- timvisee 8y agoI'm so happy to be a bunq user (which is a Dutch bank), they provide a fully featured and easy to work with API to do and build anything you want. Other banks should follow.
- VikingCoder 8y agoI hate sounding like a VC jerk, but the banking industry needs some serious disruption.
- dddddaviddddd 8y agoParticularly in user-facing aspects
- lgregg 8y agoI just had a 45 min talk with a rep of my credit union; She agrees. They don't have per transaction alerts.
- TomMarius 8y agoWhat user facing aspects bug you the most?
- TuringNYC 8y agoI'd love: - Universal, near-instantaneous per transaction alerts. It helps stop fraud right away rather than after-the-fact. It also helps solve problems while they are fresh rather than trying to resolve a month-old+ issue. Some countries overseas do this almost universally. - On-restaurant-table credit card swipe rather than taking the card away and entering tip later. Many places overseas do this almost universally. -- Side note on the txn alerts: I suppose it is due to the way Visa/MasterCard are processed, but I sometimes get alerts a day or more after i've done a credit card transaction. Some cards dont even have the option for alerts. AMEX has great alters, almost instantaneous, but I suppose they control the entire stack, so that makes sense.
- JoshTriplett 8y ago> - On-restaurant-table credit card swipe rather than taking the card away and entering tip later. Many places overseas do this almost universally. This isn't a problem you can solve by changing the banking industry. This is a point-of-sale problem.
- heinrichf 8y agoMany banks do not actually include any information in the notifications, because of the unencrypted nature of in-transit email. That is, "There has been a debit on your account" instead of "There has been a debit of X$ from merchant Y".
- exegete 8y agoIt'd be pretty neat to give a bank your public PGP key and then they could send you encrypted emails with more details.
- jsjohnst 8y agoFor the 0.001% of Nanking customers who understand PGP, let alone who can set it up. Yes, I’d love it too, but until PGP is more user friendly for the masses, I wouldn’t hold my breath on any bank bothering.
- exegete 8y agoOh I agree. I think it will never happen. But as a data point, Facebook has the option to add it: https://www.facebook.com/notes/protect-the-graph/securing-email-communications-from-facebook/1611941762379302 https://www.facebook.com/notes/protect-the-graph/securing-em... So, it's not too far fetched.
- iamdbtoo 8y agoAnyone use Firefly III? It uses the Spectre API to allow you to have a self-hosted Mint. https://firefly-iii.org/ https://firefly-iii.org/
- shiv86 8y agoThis seem like a great option!...thanks for sharing
- benbristow 8y agoIn the UK the fintech (Financial Tech) scene is becoming more prevalent, for the better. Recently I switched to a new online-only bank called Monzo. It's fully licensed and all accounts are insured up to a certain amount by the UK government. It's great. They're in the top charts for apps in the UK now on the iOS App Store. There's a few other alternatives like Starling Bank and Revolut too. They're very good. They're open about their tech stack with developer blogs and it's a modern stack which lets them iterate in a quick and agile manner unlike the legacy banks. You get push notifications on every payment (usually you get the notification before the payment has even processed on the vendor's end!). Tells you exactly how much you've been spending every day. They've got RESTful JSON APIs you can integrate with if you want to and they also have the option in the app to easily export your data into CSV or Quickbooks format. They even have IFTTT integration. Really makes you manage your money better.
- kqr 8y agoThis is, on one hand, great. I made the switch to a similar bank myself a few years back. What I failed to realise at the time was how exposed I became to the vulnerabilities associated with being cashless. Cash is not just an ancient relic. Cash is an ancient relic and a fundamental component of a free society. My country has, in practise, become nearly cashless and I used to be proud to be one of the very early adopters of that mindset. Now I lament how hard it is to deal in cash. I am planning to switch to the only bank left in my country still providing personal cash services. The only one. That should be scary, not relieving.
- poulsbohemian 8y agoI just spent 20 minutes reading about the fascinating case of Ellis vs. Grand Rapids, as linked from the poster's site (http://gduverger.com/ellis-v-city http://gduverger.com/ellis-v-city). This was like a two-for-one deal, so thanks!
- gduverger 8y agoHaha, thank you!
- joaoheleno 8y agoAre the bar charts generated with the pandas library? The ones in the report received by mail... Took a peek at the documentation but all I see is graphic based ones with plot functions. Thanks
- gduverger 8y agoThe bar charts are my own, not part of pandas. I wrote a little thing for it. It's not very sophisticated right now but I've been thinking of packaging it a little bit better and maybe sharing it.
- virusduck 8y agoAmEx sends notifications for any use of my card. It is really a great thing.
- jimmyswimmy 8y agoI guess it's not popular or all that well-known anymore, but for quite a while there's been a Quicken-led banking interface for some banks. Known as OFX or Direct Connect, it provides at least one-way (download) access to banking transactions. I think there's a way to upload as well but have never used it nor had a bank that supports it for upload. My bank has a separate enrollment - it was free - offering download-only access to my transactions. I haven't used it in awhile (just too much on my plate) but it worked well as recently as 2016. https://github.com/aqbanking/aqbanking https://github.com/aqbanking/aqbanking is one open implementation for the interface.
- mindslight 8y ago^ This. It's often called "Quicken Direct Connect" (NOT "web connect", that's a bastardization trying to push the login flow through the proprietary web interface), and often has to be specifically enabled for your account (Bank of Slum-merica is the only place I've heard charging for the functionality though). Check say https://ofx-prod-filist.intuit.com/qb2600/data/fidir.txt https://ofx-prod-filist.intuit.com/qb2600/data/fidir.txt to see if your bank is listed (that contains both direct and web connect banks, you'll figure out what the flags mean). Setup a cron or human cron to pull and save the raw OFX (QFX) query every day/week/month, and then run whatever reports/analytics you want from that. This way you'll have history to use with whatever program/scripts you move to, and can also straightforwardly integrate legacy banks that make you use the web interface (just at a much lower polling rate).
- burkemw3 8y agoHow does web connect actually work? I've never found any description of the method
- mindslight 8y agoI don't exactly know. But I'm under the impression that it performs the standard website login flow via headless browsing, then uses the website's "download transactions as OFX" functionality. So similar problems as "screen scraping" but less error prone because if it successfully downloads the data, it is in a well-defined format.
- yurishimo 8y agoPosts like this make me pretty excited to be a customer of a "tech-y" online bank. (In my case, Simple, not shilling, I just like them) Theoretically, these banks with great mobile apps are one step away from giving you API access once it's needed/wanted by enough consumers. Technically, your phone is doing it under the hood already. Simple uses React for their web interface, so I imagine they have nice JSON APIs for all of the data already. There was a site advertising developer managed bank accounts last year, https://root.co.za/ https://root.co.za/, I think it's also a cool idea.
- RickS 8y agoYou can prod the network panel to reverse engineer simple's APIs. I've done it for some visualization tools in the past. It's sloppy and undocumented, but their support staff are fantastic and surprisingly technical, so I had about a 100% success rate tweeting them things like "what's the new endpoint for what was previously /bank/transactions?" A public version of this API was one of their stated goals in 2010 or so, but they're more interested in chasing non-technical product goals like couples accounts. I expect their most technical customers don't make up enough transaction volume to justify attention. I love simple for being the least worst, but am still saddened by what they could have been if they had stripe's passion for technical execution. But hey, at least there's an API and they won't slap you with the CFAA for hitting it.
- yurishimo 8y agoSorry for the late reply here. I'm still new to HN and haven't figured out everything yet... How are you doing authentication for the API? Is it session based or can you pass some sort of API token after logging in?
- technofiend 8y agoI've had to do something similar when a bank decided that CSV history was limited to a much shorter time period than PDF documents. Nothing a little python-driven OCR can't handle. But it was stupid I had to employ such measures in the first place.
- achandlerwhite 8y agoLook into OFX for pulling down account and transaction data It’s the api that TurboTax, Quicken, and others have used for years and you can use it too. It’s a little outdated in its design and implementations are hit or miss on the banking side. But for most large banks it works.
- KirinDave 8y agoThe actual secret API of banks-and by the way this is the initial strategy Plaid pursed if rumor is to be believed (essentially without the consent of the banks)-is by reverse engineering mobile app APIs. Most of these bank APIs try to use cheesy secret token vending to prevent casual API traffic on their endpoints, but the reality is that a sufficiently instrumented Android kernel (or rooted iOS device) will let you reverse engineer those protocols and masquerade as legitimate users.
- josephh 8y agoThat rumor sounds far from plausible though. If they were to attempt to use the reverse-engineered API from their own servers without consent, banks would find out (in a matter of hours) and shut them down when they discover a huge spike in traffic from a relatively small pool of IPs. If they were to access it directly from customers' phone/browser via their (web-)apps, I expect that it would've caused a huge media storm by now when someone finds out they are storing/transmitting the password in plaintext (or its equivalent) to be used for authentication. As for this instance specifically, I believe Chase grants Mint (and few other whitelisted companies) account access via OAuth. It's a step in the right direction, although it's not clear to me what their long term goal is.
- KirinDave 8y agoHi, I founded Level Money, was one of Intuit's earliest aggcat customers and one of their last customers, and did this for a lot of people until Capital One bought my company and we did it for them. AMA, I guess. But to answer the implicit question: shutting that off can be harder than it sounds. And because it's a mobile API and iOS's store has fairly slow update cycles, it can be very hard to simply rotate your API spec fast enough without interrupting customer service: a difficult thing for a bank to get away with (lol, I'm joking they're down all the time).
- ikeboy 8y agoWhy is shutting it difficult? Any IP that logs into more than 5 accounts within a day gets a ban. They can use proxies but eventually will run out. You can also use your own canary accounts, provide them to the service you want to block, and ban any IP that tries to log in
- ctdean 8y agoI talk to (about) a person a week who wants to create a new US bank. Some are pursuing a de novo charter, some are buying a bank, and some are a quasi bank on top of another bank. The real blocker here is the Fed won't grant new charters and often won't transfer charters. I'm hoping this will change in the next few years and we can get some real competition. (Disclosure: my job is making APIs for US Banks.)
- techsupporter 8y agoAny idea why this is the situation? What about doing a state-by-state charter?
- ctdean 8y agoThe fed is still living with the fear of the 2008 crash. And you still need FDIC insurance even if you have a state charter.
- RickS 8y agoWhat's your take on new entrants at the processor level? When I worked for a debit card startup (stripe's new feature, but worse, and years earlier), it seemed like the "bank account" parts were mostly boring money buckets, and all the interesting features existed at the processor level. FIS was tragicomic to interface with, and they control something like 50% of card swipes in the US if i understand correctly? Seems like if you want to make a really major play, you do to FIS what stripe did to paypal, right?
- altmind 8y agoYou may want to try plaid. They've integrated every major US bank(and some CUs) either using bank api or scraping their web pages. You can get not only transactions, but also account info, balances and enough info to perform charges/payments(e.g. ACH routing/account numbers). Api is no-nonsence and they got decent dashboard. Its paid service, but they allow 100 accounts for free(you need to apply though). To check what are major banks they support, you can visit their status page: https://status.plaid.com/ https://status.plaid.com/
- patsplat 8y agoEmail notification is a fantastic event notification tool for services that don't / won't emit events or submit to callbacks. It's the one pixel gif of systems integration.
- btmerr 8y agoForgive the shameless self-promotion, but we at Seed offer a customer facing API for read-only transaction data. We would happily build out more API features, but we've seen very little demand, despite the frequent HN threads about bank APIs. We are a business focused bank, but we support sole proprietors and freelancers as well. API docs are here: http://docs.seed.co/v1.0.0/docs http://docs.seed.co/v1.0.0/docs https://seed.co https://seed.co p.s. We are working on Android I promise please don't yell at us (again).
- mattkopecki 8y agoIt's really unclear to me what your product is. I would have given up trying to figure out if you hadn't implied that it's somehow related to what this cool blog post accomplishes. From the FAQ, my best guess is that you offer businesses bank accounts that come bundled with the reporting you'd get with something like Mint/Quickbooks? It's OK to basically be two services glued together, but why are you better than just the two existing services, were they to be glued together?
- RyanShook 8y agoThis is cool. Will OP open source this so others can use it?
- gduverger 8y agoI've been getting a good amount of requests for open sourcing it. It would require a few hours of work but I might take the time one of those weekends. Please feel free to send me an email (georges.duverger@gmail.com) if you'd like me to keep you updated.
- RickS 8y agoThere's so much cool stuff you can do with this data. Think of all the creepy shit amazon does with your browsing data, but for your spending data, and you can browse it. We're leaving a crazy amount of insight on the table. Shameless plug: I built some personal finance viz tools based on similar data. they also just barely scratch the surface: http://rick.xxx/orchid.html http://rick.xxx/orchid.html If you're working on interesting things in the fintech space, I'd love to chat with you about strategy or execution, especially w/r/t product design. Email in profile.
- HuckFackerNews 8y agoIt's called SWIFT.
- a-b 8y agohttps://wiki.gnucash.org/wiki/OFX_Direct_Connect_Bank_Settings https://wiki.gnucash.org/wiki/OFX_Direct_Connect_Bank_Settin...
- Tharkun 8y agoMy bank allows me to export my account statements in SWIFT MT 940 format, which is pretty convenient. Sadly there's no easy/free way to automate this process.
- homero 8y agoCapital One just added oauth with mint and before that they had read only passwords
- cjpa 8y agoSo, he does not trust third-parties with his banking details, but uses a third-party to send emails with a complete detail of his account to an email-server (which might or might not be run by a third party). The logic is lost on me.
- Legogris 8y agoFurthermore, the e-mails are sent in clear-text and readable by any endpoint on the way transporting the e-mail to its destination. It's akin to sending your information back to yourself by post card rather than stored in a file with a third-party.
- gduverger 8y agoThe alert emails do not contain “detail of [my] account,” only the last 4 digits of my account (for Chase). Even if someone were to intercept those emails, they couldn't do much with them.
- vonseel 8y agoCan you expand on unwanted transactions? TBH, the only “unwanted” transactions I’ve found were indeed fradulent, and these have been immediately caught by BoFA, VISA, and other financiers. I have had problems with false positives, however.
- chrismeller 8y agoPeople would normally call out fraud if that were the case, but it's certainly possible that it's something the bank isn't aware of yet so their systems did not catch. In general I just assumed he meant things like "signed up for that trial and forgot to cancel", "accidentally got charged twice", "tip amount was clearly wrong"... Innocent things that happen from time to time, but clearly need to be resolved.
- gduverger 8y agoExactly. By “unwanted,” I meant things like subscriptions I forgot I was paying or (innocent?) merchant mistakes but that rarely happened.
- cja 8y agoBank Millennium is a normal Polish bank, with branches everywhere. Their mobile app gives me an instant notification when transactions occur. They even have an Android Wear app. Irrelevant but also impressive: their app uses fingerprint login and whenever I phone them they talk to me in English without me having to ask.