5 ms·
Ask HN: How does Mint.com work?
Mint.com connects to your bank account, credit card account, etc, and downloads your data. How do they do that? Do they need any special authorization from the banks or it's an open API? How about security? Can you please explain their back-end system?
The same question applies to Blippy and inDinero. Anyone know what these guys are doing on the back end to get the transaction data?
- pw 16y agoAt least prior to their acquisition by Intuit, Mint's backend was powered by Yodlee. This TechCrunch article provides a little background: http://techcrunch.com/2009/09/18/mint-is-yodlees-youtube/ http://techcrunch.com/2009/09/18/mint-is-yodlees-youtube/
- grep 16y agoIt says that Yodlee received $2mil/year, isn't that a bit expensive for startups like inDinero to pay? (I'm assuming inDinero uses the same system)
- smackfu 16y agoI wouldn't be surprised if it is priced based on number of accounts.
- grep 16y agoand Blippy, I assume they use Yodlee or they've created their own system?
- AlexMuir 16y agoYodlee are generally amenable to negotiating with startups. After the Mint situation they might be willing to look at some form of equity share if you could demonstrate traction.
- djb_hackernews 16y agoStraight screen scrapin' yo. I worked for a similar startup that collected more detailed information than yodlee/mint, it was a product for financial managers instead of consumers. We collected over a 1mil transactions per night from over 3000 financial institutions. It was no joke. You might think screen scraping is silly but the bottom line is if a bank had an api (OFX, and very few do offer OFX) or formatted data downloads(csv,xls) the data tended to be stale or incorrect. Reasoning behind that is more eyeballs are on the web pages and so bugs/inconsistencies are noticed quicker. There was more of an expectation for the web pages to be accurate.
- encoderer 16y agoI work for a fairly successful start-up in the advertising industry.. We've done a MASSIVE amount of screen-scraping. phpQuery ftw. I had a few months solid of nothing but scraper-building. Besides, what's Google if not a screen scraper? :D If anybody thinks it's "silly" they probably have a characterization in their mind that's not really how it works in the wild when done skillfully.
- Sukotto 16y agoWould you please share some code samples or favorite blog posts on how you're using phpQuery as part of a scrape app? What set of tools/libraries are you using? I'd really like to hear more about the current state of the art (without you telling any company secrets). I have quite a lot of experience scraping utility and government websites (no javascript) in perl + LWP... but I'm getting a little tired of perl and am looking to give a new toolset a try. Perferably one that can handle a broader range of modern websites.
- asnyder 16y agophp|architect's recently released book entirely on this topic: http://www.phparch.com/books/phparchitects-guide-to-web-scraping-with-php/ http://www.phparch.com/books/phparchitects-guide-to-web-scra...
- l4u 16y agoI have been using spynner, written in python. It uses pyQtWebkit, and jQuery. Use the trunk version at http://code.google.com/p/spynner/ http://code.google.com/p/spynner/
- justliving 16y agoI have been using quite a bit of python and beautifulsoup. http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/
- encoderer 16y agoI will throw you 1 bone, but past that, I'm careful because my work here is not my own, it's my employers. libxml_use_internal_errors(true); This little function call is the secret to solving what seem at first to be intractable memory-leaks. The trouble is the scraper uses libxml, and libxml issues a notice/warning every time the HTML is malformed. Without this call, those errs will bubble up to the PHP error handler and it'll murder performance and memory usage. One more i suppose... If you scrape inside a loop (and unless you're using a distributed job queue,if you're scraping more than one URL at a time you almost certainly are), missing a unloadDocument() call is going to cost you each time you iterate. The objects it creates, IIRC, have some circular dependency issues and if you don't explicitly unloadDocument you'll run into trouble. (Suppose it should be OK tho if you've enabled that GC feature in PHP 5.3) And, generally, a tip... sometimes it's tempting to write a simple regex instead of a chain like, say, pq($this->node->find('a')->get(0))->attr('href') Write the chain. Regex is just too brittle.
- smackfu 16y agoThey do use Yodlee but you still raise some interesting questions. Should the bank really be allowing access by a third party using stored two-factor credentials?
- grep 16y agoI don't think Blippy uses Yodlee. Maybe they are using some kind of bank API?
- niels_olson 16y agoLast I looked into this, Yodlee's security pages were a good place to start because they have a lot of the key words to look up. http://www.yodlee.com/security_overview.shtml http://www.yodlee.com/security_overview.shtml A lot of the "how" is meeting security wickets (physical, application, transport, audit, examination).
- billybob 16y ago"How do they get the data" doesn't seem mysterious once you give them your logins. "Is this safe and why or why not" is the question I'd be much rather have answered.
- grep 16y agoOnce you give them the login info, how can they get the data from the bank?
- pbhjpbhj 16y agoDon't bank user T&Cs specifically disallow sharing of your username and passwords though? Why would the bank be happy with a third party interrogating their systems in this way and allow the massive flow of traffic to a [successful] companies IP addresses? Surely there has to be some oversight from the bank? I think this would be illegal in the UK under Computer Misuse Act, or somesuch, as it's unauthorised access to a computer system; the user doesn't have the authority to grant others access.
- gcb 16y agoDid quicken back then also used screen scrapping? Most banks at the time didn't even had websites.
- gcb 16y agoDid quicken back then also used screen scrapping? Most banks at the time didn't even had websites.
- hardik 16y agoThink it should be "How mint.com works?" or "How does mint.com work?"
- angilly 16y agoIt's not screen-scraping. They use Yodlee and CashEdge.
- grep 16y agoMaybe someone from inDinero could join the conversation?
- jlm382 16y agoThis is Jessica from inDinero. We use the same technology that Mint uses -- namely, integrating with a third party service called Yodlee. They take care of aggregating financial data through various means. Screen scraping, direct OFX feeds, etc... The typical question is, why do Mint and inDinero use Yodlee instead of building the solution out themselves? 1) Security Liability. No startup should ever have to deal with the problems that go with storing passwords to financial accounts. Yodlee is in the business of security, they have direct feeds with major banks, making it much easier (and safer) to just integrate with them. 2) Mass quantity of banks. Screen scraping from so many banks is a pain in the *. It isn't standard either -- compare the bank website of Wells Fargo to that of a local credit union that asks 5 security questions upon login. In short, it's a brutal nightmare. 3) None of our businesses are in the business of screen-scraping. If Mint had to spend the first year of business integrating with banks, they wouldn't be successful. And even once the integrations are done, you have to maintain them in the event that the bank changes their login page or interface. In short, it's not worth any startup's time to do manual screen scraping themselves. Would be happy to discuss further if you DM me.
- nickadams 16y agoFeel free to ignore this if it's something you can't talk about... but can you confirm if Yodlee works with startups concerning licensing costs? Do they do something where you pay based on the number of accounts? Or do they ask for equity? I've seen a number of startups use Yodlee or CashEdge and I was always curious how they could afford this or how the setup worked. I can't imaging they pay what Mint was reported to have paid (about $2mil/yr) to license the software.
- Gianteye 16y agoWould you consider working with Iphone app developers to make money tracking and spending actionable? I've wanted this tool for a few years now. If I set my expenses, say utilities, rent, insurance, and car payments in a system that also tracks my purchases and income, I could see my daily budget for other more spontaneous expenses. I could open up my app to find what my drinking/nightlife expenses can be today, or tally up the day's spending to see if I'm still on track with that utility bill. The thing I've found frustrating about financial planning tools is that they seldom give me any useful information about the purchase I want to make right now. Instead, they tell me about the malleable resource called money, and offer me the chance to look at it from a 3rd person perspective.