10 ms·

I got control of my spending with some no-code services and 100 lines of Python

143 points by jcp2fa 7y ago

Amezarak 7y ago
Just a nitpick, but the word you’re looking for is discretionary. :) We would all do well to follow this example though - NIH and reinventing the wheel are huge problems in our profession. Great work!
sash_ 7y ago
But this way you send all your financial transactions through at least 2 additional providers (zapier and google) and your email provider. Not sure if this is worth it.
- deleted 7y ago
  [deleted]
- tlarkworthy 7y ago
  I prefer this, the finance is lost in the noise of other traffic and no creds to banks are shared unlike mint.
  - graphememes 7y ago
    credentials are sent to the bank servers, and a token is stored on the mint servers, credentials are never shared _to mint_
    - tlarkworthy 7y ago
      No they store the plaintext password. Maybe more banks have oauth2 but many don't. 1st item on their FAQ https://www.mint.com/how-mint-works/security https://www.mint.com/how-mint-works/security
loop0 7y ago
Your idea is very nice! I’m going to try it but without the use of zapier, instead I’m going to use python’s smtpd lib to directly receive the emails and parse them. Thanks for sharing
echion 7y ago
Would love to see a .procmail rule + scripts to do this without the significant zapier integration.
- woodruffw 7y ago
  I do essentially this with `imapfilter` and some cronjobs. The biggest hassle is the fact that each of my banks/financial institutions uses at least half a dozen unique addresses to send me relevant information.
spking 7y ago
I love this...homebrew account aggregation. This is basically what I use Personal Capital for.
ScoutOrgo 7y ago
I have been wanting to do this, as I set up the Chase email notifications, but never got motivated to tackle what you used zapier for. One thing that it seems you could do is use Google sheets for your constants vs coded in the repo. That way if you spend from somewhere new, you can do add that category on the fly by updating your sheet.
timwis 7y ago
Very creative. Love the idea of using gsheets as the back-end so you can leverage the power of spreadsheets (which were basically designed for use cases like aggregating transactions!)
slavoingilizov 7y ago
I admire how you've managed to do this easily (and avoid writing code for everything). But the fact that you have to do it shows how broken and lethargic banks have become. In the UK, they are starting to wake up. The new "challenger" banks which exist here all do this as a hygiene feature - see Monzo, Starling, Revolut. With Open Banking mandating open APIs for all banks, this will become the norm soon. The fact that American banks and regulation hasn't jumped on this is a symptom of something very wrong with the system. I sincerely hope the launch of Apple Card or similar services disrupt consumer finance in the US as well and set a new bar. And when the above mentioned banks come to the US (which they are currently working on), you should absolutely switch and delete the 100 lines of python as well.
- jedberg 7y ago
  Banking in the US is utterly ridiculous. In Europe, if you need to move up to 100,000E in a day, you can do it with a few clicks or taps, and maybe a card reader that they already gave you. In the US, you have to use a wire transfer is you're moving more than about $4,000. And wires cost a lot of money. And they aren't even instant even though they are billed that way. I had to move some money from one account I own to another, and I needed it there the same day. The only way to make it happen was to drive 15 minutes to the origin bank, spend 15 minutes filling out a paper form in the branch, which was then faxed to the central processing center. Then I had to call the center center every hour to see if they'd gotten to it yet. When they finally did do it, I had to call the receiving bank with the tracking number and tell them, and then they had to find it on their side, which took a couple minutes, before it was finally processed. From the time I finished filling out the "instant" wire form (and paying $45!) to the time it hit my other account was over three hours! And that was after the initial 30 minute delay of driving to the bank and filling out a paper form.
  - reaperducer 7y ago
    You're lucky it went in three hours. A couple of times I've moved between $17,000 and $40,000, and Chase sits on it for up to TEN DAYS before it will release received the funds. Chase claims it's a federal regulation, but no one in the branch or on the phone can tell me which one.
dguo 7y ago
Very cool. Using email notifications to get transaction data is a nice hack. If you do find yourself wanting a more formal API, check out Plaid (https://plaid.com/ https://plaid.com/). I've never used it myself, but I believe the free plan is enough to get your personal transactions.
- heavenlyblue 7y ago
  My bank doesn't provide an API. So I just automated a headless browser to log into my account and download the statements.
  - bronco21016 7y ago
    This has long been on my todo list. Do you find it breaks often? Or was it kind of a build it, set it, and (mostly) forget it type of project?
    - heavenlyblue 7y ago
      I just need to supply my bank statements to the accountant annualy (bank is TSB). They have a set of rules (less that 150 lines, not more than 3 months) that constrain how you can get the statements, and a 5-step process to open up the UI to download one, then select the dates, etc. You may imagine it's simply error-prone and repetitive. A few years I didn't even need to change the script. But even when I needed it wasn't something too complex. As long as you constrain the expected behaviour and error early - it works perfectly well.
  - graphememes 7y ago
    Over 18,000 banks provide APIs, what bank do you use? Is it a small credit union?
    - Spinosaurus 7y ago
      > Over 18,000 banks provide APIs Please consider adding a source for this when you have a moment.
alecgorge 7y ago
An easy way to take this to the next level (and be truly no-code) is: https://www.tillerhq.com/ https://www.tillerhq.com/ I love being able to continue using Google Sheets shortcuts + automation features. It allows me to generate well-formed reports that I can analyze with pivot tables or any of my normal processes that I use for analyzing datasets.
- bluetidepro 7y ago
  Tiller costs $59/year, though. So yes, no-code, but it also costs money.
diminoten 7y ago
What about Mint?
- justusthane 7y ago
  From the article: > I looked into existing budgeting software such as Mint or YNAB; they were feature packed but cumbersome to setup and complex to use. All I wanted was a simple, unified view of my transaction history across all payment mechanisms with a few spending categories from which I could do my own analysis.
neilv 7y ago
For anyone who wants to do things like double-entry accounting with transaction splits (e.g., properly encode an Amazon order for 5 different items from different expense categories, that're paid for by a mix of rewards points, Gift Card balance, and CC), there's GnuCash. You can also do things like record transactions yourself, update with downloaded data, and reconcile against monthly statements (as another way of catching discrepancies). You can make custom reports with charts. I used these, for example, for comparing how much I spent buying coffee every day, vs. how much I spent on every cost that goes into making coffee at home, when I tried that for a while. There's also features for tracking investments, including automatic updates of market prices. And, if you're a small business, you can build and generate invoices (including building the invoice over multiple days), and track accounts receivable. Warning: GnuCash can be a significant investment of time. I eventually decided it was no longer worth it for me, though I might someday resume it. Other Warning: If you have retirement accounts or other investments in GnuCash, consider hiding them from view, which GnuCash lets you do with any account. Bogleheads-style, US people probably want to keep contributing to a predetermined balance of total-market stocks and bonds, automatically, and not think about it, especially when there's market fluctuations.
- majewsky 7y ago
  > You can make custom reports with charts. I used these, for example, for comparing how much I spent buying coffee every day, vs. how much I spent on every cost that goes into making coffee at home, when I tried that for a while. Nice! But why do you need a separate report for that? Just put all the coffee-related expenses into their own subaccount of the expense account for Food, and run the standard expense reports on that account. > Warning: GnuCash can be a significant investment of time. I eventually decided it was no longer worth it for me, though I might someday resume it. Oh, I know what you mean. Especially when you first set it up. For me, it's about 15 minutes per week, all things considered. I track all expenses, including cash expenses. The only thing I don't break down into individual items are grocery bills. Those get folded into one daily "Groceries" transaction.
- Havoc 7y ago
  Never really understood the need for double entry in the personal context.
garp 7y ago
pocketsmith.com does a great job of this out of the box (no affiliation, just a user and fan).
air7 7y ago
> As an engineer, it is in my DNA to build things. However, it’s too easy to get stuck in the mindset that you have to build everything. This mentality often leads one stuck with a plethora of incomplete products that grew too complex to finish before losing interest. This. I can relate so much. I feel my instinct is always to roll-my-own because I can and, perhaps more importantly, because I'm not familiar with relevant no-code services. What other such tools/services are out there that can help "to go from idea to proof-of-concept in only a few hours."?
- davidjnelson 7y ago
  The tools in the article, google sheets and zapier are quite good. A few more that might fit are airtable, webflow and stripe. Maybe even look into something like google app maker as well.
- stolksdorf 7y ago
  It's relatively new, but I've been using Coda.io to build a lot of interesting ideas out. I've built some fantasy leagues, a very powerful task tracker/organizer, and do all of my trip planning using it. Definitely check out their templates, https://coda.io/templates https://coda.io/templates
- brooklyntom 7y ago
  As luck would have it, I recently pulled together a list of no code tools into a market landscape: https://www.createwithoutcode.com/posts/no-code-development-landscape https://www.createwithoutcode.com/posts/no-code-development-... I'll be adding more in the coming days... To directly answer your question, I would recommend Bubble.is as a great place to start in the no code space.
dopeboy 7y ago
I'd like the capabilities of Mint without the creds hand off. BofA has starting doing this in their portal but it's not well done. I like the "hack" here - seems BofA supports this kind of alert too. I'd love to follow a simple guide that lets me (1) setup the zapier instance and (2) deploy the server side to a DO or Heroku.
deleted 7y ago
[deleted]
gkbrk 7y ago
I've done something similar recently; but instead of grabbing data from my bank, I chose to enter data manually from receipts. This makes it work for both cash and debit payments, and doesn't need any maintenance for API breakage. I find doing this data entry calming, to be honest. I wrote about it here https://gkbrk.com/2019/04/plaintext-budgeting/ https://gkbrk.com/2019/04/plaintext-budgeting/
- bronco21016 7y ago
  I used to do this when I lived on a much tighter budget and found the data entry was calming and provided something to do that didn’t cost money. However, now that I have a little disposable income and a family the thought of sitting down and entering financial transactions every couple of days sounds horrifying. I can barely keep up with my monthly spreadsheet tracking net worth.
- decasteve 7y ago
  > Or… I could use a shell script to run calculations with SQLite. I did exactly that and I import the data from the bank’s csv-export once per month. I have a set of sql statements to run and categorize everything and then I manually enter the stuff that isn’t caught by the script. I manually enter cash transactions. And the end of the year I run a sql statement to generate a report with all of the significant amounts for taxes.
- klohto 7y ago
  Did you consider to use some kind of OCR?
Havoc 7y ago
>[email purchase notifications ] I suspect every single major card company or payment application has the same option. Quick glance at my UK amex says nope
6ue7nNMEEbHcM 7y ago
Hi, sorry for a bit of off-topic but perhaps someone could tell me if I'm wrong here. I clicked on this article on medium.com and it shows me a dialog from Google asking if I want to continue browsing medium.com authenticated with my Google account. It shows there my name and e-mail address. I find it a bit concerning. Can medium.com technically walk through the DOM of that page to harvest my personal data such as e-mail address and send it to medium.com with some API call? I guess it must be filled by some .js code linked from google domains. Scary to see my e-mail address appear on random web page I entered.
- dwild 7y ago
  Medium doesn't have access to that part, even though you see it on your screen. It use an iframe, which in the past, was accessible, but now with crossdomain policy and all that, the iframe content is no longer accessible.
oldandboring 7y ago
The brilliance of this is the use of email receipts as the data source. There's so many services that use Yodlee (and/or the the Quicken-style "direct connect" method) to fetch your transactions but they all eventually fall over: - your credentials change - your credentials require MFA authentication - your credentials require a password reset or security question/answer intervention - the bank, Yodlee, or the service lose track of the transaction offset and you get duplicate transactions or missing transactions - the service doesn't transmit all available metadata

143 points by jcp2fa 7y ago

Just a nitpick, but the word you’re looking for is discretionary. :) We would all do well to follow this example though - NIH and reinventing the wheel are huge problems in our profession. Great work!

sash_ 7y ago

But this way you send all your financial transactions through at least 2 additional providers (zapier and google) and your email provider. Not sure if this is worth it.

deleted 7y ago
[deleted]
tlarkworthy 7y ago
I prefer this, the finance is lost in the noise of other traffic and no creds to banks are shared unlike mint.
- graphememes 7y ago
  credentials are sent to the bank servers, and a token is stored on the mint servers, credentials are never shared _to mint_
  - tlarkworthy 7y ago
    No they store the plaintext password. Maybe more banks have oauth2 but many don't. 1st item on their FAQ https://www.mint.com/how-mint-works/security https://www.mint.com/how-mint-works/security

loop0 7y ago

Your idea is very nice! I’m going to try it but without the use of zapier, instead I’m going to use python’s smtpd lib to directly receive the emails and parse them. Thanks for sharing

echion 7y ago

Would love to see a .procmail rule + scripts to do this without the significant zapier integration.

woodruffw 7y ago
I do essentially this with `imapfilter` and some cronjobs. The biggest hassle is the fact that each of my banks/financial institutions uses at least half a dozen unique addresses to send me relevant information.

spking 7y ago

I love this...homebrew account aggregation. This is basically what I use Personal Capital for.

ScoutOrgo 7y ago

I have been wanting to do this, as I set up the Chase email notifications, but never got motivated to tackle what you used zapier for. One thing that it seems you could do is use Google sheets for your constants vs coded in the repo. That way if you spend from somewhere new, you can do add that category on the fly by updating your sheet.

timwis 7y ago

Very creative. Love the idea of using gsheets as the back-end so you can leverage the power of spreadsheets (which were basically designed for use cases like aggregating transactions!)

slavoingilizov 7y ago

I admire how you've managed to do this easily (and avoid writing code for everything). But the fact that you have to do it shows how broken and lethargic banks have become. In the UK, they are starting to wake up. The new "challenger" banks which exist here all do this as a hygiene feature - see Monzo, Starling, Revolut. With Open Banking mandating open APIs for all banks, this will become the norm soon. The fact that American banks and regulation hasn't jumped on this is a symptom of something very wrong with the system. I sincerely hope the launch of Apple Card or similar services disrupt consumer finance in the US as well and set a new bar. And when the above mentioned banks come to the US (which they are currently working on), you should absolutely switch and delete the 100 lines of python as well.

jedberg 7y ago
Banking in the US is utterly ridiculous. In Europe, if you need to move up to 100,000E in a day, you can do it with a few clicks or taps, and maybe a card reader that they already gave you. In the US, you have to use a wire transfer is you're moving more than about $4,000. And wires cost a lot of money. And they aren't even instant even though they are billed that way. I had to move some money from one account I own to another, and I needed it there the same day. The only way to make it happen was to drive 15 minutes to the origin bank, spend 15 minutes filling out a paper form in the branch, which was then faxed to the central processing center. Then I had to call the center center every hour to see if they'd gotten to it yet. When they finally did do it, I had to call the receiving bank with the tracking number and tell them, and then they had to find it on their side, which took a couple minutes, before it was finally processed. From the time I finished filling out the "instant" wire form (and paying $45!) to the time it hit my other account was over three hours! And that was after the initial 30 minute delay of driving to the bank and filling out a paper form.
- reaperducer 7y ago
  You're lucky it went in three hours. A couple of times I've moved between $17,000 and $40,000, and Chase sits on it for up to TEN DAYS before it will release received the funds. Chase claims it's a federal regulation, but no one in the branch or on the phone can tell me which one.

dguo 7y ago

Very cool. Using email notifications to get transaction data is a nice hack. If you do find yourself wanting a more formal API, check out Plaid (https://plaid.com/ https://plaid.com/). I've never used it myself, but I believe the free plan is enough to get your personal transactions.

heavenlyblue 7y ago
My bank doesn't provide an API. So I just automated a headless browser to log into my account and download the statements.
- bronco21016 7y ago
  This has long been on my todo list. Do you find it breaks often? Or was it kind of a build it, set it, and (mostly) forget it type of project?
  - heavenlyblue 7y ago
    I just need to supply my bank statements to the accountant annualy (bank is TSB). They have a set of rules (less that 150 lines, not more than 3 months) that constrain how you can get the statements, and a 5-step process to open up the UI to download one, then select the dates, etc. You may imagine it's simply error-prone and repetitive. A few years I didn't even need to change the script. But even when I needed it wasn't something too complex. As long as you constrain the expected behaviour and error early - it works perfectly well.
- graphememes 7y ago
  Over 18,000 banks provide APIs, what bank do you use? Is it a small credit union?
  - Spinosaurus 7y ago
    > Over 18,000 banks provide APIs Please consider adding a source for this when you have a moment.

alecgorge 7y ago

An easy way to take this to the next level (and be truly no-code) is: https://www.tillerhq.com/ https://www.tillerhq.com/ I love being able to continue using Google Sheets shortcuts + automation features. It allows me to generate well-formed reports that I can analyze with pivot tables or any of my normal processes that I use for analyzing datasets.

bluetidepro 7y ago
Tiller costs $59/year, though. So yes, no-code, but it also costs money.

diminoten 7y ago

What about Mint?

justusthane 7y ago
From the article: > I looked into existing budgeting software such as Mint or YNAB; they were feature packed but cumbersome to setup and complex to use. All I wanted was a simple, unified view of my transaction history across all payment mechanisms with a few spending categories from which I could do my own analysis.

neilv 7y ago

For anyone who wants to do things like double-entry accounting with transaction splits (e.g., properly encode an Amazon order for 5 different items from different expense categories, that're paid for by a mix of rewards points, Gift Card balance, and CC), there's GnuCash. You can also do things like record transactions yourself, update with downloaded data, and reconcile against monthly statements (as another way of catching discrepancies). You can make custom reports with charts. I used these, for example, for comparing how much I spent buying coffee every day, vs. how much I spent on every cost that goes into making coffee at home, when I tried that for a while. There's also features for tracking investments, including automatic updates of market prices. And, if you're a small business, you can build and generate invoices (including building the invoice over multiple days), and track accounts receivable. Warning: GnuCash can be a significant investment of time. I eventually decided it was no longer worth it for me, though I might someday resume it. Other Warning: If you have retirement accounts or other investments in GnuCash, consider hiding them from view, which GnuCash lets you do with any account. Bogleheads-style, US people probably want to keep contributing to a predetermined balance of total-market stocks and bonds, automatically, and not think about it, especially when there's market fluctuations.

majewsky 7y ago
> You can make custom reports with charts. I used these, for example, for comparing how much I spent buying coffee every day, vs. how much I spent on every cost that goes into making coffee at home, when I tried that for a while. Nice! But why do you need a separate report for that? Just put all the coffee-related expenses into their own subaccount of the expense account for Food, and run the standard expense reports on that account. > Warning: GnuCash can be a significant investment of time. I eventually decided it was no longer worth it for me, though I might someday resume it. Oh, I know what you mean. Especially when you first set it up. For me, it's about 15 minutes per week, all things considered. I track all expenses, including cash expenses. The only thing I don't break down into individual items are grocery bills. Those get folded into one daily "Groceries" transaction.
Havoc 7y ago
Never really understood the need for double entry in the personal context.

garp 7y ago

pocketsmith.com does a great job of this out of the box (no affiliation, just a user and fan).

air7 7y ago

> As an engineer, it is in my DNA to build things. However, it’s too easy to get stuck in the mindset that you have to build everything. This mentality often leads one stuck with a plethora of incomplete products that grew too complex to finish before losing interest. This. I can relate so much. I feel my instinct is always to roll-my-own because I can and, perhaps more importantly, because I'm not familiar with relevant no-code services. What other such tools/services are out there that can help "to go from idea to proof-of-concept in only a few hours."?

davidjnelson 7y ago
The tools in the article, google sheets and zapier are quite good. A few more that might fit are airtable, webflow and stripe. Maybe even look into something like google app maker as well.
stolksdorf 7y ago
It's relatively new, but I've been using Coda.io to build a lot of interesting ideas out. I've built some fantasy leagues, a very powerful task tracker/organizer, and do all of my trip planning using it. Definitely check out their templates, https://coda.io/templates https://coda.io/templates
brooklyntom 7y ago
As luck would have it, I recently pulled together a list of no code tools into a market landscape: https://www.createwithoutcode.com/posts/no-code-development-landscape https://www.createwithoutcode.com/posts/no-code-development-... I'll be adding more in the coming days... To directly answer your question, I would recommend Bubble.is as a great place to start in the no code space.

dopeboy 7y ago

I'd like the capabilities of Mint without the creds hand off. BofA has starting doing this in their portal but it's not well done. I like the "hack" here - seems BofA supports this kind of alert too. I'd love to follow a simple guide that lets me (1) setup the zapier instance and (2) deploy the server side to a DO or Heroku.

deleted 7y ago

[deleted]

gkbrk 7y ago

I've done something similar recently; but instead of grabbing data from my bank, I chose to enter data manually from receipts. This makes it work for both cash and debit payments, and doesn't need any maintenance for API breakage. I find doing this data entry calming, to be honest. I wrote about it here https://gkbrk.com/2019/04/plaintext-budgeting/ https://gkbrk.com/2019/04/plaintext-budgeting/

bronco21016 7y ago
I used to do this when I lived on a much tighter budget and found the data entry was calming and provided something to do that didn’t cost money. However, now that I have a little disposable income and a family the thought of sitting down and entering financial transactions every couple of days sounds horrifying. I can barely keep up with my monthly spreadsheet tracking net worth.
decasteve 7y ago
> Or… I could use a shell script to run calculations with SQLite. I did exactly that and I import the data from the bank’s csv-export once per month. I have a set of sql statements to run and categorize everything and then I manually enter the stuff that isn’t caught by the script. I manually enter cash transactions. And the end of the year I run a sql statement to generate a report with all of the significant amounts for taxes.
klohto 7y ago
Did you consider to use some kind of OCR?

Havoc 7y ago

>[email purchase notifications ] I suspect every single major card company or payment application has the same option. Quick glance at my UK amex says nope

6ue7nNMEEbHcM 7y ago

Hi, sorry for a bit of off-topic but perhaps someone could tell me if I'm wrong here. I clicked on this article on medium.com and it shows me a dialog from Google asking if I want to continue browsing medium.com authenticated with my Google account. It shows there my name and e-mail address. I find it a bit concerning. Can medium.com technically walk through the DOM of that page to harvest my personal data such as e-mail address and send it to medium.com with some API call? I guess it must be filled by some .js code linked from google domains. Scary to see my e-mail address appear on random web page I entered.

dwild 7y ago
Medium doesn't have access to that part, even though you see it on your screen. It use an iframe, which in the past, was accessible, but now with crossdomain policy and all that, the iframe content is no longer accessible.

oldandboring 7y ago

The brilliance of this is the use of email receipts as the data source. There's so many services that use Yodlee (and/or the the Quicken-style "direct connect" method) to fetch your transactions but they all eventually fall over: - your credentials change - your credentials require MFA authentication - your credentials require a password reset or security question/answer intervention - the bank, Yodlee, or the service lose track of the transaction offset and you get duplicate transactions or missing transactions - the service doesn't transmit all available metadata