6 ms·

I've used the ledger-likes extensively, and they have two major problems: (1) One of the hard problems in personal finance is data capture and input validation

I've used the ledger-likes extensively, and they have two major problems: (1) One of the hard problems in personal finance is data capture and input validation. Programs like QuickBooks and Xero don't just hide the GL because they're being mean, they're doing it because even experienced bookkeepers make mistakes on the general ledger all the time. "Since it's a text file it's easy to read" solves exactly the wrong problem: you want it to be easy to write, and easy to write CORRECTLY. (2) They're the sort of solution that seems fantastic if you don't actually operate at scale. But once you start dealing with tens of transactions per day it simply is not sustainable. Safe to say: I'm not really a fan.

jolmg 5y ago
> One of the hard problems in personal finance is data capture With hledger and vim, I typically search and copy a previous similar transaction, then search and copy similar items. I have a keybinding to change the date, so like 3\d changes the date of the transaction I copied to 3 days ago. Overall, being proficient in vim, capture takes very few keystrokes. Since the files are plain text, I can easily e.g. list all products I've ever bought with a `grep | sort | uniq` pipeline, to check stuff like whether I put a product identifier under a wrong account or used different identifiers for the same product. Also, since the files are plain text, I can track changes with git. That gives me clarity over changes in the sense that I can make a large change (e.g. reorganizing accounts) over historical transactions and review the changes before committing. Regarding large changes, which I think is also a type of capture, I can easily make them with `sed -i ...`, vim macros, awk, etc. > and input validation After I've captured, running hb (which is a personal alias for `hledger balance`) results in hledger raising an error if a transaction is not balanced. I can also add an assertion `= X` to the end of account changes to assert what the account is supposed to hold after the change. The number comes from physically checking the account, whether that's by checking the bank app or opening my wallet and counting. hledger will raise an error when an assertion is wrong. > Programs like QuickBooks and Xero Being unfamiliar with QuickBooks and Xero, how would these programs do capture and validation better for me?
- croo 5y ago
  This is already after the data capture part. The hard part is to remember every little transaction I had until I have vim at my hand. The only solution seems to be working more or less is to take out my phone and type it as soon as the transaction happened. And I won't try to use vim on phone ever again..
  - andrepd 5y ago
    I use GnuCash. For card transactions I simply import my bank statement .csv file every other week or so. For cash transaction, I use the app on my phone, otherwise I just forget how much I spent and where, by the time I get to my PC.
    - abdullahkhalids 5y ago
      I never really got past the importing stage in GNUcash. I want to categorize the payments into different accounts, depending on how the money was spent. But there is no easy way of doing it. Or doing it iteratively.
      - andrepd 5y ago
        When you import all your new transactions are presented. You can click (or ctrl-click to choose several) and match them to an account of your choosing. It also memorises what transactions go into what category, so after you categorise, say "PURCHASE LIDL" as Expenses.Groceries, it will automatically match future transactions with the same name to groceries as well.
  - jolmg 5y ago
    > The hard part is to remember every little transaction I had until I have vim at my hand. Most of the time, there's no need to remember. Just put the receipt in your pocket, and when you get home you can further postpone it by stacking it with other receipts on a receipt spike. It's only the cash transactions that are best not left postponed for longer than a day or 2, but if you just scribble down the important bits on a scrap of paper, you can postpone it like any other receipt.
fortran77 5y ago
The reason I use Xero is the integration with my bank, credit cards, invoicing, and payroll system.
roylez 5y ago
Have you tried `ledger xact` command? This helps to create new transactions based on previous ones. I have a small wrapper[1] for this, namely "la", which enables me to do something like "la coles 5" to create a new transaction like previous supermarket transaction with the today as date, same from and to accounts, and $5 as amount. If I want to change something, it could be as complete as "la [YYYY.MM.DD/M.D/D] coles grocery 5 credit". If someone has tens of transactions per day it is very likely they are all similar and only little extra adjustment is needed. Also I use a credit card for most of my expenses, and there are only a handful of bank transfers per month from my access account. Every first day of the month I pull out last month's credit card statement and log all the transactions. It usually takes less than 30 minutes. [1]: https://github.com/roylez/dotfiles/blob/master/misc/bin/la https://github.com/roylez/dotfiles/blob/master/misc/bin/la
minikomi 5y ago
Termux on the phone with custom ledger focused org capture templates has worked well for me Now I just use a simple discord bot to capture, which is synced to a private git repository
- VoidWhisperer 5y ago
  Is your bot that does this open sourced? This seems very interesting
  - m-p-3 5y ago
    It would be nice to see how hard it would be to convert into a Matrix bot, at least the conversation would be end-to-end encrypted.
  - minikomi 5y ago
    Sorry mate it's just an easy message parser built on discord.py, pretty bodgy code and no intention to OS Messages look like &out soba shop Soba 800 Beer 700 Or &p #custom:category payee Item (int cost) Item (int cost)
tomnipotent 5y ago
> hard problems in personal finance is data capture and input validation It's also true for business accounting. Getting data into your accounting/ERP software is always a clusterfuck, and I've yet to see a process that didn't require many hours/days of reconciliation and posting manual journal entries to close a reporting period.
- ethbr0 5y ago
  This is the data representation problem. It's in the customers' best interest to have a universal format. It's in no one else's (read: especially vendors) interest for there to be a universal format. Which is why you generally only see it when it's mandated by a coordinating party in the middle (e.g. Medicare).
  - blowski 5y ago
    In finance, would this be the open banking standard?
    - ethbr0 5y ago
      It might be partly a US vs. UK/EUR thing, where the latter are more comfortable with compelling industry to just do a thing. And the former is very uncomfortable with that. In the US, open banking would probably look more like "the IRS requires a bank to have accounts readable using these formats, or otherwise be able to provide same manually, when requested" to prod industry to do it.
  - tomnipotent 5y ago
    Sort of. The challenge is that no two businesses are really alike, so it's difficult to create a standard that can capture all the nuance of running a business. GAAP/IFRS-compliant accounting requires double-entry accounting, so many ERP systems share a lot of the same things (account, debit/credit, accruals, receipt of goods, month-end close) that are needed to spit out the three sheets and other artifacts for financial reporting. A running joke in the industry is that you don't customize your ERP for your business, but customize your business for your ERP. You end up with a lot of "middleware" that essentially adapts your operational systems into the format needed by the ERP to do its job, and your staff will design workflow processes specifically centered around how the ERP works for things like closing the books or managing inventory.
smichael 5y ago
Another data point: 95% of my data entry (all but cash) is automatic CSV download (using plaid2qif) and mostly-automatic deduplicating import (I tweak the import rules when needed). So, a daily `make import`. I have also run this from a nightly cron job. For most of us, that's a little too automatic; usually we want a bit more oversight and awareness.
hiq 5y ago
1. PTA in itself does not provide you all the tools to validate these, but it's low-level enough that you can implement any validation you want on top easily, if you're proficient with CLI tools. I don't think I would use it at the enterprise level simply because they're less commonly used, but for personal finances I find it sensible enough. Which mistake do you think is more common with PTA than with other programs? 2. What exactly would not be sustainable with PTA which would be with other programs?
slavoingilizov 5y ago
100% this. Collecting and validating data should be so easy that you don't have to think about it. Otherwise I find I can't stick to any solution/tool for too long. If discipline is required, then it's too complicated. This is why I've started looking into tools which automate the data collection. In the UK (and EU more broadly), after Open Banking got introduced there are many apps popping out which basically connect to your accounts (Credit Cards, bank accounts, etc.) and combine all data into a single place. Does Plaid not enable this in the US? It's not "100% accurate" but most enable you to do auto-categorisation, smart reporting and budgeting as well. They are not open-source but do much more than I'd ever be able to do with PTA (not because it's not supported but because it requires effort). Of the 2 bad solutions, they seem to be the less bad for me. Some example apps: Emma, Yolt, Lumio.
chromatin 5y ago
I dunno, I kinda disagree: I am not writing transactions manually, they are written by my brokerage parser, bank statement parser, blockchain parser, etc. The reporting and analytics functionality of the ledger-likes is AFAICT unrivaled, and is the REAL reason to use them (not the ostensibly easy-to-use “plain text” general ledger aspect, about which I agree with you)
avodonosov 5y ago
For data capture, what can help is just not aiming to record every cheque. Record significant spendings, and account the rest as "other" - it should be the amount allowing to balance the ledger, given the current assets, income, and the significant spendings. My problem with ledger was the absence of correct support for multi currency accounting.
megous 5y ago
Yes, it's much easier to just have accounting data in a relational database. It's much easier to work with (edits and arbitrary analysis/reporting), rather than having to use/learn some bespoke tooling and data format. It's much easier to associate entries with metadata or categories, and much simpler to add editing GUI on top if you want to, or integrate with third party data sources. (I wrote a text based ledger tool myself, and then switched to storing data in PostgreSQL)
nextaccountic 5y ago
> Programs like QuickBooks and Xero don't just hide the GL because they're being mean What is the GL?
- draebek 5y ago
  General ledger https://en.wikipedia.org/wiki/General_ledger https://en.wikipedia.org/wiki/General_ledger

jolmg 5y ago

> One of the hard problems in personal finance is data capture With hledger and vim, I typically search and copy a previous similar transaction, then search and copy similar items. I have a keybinding to change the date, so like 3\d changes the date of the transaction I copied to 3 days ago. Overall, being proficient in vim, capture takes very few keystrokes. Since the files are plain text, I can easily e.g. list all products I've ever bought with a `grep | sort | uniq` pipeline, to check stuff like whether I put a product identifier under a wrong account or used different identifiers for the same product. Also, since the files are plain text, I can track changes with git. That gives me clarity over changes in the sense that I can make a large change (e.g. reorganizing accounts) over historical transactions and review the changes before committing. Regarding large changes, which I think is also a type of capture, I can easily make them with `sed -i ...`, vim macros, awk, etc. > and input validation After I've captured, running hb (which is a personal alias for `hledger balance`) results in hledger raising an error if a transaction is not balanced. I can also add an assertion `= X` to the end of account changes to assert what the account is supposed to hold after the change. The number comes from physically checking the account, whether that's by checking the bank app or opening my wallet and counting. hledger will raise an error when an assertion is wrong. > Programs like QuickBooks and Xero Being unfamiliar with QuickBooks and Xero, how would these programs do capture and validation better for me?

croo 5y ago
This is already after the data capture part. The hard part is to remember every little transaction I had until I have vim at my hand. The only solution seems to be working more or less is to take out my phone and type it as soon as the transaction happened. And I won't try to use vim on phone ever again..
- andrepd 5y ago
  I use GnuCash. For card transactions I simply import my bank statement .csv file every other week or so. For cash transaction, I use the app on my phone, otherwise I just forget how much I spent and where, by the time I get to my PC.
  - abdullahkhalids 5y ago
    I never really got past the importing stage in GNUcash. I want to categorize the payments into different accounts, depending on how the money was spent. But there is no easy way of doing it. Or doing it iteratively.
    - andrepd 5y ago
      When you import all your new transactions are presented. You can click (or ctrl-click to choose several) and match them to an account of your choosing. It also memorises what transactions go into what category, so after you categorise, say "PURCHASE LIDL" as Expenses.Groceries, it will automatically match future transactions with the same name to groceries as well.
- jolmg 5y ago
  > The hard part is to remember every little transaction I had until I have vim at my hand. Most of the time, there's no need to remember. Just put the receipt in your pocket, and when you get home you can further postpone it by stacking it with other receipts on a receipt spike. It's only the cash transactions that are best not left postponed for longer than a day or 2, but if you just scribble down the important bits on a scrap of paper, you can postpone it like any other receipt.

fortran77 5y ago

The reason I use Xero is the integration with my bank, credit cards, invoicing, and payroll system.

roylez 5y ago

Have you tried `ledger xact` command? This helps to create new transactions based on previous ones. I have a small wrapper[1] for this, namely "la", which enables me to do something like "la coles 5" to create a new transaction like previous supermarket transaction with the today as date, same from and to accounts, and $5 as amount. If I want to change something, it could be as complete as "la [YYYY.MM.DD/M.D/D] coles grocery 5 credit". If someone has tens of transactions per day it is very likely they are all similar and only little extra adjustment is needed. Also I use a credit card for most of my expenses, and there are only a handful of bank transfers per month from my access account. Every first day of the month I pull out last month's credit card statement and log all the transactions. It usually takes less than 30 minutes. [1]: https://github.com/roylez/dotfiles/blob/master/misc/bin/la https://github.com/roylez/dotfiles/blob/master/misc/bin/la

minikomi 5y ago

Termux on the phone with custom ledger focused org capture templates has worked well for me Now I just use a simple discord bot to capture, which is synced to a private git repository

VoidWhisperer 5y ago
Is your bot that does this open sourced? This seems very interesting
- m-p-3 5y ago
  It would be nice to see how hard it would be to convert into a Matrix bot, at least the conversation would be end-to-end encrypted.
- minikomi 5y ago
  Sorry mate it's just an easy message parser built on discord.py, pretty bodgy code and no intention to OS Messages look like &out soba shop Soba 800 Beer 700 Or &p #custom:category payee Item (int cost) Item (int cost)

tomnipotent 5y ago

> hard problems in personal finance is data capture and input validation It's also true for business accounting. Getting data into your accounting/ERP software is always a clusterfuck, and I've yet to see a process that didn't require many hours/days of reconciliation and posting manual journal entries to close a reporting period.

ethbr0 5y ago
This is the data representation problem. It's in the customers' best interest to have a universal format. It's in no one else's (read: especially vendors) interest for there to be a universal format. Which is why you generally only see it when it's mandated by a coordinating party in the middle (e.g. Medicare).
- blowski 5y ago
  In finance, would this be the open banking standard?
  - ethbr0 5y ago
    It might be partly a US vs. UK/EUR thing, where the latter are more comfortable with compelling industry to just do a thing. And the former is very uncomfortable with that. In the US, open banking would probably look more like "the IRS requires a bank to have accounts readable using these formats, or otherwise be able to provide same manually, when requested" to prod industry to do it.
- tomnipotent 5y ago
  Sort of. The challenge is that no two businesses are really alike, so it's difficult to create a standard that can capture all the nuance of running a business. GAAP/IFRS-compliant accounting requires double-entry accounting, so many ERP systems share a lot of the same things (account, debit/credit, accruals, receipt of goods, month-end close) that are needed to spit out the three sheets and other artifacts for financial reporting. A running joke in the industry is that you don't customize your ERP for your business, but customize your business for your ERP. You end up with a lot of "middleware" that essentially adapts your operational systems into the format needed by the ERP to do its job, and your staff will design workflow processes specifically centered around how the ERP works for things like closing the books or managing inventory.

smichael 5y ago

Another data point: 95% of my data entry (all but cash) is automatic CSV download (using plaid2qif) and mostly-automatic deduplicating import (I tweak the import rules when needed). So, a daily `make import`. I have also run this from a nightly cron job. For most of us, that's a little too automatic; usually we want a bit more oversight and awareness.

hiq 5y ago

1. PTA in itself does not provide you all the tools to validate these, but it's low-level enough that you can implement any validation you want on top easily, if you're proficient with CLI tools. I don't think I would use it at the enterprise level simply because they're less commonly used, but for personal finances I find it sensible enough. Which mistake do you think is more common with PTA than with other programs? 2. What exactly would not be sustainable with PTA which would be with other programs?

slavoingilizov 5y ago

100% this. Collecting and validating data should be so easy that you don't have to think about it. Otherwise I find I can't stick to any solution/tool for too long. If discipline is required, then it's too complicated. This is why I've started looking into tools which automate the data collection. In the UK (and EU more broadly), after Open Banking got introduced there are many apps popping out which basically connect to your accounts (Credit Cards, bank accounts, etc.) and combine all data into a single place. Does Plaid not enable this in the US? It's not "100% accurate" but most enable you to do auto-categorisation, smart reporting and budgeting as well. They are not open-source but do much more than I'd ever be able to do with PTA (not because it's not supported but because it requires effort). Of the 2 bad solutions, they seem to be the less bad for me. Some example apps: Emma, Yolt, Lumio.

chromatin 5y ago

I dunno, I kinda disagree: I am not writing transactions manually, they are written by my brokerage parser, bank statement parser, blockchain parser, etc. The reporting and analytics functionality of the ledger-likes is AFAICT unrivaled, and is the REAL reason to use them (not the ostensibly easy-to-use “plain text” general ledger aspect, about which I agree with you)

avodonosov 5y ago

For data capture, what can help is just not aiming to record every cheque. Record significant spendings, and account the rest as "other" - it should be the amount allowing to balance the ledger, given the current assets, income, and the significant spendings. My problem with ledger was the absence of correct support for multi currency accounting.

megous 5y ago

Yes, it's much easier to just have accounting data in a relational database. It's much easier to work with (edits and arbitrary analysis/reporting), rather than having to use/learn some bespoke tooling and data format. It's much easier to associate entries with metadata or categories, and much simpler to add editing GUI on top if you want to, or integrate with third party data sources. (I wrote a text based ledger tool myself, and then switched to storing data in PostgreSQL)

nextaccountic 5y ago

> Programs like QuickBooks and Xero don't just hide the GL because they're being mean What is the GL?

draebek 5y ago
General ledger https://en.wikipedia.org/wiki/General_ledger https://en.wikipedia.org/wiki/General_ledger