13 ms·

Things I learned while developing a billing system

273 points by arnon 6y ago

gerikson 6y ago
Obligatory: https://blog.plover.com/prog/Moonpig.html https://blog.plover.com/prog/Moonpig.html
- arnon 6y ago
  Thanks for that! I haven't seen this... I'm glad to see he also has the same thought about floating points. Also: > Happily, Moonpig did not have to deal with multiple currencies. That would have added tremendous complexity to the financial calculations, and I am not confident that Rik and I could have gotten it right in the time available. This is one of the things we deal with which is quite challenging.
  - gerikson 6y ago
    Yeah, I realized that this was probably the biggest difference!
bidirectional 6y ago
> A common wisdom in database design is “never using floating-point numbers for money” Common, and in my experience totally wrong. It's the most pervasive cargo culting I've experienced amongst developers, where people with 0 experience with financial applications will recoil if you argue against it. In my time developing applications for front office at an investment bank, floating point often works best. Of course in your case though, for a billing system, the method you describe is obviously the right one.
- devwastaken 6y ago
  How did you handle the properties of floating point? The reason people recoil at that more than likely has to do with lack of knowledge in how floats are wrangled to ensure they're accurate.
  - stevesimmons 6y ago
    Which properties? For front-office risk and pricing calcs, speed of calculation matters most. Hence IEEE754 float64. Fixed precision decimals really only matter for middle and back-office, for trade confirmation and settlement.
    - tantalor 6y ago
      What's the difference between front/middle/back office?
      - arthurcolle 6y ago
        In the context of a bulge bracket investment bank, it's basically like this: Front office - S&T (sales & trading), i.e. revenue generating activity. Usually includes any quants/quant developers actively working on things that make money Middle office - operations. handles settlements, confirms, and generally anything related to the post-trade flow that is "after the trade is booked" Back office - accounting, legal, engineering, IT support, HR. Anything that isn't a revenue center that also isn't even tangentially facing revenue generating operations
  - bidirectional 6y ago
    Which properties do you mean in particular? The main point is just that finance doesn't necessarily require perfect accuracy. It does when Bob sends Alice $1 and their account balances must line up perfectly, it doesn't when you're implementing models which already have greater error than floating point could imbue. Not to mention that decimals, integers and rationals cannot compute something as simple as compound interest without lack of accuracy, so they're not going to save you when you're implementing Black-Scholes.
    - lamp987 6y ago
      >rationals cannot compute something as simple as compound interest without lack of accuracy Care to elaborate?
- mamcx 6y ago
  Floating point is bad as null are, and is defended by the same apologist of null: "I don't make mistakes... so is good!". Floating point ARE a major source of errors across everyone that use them. ARE infectious. ARE semantically wrong. ARE not made for financial calculation. ARE WRONG. Period. Just because under a lot of discipline (or luck, or just "assume" is working but nobody have checked, or work before but how knows if today?) not make it a good choice for financial/money. Is the same error when people think old String types can be used in the unicode world, instead of have a proper type for that. Luck help a lot. But is not something to be proud about.
  - bidirectional 6y ago
    Well no, the key difference between null and floating point is that for all its flaws, floating point is by far the fastest way of doing non-integer numerical computation. Null is just an ugly convenience hack of arguable merit, floats are fundamental. They're not a major source of error when you're implementing a model which is already inexact to a far larger degree than the problems caused by floats. Black-Scholes does not perfectly price an option, your bootstrapped curve is not a perfect predictor of market conditions in 28 years time. These are the problems faced in front-office finance, the error is already so far beyond 1 + 0.1 not perfectly matching 1.1 that it's not worth caring about. I've worked on applications where users just wanted to see numbers to the nearest 100k so they could model out a few trades they planned to make over the phone. When you're working on a retail banking app where you need to track customer's balances, or an accounting system, or anything like that, then sure, floating point would be malpractice. That is nothing like any of the applications I've worked on in the financial field.
  - cameronh90 6y ago
    Finance isn't just banking and accounting. A huge amount of finance is modelling, simulations, signal generation, etc. where being fast is often way more important than being completely 100% accurate. In financial modelling, errors from floating point are going to be insignificant compared to all the other assumptions you make in your models. I have worked in finance for my entire career, and everyone uses floating point arithmetic for dealing with money on the modelling side (sometimes $0.01 = 1.0, sometimes $1 = 1.0, depends on the institution/currency/convention/context, but we have to deal with fractional money anyway). They do NOT use it on the accounting/back office side. That would be a spectacularly bad idea. Those systems are designed for accuracy to the penny.
- bxparks 6y ago
  You are getting a lot of down-votes, but you are correct, and I gave you some upvotes, at least one. If the application is doing financial modeling and estimations, where only 1-3 decimal place of accuracy is needed, then floating point is the right choice. It greatly simplifies the app. If the application is doing accounting, payments and billing, where people expect accuracy to the penny or more (e.g. to 1/10000 of a penny), then it needs to use a Decimal type.
- smallnamespace 6y ago
  Yes, every time someone categorically declaims that 'floats and money should never mix', I question whether they've ever met a real accountant. Accountants frequently spend all day in Excel. Excel uses floats for all computations (doubles, to be precise) [1]. Now, in all fairness, an accountant and developer's relationship to the numbers is rather different. For an accountant, the risk of floats blowing up is largely mitigated by the fact that they have a close, intimate relationship with the actual numbers. The responsible accountant should always deliver numbers they have personally reviewed, while a developer is usually automating a process, generating numbers that have yet to touch a human eye, so there is rather less room for error. Still, it's simply untrue that one should never use floats for money. Many of the cases where floats would generate bad results are also problematic for simple alternatives. For example, fixnums are simply 'floats that can't float', so you need to be able to guarantee a fixed range ahead of time. Understanding basic numerical analysis is unavoidable to writing correct code. [1] https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft...
- FabHK 6y ago
  FWIW, front office is the one place where floating points for money might be sensible, because in most cases in the front office you don't care about pennies. Instead, what you want to get right is derivatives ("risk"), which you compute with finite differences ("bumps") or somewhat more advanced methods (adjoint automatic differentiation, basically the backward propagation algorithm). So, yeah, if you deal with risk and PVs (in other words, maths) rather than actual cash flows, sure, knock yourself out, use floats. That does not invalidate the general advice to avoid floats for money (which is probably why you were downvoted).
epa 6y ago
All of these features are fairly basic and routine and should have been part of scoping. The issue is that engineers rarely speak to finance teams to gather input which leads to these 'surprises'. The article is equivalent to "how hard could it be to win my lawsuit, i researched the law".
- encoderer 6y ago
  Ok, genius, you go write a blog post then.
  - yellowyacht 6y ago
    > Ok, genius, you go write a blog post then. That was unnecessarily antagonistic
    - encoderer 6y ago
      Totally disagree. I came here for commentary on the blog post not a cliche HackerNews trope. When I replied, his was the top comment. Honestly, guys like that need a nudge every now and again to train their filter. A downvote just isn't enough. Welcome to the community.
      - dang 6y ago
        The solution to bad comments is not making the thread even worse, and the solution to "cliche HackerNews trope" is not to add another cliche trope with more aggression. That only contributes to taking us further into hell. The intended spirit of the site is just the opposite of that. If you're familiar enough with HN to complain about tropes, I assume you're aware of this? If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- spelunker 6y ago
  To be fair, this blog is apparently written by a product manager, who presumably discovered these features by doing research. And probably asking finance teams.
mmcconnell1618 6y ago
Another fun use case is when a customer is charged for their invoice via credit card and then 60 days later, you get a chargeback from the bank. Now you have to unwind the previous invoices and entitlement systems to figure out if you should fight the chargeback, cancel the subscription or some other non-standard process.
canada_dry 6y ago
Here's a lesson (I can laugh about now that I'm retired): one of the first billing systems I ever wrote (in the 90's) had invoices with a mask of '$ZZ,ZZ9.99' based on interviews with staff. About a year later I get an urgent call from the owner. It seems a client was under-billed about $100K dollars due to truncated invoice. Subsequently, whenever interviewing clients for requirements I'd mention this and it usually resulted in padding their specs.
- mleonhard 6y ago
  Did you truncate instead of checking for overflow?
  - selcuka 6y ago
    Probably human error because the input widget did not allow that many digits.
mr-wendel 6y ago
My personal favorite: how long is a month? - An even 30 days = Easy to explain and do math with. Adding/subtracting 1 month can leave you in the same month or skip February entirely! Anniversary dates bounce around from month-to-month. Maps poorly to yearly calculations. - However many days >this< month has = Harder to explain and do math with. Months without 31 days can get skipped when adding/subtracting 1 month. Recurring events get pushed away from the end of the month to cluster around the beginning of the month. It does result in a steady anniversary date for edge cases. - Closest numeric day, but prev/next month (e.g. Jan 31 -> Feb 28): Easy and hard to explain and do math with. It's great for ensuring events only ever happen once per calendar month, but adds extra ambiguity (e.g. Jan 31 > Feb 28 > March 30? ... or March 28th?) - 30.4... = Just no... except for the few times this is right and you need consistency to avoid unfairly comparing 28 days vs 31 days. The consequence is some very surprising things the first time you see them. Some billing systems simply avoid doing work past the 28th of each month (either doing it a bit early or a bit late). Some just embrace the weirdness (whichever you flavor you pick) and you get used to the quirks (e.g. end-of-month lulls and start-of-month spikes).
- gergely 6y ago
  These use cases are also "fun" if you add timezones as well into the mix!
  - deckard1 6y ago
    shudder I can only imagine needing to toss in standard/daylight timezone switch in there. Billing code really is the pinnacle of developer pain. It mixes the arbitrariness of special case business and customer rules with the absolute horror of time and date math.
    - pc86 6y ago
      Not only that but in T-SQL, `AT TIME ZONE` takes DST into account but not in the names. So something like: SELECT CreatedDate AT TIME ZONE 'UTC' AT TIME ZONE 'Eastern Standard Time' FROM MyTable would in fact return `CreatedDate` at Eastern Daylight Time if executed today.
kjrose 6y ago
Having worked on billing and accounting systems for decades now. I am constantly amazed by all of the edge cases and situations that have to be handled to ensure that every bill is correct and nothing is missed or misbilled. Especially since the oil firms out here will reject an invoice if it is off by as little as a penny. At first it seems like it's so easy and straightforward but when you are looking at thousands if not millions of bills a year and the myriad methods you need to compensate for to ensure its accepted by large firms.... well. I can say nothing surprises me anymore. Including hardware level errors causing an issue no one expected with a final number.
- throwawayboise 6y ago
  Right -- accounting seems simple. It's just debits and credits and adding it all up. Until you get into it, and you understand why "Accounting" can easily be a department of dozens or hundreds of people.
  - kjrose 6y ago
    There's a reason to separate your accounts receivable from payables. To separate invoicing from receipts. And so on and so on. You don't realize it until you suddenly realize why it was such a bad idea to have one point of failure for everything.
- cosmodisk 6y ago
  The principles behind it all are pretty simple and well established. What makes it complicated are the people and poorly designed systems.
- flukus 6y ago
  > Especially since the oil firms out here will reject an invoice if it is off by as little as a penny. Half of them will reject you for being wrong, the other half will reject you for being right because it doesn't line up with their floating point errors in excel.
ehutch79 6y ago
There are absolutely time when you need to deal with amounts smaller than the smallest division of a currency. For example: 1 ea @ $0.00589 Normally this is when ordering in the thousands or millions of small things, but you need to record that fractional size.
- Kiro 6y ago
  Yeah, this immediately broke my idea of only dealing with cents and integers. All of a sudden I wanted to price something $0.001 and it all came crashing down.
  - ranguna 6y ago
    In the real world when you place a transaction, how do you pay a fraction of a cent?
    - Kiro 6y ago
      As the parent says it's the unit price (with quantity of millions) so the end amount will be rounded but all the steps before that need to be able to handle fractions.
berkes 6y ago
In my last financial product we added 'bitcoin' and 'festival tokens' as currencies in the backend. At first for fun. But we found out the client devs (web, mobile apps) had severe difficulties and wanted it removed. Turned out they all implemented currencies wrong. Either with hardcoded decimals or with localised in and outputs that would break if a client changed their locales. So now, my favorite best practice for any financial data handling is that: ensure your system can handle Bitcoin (8 decimal places) and festival tokens (missing currency symbol, zero decimals). Anywhere this leads to trouble is a red flag and will probably cause trouble later on. Now at least you are aware.
- michael1999 6y ago
  That's great advice. I'll add it to my list of round-tripping German, and Japanese text with emojis, and changing the timezone.
  - arthurcolle 6y ago
    What do you mean by round tripping German here? I think I get the issues with the other two, but curious about the first thing!
    - deleted 6y ago
      [deleted]
    - dan-robertson 6y ago
      Not sure about round tripping but perhaps the longer words might stress test webpage layouts.
      - FabHK 6y ago
        Round trip = sending it from the input/front end into the system/back end (whatever it is, database/file system/???) and back to the front, to confirm that what comes out is what you put in. For German, I'd assume it's just about the special characters (äöüß), ie testing that encodings are somewhat correct (at least beyond ASCII).
brixon 6y ago
"Whatever limitations you plan for, plan for how to bypass them too. This will happen." The world is not straightforward, allow an admin to fix/change anything and when they get tired of making some change then code that new path in the system. Working with or writing systems that do not allow admin overriding is painful.
pfranz 6y ago
One edge-case I've noticed with Apple's subscription billing and heard other people talk about when implementing billing systems is changing currencies/countries. Apple, in line with most people's expectations, when you cancel a subscription it doesn't cancel immediately and prorate a refund. It just stops future billing and lets you serve out your remaining cycle (gyms and other meatspace places do this). The problem is you then have to wait until your subscription expires to transfer your account to a new country/currency.
loop0 6y ago
I went through the same issues 6 years ago, had to create the entire billing system from scratch, and there are so many corner cases I had to deal with, specially with plan changes on recurring subscriptions, had to deal with world wide timezones for the invoice creations and lots of other stuff. I just wish there was a pluggable, robust and flexible solution to use instead of reinventing the wheel.
deleted 6y ago
[deleted]
kube-system 6y ago
> Store money including the smallest possible subdivisions. > For example: > $100 US may be stored as 10000 Eh, the US dollar is often subdivided further than that. Here's four examples, in different industries: http://cdn.radioiowa.com/wp-content/uploads/2011/12/gas-pump-12-19-2011.jpg http://cdn.radioiowa.com/wp-content/uploads/2011/12/gas-pump... https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ https://finance.yahoo.com/u/yahoo-finance/watchlists/most-active-penny-stocks https://finance.yahoo.com/u/yahoo-finance/watchlists/most-ac... https://www.digikey.com/en/products/detail/microchip-technology/ATMEGA328P-AUR/3789455 https://www.digikey.com/en/products/detail/microchip-technol...
- kbenson 6y ago
  Those are unit prices or interval prices. Any transaction will almost definitely be in an amount of dollars and centers if dealing in US, rounding to a whole cent based on the total units or interval. For example, if you're paying #3.089 a gallon at the pump, and you pump exactly one gallon, you don't pay $3.089, you'll pay $3.09, and neither your account or the account you're paying into will need to know or care about the tenth of a cent difference, because our monetary system doesn't really deal with denominations that small in transactions.
  - deckard1 6y ago
    yeah but this is where you really have to be careful where you do your rounding. 3.089 * 10,000 = 30890 3.09 * 10,000 = 30900 A one cent rounding added up to $10 difference. It's easy to screw this up in code, sending values to databases or across APIs, etc. Also reminds me of the plot of Office Space. Which is a bit humorous since they screwed up their own scamming scheme in similar fashion. https://www.youtube.com/watch?v=yZjCQ3T5yXo https://www.youtube.com/watch?v=yZjCQ3T5yXo
    - kbenson 6y ago
      There a subtlety in what I was saying that's being lost (which means you aren't wrong, but what you're stating doesn't necessarily apply specifically to what I was trying to say). You don't round rates, or prices, but you do round accounts because those represent real money. Your account can't actually have a fraction of a cent in it. Either the account it came from keeps the cent, or your account gets it. In a very real way, an account can be thought of as an integer number of cents (because often it's implemented that way). That may or may not make sense for your inventory or service system, where sub-cent rates make sense because the number of units is often greater than one. To bring this back to the original comment and the items it was referencing, store money (real money, like accounts) in the smallest possible subdivision (cents for USD), but that doesn't need to apply to pricing, which is a potential amount of money. Potential money needs to be changed into real money at the time of a transaction, and real money cares about quantities it's possible to have, and it's not really possible (or at least useful, in most cases) to have less than the smallest possible subdivisible amount of a currency. You're entirely correct though that you can't assume that cents is enough to accurately model everything to do with a business that works in USD, and ignoring that will result in problems like you showed.
sorbits 6y ago
I would base anything dealing with money on a double entry accounting system. The issues that arise are then a question about how to translate them to journal entries, which will require accounting experience, and maybe your chart of accounts needs to be revised, but the core system should be fairly stable, and voiding an invoice by issuing a credit note comes automatically, as you’re working with an append-only ledger. The issues about prepaid plans should also be handled, as payments for services not rendered yet should not be recognized as income, but instead kept as a liability: The OP mentions their system is used by 15,000 customers, so I would give each customer their own account (in the chart of accounts). The main technical issue is what datatype to use for money, the rest are problems solved by following general accounting principles, though I fear that a lot of programmers out there are reinventing the wheel (in suboptimal ways).
- throwawayboise 6y ago
  > payments for services not rendered yet should not be recognized as income, but instead kept as a liability This entirely depends on whether you are operating on a cash or accrual basis. Both approach are valid (at least in the USA) and cash basis is often used by small businesses.
  - sorbits 6y ago
    I use the word “should” as per RFC 2119, i.e. recommended (as opposed to “must” = required). Though the IRS does limit the types of businesses that can do cash-basis accounting, it would be businesses without inventory, and who does not offer credit to their customers, e.g. a hairdresser would probably use cash-basis accounting, but more complicated businesses would not, certainly not a business that needs its own billing system :)
- Edmond 6y ago
  I would second this advice :) I fumbled and bumbled my way to this realization while trying to build a billing system intended to tolerate all sorts of invoicing and reversal scenarios. More precisely, it doesn't matter what the nature of the billing is, ie recurring (monthly, quarterly..etc) vs one-time; You want to build a system around invoicing for discrete items and resolving those invoices against various criteria (payment received, credits issued, cancelled plans...etc).
breischl 6y ago
My favorite: when do you do rounding? For instance, with proration. Take each line item, prorate it, round it, then add them all together. Totally reasonable. Now add all the line items together, prorate the total, and round it. Also reasonable, but there's a decent chance the number is different by a few pennies because of rounding differences. If you chained more calculations the differences would compound. Whichever way you choose, somebody is going to whip out their calculator and tell you that you did it wrong (but only when they come out better the other way). This applies anywhere you're doing multiplication or division. Discounts, proration, taxes, "cashback", "store credit dividend", whatever.
- ianmcgowan 6y ago
  In my world, we keep a running total of the rounded prorated amounts, and then the last item (n) becomes (total - total up to n-1) to make sure amounts match. It can still be a problem if the last value is very small however.
  - breischl 6y ago
    Yeah, that's another way to do it. Another way to look at that is you're doing the proration on the final amount, and then distributing the rounding difference across the line items. Which is nice because you get the right total, though it can look weird when line items for the exact same thing are off by a penny from each other.
unixhero 6y ago
Billing is very hard. That is why systems like Geneva is used.
hn_throwaway_99 6y ago
OK, I read the first bulletpoint and just thought "How did this get upvoted this much, this makes no sense": > Money isn’t always decimal > A common wisdom in database design is “never using floating-point numbers for money”. Some recommend using the MONEY datatype, while others tell you to use DECIMAL. > Both of these are wrong. Sure, yeah, in the US and most of Europe, money is decimal. > That’s certainly not the case in Japan – you can’t charge 2500.50 JP¥. The author has a very odd definition of "Decimal". Integers are most certainly decimals. The fact that in Japan you can't charge fractions of a a yen doesn't mean that Decimal isn't actually an excellent choice to represent that currency, because it certainly can represent any yen value you would need. The whole reason you don't want to use float is because it cannot represent certain valid monetary values. As others have pointed out, storing values as "the smallest division", e.g. pennies for USD, is just wrong, because there are many contexts where you have to represent fractions of a cent <insert Superman 3 joke here>.
- scrollaway 6y ago
  There is a larger point here, that 2500.5 JPY is not a valid amount to charge, thus having it be possible is a problem. Whereas 10000 USD cents is a valid value. Integer storage of money is my favourite as well to be honest. It's easy to work with and reason about, and it just makes decimalization a display issue.
  - hn_throwaway_99 6y ago
    But as others have pointed out, actual value can be fractional, it's just not priced that way. I mean, in the US it's actually pretty standard for accounting applications to use a scale of 4 (4 units to the right of the decimal point, the default for MONEY), even though people are never charged fractions of a cent. Simplest example someone gave is gas station pricing, in which prices often end in 9/10ths of a cent - the price is calculated that way, and only rounded when the customer pays.
- numpad0 6y ago
  Isn't the "all prices are pennies" approach the way USD is handled in practice, like "$399.99"? Certainly no one does irrational divisions like "$24 1/3rd". There might be a small paradigm shift that USD or EUR users might feel if they are used to think of supplementary units genuinely as divisions of full units, but I think most systems do consider the main unit as a sum of smallest denomination and not the or any other ways around.
Rafert 6y ago
> Store money including the smallest possible subdivisions. This is fine as long as everybody follows ISO4217. In my experience many 3rd party systems _mostly_ do it and of course you get bitten by the edge cases. For example, Stripe decided HUF doesn't have 2 decimals (I know that's how it's used in practice today but we're talking standards and system interoperability here), or that ISK subdivides into hundreds ("cents") in contrast to the ISO tables. Compare https://stripe.com/docs/currencies https://stripe.com/docs/currencies with http://currency-iso.org/en/home/tables/table-a1.html http://currency-iso.org/en/home/tables/table-a1.html. As Stripe notes for UGX, backing out of this is nearly impossible because of the subtle break in backwards compatibility. And who knows how other payment providers deviate in their own way. For my own sanity and that of my coworkers (e.g. easier analysis for extracted tables in the data warehouse) I strongly prefer avoiding this headache and use decimal instead. You can do the appropriate rounding for a currency in your money object as for example https://www.martinfowler.com/eaaCatalog/money.html https://www.martinfowler.com/eaaCatalog/money.html does.
bellttyler 6y ago
Excellent article! I've personally been working with billing systems for 2+ years and have felt all those pains.
_orcaman_ 6y ago
Super interesting!
xyzzy21 6y ago
Great article! And it only schemes the surface. There's also PO-based vs. CC-based payment which have different sequencing of the details. You have one I run into all the time: Net-30 vs. Net-60 vs. Net-90. As well as linkages to quotations and terms. B2B takes all this to another level.

273 points by arnon 6y ago

gerikson 6y ago

Obligatory: https://blog.plover.com/prog/Moonpig.html https://blog.plover.com/prog/Moonpig.html

arnon 6y ago
Thanks for that! I haven't seen this... I'm glad to see he also has the same thought about floating points. Also: > Happily, Moonpig did not have to deal with multiple currencies. That would have added tremendous complexity to the financial calculations, and I am not confident that Rik and I could have gotten it right in the time available. This is one of the things we deal with which is quite challenging.
- gerikson 6y ago
  Yeah, I realized that this was probably the biggest difference!

bidirectional 6y ago

> A common wisdom in database design is “never using floating-point numbers for money” Common, and in my experience totally wrong. It's the most pervasive cargo culting I've experienced amongst developers, where people with 0 experience with financial applications will recoil if you argue against it. In my time developing applications for front office at an investment bank, floating point often works best. Of course in your case though, for a billing system, the method you describe is obviously the right one.

devwastaken 6y ago
How did you handle the properties of floating point? The reason people recoil at that more than likely has to do with lack of knowledge in how floats are wrangled to ensure they're accurate.
- stevesimmons 6y ago
  Which properties? For front-office risk and pricing calcs, speed of calculation matters most. Hence IEEE754 float64. Fixed precision decimals really only matter for middle and back-office, for trade confirmation and settlement.
  - tantalor 6y ago
    What's the difference between front/middle/back office?
    - arthurcolle 6y ago
      In the context of a bulge bracket investment bank, it's basically like this: Front office - S&T (sales & trading), i.e. revenue generating activity. Usually includes any quants/quant developers actively working on things that make money Middle office - operations. handles settlements, confirms, and generally anything related to the post-trade flow that is "after the trade is booked" Back office - accounting, legal, engineering, IT support, HR. Anything that isn't a revenue center that also isn't even tangentially facing revenue generating operations
- bidirectional 6y ago
  Which properties do you mean in particular? The main point is just that finance doesn't necessarily require perfect accuracy. It does when Bob sends Alice $1 and their account balances must line up perfectly, it doesn't when you're implementing models which already have greater error than floating point could imbue. Not to mention that decimals, integers and rationals cannot compute something as simple as compound interest without lack of accuracy, so they're not going to save you when you're implementing Black-Scholes.
  - lamp987 6y ago
    >rationals cannot compute something as simple as compound interest without lack of accuracy Care to elaborate?
mamcx 6y ago
Floating point is bad as null are, and is defended by the same apologist of null: "I don't make mistakes... so is good!". Floating point ARE a major source of errors across everyone that use them. ARE infectious. ARE semantically wrong. ARE not made for financial calculation. ARE WRONG. Period. Just because under a lot of discipline (or luck, or just "assume" is working but nobody have checked, or work before but how knows if today?) not make it a good choice for financial/money. Is the same error when people think old String types can be used in the unicode world, instead of have a proper type for that. Luck help a lot. But is not something to be proud about.
- bidirectional 6y ago
  Well no, the key difference between null and floating point is that for all its flaws, floating point is by far the fastest way of doing non-integer numerical computation. Null is just an ugly convenience hack of arguable merit, floats are fundamental. They're not a major source of error when you're implementing a model which is already inexact to a far larger degree than the problems caused by floats. Black-Scholes does not perfectly price an option, your bootstrapped curve is not a perfect predictor of market conditions in 28 years time. These are the problems faced in front-office finance, the error is already so far beyond 1 + 0.1 not perfectly matching 1.1 that it's not worth caring about. I've worked on applications where users just wanted to see numbers to the nearest 100k so they could model out a few trades they planned to make over the phone. When you're working on a retail banking app where you need to track customer's balances, or an accounting system, or anything like that, then sure, floating point would be malpractice. That is nothing like any of the applications I've worked on in the financial field.
- cameronh90 6y ago
  Finance isn't just banking and accounting. A huge amount of finance is modelling, simulations, signal generation, etc. where being fast is often way more important than being completely 100% accurate. In financial modelling, errors from floating point are going to be insignificant compared to all the other assumptions you make in your models. I have worked in finance for my entire career, and everyone uses floating point arithmetic for dealing with money on the modelling side (sometimes $0.01 = 1.0, sometimes $1 = 1.0, depends on the institution/currency/convention/context, but we have to deal with fractional money anyway). They do NOT use it on the accounting/back office side. That would be a spectacularly bad idea. Those systems are designed for accuracy to the penny.
bxparks 6y ago
You are getting a lot of down-votes, but you are correct, and I gave you some upvotes, at least one. If the application is doing financial modeling and estimations, where only 1-3 decimal place of accuracy is needed, then floating point is the right choice. It greatly simplifies the app. If the application is doing accounting, payments and billing, where people expect accuracy to the penny or more (e.g. to 1/10000 of a penny), then it needs to use a Decimal type.
smallnamespace 6y ago
Yes, every time someone categorically declaims that 'floats and money should never mix', I question whether they've ever met a real accountant. Accountants frequently spend all day in Excel. Excel uses floats for all computations (doubles, to be precise) [1]. Now, in all fairness, an accountant and developer's relationship to the numbers is rather different. For an accountant, the risk of floats blowing up is largely mitigated by the fact that they have a close, intimate relationship with the actual numbers. The responsible accountant should always deliver numbers they have personally reviewed, while a developer is usually automating a process, generating numbers that have yet to touch a human eye, so there is rather less room for error. Still, it's simply untrue that one should never use floats for money. Many of the cases where floats would generate bad results are also problematic for simple alternatives. For example, fixnums are simply 'floats that can't float', so you need to be able to guarantee a fixed range ahead of time. Understanding basic numerical analysis is unavoidable to writing correct code. [1] https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft...
FabHK 6y ago
FWIW, front office is the one place where floating points for money might be sensible, because in most cases in the front office you don't care about pennies. Instead, what you want to get right is derivatives ("risk"), which you compute with finite differences ("bumps") or somewhat more advanced methods (adjoint automatic differentiation, basically the backward propagation algorithm). So, yeah, if you deal with risk and PVs (in other words, maths) rather than actual cash flows, sure, knock yourself out, use floats. That does not invalidate the general advice to avoid floats for money (which is probably why you were downvoted).

epa 6y ago

All of these features are fairly basic and routine and should have been part of scoping. The issue is that engineers rarely speak to finance teams to gather input which leads to these 'surprises'. The article is equivalent to "how hard could it be to win my lawsuit, i researched the law".

encoderer 6y ago
Ok, genius, you go write a blog post then.
- yellowyacht 6y ago
  > Ok, genius, you go write a blog post then. That was unnecessarily antagonistic
  - encoderer 6y ago
    Totally disagree. I came here for commentary on the blog post not a cliche HackerNews trope. When I replied, his was the top comment. Honestly, guys like that need a nudge every now and again to train their filter. A downvote just isn't enough. Welcome to the community.
    - dang 6y ago
      The solution to bad comments is not making the thread even worse, and the solution to "cliche HackerNews trope" is not to add another cliche trope with more aggression. That only contributes to taking us further into hell. The intended spirit of the site is just the opposite of that. If you're familiar enough with HN to complain about tropes, I assume you're aware of this? If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
spelunker 6y ago
To be fair, this blog is apparently written by a product manager, who presumably discovered these features by doing research. And probably asking finance teams.

mmcconnell1618 6y ago

Another fun use case is when a customer is charged for their invoice via credit card and then 60 days later, you get a chargeback from the bank. Now you have to unwind the previous invoices and entitlement systems to figure out if you should fight the chargeback, cancel the subscription or some other non-standard process.

canada_dry 6y ago

Here's a lesson (I can laugh about now that I'm retired): one of the first billing systems I ever wrote (in the 90's) had invoices with a mask of '$ZZ,ZZ9.99' based on interviews with staff. About a year later I get an urgent call from the owner. It seems a client was under-billed about $100K dollars due to truncated invoice. Subsequently, whenever interviewing clients for requirements I'd mention this and it usually resulted in padding their specs.

mleonhard 6y ago
Did you truncate instead of checking for overflow?
- selcuka 6y ago
  Probably human error because the input widget did not allow that many digits.

mr-wendel 6y ago

My personal favorite: how long is a month? - An even 30 days = Easy to explain and do math with. Adding/subtracting 1 month can leave you in the same month or skip February entirely! Anniversary dates bounce around from month-to-month. Maps poorly to yearly calculations. - However many days >this< month has = Harder to explain and do math with. Months without 31 days can get skipped when adding/subtracting 1 month. Recurring events get pushed away from the end of the month to cluster around the beginning of the month. It does result in a steady anniversary date for edge cases. - Closest numeric day, but prev/next month (e.g. Jan 31 -> Feb 28): Easy and hard to explain and do math with. It's great for ensuring events only ever happen once per calendar month, but adds extra ambiguity (e.g. Jan 31 > Feb 28 > March 30? ... or March 28th?) - 30.4... = Just no... except for the few times this is right and you need consistency to avoid unfairly comparing 28 days vs 31 days. The consequence is some very surprising things the first time you see them. Some billing systems simply avoid doing work past the 28th of each month (either doing it a bit early or a bit late). Some just embrace the weirdness (whichever you flavor you pick) and you get used to the quirks (e.g. end-of-month lulls and start-of-month spikes).

gergely 6y ago
These use cases are also "fun" if you add timezones as well into the mix!
- deckard1 6y ago
  shudder I can only imagine needing to toss in standard/daylight timezone switch in there. Billing code really is the pinnacle of developer pain. It mixes the arbitrariness of special case business and customer rules with the absolute horror of time and date math.
  - pc86 6y ago
    Not only that but in T-SQL, `AT TIME ZONE` takes DST into account but not in the names. So something like: SELECT CreatedDate AT TIME ZONE 'UTC' AT TIME ZONE 'Eastern Standard Time' FROM MyTable would in fact return `CreatedDate` at Eastern Daylight Time if executed today.

kjrose 6y ago

Having worked on billing and accounting systems for decades now. I am constantly amazed by all of the edge cases and situations that have to be handled to ensure that every bill is correct and nothing is missed or misbilled. Especially since the oil firms out here will reject an invoice if it is off by as little as a penny. At first it seems like it's so easy and straightforward but when you are looking at thousands if not millions of bills a year and the myriad methods you need to compensate for to ensure its accepted by large firms.... well. I can say nothing surprises me anymore. Including hardware level errors causing an issue no one expected with a final number.

throwawayboise 6y ago
Right -- accounting seems simple. It's just debits and credits and adding it all up. Until you get into it, and you understand why "Accounting" can easily be a department of dozens or hundreds of people.
- kjrose 6y ago
  There's a reason to separate your accounts receivable from payables. To separate invoicing from receipts. And so on and so on. You don't realize it until you suddenly realize why it was such a bad idea to have one point of failure for everything.
cosmodisk 6y ago
The principles behind it all are pretty simple and well established. What makes it complicated are the people and poorly designed systems.
flukus 6y ago
> Especially since the oil firms out here will reject an invoice if it is off by as little as a penny. Half of them will reject you for being wrong, the other half will reject you for being right because it doesn't line up with their floating point errors in excel.

ehutch79 6y ago

There are absolutely time when you need to deal with amounts smaller than the smallest division of a currency. For example: 1 ea @ $0.00589 Normally this is when ordering in the thousands or millions of small things, but you need to record that fractional size.

Kiro 6y ago
Yeah, this immediately broke my idea of only dealing with cents and integers. All of a sudden I wanted to price something $0.001 and it all came crashing down.
- ranguna 6y ago
  In the real world when you place a transaction, how do you pay a fraction of a cent?
  - Kiro 6y ago
    As the parent says it's the unit price (with quantity of millions) so the end amount will be rounded but all the steps before that need to be able to handle fractions.

berkes 6y ago

In my last financial product we added 'bitcoin' and 'festival tokens' as currencies in the backend. At first for fun. But we found out the client devs (web, mobile apps) had severe difficulties and wanted it removed. Turned out they all implemented currencies wrong. Either with hardcoded decimals or with localised in and outputs that would break if a client changed their locales. So now, my favorite best practice for any financial data handling is that: ensure your system can handle Bitcoin (8 decimal places) and festival tokens (missing currency symbol, zero decimals). Anywhere this leads to trouble is a red flag and will probably cause trouble later on. Now at least you are aware.

michael1999 6y ago
That's great advice. I'll add it to my list of round-tripping German, and Japanese text with emojis, and changing the timezone.
- arthurcolle 6y ago
  What do you mean by round tripping German here? I think I get the issues with the other two, but curious about the first thing!
  - deleted 6y ago
    [deleted]
  - dan-robertson 6y ago
    Not sure about round tripping but perhaps the longer words might stress test webpage layouts.
    - FabHK 6y ago
      Round trip = sending it from the input/front end into the system/back end (whatever it is, database/file system/???) and back to the front, to confirm that what comes out is what you put in. For German, I'd assume it's just about the special characters (äöüß), ie testing that encodings are somewhat correct (at least beyond ASCII).

brixon 6y ago

"Whatever limitations you plan for, plan for how to bypass them too. This will happen." The world is not straightforward, allow an admin to fix/change anything and when they get tired of making some change then code that new path in the system. Working with or writing systems that do not allow admin overriding is painful.

pfranz 6y ago

One edge-case I've noticed with Apple's subscription billing and heard other people talk about when implementing billing systems is changing currencies/countries. Apple, in line with most people's expectations, when you cancel a subscription it doesn't cancel immediately and prorate a refund. It just stops future billing and lets you serve out your remaining cycle (gyms and other meatspace places do this). The problem is you then have to wait until your subscription expires to transfer your account to a new country/currency.

loop0 6y ago

I went through the same issues 6 years ago, had to create the entire billing system from scratch, and there are so many corner cases I had to deal with, specially with plan changes on recurring subscriptions, had to deal with world wide timezones for the invoice creations and lots of other stuff. I just wish there was a pluggable, robust and flexible solution to use instead of reinventing the wheel.

deleted 6y ago

[deleted]

kube-system 6y ago

> Store money including the smallest possible subdivisions. > For example: > $100 US may be stored as 10000 Eh, the US dollar is often subdivided further than that. Here's four examples, in different industries: http://cdn.radioiowa.com/wp-content/uploads/2011/12/gas-pump-12-19-2011.jpg http://cdn.radioiowa.com/wp-content/uploads/2011/12/gas-pump... https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ https://finance.yahoo.com/u/yahoo-finance/watchlists/most-active-penny-stocks https://finance.yahoo.com/u/yahoo-finance/watchlists/most-ac... https://www.digikey.com/en/products/detail/microchip-technology/ATMEGA328P-AUR/3789455 https://www.digikey.com/en/products/detail/microchip-technol...

kbenson 6y ago
Those are unit prices or interval prices. Any transaction will almost definitely be in an amount of dollars and centers if dealing in US, rounding to a whole cent based on the total units or interval. For example, if you're paying #3.089 a gallon at the pump, and you pump exactly one gallon, you don't pay $3.089, you'll pay $3.09, and neither your account or the account you're paying into will need to know or care about the tenth of a cent difference, because our monetary system doesn't really deal with denominations that small in transactions.
- deckard1 6y ago
  yeah but this is where you really have to be careful where you do your rounding. 3.089 * 10,000 = 30890 3.09 * 10,000 = 30900 A one cent rounding added up to $10 difference. It's easy to screw this up in code, sending values to databases or across APIs, etc. Also reminds me of the plot of Office Space. Which is a bit humorous since they screwed up their own scamming scheme in similar fashion. https://www.youtube.com/watch?v=yZjCQ3T5yXo https://www.youtube.com/watch?v=yZjCQ3T5yXo
  - kbenson 6y ago
    There a subtlety in what I was saying that's being lost (which means you aren't wrong, but what you're stating doesn't necessarily apply specifically to what I was trying to say). You don't round rates, or prices, but you do round accounts because those represent real money. Your account can't actually have a fraction of a cent in it. Either the account it came from keeps the cent, or your account gets it. In a very real way, an account can be thought of as an integer number of cents (because often it's implemented that way). That may or may not make sense for your inventory or service system, where sub-cent rates make sense because the number of units is often greater than one. To bring this back to the original comment and the items it was referencing, store money (real money, like accounts) in the smallest possible subdivision (cents for USD), but that doesn't need to apply to pricing, which is a potential amount of money. Potential money needs to be changed into real money at the time of a transaction, and real money cares about quantities it's possible to have, and it's not really possible (or at least useful, in most cases) to have less than the smallest possible subdivisible amount of a currency. You're entirely correct though that you can't assume that cents is enough to accurately model everything to do with a business that works in USD, and ignoring that will result in problems like you showed.

sorbits 6y ago

I would base anything dealing with money on a double entry accounting system. The issues that arise are then a question about how to translate them to journal entries, which will require accounting experience, and maybe your chart of accounts needs to be revised, but the core system should be fairly stable, and voiding an invoice by issuing a credit note comes automatically, as you’re working with an append-only ledger. The issues about prepaid plans should also be handled, as payments for services not rendered yet should not be recognized as income, but instead kept as a liability: The OP mentions their system is used by 15,000 customers, so I would give each customer their own account (in the chart of accounts). The main technical issue is what datatype to use for money, the rest are problems solved by following general accounting principles, though I fear that a lot of programmers out there are reinventing the wheel (in suboptimal ways).

throwawayboise 6y ago
> payments for services not rendered yet should not be recognized as income, but instead kept as a liability This entirely depends on whether you are operating on a cash or accrual basis. Both approach are valid (at least in the USA) and cash basis is often used by small businesses.
- sorbits 6y ago
  I use the word “should” as per RFC 2119, i.e. recommended (as opposed to “must” = required). Though the IRS does limit the types of businesses that can do cash-basis accounting, it would be businesses without inventory, and who does not offer credit to their customers, e.g. a hairdresser would probably use cash-basis accounting, but more complicated businesses would not, certainly not a business that needs its own billing system :)
Edmond 6y ago
I would second this advice :) I fumbled and bumbled my way to this realization while trying to build a billing system intended to tolerate all sorts of invoicing and reversal scenarios. More precisely, it doesn't matter what the nature of the billing is, ie recurring (monthly, quarterly..etc) vs one-time; You want to build a system around invoicing for discrete items and resolving those invoices against various criteria (payment received, credits issued, cancelled plans...etc).

breischl 6y ago

My favorite: when do you do rounding? For instance, with proration. Take each line item, prorate it, round it, then add them all together. Totally reasonable. Now add all the line items together, prorate the total, and round it. Also reasonable, but there's a decent chance the number is different by a few pennies because of rounding differences. If you chained more calculations the differences would compound. Whichever way you choose, somebody is going to whip out their calculator and tell you that you did it wrong (but only when they come out better the other way). This applies anywhere you're doing multiplication or division. Discounts, proration, taxes, "cashback", "store credit dividend", whatever.

ianmcgowan 6y ago
In my world, we keep a running total of the rounded prorated amounts, and then the last item (n) becomes (total - total up to n-1) to make sure amounts match. It can still be a problem if the last value is very small however.
- breischl 6y ago
  Yeah, that's another way to do it. Another way to look at that is you're doing the proration on the final amount, and then distributing the rounding difference across the line items. Which is nice because you get the right total, though it can look weird when line items for the exact same thing are off by a penny from each other.

unixhero 6y ago

Billing is very hard. That is why systems like Geneva is used.

hn_throwaway_99 6y ago

OK, I read the first bulletpoint and just thought "How did this get upvoted this much, this makes no sense": > Money isn’t always decimal > A common wisdom in database design is “never using floating-point numbers for money”. Some recommend using the MONEY datatype, while others tell you to use DECIMAL. > Both of these are wrong. Sure, yeah, in the US and most of Europe, money is decimal. > That’s certainly not the case in Japan – you can’t charge 2500.50 JP¥. The author has a very odd definition of "Decimal". Integers are most certainly decimals. The fact that in Japan you can't charge fractions of a a yen doesn't mean that Decimal isn't actually an excellent choice to represent that currency, because it certainly can represent any yen value you would need. The whole reason you don't want to use float is because it cannot represent certain valid monetary values. As others have pointed out, storing values as "the smallest division", e.g. pennies for USD, is just wrong, because there are many contexts where you have to represent fractions of a cent <insert Superman 3 joke here>.

scrollaway 6y ago
There is a larger point here, that 2500.5 JPY is not a valid amount to charge, thus having it be possible is a problem. Whereas 10000 USD cents is a valid value. Integer storage of money is my favourite as well to be honest. It's easy to work with and reason about, and it just makes decimalization a display issue.
- hn_throwaway_99 6y ago
  But as others have pointed out, actual value can be fractional, it's just not priced that way. I mean, in the US it's actually pretty standard for accounting applications to use a scale of 4 (4 units to the right of the decimal point, the default for MONEY), even though people are never charged fractions of a cent. Simplest example someone gave is gas station pricing, in which prices often end in 9/10ths of a cent - the price is calculated that way, and only rounded when the customer pays.
numpad0 6y ago
Isn't the "all prices are pennies" approach the way USD is handled in practice, like "$399.99"? Certainly no one does irrational divisions like "$24 1/3rd". There might be a small paradigm shift that USD or EUR users might feel if they are used to think of supplementary units genuinely as divisions of full units, but I think most systems do consider the main unit as a sum of smallest denomination and not the or any other ways around.

Rafert 6y ago

> Store money including the smallest possible subdivisions. This is fine as long as everybody follows ISO4217. In my experience many 3rd party systems _mostly_ do it and of course you get bitten by the edge cases. For example, Stripe decided HUF doesn't have 2 decimals (I know that's how it's used in practice today but we're talking standards and system interoperability here), or that ISK subdivides into hundreds ("cents") in contrast to the ISO tables. Compare https://stripe.com/docs/currencies https://stripe.com/docs/currencies with http://currency-iso.org/en/home/tables/table-a1.html http://currency-iso.org/en/home/tables/table-a1.html. As Stripe notes for UGX, backing out of this is nearly impossible because of the subtle break in backwards compatibility. And who knows how other payment providers deviate in their own way. For my own sanity and that of my coworkers (e.g. easier analysis for extracted tables in the data warehouse) I strongly prefer avoiding this headache and use decimal instead. You can do the appropriate rounding for a currency in your money object as for example https://www.martinfowler.com/eaaCatalog/money.html https://www.martinfowler.com/eaaCatalog/money.html does.

bellttyler 6y ago

Excellent article! I've personally been working with billing systems for 2+ years and have felt all those pains.

_orcaman_ 6y ago

Super interesting!

xyzzy21 6y ago

Great article! And it only schemes the surface. There's also PO-based vs. CC-based payment which have different sequencing of the details. You have one I run into all the time: Net-30 vs. Net-60 vs. Net-90. As well as linkages to quotations and terms. B2B takes all this to another level.