7 ms·

Engineering principles for building financial systems

403 points by KothuRoti 2y ago

caseysoftware 2y ago
Years ago, I worked on APIs for a top 10 bank in the US. In our discovery/delving into their logic, the most fascinating thing was interest rate. Some of their systems stored - and therefore the APIs - presented interest rate as a simple number (5% was 5) where other used a decimal (5% was 0.05) and still others used basis points (5% was 500). One of their VPs told me they once saw an unexpected uptick in loan applications and dug in to find out the quoted rate was 0.05% Consistency is key.
- deleted 2y ago
  [deleted]
jroseattle 2y ago
What I find more important than the timestamp format -- the timestamp source. Centralize from where the ts is set (the db is a great place to do this.) Don't let client-code set the timestamp.
game_the0ry 2y ago
Is it just me or did the author just describe block chain, except without the cryptography and not distributed? Or perhaps a Kafka partition / commit logs with infinite retention.
ricericebaby 2y ago
I'll have this saved for the day IRS comes knocking
Terr_ 2y ago
> Use consistent rounding methodologies This may also mean promoting them to a named part of the business-domain in code, with their own functions, unit-tests, stuff like "fetch Rounding Strategy Suite by country code", etc. > Use integer representations of time. This one is a little controversial but I stand by it. There are so many libraries in different technologies that parse timestamps into objects, and they all do them differently. Avoid this headache and just use integers. Unix timestamp, or even integer based UTC datetimes work perfectly fine. Warning: This only works for times that are either firmly in the past or which represent a future delay-offset which is totally under your own exclusive control. For example, suppose your company has a 48 hour cancellation policy. You can probably compute a future timestamp and use that, it won't depend on whether a customer is about to hit a DST transition or leap-seconds or whatever. nsition. In contrast, the nation of Ruritania may have their tax year ending at December 30th 17:30 Ruritania Time, but you don't reeeeealy know how many seconds from now that will happen: Tomorrow the Ruritania congress may pass a law altering their tax-schedule or their entire national time-keeping. This means your underlying source-of-truth needs to be the matching conditions, and that often means storing the time-zone.
- darby_nine 2y ago
  Honestly we should just abandon all that bullshit and just use TAI for internal representation. Would probably save billions.
- amelius 2y ago
  > but you don't reeeeealy know how many seconds from now that will happen So don't use a time-like object. Instead use a condition.
- szundi 2y ago
  I like this answer as usually the world changes that cannot be easily ported to code makes software shit at the end.
  - Terr_ 2y ago
    I like to quip that many hard technical-problems are actually three business-problems in a trenchcoat. In this case, the technical outgrowth might be heroic over-weekend crunch: Write some code that finds every affected timestamp in the DB, reconstructs the various code-chains that created them, divines an "original meaning", calculates the difference, and somehow apply changes in a way that doesn't make auditors angry or lock a key table for and hour or leads to data-corruption from race conditions...
    - funcDropShadow 2y ago
      If you have stored future times as timestamp in the database, then you also need to know with which version of the IANA timezone database those timestamps were computed. Fun ensues. Or one could use the correct data time format for the application use case. It depends on how much you value your sleep and weekends.
- bigiain 2y ago
  Also: > Granularity of your financial amounts should ... ... comply with relevant regulations. I worked on a gambling webapp a long time back - we were bound by government regulations on gambling licences to perform all calculations at a precision of one ten thousandth of a cent, and were only permitted to round to cents once at the very end just before displaying a dollar amount to a person. (This suited me down to the ground because I pointed out to everybody that Javascript couldn't be guaranteed to treat numbers and calculations in a way that would keep the regulators auditors happy, and managed to push _every_ calculation involving dollar amounts off to the Java backend team.) We also used integer UTC milliseconds for time, so we'd have good enough for the auditors timestamps showing who placed a valid bet at (say) 12:34:59.9997 and who else place an invalid bet at 12:35:00.0003 for a wager with a cut off time of 12:35. (We asked for and got a ruling from the regulators that the time that mattered was the time the API call was processed by the backend, since network latencies between the webapp and the backend platform were impossible to predict. I have no idea if the backend had millisecond accurate time, that wasn't my problem and I _so_ didn't want to have them make it my problem by asking the question.)
Galanwe 2y ago
I wouldn't endorse many of the things stated there. A lot of the points are inaccurate or straight up misleading. > The three main goals of your accounting system are to be (1) Accurate, (2) Auditable and (3) Timely. You forgot "consistent", which is very different from "accurate". Most financial transactions have multiple time dimensions: a trade is requested, booked, filled, matched, reconciliated, paid, settled at different times. These times often don't even follow a perfectly logical order (you can request and fill a trade and book it an hour later). These time dimensions are often why there are multiple, layered, ledgers to store transactions. An accounting system needs to provide consistent views that depend on the dimension you want to look at (e.g. a trade that is filled but not settled counts for front office accounting, not for back office accounting). > If there is an event that occurred but is not in the financial record, than the system is not complete. The base assumption of every (working) accounting system I have worked with is the reverse. There WILL be transactions that flow through the system in an untimely manner, your system needs to handle that, thus the need for multiple layers of ledgers, and the necessity to batch reconciliation processes between these layers. You will book trades after they are filled. You will pay materials before inputing the bill. Etc. > If you are only working with dollars, representing values in cents might be sufficient. If you are a global business, prefer micros or a DECIMAL(19, 4). If you are a global business only working with euro and dollar maybe. Otherwise most FX markets quote to 8 decimals, and that's not counting swaps. I would recommend using at least 8 decimal places for storage. > Delay currency conversion as long as you can. Preemptively converting currencies can cause loss of precision. No! Delay currency conversion _until conversion occurs_! An accounting system does not convert currencies at will. There is no dollar equivalent to euro, you either have dollars or euro. And it's not a matter of rounding errors, cash management is part of legal and accounting duties.
- withinboredom 2y ago
  > No! Delay currency conversion _until conversion occurs_! Sometimes these are legal requirements one way or the other. In my country, invoices in other currencies are based on the ECB rate of the date of the invoice, no matter what the actual value is when converted.
  - 2y ago
JetSetIlly 2y ago
Good summary but the article doesn't mention the engineering of the user interface. In this field of accountancy, I think the UI is an engineering problem at least as important as anything else mentioned in the article. As someone who works in accountancy (as a bookkeeper/accountant) I have to be brutally honest and say that I've been very disappointed in the UI of all the accounting software packages I've used. None of them give me the immediacy or simplicity of a well organised filing cabinet, they really don't. I'm not sure what a good solution would look like, but I can't help thinking that making the double-entry system more visible and apparent to the user would be a good start.
- jakjak123 2y ago
  I agree, but also our accountants kept asking for the craziest of things. I guess its a problem that the real world of transactions is full of async processes and weirdness. There is always something new weird that will happen, so it always ends up back in a spreadsheet software.
- jsnk 2y ago
  This was the same perspective i had as an engineer working on a finance problem for accountants and sales. So i would build a nice reporting tool that had modern ui. The people liked it at first but they always regress to spreadsheet. They just ask me to export the data in spreadsheet.
- Ratelman 2y ago
  As someone not in accountancy but having worked in corporate and been forced to work on accounting software for managing budgets, approving transactions etc. - you have really gotten the short end of the stick from general usability perspective. There are so many things that don't make sense - even for the accountants that use the system on day-to-day basis - they have workarounds for (what I would consider) the most basic of things (or over-complicated solutions for something basic like manager approvals for logging expenses - why do I have to click through several screens that don't follow any sort of clear, logical flow). And it's so embedded in an organization - that you can rest assured once that client is signed up they're not switching that system. Same for HR software...
rwieruch 2y ago
Related [0], but one week ago I have written about my experiences using TypeScript for an invoicing system, if anyone is interested. It's especially about rounding errors and how to prevent them. Since then we have created hundreds of invoices (and cancellations) and everything worked as expected with minor hiccups in between. [0] https://www.robinwieruch.de/javascript-rounding-errors/ https://www.robinwieruch.de/javascript-rounding-errors/
- blackhaj7 2y ago
  Superb article Robin Sounds like you really went though a rough time building the invoicing system! I have read many times about the difficulties of dealing with decimals/rounding but never found an article that nailed the explanation as well as you did
alphalima 2y ago
On the best practices, as noted by others, there are probably classes in your standard lib/ commons lib that cover this stuff better than storing in integers (e.g. BigDecimal in Java and Python Decimal have precision and rounding mode concepts built in). Something I've found valuable is on is managing euler units/ratios (e.g. proportions 10^1, percentages 10^-2, basis points 10^-4) . Enforcing a standard that all interchange, data storage has either the same scale, _or is stored with its scale (and not in the name)_ will reduce errors hugely significantly.
- dennis_jeeves2 2y ago
  Just FYI here: To the extent that I explored even BigDecimal in Java falls short when it comes to the calculations. It does do well on the rounding. There might be other libraries that are better when it comes to calculations. If those libraries don't exist integers are the way the go.
- eru 2y ago
  You standard library might even already have libraries for eg storing currencies or time, too.
0wis 2y ago
Another good engineering principle is to write a lot of tests. I know it is a basic rule of engineering but it is not always followed. However, beware as the result of your test will also be seen by auditors, so if you refactor a system, it is better to have a write/solve approach than write all your tests firsts and solve them afterwards.
noelwelsh 2y ago
> Use integer representations of time This sounds like a type system issue (and the linked Etsy post is also a type system issue), or more precisely languages / systems with a loose type system where you can just pretend a time is a number without having to explicitly say what that number means. Conventions are good, but (real) type systems are better because they check everything every time.
- eru 2y ago
  Yes. And in this context, it doesn't even matter (too much) whether your type system is statically checked or done at runtime. The main difference is whether the compiler will yell at you, or whether it'll be a runtime error. But the important point is that your mixing up won't go through and silently do the wrong thing.
mgaunard 2y ago
Representing prices is difficult because a lot of financial instruments have widely different values and price increments. Decimal floating-point gives you the range to cover everything but is generally very inefficient to process. What you usually want is fixed-point decimal, with the caveat that the scaling factor will be different per asset.
- eru 2y ago
  Depending on how many numbers you process, inefficiency might not matter. You could probably use arbitrary precision rational numbers (eg as implemented in Python https://docs.python.org/3/library/fractions.html https://docs.python.org/3/library/fractions.html) for all your accounting needs, and it would still be fast enough.
  - mgaunard 2y ago
    There are millions of financial instruments and their price change 100 million times per day. That's a lot to process, so yes, it matters.
    - eru 2y ago
      Yes, it depends on what you do. If you want to do financial pricing of lots of instruments, performance is important. If you want to do the accounting for one company (that is not trading in millions of financial instruments but, say, is building and selling widgets), you have far fewer numbers on your books.
hippich 2y ago
Another interesting head scratcher (thank God I never had to deal with myself, but saw the mess it can make in the data easily) are government-enacted devaluations. And probably very similar to the concept, stock splits.
47 2y ago
A great companion read is Martin Fowler’s “Accounting Patterns”[1]. Having built and maintained systems that manage financial events for over a decade, I wish I had read these patterns earlier. [1] https://martinfowler.com/apsupp/accounting.pdf https://martinfowler.com/apsupp/accounting.pdf
- arsenico 2y ago
  I think Fowler's work is an underrated must-read for anyone who works in domains related to moving money. Makes any kind of engineering practices and architecture principles logical and make sense.
  - jakjak123 2y ago
    Yes, and I was lucky enough to read his stuff on finance within 6 months of starting work! He has some very good design ideas for many things, just as always treat it like things in your toolbox and not dogma.
willtemperley 2y ago
> Use integers to represent financial amounts I followed this mantra when building a trading system. This was the worst engineering decision I ever made - it added complexity and runtime overhead, plus readability was reduced when debugging. If your language has a proper decimal type like BigDecimal in Java or Decimal in Swift, I'd suggest using it. They offer perfect precision for financial applications, are battle tested and are a natural way to represent financial amounts. Yes, Java BigDecimal is unweildy, but it works. If you're using JavaScript however, definitely use integers to represent financial amounts.
- erik_seaberg 2y ago
  There's also javax.money.
- dennis_jeeves2 2y ago
  >BigDecimal in Java Correct me if I'm wrong. The last I checked, BigDecimal in Java falls short when it comes to the calculations. ( very big and very small numbers). It does do well on rounding. There might be other libraries that are better when it comes to calculations. If those libraries don't exist integers are the way the go.
  - willtemperley 2y ago
    If you've got any specific examples I'd be interested, but whilst the BigDecimal API is not pretty and it's definitely not fast, I don't know of any actual problems with calculations. If you could describe how to represent all possible financial values using integers in a clean manner, I'd love to know. It gets messy very quickly if you're dealing with dollars at two decimal places and BTC at 8 decimal places.
ibejoeb 2y ago
We're basically talking about bookkeeping systems here, rather than high finance. Use a good relational database. 1. ACID, so you don't have to invent it. 2. Arbitrary precision numeric data types with vetted operations and rounding modes. 3. It does time as well as anyone can 4. Your computations and reporting can be done entirely in SQL. 5. When you get good at SQL or hire someone who is, the reporting is elegant. 6. It's as fast as it needs to be, and a good database architect can make it extremely fast. 7. It has all of the tooling to do disaster prevention and recovery. I've built so many financial systems for the biggest multinationals in the world, and I have never regretted doing the bulk of the work in the database. Coca-Cola, GE, UPS, AIG, every iteration of the Tycos, basically every pharma with their crazy transfer pricing schemes... Whenever I experienced a performance problem, there have been way to address it that were far easier than what would have been to reinvent all of the hard tech that underlies consistent, accurate financial systems.
- nyrikki 2y ago
  ACID and DTCs will get you in legal trouble with financial systems. That is part of the reason financial systems backed by SQL are so complicated and fragile while old COBOL code keeps going. RDBMS tutorials always use ATMs as an example, but you can't 'roll back' with ATMs, you have to use compensating actions. ACID is useful in many contexts, but actually adds complications to financial transactions. Think of the ledger as being the source of truth, with the DB as simply a materialized view. It has applications with edge caching in web apps too. Events as the source of record can help get rid of cache invalidation challenges compared to cache aside or write back. Two sides errors that don't impact the trial balance is something to search for if you want to know why an ACID DB as the source of truth is problematic in accounting.
- tumetab1 2y ago
  > 2. Arbitrary precision numeric data types with vetted operations and rounding modes. Be aware that a good rationale for choosing the database but you should enforce a specific precision per table otherwise you can get nasty accounting bugs because you're adding a 3.0095 amount with an 9.99 amount.
gushogg-blake 2y ago
Regarding the types, I think it's easy to fall into the trap of thinking you have to try and force the domain concept into one of the commonly available data structures at the point of creation, i.e. the price "£3.99" has to become (int) 399 or (float) 3.99000000001. Actually, as another commentor mentioned, you can and should just choose something that represents what you want. In many cases it will be an object with various domain-specific fields. If you need to be sure that "3.99" always comes back as "3.99", you can use a string, which doesn't have any issues with precision in the storage format, and will usually map exactly to how humans deal with the concept. (Not necessarily recommending this, but it's worth considering.) Similarly with JavaScript precision issues, to take another example that's been mentioned here - if you need numbers to be precise and the performance cost is acceptable, JavaScript can compute to arbitrary precision just as well as any other language. Just don't assume you're forced to convert your concept of a number to use the built-in numeric types and operations.
- ibejoeb 2y ago
  The string "3.99" always comes back as a string, but when you need do anything at all to it, it resolves down to the same problem.
  - langcss 2y ago
    Ideally, one should make illegal state unrepresentable. The problem with a string is there is no guarantee it represents a money amount.
- a_c_s 2y ago
  If you have a decimal then you should store it as a Decimal type, not a float nor an int. This is why Decimal types exist - the fact that most programming languages don't have a native Decimal type baffles me.
gwright 2y ago
The biggest misunderstanding I see in these types of discussions is that there is a lossless conversion between binary floating point representation and decimal representation. Very few values have exact decimal and binary floating point representations.
- RaftPeople 2y ago
  Note: I think you meant to say "lossy" instead of "lossless"
  - gwright 2y ago
    Nope. I was asserting that confusion comes from people believing that a "lossless" conversion exists. Sorry if that wasn't clear.
    - RaftPeople 2y ago
      Ok, that makes sense.
DavidJowel 2y ago
[dead]
Havoc 2y ago
Mostly agree though the concept of materiality seems out of place here. Decision making factors that in sure but system design absolutely shouldn’t. There is no reason why accounting systems can’t be accurate down to the penny. That’s what computers are good at after all. Only place where I could see system like relevance is in presentation- showing revenue on millions etc
- infecto 2y ago
  Agree. I mostly ignored the rest of the post because this threw red flags that the poster does not build real financial systems. Financial systems should be capturing everything, now on the reporting end there is a question on materiality but most definitely not on the system design.
DarkContinent 2y ago
> Batch is just a special case of streaming No. Designing a system that is always up and running and can process small amounts of data constantly is a completely different problem from designing a system that runs occasionally with a lot of data. For one thing, your output formats are usually different in the latter case (maybe you're creating a PDF for example). Also the high availability requirement just makes things different at the design level. Finally, the author claims it's not hard to switch between batch and streaming. With a large volume of preexisting data, this is just not true. For example, if you make a REST API call for each document in a DB, it can take days or months to load that. If batching together documents isn't a possibility, how do you move data between stores easily? (This data movement is often required when switching between batch and streaming.)
- erik_seaberg 2y ago
  The key is working incrementally, not sitting idle for months and then hammering production as hard as possible trying to get all the deferred work done in exactly one batch.
- lkdfjlkdfjlg 2y ago
  I'm seconding this, and I have first hand experience in exactly this problem, in finance. My first boss also had the view that "batching is a special case of streaming where you stream N and streaming is also a special case of batching where the batch size is 1 and so it doesn't matter which one you implement". This was never performant enough and he was eventually asked to leave.
woah 2y ago
Cryptocurrencies tend to use very large integers (128 or 256 bit) for everything throughout the system. Rounding doesn't happen until a number hits the UI. Why would you design a system that intentionally destroys data several times internally?
- deleted 2y ago
  [deleted]
- makeitdouble 2y ago
  That's usually how it's done in systems using int as well: internal calculations are done with as much precision as needed, and only the significant results are converted back to int using a defined rounding strategy.
- RaftPeople 2y ago
  > Why would you design a system that intentionally destroys data several times internally? Because it matches the requirements of the domain being modeled. Most business to business and business to consumer transactions require that amounts are limited to some number of decimal positions (e.g. in US it's 2 decimal positions). If you create a system document/transaction like an invoice that doesn't adhere to this rule then your software is not properly modeling the domain (i.e. it's a design flaw). If you decided to leave it unrounded internally, then you are basically storing an incorrect value and delaying the calculation of the correct value until consumption (which is typically hundreds to thousands of consumers). But what is the gain to storing the wrong value? Nobody wants it and it just complicates the entire system.
- HWR_14 2y ago
  Sometimes precision (and thus data) needs to be destroyed. Say you owe me $1,000.211 and write me a check for $1,000.21 Do you still owe me a tenth-of-a-cent? Even if you do, does anyone (me, you, the taxing authorities) care? In fact, does anyone want to go through the effort of maintaining that debt? Even if I repeat this with 100,000 customers, I'm out $100. Again, does anyone care?
cvdub 2y ago
Managing rounding and ensuring each set of entries balance can be tricky, especially if you have to share data with a system that can only handle currencies with two decimal places. There are scenarios where it’s actually not possible to have every set of entries balance, and have the total sum of all entries equal the correct balance. For example, if you had three monthly payments of $5, $10, and $10, you might book something like: Cash (5) Expense 8.33 Deferred (3.33) Cash (10) Expense 8.33 Deferred 1.67 Cash (10) Expense 8.33 Deferred 1.67 All three of those blocks of entries balance, but the sum of expenses is 24.99 instead of 25. I’m not sure there’s a way around this issue if you’re forced to use two decimal places. Luckily the discrepancy is immaterial. I’d love to know if anyone else has encountered this problem.

403 points by KothuRoti 2y ago

Years ago, I worked on APIs for a top 10 bank in the US. In our discovery/delving into their logic, the most fascinating thing was interest rate. Some of their systems stored - and therefore the APIs - presented interest rate as a simple number (5% was 5) where other used a decimal (5% was 0.05) and still others used basis points (5% was 500). One of their VPs told me they once saw an unexpected uptick in loan applications and dug in to find out the quoted rate was 0.05% Consistency is key.

deleted 2y ago
[deleted]

jroseattle 2y ago

What I find more important than the timestamp format -- the timestamp source. Centralize from where the ts is set (the db is a great place to do this.) Don't let client-code set the timestamp.

game_the0ry 2y ago

Is it just me or did the author just describe block chain, except without the cryptography and not distributed? Or perhaps a Kafka partition / commit logs with infinite retention.

ricericebaby 2y ago

I'll have this saved for the day IRS comes knocking

Terr_ 2y ago

> Use consistent rounding methodologies This may also mean promoting them to a named part of the business-domain in code, with their own functions, unit-tests, stuff like "fetch Rounding Strategy Suite by country code", etc. > Use integer representations of time. This one is a little controversial but I stand by it. There are so many libraries in different technologies that parse timestamps into objects, and they all do them differently. Avoid this headache and just use integers. Unix timestamp, or even integer based UTC datetimes work perfectly fine. Warning: This only works for times that are either firmly in the past or which represent a future delay-offset which is totally under your own exclusive control. For example, suppose your company has a 48 hour cancellation policy. You can probably compute a future timestamp and use that, it won't depend on whether a customer is about to hit a DST transition or leap-seconds or whatever. nsition. In contrast, the nation of Ruritania may have their tax year ending at December 30th 17:30 Ruritania Time, but you don't reeeeealy know how many seconds from now that will happen: Tomorrow the Ruritania congress may pass a law altering their tax-schedule or their entire national time-keeping. This means your underlying source-of-truth needs to be the matching conditions, and that often means storing the time-zone.

darby_nine 2y ago
Honestly we should just abandon all that bullshit and just use TAI for internal representation. Would probably save billions.
amelius 2y ago
> but you don't reeeeealy know how many seconds from now that will happen So don't use a time-like object. Instead use a condition.
szundi 2y ago
I like this answer as usually the world changes that cannot be easily ported to code makes software shit at the end.
- Terr_ 2y ago
  I like to quip that many hard technical-problems are actually three business-problems in a trenchcoat. In this case, the technical outgrowth might be heroic over-weekend crunch: Write some code that finds every affected timestamp in the DB, reconstructs the various code-chains that created them, divines an "original meaning", calculates the difference, and somehow apply changes in a way that doesn't make auditors angry or lock a key table for and hour or leads to data-corruption from race conditions...
  - funcDropShadow 2y ago
    If you have stored future times as timestamp in the database, then you also need to know with which version of the IANA timezone database those timestamps were computed. Fun ensues. Or one could use the correct data time format for the application use case. It depends on how much you value your sleep and weekends.
bigiain 2y ago
Also: > Granularity of your financial amounts should ... ... comply with relevant regulations. I worked on a gambling webapp a long time back - we were bound by government regulations on gambling licences to perform all calculations at a precision of one ten thousandth of a cent, and were only permitted to round to cents once at the very end just before displaying a dollar amount to a person. (This suited me down to the ground because I pointed out to everybody that Javascript couldn't be guaranteed to treat numbers and calculations in a way that would keep the regulators auditors happy, and managed to push _every_ calculation involving dollar amounts off to the Java backend team.) We also used integer UTC milliseconds for time, so we'd have good enough for the auditors timestamps showing who placed a valid bet at (say) 12:34:59.9997 and who else place an invalid bet at 12:35:00.0003 for a wager with a cut off time of 12:35. (We asked for and got a ruling from the regulators that the time that mattered was the time the API call was processed by the backend, since network latencies between the webapp and the backend platform were impossible to predict. I have no idea if the backend had millisecond accurate time, that wasn't my problem and I _so_ didn't want to have them make it my problem by asking the question.)

Galanwe 2y ago

I wouldn't endorse many of the things stated there. A lot of the points are inaccurate or straight up misleading. > The three main goals of your accounting system are to be (1) Accurate, (2) Auditable and (3) Timely. You forgot "consistent", which is very different from "accurate". Most financial transactions have multiple time dimensions: a trade is requested, booked, filled, matched, reconciliated, paid, settled at different times. These times often don't even follow a perfectly logical order (you can request and fill a trade and book it an hour later). These time dimensions are often why there are multiple, layered, ledgers to store transactions. An accounting system needs to provide consistent views that depend on the dimension you want to look at (e.g. a trade that is filled but not settled counts for front office accounting, not for back office accounting). > If there is an event that occurred but is not in the financial record, than the system is not complete. The base assumption of every (working) accounting system I have worked with is the reverse. There WILL be transactions that flow through the system in an untimely manner, your system needs to handle that, thus the need for multiple layers of ledgers, and the necessity to batch reconciliation processes between these layers. You will book trades after they are filled. You will pay materials before inputing the bill. Etc. > If you are only working with dollars, representing values in cents might be sufficient. If you are a global business, prefer micros or a DECIMAL(19, 4). If you are a global business only working with euro and dollar maybe. Otherwise most FX markets quote to 8 decimals, and that's not counting swaps. I would recommend using at least 8 decimal places for storage. > Delay currency conversion as long as you can. Preemptively converting currencies can cause loss of precision. No! Delay currency conversion _until conversion occurs_! An accounting system does not convert currencies at will. There is no dollar equivalent to euro, you either have dollars or euro. And it's not a matter of rounding errors, cash management is part of legal and accounting duties.

withinboredom 2y ago
> No! Delay currency conversion _until conversion occurs_! Sometimes these are legal requirements one way or the other. In my country, invoices in other currencies are based on the ECB rate of the date of the invoice, no matter what the actual value is when converted.
- 2y ago

JetSetIlly 2y ago

Good summary but the article doesn't mention the engineering of the user interface. In this field of accountancy, I think the UI is an engineering problem at least as important as anything else mentioned in the article. As someone who works in accountancy (as a bookkeeper/accountant) I have to be brutally honest and say that I've been very disappointed in the UI of all the accounting software packages I've used. None of them give me the immediacy or simplicity of a well organised filing cabinet, they really don't. I'm not sure what a good solution would look like, but I can't help thinking that making the double-entry system more visible and apparent to the user would be a good start.

jakjak123 2y ago
I agree, but also our accountants kept asking for the craziest of things. I guess its a problem that the real world of transactions is full of async processes and weirdness. There is always something new weird that will happen, so it always ends up back in a spreadsheet software.
jsnk 2y ago
This was the same perspective i had as an engineer working on a finance problem for accountants and sales. So i would build a nice reporting tool that had modern ui. The people liked it at first but they always regress to spreadsheet. They just ask me to export the data in spreadsheet.
Ratelman 2y ago
As someone not in accountancy but having worked in corporate and been forced to work on accounting software for managing budgets, approving transactions etc. - you have really gotten the short end of the stick from general usability perspective. There are so many things that don't make sense - even for the accountants that use the system on day-to-day basis - they have workarounds for (what I would consider) the most basic of things (or over-complicated solutions for something basic like manager approvals for logging expenses - why do I have to click through several screens that don't follow any sort of clear, logical flow). And it's so embedded in an organization - that you can rest assured once that client is signed up they're not switching that system. Same for HR software...

rwieruch 2y ago

Related [0], but one week ago I have written about my experiences using TypeScript for an invoicing system, if anyone is interested. It's especially about rounding errors and how to prevent them. Since then we have created hundreds of invoices (and cancellations) and everything worked as expected with minor hiccups in between. [0] https://www.robinwieruch.de/javascript-rounding-errors/ https://www.robinwieruch.de/javascript-rounding-errors/

blackhaj7 2y ago
Superb article Robin Sounds like you really went though a rough time building the invoicing system! I have read many times about the difficulties of dealing with decimals/rounding but never found an article that nailed the explanation as well as you did

alphalima 2y ago

On the best practices, as noted by others, there are probably classes in your standard lib/ commons lib that cover this stuff better than storing in integers (e.g. BigDecimal in Java and Python Decimal have precision and rounding mode concepts built in). Something I've found valuable is on is managing euler units/ratios (e.g. proportions 10^1, percentages 10^-2, basis points 10^-4) . Enforcing a standard that all interchange, data storage has either the same scale, _or is stored with its scale (and not in the name)_ will reduce errors hugely significantly.

dennis_jeeves2 2y ago
Just FYI here: To the extent that I explored even BigDecimal in Java falls short when it comes to the calculations. It does do well on the rounding. There might be other libraries that are better when it comes to calculations. If those libraries don't exist integers are the way the go.
eru 2y ago
You standard library might even already have libraries for eg storing currencies or time, too.

0wis 2y ago

Another good engineering principle is to write a lot of tests. I know it is a basic rule of engineering but it is not always followed. However, beware as the result of your test will also be seen by auditors, so if you refactor a system, it is better to have a write/solve approach than write all your tests firsts and solve them afterwards.

noelwelsh 2y ago

> Use integer representations of time This sounds like a type system issue (and the linked Etsy post is also a type system issue), or more precisely languages / systems with a loose type system where you can just pretend a time is a number without having to explicitly say what that number means. Conventions are good, but (real) type systems are better because they check everything every time.

eru 2y ago
Yes. And in this context, it doesn't even matter (too much) whether your type system is statically checked or done at runtime. The main difference is whether the compiler will yell at you, or whether it'll be a runtime error. But the important point is that your mixing up won't go through and silently do the wrong thing.

mgaunard 2y ago

Representing prices is difficult because a lot of financial instruments have widely different values and price increments. Decimal floating-point gives you the range to cover everything but is generally very inefficient to process. What you usually want is fixed-point decimal, with the caveat that the scaling factor will be different per asset.

eru 2y ago
Depending on how many numbers you process, inefficiency might not matter. You could probably use arbitrary precision rational numbers (eg as implemented in Python https://docs.python.org/3/library/fractions.html https://docs.python.org/3/library/fractions.html) for all your accounting needs, and it would still be fast enough.
- mgaunard 2y ago
  There are millions of financial instruments and their price change 100 million times per day. That's a lot to process, so yes, it matters.
  - eru 2y ago
    Yes, it depends on what you do. If you want to do financial pricing of lots of instruments, performance is important. If you want to do the accounting for one company (that is not trading in millions of financial instruments but, say, is building and selling widgets), you have far fewer numbers on your books.

hippich 2y ago

Another interesting head scratcher (thank God I never had to deal with myself, but saw the mess it can make in the data easily) are government-enacted devaluations. And probably very similar to the concept, stock splits.

47 2y ago

A great companion read is Martin Fowler’s “Accounting Patterns”[1]. Having built and maintained systems that manage financial events for over a decade, I wish I had read these patterns earlier. [1] https://martinfowler.com/apsupp/accounting.pdf https://martinfowler.com/apsupp/accounting.pdf

arsenico 2y ago
I think Fowler's work is an underrated must-read for anyone who works in domains related to moving money. Makes any kind of engineering practices and architecture principles logical and make sense.
- jakjak123 2y ago
  Yes, and I was lucky enough to read his stuff on finance within 6 months of starting work! He has some very good design ideas for many things, just as always treat it like things in your toolbox and not dogma.

willtemperley 2y ago

> Use integers to represent financial amounts I followed this mantra when building a trading system. This was the worst engineering decision I ever made - it added complexity and runtime overhead, plus readability was reduced when debugging. If your language has a proper decimal type like BigDecimal in Java or Decimal in Swift, I'd suggest using it. They offer perfect precision for financial applications, are battle tested and are a natural way to represent financial amounts. Yes, Java BigDecimal is unweildy, but it works. If you're using JavaScript however, definitely use integers to represent financial amounts.

erik_seaberg 2y ago
There's also javax.money.
dennis_jeeves2 2y ago
>BigDecimal in Java Correct me if I'm wrong. The last I checked, BigDecimal in Java falls short when it comes to the calculations. ( very big and very small numbers). It does do well on rounding. There might be other libraries that are better when it comes to calculations. If those libraries don't exist integers are the way the go.
- willtemperley 2y ago
  If you've got any specific examples I'd be interested, but whilst the BigDecimal API is not pretty and it's definitely not fast, I don't know of any actual problems with calculations. If you could describe how to represent all possible financial values using integers in a clean manner, I'd love to know. It gets messy very quickly if you're dealing with dollars at two decimal places and BTC at 8 decimal places.

ibejoeb 2y ago

We're basically talking about bookkeeping systems here, rather than high finance. Use a good relational database. 1. ACID, so you don't have to invent it. 2. Arbitrary precision numeric data types with vetted operations and rounding modes. 3. It does time as well as anyone can 4. Your computations and reporting can be done entirely in SQL. 5. When you get good at SQL or hire someone who is, the reporting is elegant. 6. It's as fast as it needs to be, and a good database architect can make it extremely fast. 7. It has all of the tooling to do disaster prevention and recovery. I've built so many financial systems for the biggest multinationals in the world, and I have never regretted doing the bulk of the work in the database. Coca-Cola, GE, UPS, AIG, every iteration of the Tycos, basically every pharma with their crazy transfer pricing schemes... Whenever I experienced a performance problem, there have been way to address it that were far easier than what would have been to reinvent all of the hard tech that underlies consistent, accurate financial systems.

nyrikki 2y ago
ACID and DTCs will get you in legal trouble with financial systems. That is part of the reason financial systems backed by SQL are so complicated and fragile while old COBOL code keeps going. RDBMS tutorials always use ATMs as an example, but you can't 'roll back' with ATMs, you have to use compensating actions. ACID is useful in many contexts, but actually adds complications to financial transactions. Think of the ledger as being the source of truth, with the DB as simply a materialized view. It has applications with edge caching in web apps too. Events as the source of record can help get rid of cache invalidation challenges compared to cache aside or write back. Two sides errors that don't impact the trial balance is something to search for if you want to know why an ACID DB as the source of truth is problematic in accounting.
tumetab1 2y ago
> 2. Arbitrary precision numeric data types with vetted operations and rounding modes. Be aware that a good rationale for choosing the database but you should enforce a specific precision per table otherwise you can get nasty accounting bugs because you're adding a 3.0095 amount with an 9.99 amount.

gushogg-blake 2y ago

Regarding the types, I think it's easy to fall into the trap of thinking you have to try and force the domain concept into one of the commonly available data structures at the point of creation, i.e. the price "£3.99" has to become (int) 399 or (float) 3.99000000001. Actually, as another commentor mentioned, you can and should just choose something that represents what you want. In many cases it will be an object with various domain-specific fields. If you need to be sure that "3.99" always comes back as "3.99", you can use a string, which doesn't have any issues with precision in the storage format, and will usually map exactly to how humans deal with the concept. (Not necessarily recommending this, but it's worth considering.) Similarly with JavaScript precision issues, to take another example that's been mentioned here - if you need numbers to be precise and the performance cost is acceptable, JavaScript can compute to arbitrary precision just as well as any other language. Just don't assume you're forced to convert your concept of a number to use the built-in numeric types and operations.

ibejoeb 2y ago
The string "3.99" always comes back as a string, but when you need do anything at all to it, it resolves down to the same problem.
- langcss 2y ago
  Ideally, one should make illegal state unrepresentable. The problem with a string is there is no guarantee it represents a money amount.
a_c_s 2y ago
If you have a decimal then you should store it as a Decimal type, not a float nor an int. This is why Decimal types exist - the fact that most programming languages don't have a native Decimal type baffles me.

gwright 2y ago

The biggest misunderstanding I see in these types of discussions is that there is a lossless conversion between binary floating point representation and decimal representation. Very few values have exact decimal and binary floating point representations.

RaftPeople 2y ago
Note: I think you meant to say "lossy" instead of "lossless"
- gwright 2y ago
  Nope. I was asserting that confusion comes from people believing that a "lossless" conversion exists. Sorry if that wasn't clear.
  - RaftPeople 2y ago
    Ok, that makes sense.

DavidJowel 2y ago

[dead]

Havoc 2y ago

Mostly agree though the concept of materiality seems out of place here. Decision making factors that in sure but system design absolutely shouldn’t. There is no reason why accounting systems can’t be accurate down to the penny. That’s what computers are good at after all. Only place where I could see system like relevance is in presentation- showing revenue on millions etc

infecto 2y ago
Agree. I mostly ignored the rest of the post because this threw red flags that the poster does not build real financial systems. Financial systems should be capturing everything, now on the reporting end there is a question on materiality but most definitely not on the system design.

DarkContinent 2y ago

> Batch is just a special case of streaming No. Designing a system that is always up and running and can process small amounts of data constantly is a completely different problem from designing a system that runs occasionally with a lot of data. For one thing, your output formats are usually different in the latter case (maybe you're creating a PDF for example). Also the high availability requirement just makes things different at the design level. Finally, the author claims it's not hard to switch between batch and streaming. With a large volume of preexisting data, this is just not true. For example, if you make a REST API call for each document in a DB, it can take days or months to load that. If batching together documents isn't a possibility, how do you move data between stores easily? (This data movement is often required when switching between batch and streaming.)

erik_seaberg 2y ago
The key is working incrementally, not sitting idle for months and then hammering production as hard as possible trying to get all the deferred work done in exactly one batch.
lkdfjlkdfjlg 2y ago
I'm seconding this, and I have first hand experience in exactly this problem, in finance. My first boss also had the view that "batching is a special case of streaming where you stream N and streaming is also a special case of batching where the batch size is 1 and so it doesn't matter which one you implement". This was never performant enough and he was eventually asked to leave.

woah 2y ago

Cryptocurrencies tend to use very large integers (128 or 256 bit) for everything throughout the system. Rounding doesn't happen until a number hits the UI. Why would you design a system that intentionally destroys data several times internally?

deleted 2y ago
[deleted]
makeitdouble 2y ago
That's usually how it's done in systems using int as well: internal calculations are done with as much precision as needed, and only the significant results are converted back to int using a defined rounding strategy.
RaftPeople 2y ago
> Why would you design a system that intentionally destroys data several times internally? Because it matches the requirements of the domain being modeled. Most business to business and business to consumer transactions require that amounts are limited to some number of decimal positions (e.g. in US it's 2 decimal positions). If you create a system document/transaction like an invoice that doesn't adhere to this rule then your software is not properly modeling the domain (i.e. it's a design flaw). If you decided to leave it unrounded internally, then you are basically storing an incorrect value and delaying the calculation of the correct value until consumption (which is typically hundreds to thousands of consumers). But what is the gain to storing the wrong value? Nobody wants it and it just complicates the entire system.
HWR_14 2y ago
Sometimes precision (and thus data) needs to be destroyed. Say you owe me $1,000.211 and write me a check for $1,000.21 Do you still owe me a tenth-of-a-cent? Even if you do, does anyone (me, you, the taxing authorities) care? In fact, does anyone want to go through the effort of maintaining that debt? Even if I repeat this with 100,000 customers, I'm out $100. Again, does anyone care?

cvdub 2y ago

Managing rounding and ensuring each set of entries balance can be tricky, especially if you have to share data with a system that can only handle currencies with two decimal places. There are scenarios where it’s actually not possible to have every set of entries balance, and have the total sum of all entries equal the correct balance. For example, if you had three monthly payments of $5, $10, and $10, you might book something like: Cash (5) Expense 8.33 Deferred (3.33) Cash (10) Expense 8.33 Deferred 1.67 Cash (10) Expense 8.33 Deferred 1.67 All three of those blocks of entries balance, but the sum of expenses is 24.99 instead of 25. I’m not sure there’s a way around this issue if you’re forced to use two decimal places. Luckily the discrepancy is immaterial. I’d love to know if anyone else has encountered this problem.