7 ms·
Fintech Engineering Handbook
- danielabinav160 4mo agoThe idempotency keys section alone is worth the read most devs learn that lesson the hard way.
- __natty__ 4mo agoAlso audit trails. Good audit trail can save company (and you) in emergency as well. Useful for debugging and last resort of compliance data source.
- mrkeen 4mo agoI just build the audit trails and skip the non audit trails. That way I can debug and have a last resort of compliance, but also save time by not building the first resort of compliance.
- pards 4mo ago100%. It deserves more detail, too. I've spent many hours explaining how idempotency is supposed to work, and why it's important. Most teams understand the need for it, but very few thought about it up front.
- lxgr 4mo agoI just wish the financial industry itself had known about these when the core banking systems and financial communication protocols of the 60s and 70s were invented that are still being used to this day... Many of these predate the widespread knowledge of idempotency, so often idempotency keys are hacked together by joining various, hopefully globally unique fields, except that they never quite are. (You can look behind the curtain sometimes, e.g. when your bank does not let you transfer the same amount to the same recipient account on the same calendar day.)
- dc_giant 4mo agoSorry have to ask these days. Is this carefully written down information from years of experience in the field or AI slop?
- thewisenerd 4mo agofrom the author's mastodon post [0] I just published Fintech Engineering Handbook distilled from 6 years of tears, sweat and swears. It’s a free ~25-page resource with various hints and patterns around handling money. Tell me what you think! other than that, peruse the commits on the source [1], or wait for the author to respond. [0]: https://mas.to/@krever/116814803588993437 https://mas.to/@krever/116814803588993437 [1]: https://github.com/Krever/fintech-engineering-handbook/commits/main/ https://github.com/Krever/fintech-engineering-handbook/commi...
- jagged-chisel 4mo agoAppears that the author got some help organizing the document, but wrote it all themselves.
- zipy124 4mo agoWhilst I wouldn't say anything in it requires years of experience to know, this would be helpful for someone who hasn't considered anything about monetary systems. It doesn't read like slop, but I could be wrong but even so it all seems fairly reasonable (I've only fully read about 50% before realising there's nothing new here for me, and then skimmed to rest).
- manwithopinions 4mo agoSkimmed it and based on my experience in fintech, it looks good, accurately represents the real world. I guess there’s still a chance it is AI generated but it doesn’t seem like vacuous slop, it has substance!
- krever 4mo agoHey, author here :) Its at least 80% organic artisanal writing and maybe 20% AI when I needed help with grammar, completeness, broader perspective and everything around.
- lxgr 4mo agoWord of advice to anyone considering the "minor-units precision" strategy for representing monetary amounts: Don't (or at least, don't use it as an interchange/API data format). It seems like a clever idea (fast integer math, no rounding problems for addition and subtraction), but it'll bite you incredibly hard if you ever stumble upon an edge case such as working with a partner that has a different implied number of digits for a given currency. This is especially relevant for stablecoins, which often have a different number of implied decimal digits than the "fiat" currency they represent. Also, consider representing amounts as a string type in JSON-based APIs. JSON does not specify decimal precision, so you (and all your users/vendors) will always have to make sure your parser/serializer doesn't internally lose precision by going via floating point. This can get ugly fast, and while a string seems conceptually less neat, it completely bypasses that problem. (Some will call this an anti-pattern [1], but I'd rather not fight this particular battle for ideological purity on the shoulders of my users or shareholders.) [1] https://blog.json-everything.net/posts/numbers-are-numbers-not-strings/ https://blog.json-everything.net/posts/numbers-are-numbers-n...
- gucci-on-fleek 4mo agoWhat do you recommend instead? Standard floating-point ("float"/"double"), fixed-point arithmetic with thousandths (or smaller) of the minor unit, arbitrary-precision decimal numbers, or something else entirely?
- lxgr 4mo agoI think what matters most is your database and API representation, as well as having consistent and well-defined rounding rules. I largely agree with TFA: Round explicitly and consistently whenever you cross a boundary, i.e. database persistence and internal API calls. Use whatever works for your required business case internally (i.e. inside of procedures calculating some function of one or more input amounts). This can be regular old floats/doubles if you absolutely know what you're doing, or BigDecimal if you aren't and would rather suffer slightly slower performance than having to talk to an auditor about IEEE 754 rounding modes, or even minor-amount integers (yes, even though I just said to not use them – but you'll want to ABSOLUTELY NEVER leak them outside of your system, including your data/analytics pipeline, which might have different ideas about financial amounts than your business logic implementing a nice custom monetary type).
- dapperdrake 4mo agoFirst half didn’t sound so bad.
- ricardobayes 4mo agoDoes anyone have more learning resources in this field? Any model implementations, pet projects, anything to get going?
- nvlled 4mo agoHuh, that's strange, where did all comments to this post go? I remember seeing three, then a few more the day after. Were they bots? They seemed legit recommendations, like the Pricing Money book.
- cirrhosis 4mo agoI have just left a fintech company after 5 years and I can say after reading this, it looks legit to me (not AI slop as someone asked). These are the same sort of lessons I learned during my time in the industry. I would recommend anyone starting in fintech to take some time to understand accounting principles and the ledger in a bit more depth than just debits vs credits - this is likely what is most unfamiliar to programmers. Also financial software is very data-heavy and I learned more about databases in my time working in fintech than the 15 years before that. I think going into a bit more detail about even the basics (indexes) will save a lot of headaches.
- sdevonoes 4mo ago> I would recommend anyone starting in fintech to take some time to understand accounting principles and the ledger in a bit more depth than just debits vs credits Any good resources you would recommend to learn more about this?
- cirrhosis 4mo agoI've seen a few attempts at "accounting for programmers" guides, some of which I came across while browsing HN (if you search you'll find plenty). The one I used to send new developers was one such guide[1]. They also have another on building a ledger[2]. These are a good start and I wish I could recommend a proper textbook that goes into more detail - detail that I wish I had known early on - but I learned mostly as I went along. [1] https://www.moderntreasury.com/journal/accounting-for-developers-part-i https://www.moderntreasury.com/journal/accounting-for-develo... [2] https://www.moderntreasury.com/journal/how-to-scale-a-ledger-part-i https://www.moderntreasury.com/journal/how-to-scale-a-ledger...
- FabHK 4mo agoTFA, in Appendix A: Resources, lists many.
- bayarearefugee 4mo ago> I would recommend anyone starting in fintech to take some time to understand accounting principles and the ledger in a bit more depth than just debits vs credits Probably good advice (for everyone in fintech, not just programmers) considering the absolute disaster that happened at Synapse. Kind of wild nobody has gone to jail for that.
- benashford 4mo agoI think most of this applies to software engineering generally, not just fintech. For example the parts talking of retries, idempotency, event ordering, etc. This applies to all systems that require any degree of accuracy, even if no money is directly involved. I've seen so many systems built on the assumption that "we can always retry", but you can only retry if you fail cleanly in the first place, and if the downstream system offers the same level of idempotency that you think it does. Quite often these are not put to the test.
- jimmypk 4mo ago[dead]
- jappgar 4mo agoI agree. Very little in here specifically applies to fintech except the ledgering and rounding parts, which are pretty light. I would prefer to read a defense of something more radical like "database per account." Something that has unique tradeoffs within fintech. Also, the main advice I would give to fintech engineers/founders is to take risk and compliance seriously from day one. Financial systems are based around trust. If you don't provably mitigate risks you will lose trust and, eventually, your entire business.
- charlieirish 4mo agoSimilar style and message to https://shapeofthesystem.com/ https://shapeofthesystem.com/
- dxdm 4mo agoChunks of good advice, tossed and blended into a soup of verbiage generated by a helpful LLM. It sounds smooth and resists being read at the same time. I hope we can get to the point where actually good, helpful prose is generated more often than not.
- krever 4mo agoHey, author here. Happy to take feedback or answer questions. P.S. I have no clue how HN works, I posted it myself yesterday and it got 6 points. ¯\_(ツ)_/¯ Anyway, glad for the reach.
- xlii 4mo agoI glanced, and I found this handbook shallow and - in some areas - even bad advice. E.g. If I ever see a monetary value stored in something else than integers I'm going to run away screaming (thank you Rust decimals represented as JSON floats). It's always integers unless you have a VERY good reason to do otherwise (though exported view can be in anything, even in weird bitcoded formats). FX exchange. Resolution of FX isn't a point-in-time thing, things like buyer rate-in-time, seller rate-in-time, agreement, agreement tolerance, agreed upon resolution timestamp come in the effect. Immutability - that's why you want to have event sourcing everywhere that touches money: # Resolved stream A -> B -> E # Actual stream A0 -> Edit(A0, A) -> B -> C -> D -> Rollback(B) -> E Though in the end Fintech != Fintech. I worked at Fintech where money was treated like a baggage, and in other where money was a central point of everything.
- lxgr 4mo ago> thank you Rust decimals represented as JSON floats What do you mean? JSON doesn’t have floats, it has numbers, and how they’re used after being parsed is not part of the spec. > If I ever see a monetary value stored in something else than integers I'm going to run away screaming That’s good, then we’ll likely not be working on the same system :) I consider running from “amounts as integer” systems these days (but usually unfortunately can’t). In an idealized codebase that only seasoned financial programmers are allowed to touch, it can go well, but such a system is usually either overly exclusive or risks becoming brittle.
- lawlorino 4mo ago> I consider running from “amounts as integer” systems these days (but usually unfortunately can’t). In the context of Fintech, how do you otherwise resolve floating point rounding issues if not representing amounts with integers?
- lxgr 4mo agoNative decimal types, if your system has them. Many languages and databases used in financial contexts do.
- jgalt212 4mo ago> Webhooks are the most common way to receive signals from external systems, but processing them safely is not trivial I see webhooks documented all the time, but I have yet to use them in practice, nor have my customers requested them. Is the above not true, or are they widely used in some sectors and not others?
- intelcoders 4mo agoIn payment gateway integration, webhooks are usually considered a single source of truth for updating the payment status, with status check api as a fallback.
- weatherlight 4mo agoI receive webhooks all the time as ack that something was processed/ or a notification of the status of some sort of thing in an external system that we don't control.
- f3408fh 4mo agoThey are indeed everywhere, but it's possible you don't have a need for them. For example, every time you buy something online using Stripe, the seller receives a webhook indicating that a purchase was completed.
- senthil_rajasek 4mo agoDoes fintech here mean "crypto" and central bank currencies transactions?
- koliber 4mo agoAnyone know of resources like this but for capital markets? Things that would allow engineers new to trading equities, options, FX, bonds, and commodities to learn about different flows, market structure, common architectures, and other things that normally you learn from years of experience.
- mhh__ 4mo agoEquities and (listed) commodities are relatively easy to get a handle on but it genuinely takes months / years even at the frontline to understand how fixed income and FX works because its still almost entirely an OTC market. There is more central clearing than ever before but e.g. if I (say) buy a bond, fund it using a repo, swap my loot back some other currency, quite a lot of this could easily still be relying on humans pressing buttons and wiring money around. To learn how and why these things are traded, however, read this book, the only (good) truly beginners guide to fixed income: https://www.jdawiseman.com/books/pricing-money/Pricing_Money_JDAWiseman.html https://www.jdawiseman.com/books/pricing-money/Pricing_Money...
- FabHK 4mo agoEven with equities, things are not entirely trivial. A few examples: Some stocks trade in one currency, but pay dividends in a different currency. Some stocks go ex-div before the dividend amount has been determined (e.g. in Japan). Stocks trade on certain days, and trades settle on certain days, and they might be subject to different holiday calendars. Corporate actions are not entirely trivial, either (with rights issues, you need to issue two temporary securities for accounting purposes, etc.).
- dabauws 4mo agoabsolutely - and there's more too, at least when it comes to historical & live data for stocks. for example: stocks have multiple 'prices' - a bid, ask, last, mid, vwap, twap, etc. a stock's historical 'close' price can be split-adjusted, dividend-adjusted, unadjusted, split and dividend adjusted.. and more. a stock's ticker can change to a ticker used before. a stock can change ticker, but maintain a CIK identifier; a stock can change a CIK identifier, but keep the same ticker.
- anchorapi 4mo ago[flagged]
- ermantrout 4mo ago[flagged]
- rTX5CMRXIfFG 4mo agoI don’t work in fintech (yet) but I’ve studied finance recently and quite a lot of these pieces of advice are just intuitive when you know the business domain. Learning the “customer” of your software helps too
- jdw64 4mo agoAs a programmer, what I feel when I see fintech programmers each speaking from their own different experiences and perspectives is that it makes me wonder what it really means to be good at programming. What user xlii said about not storing monetary amounts as floats is a common IEEE 754 issue. And while it's true that financial tracking should be done through immutable logs or event-based records, I don't think every surrounding service needs to be built with event sourcing. I think it's enough to apply it only to core logic like ledgers, settlements, orders, and executions. Looking at xlii's comment, it seems like a technique that only becomes viable when the modeling is successful. User lxgr's comment points out that it's a minor-unit issue. If JSON numbers are parsed as floats by the language or parser, precision can be lost. Usually people send values with a separate decimal places field. However, I've heard that in HFT, they don't do that because the overhead itself is too costly. And antonymoose's comment aligns with what many books say. That's why designs like this are common in FX or API contexts. It feels like protocol design, doesn't it? Putting it all together, everyone's right within their own domain. While I think it'd be great to have someone like xlii as my senior programmer, I also feel like I wouldn't be able to design such a complex system myself. In that sense, everyone's statements are valid, and it's interesting to see how opinions diverge depending on the domain. Is this what expertise looks like Looking at all this, it seems like you can roughly infer where a programmer is coming from based on their experience. Sometimes programming doesn't feel like finding the right answer, but more like choosing a worldview Watching how programmers model their domains on HN is always fascinating. Sometimes I click on their profiles and add their domain knowledge to my own personal wiki, thinking I might use it someday
- belmarca 4mo agoNice. The book contains a bunch of good information that could already be found elsewhere but collecting it is quite practical. I highly suggest to read Kleppmann's Designing Data-Intensive Applications. The first edition was very good, a second one came out recently. I was CTO of a FinTech where I built the whole software stack from scratch: the lessons in the book are mostly correct. I say mostly, because as always, there is a lot of "it depends" to take into consideration for your particular project. For example, I chose to not use event-sourcing to avoid the whole state computation issue. A standard append-only audit trail can do the job. You can't guarantee exactly-once delivery but you can construct effectively-once processing, and that is what you really want. Store every request and response : absolutely, and not only when consuming APIs, but when collecting any information from the outside world (and, if you can, also log every intermediate transformation step within your perimeter). Content-adressed buckets + a relational table are great for this. The text also does not mention anything about data lineage. What happens if a vendor updates some data mid-day that you absolutely need to be aware of? You need to be able to account for that, while also re-playing computations that used the old values and get the same result. It's not a particularly hard problem to solve, but it takes some thought.
- potamic 4mo agoWhy content-addressed storage? I'm assuming this is mostly for auditability after the fact? Wouldn't it make sense to use some business identifier to be able to refer it back easily?
- belmarca 4mo agoYou will need a combination of the two. The filename should contain a compound business identifier, essentially a tuple of appropriate facts, as well as a (truncated) hash of the content. And yes, the reason is auditability but also bare necessity. Imagine vendor A publishes fresh files every day, but you don't exactly know when. And suppose that the vendor can (and does) republish some of them at any time, with or without notice. You need to watch those files for changes and update your own sources accordingly. For the collection and dedup, you can construct a tuple of meta-information that will give you a good idea if the file is the same (name, time last modified, file size, etc). Then only if you're still unsure do you download the file and hash it, compare with what you already have, and make a decision. (What if the file name changed slightly and now you must re-categorize the data in the file, or what if a single row was added or removed, etc). It's also useful when auditing (tamper-proof) or when re-running or debugging calculations that used the data in question. If a user used the data available at 9AM, which was unfortunately incomplete and led to issues, having a full data lineage really helps. You can trivially re-run the computation with the information available at that time to see what happened. Or you can mark some results as stale, etc. There are different ways to do this and one is keeping track of your content-adressed files in a relational table and give each row a unique identifier, so that you can tell that this particular row in your AAPL OHLCV data comes from this particular file which was fetched at this particular time etc, etc. Pair that ID with a timestamp and when doing computations you can query for "the latest" AAPL data while storing the exact file ID (time is fuzzy, content hash is not). Look into temporal tables and the like.
- traceroute66 4mo ago> even bad advice That's putting it politely. Honestly, I think this "handbook" was mostly written by an LLM. For example, in the immutability section we have this: "Separating PII from financial data lets you honor erasure without losing the financial history you’re obliged to keep." In a financial organisation the two go hand-in-hand for obvious KYC/AML reasons. Keeping the financial data whilst trashing the customer names, addresses etc. instantly on-demand before the expiry of the relevant time periods is going to leave your entire organisation with a very bad day in the office if a $lawful_body comes knocking for the data to trace a crime. People going to work in a Fintech should not be relying on a random "Handbook" written by an unknown person in an unknown jurisdiction. People going to work in a Fintech should only ever work in accordance with their employer's internal handbooks/guidelines/etc which will have been written in conjunction with their firm's lawyers and compliance people to ensure it complies with the laws and reporting requirements in the jurisdiction(s) in which their employer operates.
- lxgr 4mo ago> Keeping the financial data whilst trashing the customer names, addresses etc. instantly on-demand before the expiry of the relevant time periods [...] Where does TFA recommend that? As I see it, it recommends separating PII data you'll eventually have to delete from that you'd probably want to keep forever (including data factoring into your accounting equations/invariants), so that you can delete the former after the relevant recordkeeping periods have elapsed. > People going to work in a Fintech should not be relying on a "Handbook" written by an unknown person in an unknown jurisdiction. Sure, but they should also not blindly ignore any ideas and practices presented, or avoid looking beyond their own organization. Ideally, they'll then try to reconcile what they saw with their own knowledge and local regulations etc. > People going to work in a Fintech should only ever work in accordance with their employer's internal handbooks/guidelines/etc which will have been written in conjunction with their firm's lawyers and compliance people to ensure it complies with the laws and reporting requirements in the jurisdiction(s) in which their employer operates. Sure, in a world in with only perfect and error-free organizations, that seems like a reasonable approach. But how does one get there without having a conversation such as this one?
- morpheuskafka 4mo agoA Plaid balance check is NOT a guarantee that the ACH debit you're about to submit will go through. I don't care if the balance is one million, before that ACH can process, every single dollar can be (a) wired out, (b) cleared out by yesterday's ACHs (bills, autopay, whatever) and checks, or (c) spent at debit/ATM. I probably shouldn't tell you why I know that some fintechs don't address this.
- Ajpop3y 4mo agoThank you. It came at a much needed time.
- Ajpop3y 4mo agoThank you so much for this. It came at the needed time
- wavemode 4mo ago"Fintech" is extremely broad, and most of what gets called "Fintech" is really communication. Communication between firms, between traders, between systems, between ledgers, etc. There is no industry-wide "right" way to program anything, because the right way is ultimately whatever way the other party you're communicating with will understand. If you're dealing with a party who tracks currency in cents, then tracking currency with more precision than that is going to lead to rounding disagreements. Vice versa if you deal in cents but they deal in tenths of cents. And so on for all the other advice in this document.
- penguin_booze 4mo ago> For whom? It's refreshing to see someone using the correct phrasing. The often-seen, stupid way is 'who is this book for'.
- vshulcz 4mo ago[flagged]
- fvdessen 4mo agoThe whole avoid floats thing just isn't true. I have 20years experience in fintech and most of it used doubles. Excel uses doubles. Your frontend will use doubles. All your db supports doubles. Your stdlib knows how to parse doubles. Json uses doubles (not in theory but in practice). Many ERP system uses doubles The thing for working with currency with doubles is that you have to keep in mind that it can hold 15 digits of precision in total. As long as your numbers don't use more digits than that, like 123456789.01 or 123.456789, you can have perfect decimal precision in your financial math. You just have to always round the result to within 15 digits of precision after each computation, and before each comparison. That's what excel does. The biggest advantage of doubles is that 1) they're widely supported and 2) you can mix different precision in your system, which will appear if you do international finance or advanced financial products. Some accounting require precision up to the thousandth, some need to be rounded to multiples of 0.25. So at the end of the day you'll never use basic math but some specialised accounting math library and that library can perfectly use float as a backend.
- tukantje 4mo agoI think his experience in the crypto space is part of the reason why he is against it.
- kevinten10 4mo ago[flagged]
- lanstin 4mo agoAll this debate about floats or ints or decimals, obscures the fact “reconciliation” is an essential part of dealing with money - essentially some separate implementation of the money movement that detects when your rounding strategy or floats end up creating or destroying money, but all the accounts balance at then of the day.
- jvuygbbkuurx 4mo agoI wanted to say thanks for this resource. I'm founding a fintech startup and have been thinking about these things a lot. It is very helpful to have some general validation and guidance to make sure I'm not making huge mistakes. I really wanted event sourcing, but thought it gets complicated when events have complex validation rules. I ended up with a mix of state and audit logs which works ok for now.
- trixn 4mo ago> Money can’t be created out of nowhere Well, in case of a bank it can. And it happens on a daily basis. In fact thats how most private bank money enters the system in the first place either when a bank makes a loan or when it buys any other asset like e.g. a corporate bond (you can conceptualize a loan as an asset purchase as well, the bank buys a promise to pay from the borrower which is recorded as an asset of the bank). Both are a balance sheet extension from a double-entry bookkeeping point of view. The 4 operations double-entry bookkeeping allows are: 1. balance sheet extension (e.g. making a loan) 2. balance sheet reduction (e.g. loan repayment) 3. asset swap (e.g. a bank buying a government bond with central bank reserves) 4. liability swap (e.g. transferring money from one account to another) So it may be more accurate to just say that every event that affects the balance sheet must be one of those 4 operations. Also see "Money creation in the modern economy" - Bank of England Quarterly Bulletin 2014 Q1
- red_trumpet 4mo ago> Well, in case of a bank it can. And it happens on a daily basis. You mean it happens when a bank borrows money from a central bank, right? Not every balance sheet expansion constitutes the creation of money, e.g. when you put money on your bank account, the bank's balance sheet grows, but the money is not newly created.
- trixn 4mo ago> You mean it happens when a bank borrows money from a central bank, right? No I mean in the moment the bank grants a loan. The money the borrower receives is new money, its not taken from anywhere else. But at the same time the bank records the credit as an asset which makes it a balance sheet extension. > when you put money on your bank account, the bank's balance sheet grows That's true, you hand over cash to the bank which becomes its asset and your account balance is marked up for the same amount. Not all money in circulation is accounted on a bank balance sheet, but most of it is bank deposits. cash is an exception, it's public money, not private bank money. When you "put money in a bank account", i.e. you bring cash to the bank you are essentially selling your public government money to the bank and it pays you with its own private bank money (which it "creates" as well). But this is only a micro-view of a single event. In the aggregate money is created when either the government runs a deficit or when banks make loans and money is destroyed when taxes are paid or loans are repaid.
- deleted 4mo ago[deleted]
- nmcost16 4mo agoI really enjoyed this. This year I joined a fintech company, coming from a non-financial domain. This is the kind of information that I had to learn from experience, other devs, and random blogs/articles. The fact that this is a centralized location is awesome! I wish I knew about this when I was onboarding.
- john-frandsen 4mo ago[flagged]