10 ms·

Problems with homemade billing systems

179 points by thibo_skabgia 3y ago

pantulis 3y ago
Billing is one of those problems that seem deceivingly easy. Just talk to someone who has worked in telco billing and learn about the 4 horsemen of billpocalypse: metering, mediation, accounting, billing.
- trollied 3y ago
  Rating can be great fun. US taxes (geocoding, urgh) are an absolute nightmare. Reversed charged calls, revenue sharing. Roaming is a pain (ASN.1 files). Even document numbering is a nightmare in some countries, as are legalities surrounding incorrect bills (corrective invoices) and crediting. Did 15 years in billing (now do finance software), and whenever I thought I knew it all there would be another curve ball thrown.
- Rafsark 3y ago
  You can add taxes and negotiated contracts to the list tbh
footlose_3815 3y ago
This is an ad
542458 3y ago
Good article, and I generally agree that people are too quick to say “Ahh, this looks like a two week project” and ignore broader complexity. I’ve definitely seen a fair amount of that. My question is, how do I know that the OTS solution would actually be much better? For example, OP describes some tricky migrations from one type of billing to another, or complex grandfathering schemes - how would you have any guarantee that your third party solution would be able to support those? At least if it’s in-house you can implement the needed functionality eventually - if it’s third-party it might just be flat-out impossible.
- emilfihlman 3y ago
  Indeed. Very often organisations tend towards buying solutions without appropriately gauging the situation. One easy example from IT is buying servers vs renting them, but fortunately it seems that IT people have found the, for some uncomfortable, balance of "it always depends", and the analysis might turn multifaceted pretty quickly. I also feel that people often discount the cost of up top decisions that people below don't like.
- 6510 3y ago
  HAHAHA, my estimate was 2 weeks. This was 6 months ago. It is now a marvel of engineering with most of the 100 000 edge cases covered. You should read that as: not battle tested on (insane) users. (I'm so looking forwards to that) I estimate 2 years to completely finish it, give or take a decade or two.
  - MightyBuzzard 3y ago
    [dead]
- ineedasername 3y ago
  >how would you have any guarantee that your third party solution would be able to support those... You could say the same thing about literally any 3rd party software or service you purchase. "Why use cloud services to do anything?" etc. Sure this thinking will hold for some things, but chances are that for something as standard as billing you're not going to be implementing some revolutionary feature that gets you more customers. On the other hand, if you mess up billing because you rolled your own system that doesn't have many standard options that a mature OTS system would offer it can lose you potential customers, because for months at a time you're stuck building the most mundane and standard things like an annual plan that could be used to better appeal to buyers. For billing, a one of the most standard things ever for a business, you simply take a small amount of time to survey the OTS options and compare features. Any you know what? There's an excellent chance that buy actually getting a comprehensive look at features offered by such options you will learn about the types of things you will absolutely want to do in the future. Things that, in retrospect, seem like obvious oversights, such as an annual plan. You'll see features like that in your survey and say to yourself "Oh, yeah, that's something we may definitely want in the future, lets put that on our requirement list for a final choice."
hydroid7 3y ago
I think it depends on the business you run. To be honest, I would not mind building a billing system if my company grows to an unicorn!
- Rafsark 3y ago
  How do you know in advance it will grow into a unicorn? I tend to agree with you, but that's a shame to have a team of 30 engineers working only on billing, even if you are a unicorn.. I would prefer to have them working on my product
hrunt 3y ago
Having gone through /exactly/ what this article is about, I know the pain. But I will point those that don't read the article to the last paragraph: > We considered implementing an off-the-shelf billing solution but there was nothing flexible enough and the switching costs were too high. Algolia also tried to migrate to Zuora before backing out and rebuilding their billing system for the fourth time. Most (all? I have yet to find one) off-the-shelf systems are not geared towards everything businesses want to do with their billing. If you take the author's advice and choose one of these systems early on, you will have the exact same headaches. Either the system simply won't let you do what you want to do or the system /will/ let you do what you want to do with custom or hack-ish solutions that reproduce the custom-solution problem, but in someone else's system. Those that implement broad swathes of billing functionality are so complex, they make /everything/, even the most basic stuff, hard. BTW, when I say I know exactly how the author feels, all of what they described we encountered and implemented. We even looked at a migration to Zuora and came away with the same conclusion (also: really freaking expensive). We even had a new product that we setup on a third-party billing system, and we ran into the flip-side of the problems; we were not able to implement some billing functions we needed and we had to migrate off.
- 6510 3y ago
  > there was nothing flexible enough They would have to solve the same problem in a generic way. (add 10 more years) > we had to migrate off Adds to the fun doesn't it? My missing features hacked around previously exploded into an almost impossible puzzle.
- akitzmiller 3y ago
  I came here to pretty much make the same comment. If you have a business that can get away with OTS software, more power to you. But complicated things are complicated and complex billing rules can be make-or-break for a company. Whether you start with 3rd party or start home grown, billing systems often need humans and there really isn't a substitute.
freefaler 3y ago
Like with everything, wrap your billing in a layer and when you grow enough you put your billing under it if you need it or switch to a different provider without changing your codebase. It's not wise to outsource parts that are essential for your business, because: 1. you depend on 3rd party service for a getting paid 2. your operations cost may rise or the service may change terms/functionality
- deleted 3y ago
  [deleted]
- bluGill 3y ago
  You always need to outsource things. Unless you live on a desert island alone some things are out sourced. Most of us out source barter to a government run cash system, which we then further outsource to various banks. While out sourcing is a risk, it is also an opportunity to offload the hard work. You need to figure out what is worth doing yourself; what you outsource completely and ignore; and what you out source and audit. Accounting - including billing, is normally something you should out source and audit. Let someone else worry about all the hard and weird rules. However you need to audit it because if you do not someone corrupt will steal your money and you are still required to pay whoever you owe money after that happens - this is one way your company can go bankrupt.
  - freefaler 3y ago
    That's why you need an abstraction layer for each 3rd party provider. I've been doing online businesses 20+ years and every time we didn't create an abstraction layer our switching cost rose. Many "startups" who are hungry enough to provide good value for money switch to "enterprise" pricing when investor money run out and your operations costs may rise a lot or you get "discontinued". Look what Atlassian did with self-hosted versions. Also for tracking/logging layers it may be beneficial to mirror the data you're sending to 3rd party providers to your own db/logs, because you won't get data locked when switching. So my point is that you need to outsource smartly and keep in mind that the 3rd party dependencies increase operations risk. BTW that's why any user request shouldn't depend on 3rd party resources in a blocking matter (e.g. synchronous JS from vendors' CDN). You might pay more for your hosting but if you factor the service disruption risks you may be better off for mission critical parts.
mattboardman 3y ago
It looks like one core issue was designing the system around pre-determined bounds (monthly, yearly, etc). This happens in lots of systems and isn't specific to billing. A job scheduling system will have the same problem (we built it to track seconds and now they want us to track calendar dates!).
- Rafsark 3y ago
  Navigating this challenge involves the ever-changing boundaries of billing. Initially, you build a monthly billing system, only to find that your team requires a yearly one, which seems relatively straightforward. However, complications arise when you introduce usage-based billing, incorporating weighted values akin to storage, layered on top of a quarterly plan. This complexity is further exacerbated when custom billing structures are needed for negotiated contracts, adding another layer of intricacy to the mix.
m3nu 3y ago
I built my billing system on top of Django and it's used in 2 online services for a few thousand users and hundreds of invoices a month. You have to be mindful of some edge cases, but if the scope is kept narrow, it's doable. The added benefit is the flexibility to fit it to your use case. And with Python, you get many high-level libraries, e.g. for decimal calculations and PDF generation.
- yugarinn 3y ago
  My experience is that the scope is never kept narrow, which is exactly the same that the author is describing.
  - acutesoftware 3y ago
    Well, you can - you can have a policy that forces payments from a select group. Some users will complain when "No, you cant pay with 2 chickens every 3rd full moon", but that is too bad.
    - dagw 3y ago
      Of course you then have to be willing to turn down money/customers. As someone who has worked for quite large traditional companies, we've ended up not buying from certain companies because they couldn't invoice us like we wanted to be invoiced.
      - 6510 3y ago
        And miss out on all the fun with buyer-created invoices?
- icedchai 3y ago
  I built a billing system for a web hosting company in PHP, almost 20 years ago. This was for a few hundred users. There were a couple of edge cases, but nothing incredibly challenging.
  - 3y ago
vidarh 3y ago
I've built several billing systems and managed a team that built a much larger one, and it's not that hard, but it does require an attitude and attention to detail that a lot of people lack. E.g. one thing that worked well in the systems I've worked on was to break things into small state transitions, very firmly document the allowed states and their transitions, and log every transition to the database. Wherever possible (anywhere that didn't interact with external API's, and sometimes even when they did) we'd aim for state transitions to be idempotent. When they weren't, we'd seek to reduce the non-idempotent call to an external dependency to just that call as a separate state transition. When something very occasionally went wrong, we could trace things in detail, and pinpoint it, and most of the time we could safely replay transitions around whatever went wrong until we could see what had happened. It wasn't difficult to build these systems that way, but it was tedious, and something where it's easy for people to be tempted into shortcuts. As for scaling, you spit out events for rollups, and can shard invoice generation. We handled millions of revenue on a machine far less powerful than my laptop today. Scaling really would not have been up there for me. To #3, maintaining old pricing isn't generally that hard. Very few people drastically change how they account for usage, which tends to be the most complicated billing scenario. They tend to change amounts and periods and thresholds, which should be data, not code. With respect to a team, I agree. One of the places I worked on billing was Yahoo. At the time it wasn't just one team, but several - my team (responsible for Europe) existed largely because the European finance and product teams didn't trust the US payment services team to take their requirements seriously enough and so wouldn't let them near the European premium services... Billing isn't something you want to do yourself unless it's either a core competency or you're big enough to have to deal with that kind of bullshit. (As for size of teams, I've built a billing system with 2 people, but I've also worked on billing systems managed by 50... You really want to have a good idea whether you're likely to end up towards the former or latter before you go ahead...)
- 6510 3y ago
  > #3, maintaining old pricing I copy the entire product into the order. The thing one buys should be the thing on the screen when the decision is made. Poor pictures, description with typos, wrong categories, old/wrong pricing etc
ftxbro 3y ago
This is an ad by a company that sells billing systems. This is what ads look like now.
- tylergetsay 3y ago
  Same with "Role your own auth"
  - dboreham 3y ago
    I see what you did there.
- danenania 3y ago
  A lot of useful content gets produced by companies as a form of marketing. Just like independent devs are often blogging with the goal of getting consulting gigs or advancing their careers. There's a lot of garbage too, but if the content is good and reasonably objective, what's the problem?
brianmcc 3y ago
I would instead say - commit properly to building your billing systems as a major and ongoing project, don't dismiss it as a 2 week "fun" side thing. Not convinced any 3rd party product is going to have the flexibility to support ongoing innovation. If anything it's more of a case for in-house expertise and resource. Find some folks who want to be the billing guys! Disclaimer: I built a large in-house billing system :-)
- agos 3y ago
  see also: e-commerce systems. Hard to build, a lot of work, but way better than any 3rd party
- calvinmorrison 3y ago
  Interestingly, fastmail just flipped to having an external billing system after years of maintaining their own. AND when they bought POBOX they got that whole billing stack as well. I can imagine the nightmare but also the relief after exporting all of that work to another company. perl billing system: https://github.com/fastmail/Moonpig https://github.com/fastmail/Moonpig
  - re-thc 3y ago
    > Interestingly, fastmail just flipped to having an external billing system after years of maintaining their own. Would have nothing to do with your or someone else's decision. And Fastmail may or may not have made the right decision. It's hard to say. There are no simulators or time machines to gauge the impact anyway. The most likely answer is preference and available skillset in the area. I've seen managers that absolutely want to self build and those that never want to maintain anything as much as possible (by offloading).
madjam002 3y ago
I’ve found Temporal and Xstate are helpful at building a reliable billing system
- Rafsark 3y ago
  Could you explain how? I know temporal has not built the billing engine themselves, so wondering how the tool can help with this..
  - lorendsr 3y ago
    Using Temporal is in the category of building it yourselves, not a billing service. But it makes it take much less time to build, because it makes the changes, scaling, and grandfathering quick to do. Especially with the new Scheduled workflows feature: https://github.com/temporalio/samples-typescript/tree/main/schedules https://github.com/temporalio/samples-typescript/tree/main/s...
lowercased 3y ago
Why do I have the feeling that many people starting projects like this get in to situations where functionality is identified and they're beat down with "YAGNI!"? I've been in situations where 'build v buy' comes up - accounting, inventory, and similar domains. Identifying "hey, we need to keep XYZ data to allow for future reporting..." has been met with chants of "YAGNI" from people blind to the complexities. It's just one of the things that might lead me to opt for 'buy' in 'build v buy', but... it's hard to trust the sales people involved in the process as well. And... taking the time to do real and full analysis... is time people often don't want to pay for. And... once you're "in" to a COTS system... you're almost never leaving. The companies know that, the salesfolks know that, and have a lot of incentive to handwave away concerns, or just outright lie. Once you discover the lies... it's likely too late to switch.
- hk1337 3y ago
  That situation is why I don't like about the software development process. YAGNI gets abused to go with the sloppy shortcut that only really saved maybe a day in creation but in reality wastes months worth of maintenance.
- wredue 3y ago
  >Once you’re “in”, you’re almost never leaving Yeah. This problem is real. It’s so expensive to get out that we have had to make business process concessions because the COTS ended up not supporting, what I consider, basic functionality. We only found out years later on the basis of changing other business processes. And vendors will bleed you dry if they can. What some vendors consider to be “customizations” is insane.
say_it_as_it_is 3y ago
Taxes alone are complicated enough to warrant the existence of TaxJar. Billing systems should not be underestimated.
- Rafsark 3y ago
  This is completely true. Taxes, entitlements and billing can be 3 different products. I don't understand how people think it's a 3 week effort. Avoiding this separation can lead to a messy setup down the road.
TomNguy 3y ago
How do you monitor for tax compliance changes, especially for international usage?
- Rafsark 3y ago
  I think billing, taxes and entitlements are 3 different products, with obviously strong relationships. I would rather give the advice to rely on a third party tax tool that is connected to the billing engine instead of the "all in one" if taxes are too complex. Otherwise, just use the basic tax codes of Lago's billing engine
swader999 3y ago
Now take your system and make it multi tenant. Imagine the horror of doing that. This is what you often get with "off the shelf". There's a lot of broken dreams either way you go.
zer0x4d 3y ago
Billing in general is very hard. Especially subscription billing. Just from my experience in building billing and also later on using a third party system for another projects, I can mention so many edge cases: 1. Grandfathering 2. Upgrading across different length memberships (monthly tier 1 to yearly tier 2) 3. Crossgrading across different lengths (monthly tier 1 to quarterly tier 1) 4. Offering free trial to tier 2 when user has tier 1. 5. Promotional offers to non-paying users (half off first month) 6. Downgrades 7. Applying coupons to accounts (customer support wants to offer a specific user a free month of service when user has a quarterly subscription: billing schedule change) 8. Differences in tax regulations across countries (VAT vs whole value tax) 9. Differences in display prices (in some countries displayed prices should include tax) 10. Handling currency exchange rate changes.
- 1-6 3y ago
  Sounds a lot like Autodesk.
- Rafsark 3y ago
  I could add: - metering - relations with accounting/finance softwares - timezones (if your customers are spread around the globe) - proration - payments and dunnings
garganzol 3y ago
I wrote a billing system myself, and my first impression was that it couldn't get past 1000 lines of code. However, it had become a ~5000 lines monster and keeps on growing. Never underestimate details and edge cases.
mlhpdx 3y ago
The point the article makes to me (as a leader) is “keep the billing model simple, honest and stick with it”. The cost of change is high, and the value isn’t to the people getting the work done. My peers would likely see this as “naive” and extoll the need to “sell to the buyer”. I see it as being value focused, and keeping the “business needs” rational, and in line.
- Rafsark 3y ago
  When you don't understand the complexity of billing (revenue ops, top management or marketing), you often end up adding more and more complexity, and engineering teams have no choice. They have to build new billing features
1970-01-01 3y ago
Creating and maintaining a billing system would be another good test for AI.
- Rafsark 3y ago
  What's your vision? Can AI handle revenue prediction and billing engineering tasks? Both? How?
  - 1970-01-01 3y ago
    Resolving the billing edge cases would be a test. There is no vision, it's a question: Is your flavor-of-the-week AI 'smart' enough to design and code a billing program for my startup? How about my medium-sized company with 4 offices in 3 different countries and 4 timezones?
somsak2 3y ago
Should have a (2022) label https://web.archive.org/web/20220715000000*/https://www.getlago.com/blog/the-4-biggest-problems-with-homemade-billing-systems https://web.archive.org/web/20220715000000*/https://www.getl...
agentultra 3y ago
In my experience this happens frequently enough that there ought to be an article like this for every problem domain that software developers and executives under-estimate the complexity of. I work on card transaction processing and it's absolutely more complicated than you can imagine looking at it from the outside. It's a vast, distributed peer-to-peer system where trust is built into liabilities outside of the network itself. You want to build a system that is correct with regards to some specification in order to protect peoples' money... but the problem is that there is no specification that enumerates every possible transaction state because while there are "typical" sequences of messages to handle, your system also has to handle unexpected sequences that can't be specified: there are no guarantees that systems on the other end are emitting properly formatted events let alone behaving as expected. Card payment handling systems have to have the ability to manually correct and adjust state by humans; reconcile with external systems (because yay, payment systems in the US aren't based on instantaneous settlement until the roll-out of FedNow is complete), etc. I've watched many businesses walk straight into, it's just X, how hard can it be? Only to watch their ARR shrink and stress levels rise when a project they thought would take a month turns into a year-long journey of discovery, reflection, and increased head-count.
jabart 3y ago
When we launched in 2017, we went "in-house" with our billing. We found an open-source project and uses that for all billing as it had a recurring option and a billing portal. Fast forward a number of years, we now sync invoice data in the app and have a tiny billing portal there. We can still adjust invoices, keep old pricing, setup new pricing, auto-bill, check balances to see if a customer hasn't paid yet. Total Lines of Code here are like under 2000 for this integration and it's great. Also zero vendor lock in as well. Just another way to approach a billing system.
- jasonjayr 3y ago
  What open source project did you use?
  - jabart 3y ago
    InvoiceNinja. The install instructions had a few misses but has an API and webhooks so it's great to integrate with.
robaato 3y ago
Worked in mobile Telecoms billing back in the relatively early days. Remember discovering that Orange at the time made a good feature of competing (and acquired lots of new customers) by offering "competitor match" billing options. They made this as a business choice, but it cost them a team of over 100 contractors developing that system. Other telcos had packages but were constrained by what the packages offered as to what their marketing could offer. You pays your money and makes your choice! p.s. things learned the hard way in those early days - customers realised that a call held open for over 24 hours resulted in a reset of billing due to counter wrap around!
jerf 3y ago
Another problem with billing systems I've seen repeatedly is that the billing demands flows one way; marketing or product management basically scribbles on a napkin how they'd like to bill the customer, and that napkin gets passed to the billing team as a specification carved into stone. Basically, in practice it gets run as Waterfall, actual factual Waterfall, except the design phase is skipped. It goes straight from features to implementation. Now, on a business level I acknowledge that frequently the monetary costs of the billing system are negligible compared to the business advantages of the "correct" billing system. My complaint is more that the scribbled note often ignores what the rest of the company is doing for billing, e.g, "yes usage based billing is great but do you really need to bill on furlongs per fortnight when the rest of the company is billing based on seats or some basic, easy-to-understand resource allocation?", ignores that maybe there's an easy thing that's close to what you said but since it reuses all the existing stuff you can have it in two weeks instead of four months, etc. Do you really need to issue a discount coupon that takes off a percentage proportional to the length of the customer's first name, unless the customer is in a jurisdiction where that's illegal in which case we roll a die, unless the customer is in a jurisdiction where that's illegal in which case we give them $10 off their first $100 and take off every prime-numbered dollar after that? Because I've got code that just takes 10% off their first month right here. Do you really need to bill on the first of the month unconditionally when the rest of the company bills based on subscription start date, or vice versa? Even if you're not too worried about the costs of paying 6 months of developers, I bet you are worried about those 6 months of opportunity cost. I can't even count the months of delays caused by treating napkin scribbles as carved stone I've personally witnessed, and I'm only tangentially involved in billing over all. And somehow managers who are deeply familiar with costs and benefits and tradeoffs and make sensible decisions all the time just become the most obstinate customers when it comes to billing. No matter how small the tweak proposed and how many months forward it may bring your release date to do something just slightly different (and consistent with the rest of the company's existing policies), they will go to the wall for their billing deviation, even when it was frankly clearly nothing more than a whim or a transient thought at some point by somebody somewhere of no strategic or marketing consequence. It isn't just engineers who underestimate the complexity of billing until it's too late; it's everyone, really. "Just" print an invoice turns out not to be so simple.
deleted 3y ago
[deleted]
cmilton 3y ago
Why are we against modifying our processes to align with a particular software platform? Maybe it is much easier to follow such guidelines during the early stages of a company, but much harder later on. Do standards exist in billing systems?
baq 3y ago
I'm working on an in-house billing system right now. My only comment is 'yes'.
siva7 3y ago
I was that 20-something guy tasked with building a homemade billing system. Hint: Don't.
- marcosdumay 3y ago
  If you are not afraid of it, it means you don't know nearly enough to be successful. Maybe there are 20-something people out there with enough world knowledge to make one. But it's not a safe bet at all.
- Rafsark 3y ago
  Even if you find that guy, you'll have to find 20 more in the next 3 years if the company grows
kosolam 3y ago
How about accounting, would anyone recommend a modern open source accounting system that would be better than in-house?
- Rafsark 3y ago
  Not necessarily open source, but there are a lot of third party tools for accounting (netsuite, quickbooks, xero). And I would say they scale 100x better than an in house tool
  - kosolam 3y ago
    Not open source is not interesting. You could just go with SAP and be done with it.
pmontra 3y ago
I worked for a mobile phone operator many years ago. I was there when the company was bootstrapped. We built our own billing system because we would be billing for stuff that incumbents didn't bill for, because they didn't offer those services: we were the first 3G operator of the country and we also had to create and sell contents (example: YouTube was not born yet.) I was not in the billing team but we definitely followed rule #4 of the post: "You must be prepared to staff an entire team". And yes, #1 "Pricing changes all the time and billing needs to follow". About #2 "Your billing system needs to scale with your user base", we had to go from 0 to 3 million customers in 9 months to be viable. We were not typical, we made it. No idea about #3 "Grandfathering causes headaches" but there were probably many headaches, that one and others.
fmajid 3y ago
I've worked on telco billing systems in the past and the generally accepted metric in the industry is 70% of billing systems projects fail. Not fail as in "did not completely meet requirements or expectations", fail as in "did not deliver anything whatsoever".
- PeterStuer 3y ago
  That is the standard for any type of large enterprise software project though.
oblib 3y ago
I've been working on an simple invoicing app for small businesses for over 20 years. It has become a somewhat complex project and my app does not do much of what's cited in this article. >> Now, when someone asks for advice about their billing system, my answer is clear: DO NOT build it yourself. Moving from using CGI.pm to CouchDB/PouchDB about 10 years ago made it much easier to manage users and their data, and implement the UI. It also moved most of the workload to the user's web browser and made the app much faster for users. CouchDB was practically designed for this. Early on their developers used "invoicing" as an example for using CouchDB. But... if you have a marketing team that keeps moving the goals it wouldn't matter what you built the app with, it's going to take time to build and debug it. Could be that CouchDB/PouchDB could make that easier, but my experience is developers who've only been using SQL may get frustrated with it.
jwsteigerwalt 3y ago
The number backend engineers with competency in recurring revenue accounting is too small. Strange in our world that is more and more subscription driven.
PeterStuer 3y ago
I was waiting for them to discover the rabbit hole that is 'day counters'. (I wrote a lease payments calculator for the financial services industry. Many gotcha's lurk there.)
eternityforest 3y ago
I'm amazed anyone would ever want to DIY something that touches money. People tend to get pissy about money. Almost as much as water plant scada systems and railway safety. Maybe moreso. And if you're dealing with money, you're already inherently doing cloud stuff, because you're talking to the payment services, plus you already by definition have money to pay for a real billing system... It seems as crazy as people on r/wallstreetbets advising their parents on how to put their retirement money into hand picked stocks or something.
Pesky_Bees 3y ago
That's why you hire Product and UX designers so the engineers don't run wild into endless attempts to find an initial solution through the rabbit's hole.

179 points by thibo_skabgia 3y ago

Billing is one of those problems that seem deceivingly easy. Just talk to someone who has worked in telco billing and learn about the 4 horsemen of billpocalypse: metering, mediation, accounting, billing.

trollied 3y ago
Rating can be great fun. US taxes (geocoding, urgh) are an absolute nightmare. Reversed charged calls, revenue sharing. Roaming is a pain (ASN.1 files). Even document numbering is a nightmare in some countries, as are legalities surrounding incorrect bills (corrective invoices) and crediting. Did 15 years in billing (now do finance software), and whenever I thought I knew it all there would be another curve ball thrown.
Rafsark 3y ago
You can add taxes and negotiated contracts to the list tbh

footlose_3815 3y ago

This is an ad

542458 3y ago

Good article, and I generally agree that people are too quick to say “Ahh, this looks like a two week project” and ignore broader complexity. I’ve definitely seen a fair amount of that. My question is, how do I know that the OTS solution would actually be much better? For example, OP describes some tricky migrations from one type of billing to another, or complex grandfathering schemes - how would you have any guarantee that your third party solution would be able to support those? At least if it’s in-house you can implement the needed functionality eventually - if it’s third-party it might just be flat-out impossible.

emilfihlman 3y ago
Indeed. Very often organisations tend towards buying solutions without appropriately gauging the situation. One easy example from IT is buying servers vs renting them, but fortunately it seems that IT people have found the, for some uncomfortable, balance of "it always depends", and the analysis might turn multifaceted pretty quickly. I also feel that people often discount the cost of up top decisions that people below don't like.
6510 3y ago
HAHAHA, my estimate was 2 weeks. This was 6 months ago. It is now a marvel of engineering with most of the 100 000 edge cases covered. You should read that as: not battle tested on (insane) users. (I'm so looking forwards to that) I estimate 2 years to completely finish it, give or take a decade or two.
- MightyBuzzard 3y ago
  [dead]
ineedasername 3y ago
>how would you have any guarantee that your third party solution would be able to support those... You could say the same thing about literally any 3rd party software or service you purchase. "Why use cloud services to do anything?" etc. Sure this thinking will hold for some things, but chances are that for something as standard as billing you're not going to be implementing some revolutionary feature that gets you more customers. On the other hand, if you mess up billing because you rolled your own system that doesn't have many standard options that a mature OTS system would offer it can lose you potential customers, because for months at a time you're stuck building the most mundane and standard things like an annual plan that could be used to better appeal to buyers. For billing, a one of the most standard things ever for a business, you simply take a small amount of time to survey the OTS options and compare features. Any you know what? There's an excellent chance that buy actually getting a comprehensive look at features offered by such options you will learn about the types of things you will absolutely want to do in the future. Things that, in retrospect, seem like obvious oversights, such as an annual plan. You'll see features like that in your survey and say to yourself "Oh, yeah, that's something we may definitely want in the future, lets put that on our requirement list for a final choice."

hydroid7 3y ago

I think it depends on the business you run. To be honest, I would not mind building a billing system if my company grows to an unicorn!

Rafsark 3y ago
How do you know in advance it will grow into a unicorn? I tend to agree with you, but that's a shame to have a team of 30 engineers working only on billing, even if you are a unicorn.. I would prefer to have them working on my product

hrunt 3y ago

Having gone through /exactly/ what this article is about, I know the pain. But I will point those that don't read the article to the last paragraph: > We considered implementing an off-the-shelf billing solution but there was nothing flexible enough and the switching costs were too high. Algolia also tried to migrate to Zuora before backing out and rebuilding their billing system for the fourth time. Most (all? I have yet to find one) off-the-shelf systems are not geared towards everything businesses want to do with their billing. If you take the author's advice and choose one of these systems early on, you will have the exact same headaches. Either the system simply won't let you do what you want to do or the system /will/ let you do what you want to do with custom or hack-ish solutions that reproduce the custom-solution problem, but in someone else's system. Those that implement broad swathes of billing functionality are so complex, they make /everything/, even the most basic stuff, hard. BTW, when I say I know exactly how the author feels, all of what they described we encountered and implemented. We even looked at a migration to Zuora and came away with the same conclusion (also: really freaking expensive). We even had a new product that we setup on a third-party billing system, and we ran into the flip-side of the problems; we were not able to implement some billing functions we needed and we had to migrate off.

6510 3y ago
> there was nothing flexible enough They would have to solve the same problem in a generic way. (add 10 more years) > we had to migrate off Adds to the fun doesn't it? My missing features hacked around previously exploded into an almost impossible puzzle.
akitzmiller 3y ago
I came here to pretty much make the same comment. If you have a business that can get away with OTS software, more power to you. But complicated things are complicated and complex billing rules can be make-or-break for a company. Whether you start with 3rd party or start home grown, billing systems often need humans and there really isn't a substitute.

freefaler 3y ago

Like with everything, wrap your billing in a layer and when you grow enough you put your billing under it if you need it or switch to a different provider without changing your codebase. It's not wise to outsource parts that are essential for your business, because: 1. you depend on 3rd party service for a getting paid 2. your operations cost may rise or the service may change terms/functionality

deleted 3y ago
[deleted]
bluGill 3y ago
You always need to outsource things. Unless you live on a desert island alone some things are out sourced. Most of us out source barter to a government run cash system, which we then further outsource to various banks. While out sourcing is a risk, it is also an opportunity to offload the hard work. You need to figure out what is worth doing yourself; what you outsource completely and ignore; and what you out source and audit. Accounting - including billing, is normally something you should out source and audit. Let someone else worry about all the hard and weird rules. However you need to audit it because if you do not someone corrupt will steal your money and you are still required to pay whoever you owe money after that happens - this is one way your company can go bankrupt.
- freefaler 3y ago
  That's why you need an abstraction layer for each 3rd party provider. I've been doing online businesses 20+ years and every time we didn't create an abstraction layer our switching cost rose. Many "startups" who are hungry enough to provide good value for money switch to "enterprise" pricing when investor money run out and your operations costs may rise a lot or you get "discontinued". Look what Atlassian did with self-hosted versions. Also for tracking/logging layers it may be beneficial to mirror the data you're sending to 3rd party providers to your own db/logs, because you won't get data locked when switching. So my point is that you need to outsource smartly and keep in mind that the 3rd party dependencies increase operations risk. BTW that's why any user request shouldn't depend on 3rd party resources in a blocking matter (e.g. synchronous JS from vendors' CDN). You might pay more for your hosting but if you factor the service disruption risks you may be better off for mission critical parts.

mattboardman 3y ago

It looks like one core issue was designing the system around pre-determined bounds (monthly, yearly, etc). This happens in lots of systems and isn't specific to billing. A job scheduling system will have the same problem (we built it to track seconds and now they want us to track calendar dates!).

Rafsark 3y ago
Navigating this challenge involves the ever-changing boundaries of billing. Initially, you build a monthly billing system, only to find that your team requires a yearly one, which seems relatively straightforward. However, complications arise when you introduce usage-based billing, incorporating weighted values akin to storage, layered on top of a quarterly plan. This complexity is further exacerbated when custom billing structures are needed for negotiated contracts, adding another layer of intricacy to the mix.

m3nu 3y ago

I built my billing system on top of Django and it's used in 2 online services for a few thousand users and hundreds of invoices a month. You have to be mindful of some edge cases, but if the scope is kept narrow, it's doable. The added benefit is the flexibility to fit it to your use case. And with Python, you get many high-level libraries, e.g. for decimal calculations and PDF generation.

yugarinn 3y ago
My experience is that the scope is never kept narrow, which is exactly the same that the author is describing.
- acutesoftware 3y ago
  Well, you can - you can have a policy that forces payments from a select group. Some users will complain when "No, you cant pay with 2 chickens every 3rd full moon", but that is too bad.
  - dagw 3y ago
    Of course you then have to be willing to turn down money/customers. As someone who has worked for quite large traditional companies, we've ended up not buying from certain companies because they couldn't invoice us like we wanted to be invoiced.
    - 6510 3y ago
      And miss out on all the fun with buyer-created invoices?
icedchai 3y ago
I built a billing system for a web hosting company in PHP, almost 20 years ago. This was for a few hundred users. There were a couple of edge cases, but nothing incredibly challenging.
- 3y ago

vidarh 3y ago

I've built several billing systems and managed a team that built a much larger one, and it's not that hard, but it does require an attitude and attention to detail that a lot of people lack. E.g. one thing that worked well in the systems I've worked on was to break things into small state transitions, very firmly document the allowed states and their transitions, and log every transition to the database. Wherever possible (anywhere that didn't interact with external API's, and sometimes even when they did) we'd aim for state transitions to be idempotent. When they weren't, we'd seek to reduce the non-idempotent call to an external dependency to just that call as a separate state transition. When something very occasionally went wrong, we could trace things in detail, and pinpoint it, and most of the time we could safely replay transitions around whatever went wrong until we could see what had happened. It wasn't difficult to build these systems that way, but it was tedious, and something where it's easy for people to be tempted into shortcuts. As for scaling, you spit out events for rollups, and can shard invoice generation. We handled millions of revenue on a machine far less powerful than my laptop today. Scaling really would not have been up there for me. To #3, maintaining old pricing isn't generally that hard. Very few people drastically change how they account for usage, which tends to be the most complicated billing scenario. They tend to change amounts and periods and thresholds, which should be data, not code. With respect to a team, I agree. One of the places I worked on billing was Yahoo. At the time it wasn't just one team, but several - my team (responsible for Europe) existed largely because the European finance and product teams didn't trust the US payment services team to take their requirements seriously enough and so wouldn't let them near the European premium services... Billing isn't something you want to do yourself unless it's either a core competency or you're big enough to have to deal with that kind of bullshit. (As for size of teams, I've built a billing system with 2 people, but I've also worked on billing systems managed by 50... You really want to have a good idea whether you're likely to end up towards the former or latter before you go ahead...)

6510 3y ago
> #3, maintaining old pricing I copy the entire product into the order. The thing one buys should be the thing on the screen when the decision is made. Poor pictures, description with typos, wrong categories, old/wrong pricing etc

ftxbro 3y ago

This is an ad by a company that sells billing systems. This is what ads look like now.

tylergetsay 3y ago
Same with "Role your own auth"
- dboreham 3y ago
  I see what you did there.
danenania 3y ago
A lot of useful content gets produced by companies as a form of marketing. Just like independent devs are often blogging with the goal of getting consulting gigs or advancing their careers. There's a lot of garbage too, but if the content is good and reasonably objective, what's the problem?

brianmcc 3y ago

I would instead say - commit properly to building your billing systems as a major and ongoing project, don't dismiss it as a 2 week "fun" side thing. Not convinced any 3rd party product is going to have the flexibility to support ongoing innovation. If anything it's more of a case for in-house expertise and resource. Find some folks who want to be the billing guys! Disclaimer: I built a large in-house billing system :-)

agos 3y ago
see also: e-commerce systems. Hard to build, a lot of work, but way better than any 3rd party
calvinmorrison 3y ago
Interestingly, fastmail just flipped to having an external billing system after years of maintaining their own. AND when they bought POBOX they got that whole billing stack as well. I can imagine the nightmare but also the relief after exporting all of that work to another company. perl billing system: https://github.com/fastmail/Moonpig https://github.com/fastmail/Moonpig
- re-thc 3y ago
  > Interestingly, fastmail just flipped to having an external billing system after years of maintaining their own. Would have nothing to do with your or someone else's decision. And Fastmail may or may not have made the right decision. It's hard to say. There are no simulators or time machines to gauge the impact anyway. The most likely answer is preference and available skillset in the area. I've seen managers that absolutely want to self build and those that never want to maintain anything as much as possible (by offloading).

madjam002 3y ago

I’ve found Temporal and Xstate are helpful at building a reliable billing system

Rafsark 3y ago
Could you explain how? I know temporal has not built the billing engine themselves, so wondering how the tool can help with this..
- lorendsr 3y ago
  Using Temporal is in the category of building it yourselves, not a billing service. But it makes it take much less time to build, because it makes the changes, scaling, and grandfathering quick to do. Especially with the new Scheduled workflows feature: https://github.com/temporalio/samples-typescript/tree/main/schedules https://github.com/temporalio/samples-typescript/tree/main/s...

lowercased 3y ago

Why do I have the feeling that many people starting projects like this get in to situations where functionality is identified and they're beat down with "YAGNI!"? I've been in situations where 'build v buy' comes up - accounting, inventory, and similar domains. Identifying "hey, we need to keep XYZ data to allow for future reporting..." has been met with chants of "YAGNI" from people blind to the complexities. It's just one of the things that might lead me to opt for 'buy' in 'build v buy', but... it's hard to trust the sales people involved in the process as well. And... taking the time to do real and full analysis... is time people often don't want to pay for. And... once you're "in" to a COTS system... you're almost never leaving. The companies know that, the salesfolks know that, and have a lot of incentive to handwave away concerns, or just outright lie. Once you discover the lies... it's likely too late to switch.

hk1337 3y ago
That situation is why I don't like about the software development process. YAGNI gets abused to go with the sloppy shortcut that only really saved maybe a day in creation but in reality wastes months worth of maintenance.
wredue 3y ago
>Once you’re “in”, you’re almost never leaving Yeah. This problem is real. It’s so expensive to get out that we have had to make business process concessions because the COTS ended up not supporting, what I consider, basic functionality. We only found out years later on the basis of changing other business processes. And vendors will bleed you dry if they can. What some vendors consider to be “customizations” is insane.

say_it_as_it_is 3y ago

Taxes alone are complicated enough to warrant the existence of TaxJar. Billing systems should not be underestimated.

Rafsark 3y ago
This is completely true. Taxes, entitlements and billing can be 3 different products. I don't understand how people think it's a 3 week effort. Avoiding this separation can lead to a messy setup down the road.

TomNguy 3y ago

How do you monitor for tax compliance changes, especially for international usage?

Rafsark 3y ago
I think billing, taxes and entitlements are 3 different products, with obviously strong relationships. I would rather give the advice to rely on a third party tax tool that is connected to the billing engine instead of the "all in one" if taxes are too complex. Otherwise, just use the basic tax codes of Lago's billing engine

swader999 3y ago

Now take your system and make it multi tenant. Imagine the horror of doing that. This is what you often get with "off the shelf". There's a lot of broken dreams either way you go.

zer0x4d 3y ago

Billing in general is very hard. Especially subscription billing. Just from my experience in building billing and also later on using a third party system for another projects, I can mention so many edge cases: 1. Grandfathering 2. Upgrading across different length memberships (monthly tier 1 to yearly tier 2) 3. Crossgrading across different lengths (monthly tier 1 to quarterly tier 1) 4. Offering free trial to tier 2 when user has tier 1. 5. Promotional offers to non-paying users (half off first month) 6. Downgrades 7. Applying coupons to accounts (customer support wants to offer a specific user a free month of service when user has a quarterly subscription: billing schedule change) 8. Differences in tax regulations across countries (VAT vs whole value tax) 9. Differences in display prices (in some countries displayed prices should include tax) 10. Handling currency exchange rate changes.

1-6 3y ago
Sounds a lot like Autodesk.
Rafsark 3y ago
I could add: - metering - relations with accounting/finance softwares - timezones (if your customers are spread around the globe) - proration - payments and dunnings

garganzol 3y ago

I wrote a billing system myself, and my first impression was that it couldn't get past 1000 lines of code. However, it had become a ~5000 lines monster and keeps on growing. Never underestimate details and edge cases.

mlhpdx 3y ago

The point the article makes to me (as a leader) is “keep the billing model simple, honest and stick with it”. The cost of change is high, and the value isn’t to the people getting the work done. My peers would likely see this as “naive” and extoll the need to “sell to the buyer”. I see it as being value focused, and keeping the “business needs” rational, and in line.

Rafsark 3y ago
When you don't understand the complexity of billing (revenue ops, top management or marketing), you often end up adding more and more complexity, and engineering teams have no choice. They have to build new billing features

1970-01-01 3y ago

Creating and maintaining a billing system would be another good test for AI.

Rafsark 3y ago
What's your vision? Can AI handle revenue prediction and billing engineering tasks? Both? How?
- 1970-01-01 3y ago
  Resolving the billing edge cases would be a test. There is no vision, it's a question: Is your flavor-of-the-week AI 'smart' enough to design and code a billing program for my startup? How about my medium-sized company with 4 offices in 3 different countries and 4 timezones?

somsak2 3y ago

Should have a (2022) label https://web.archive.org/web/20220715000000*/https://www.getlago.com/blog/the-4-biggest-problems-with-homemade-billing-systems https://web.archive.org/web/20220715000000*/https://www.getl...

agentultra 3y ago

In my experience this happens frequently enough that there ought to be an article like this for every problem domain that software developers and executives under-estimate the complexity of. I work on card transaction processing and it's absolutely more complicated than you can imagine looking at it from the outside. It's a vast, distributed peer-to-peer system where trust is built into liabilities outside of the network itself. You want to build a system that is correct with regards to some specification in order to protect peoples' money... but the problem is that there is no specification that enumerates every possible transaction state because while there are "typical" sequences of messages to handle, your system also has to handle unexpected sequences that can't be specified: there are no guarantees that systems on the other end are emitting properly formatted events let alone behaving as expected. Card payment handling systems have to have the ability to manually correct and adjust state by humans; reconcile with external systems (because yay, payment systems in the US aren't based on instantaneous settlement until the roll-out of FedNow is complete), etc. I've watched many businesses walk straight into, it's just X, how hard can it be? Only to watch their ARR shrink and stress levels rise when a project they thought would take a month turns into a year-long journey of discovery, reflection, and increased head-count.

jabart 3y ago

When we launched in 2017, we went "in-house" with our billing. We found an open-source project and uses that for all billing as it had a recurring option and a billing portal. Fast forward a number of years, we now sync invoice data in the app and have a tiny billing portal there. We can still adjust invoices, keep old pricing, setup new pricing, auto-bill, check balances to see if a customer hasn't paid yet. Total Lines of Code here are like under 2000 for this integration and it's great. Also zero vendor lock in as well. Just another way to approach a billing system.

jasonjayr 3y ago
What open source project did you use?
- jabart 3y ago
  InvoiceNinja. The install instructions had a few misses but has an API and webhooks so it's great to integrate with.

robaato 3y ago

Worked in mobile Telecoms billing back in the relatively early days. Remember discovering that Orange at the time made a good feature of competing (and acquired lots of new customers) by offering "competitor match" billing options. They made this as a business choice, but it cost them a team of over 100 contractors developing that system. Other telcos had packages but were constrained by what the packages offered as to what their marketing could offer. You pays your money and makes your choice! p.s. things learned the hard way in those early days - customers realised that a call held open for over 24 hours resulted in a reset of billing due to counter wrap around!

jerf 3y ago

Another problem with billing systems I've seen repeatedly is that the billing demands flows one way; marketing or product management basically scribbles on a napkin how they'd like to bill the customer, and that napkin gets passed to the billing team as a specification carved into stone. Basically, in practice it gets run as Waterfall, actual factual Waterfall, except the design phase is skipped. It goes straight from features to implementation. Now, on a business level I acknowledge that frequently the monetary costs of the billing system are negligible compared to the business advantages of the "correct" billing system. My complaint is more that the scribbled note often ignores what the rest of the company is doing for billing, e.g, "yes usage based billing is great but do you really need to bill on furlongs per fortnight when the rest of the company is billing based on seats or some basic, easy-to-understand resource allocation?", ignores that maybe there's an easy thing that's close to what you said but since it reuses all the existing stuff you can have it in two weeks instead of four months, etc. Do you really need to issue a discount coupon that takes off a percentage proportional to the length of the customer's first name, unless the customer is in a jurisdiction where that's illegal in which case we roll a die, unless the customer is in a jurisdiction where that's illegal in which case we give them $10 off their first $100 and take off every prime-numbered dollar after that? Because I've got code that just takes 10% off their first month right here. Do you really need to bill on the first of the month unconditionally when the rest of the company bills based on subscription start date, or vice versa? Even if you're not too worried about the costs of paying 6 months of developers, I bet you are worried about those 6 months of opportunity cost. I can't even count the months of delays caused by treating napkin scribbles as carved stone I've personally witnessed, and I'm only tangentially involved in billing over all. And somehow managers who are deeply familiar with costs and benefits and tradeoffs and make sensible decisions all the time just become the most obstinate customers when it comes to billing. No matter how small the tweak proposed and how many months forward it may bring your release date to do something just slightly different (and consistent with the rest of the company's existing policies), they will go to the wall for their billing deviation, even when it was frankly clearly nothing more than a whim or a transient thought at some point by somebody somewhere of no strategic or marketing consequence. It isn't just engineers who underestimate the complexity of billing until it's too late; it's everyone, really. "Just" print an invoice turns out not to be so simple.

deleted 3y ago

[deleted]

cmilton 3y ago

Why are we against modifying our processes to align with a particular software platform? Maybe it is much easier to follow such guidelines during the early stages of a company, but much harder later on. Do standards exist in billing systems?

baq 3y ago

I'm working on an in-house billing system right now. My only comment is 'yes'.

siva7 3y ago

I was that 20-something guy tasked with building a homemade billing system. Hint: Don't.

marcosdumay 3y ago
If you are not afraid of it, it means you don't know nearly enough to be successful. Maybe there are 20-something people out there with enough world knowledge to make one. But it's not a safe bet at all.
Rafsark 3y ago
Even if you find that guy, you'll have to find 20 more in the next 3 years if the company grows

kosolam 3y ago

How about accounting, would anyone recommend a modern open source accounting system that would be better than in-house?

Rafsark 3y ago
Not necessarily open source, but there are a lot of third party tools for accounting (netsuite, quickbooks, xero). And I would say they scale 100x better than an in house tool
- kosolam 3y ago
  Not open source is not interesting. You could just go with SAP and be done with it.

pmontra 3y ago

I worked for a mobile phone operator many years ago. I was there when the company was bootstrapped. We built our own billing system because we would be billing for stuff that incumbents didn't bill for, because they didn't offer those services: we were the first 3G operator of the country and we also had to create and sell contents (example: YouTube was not born yet.) I was not in the billing team but we definitely followed rule #4 of the post: "You must be prepared to staff an entire team". And yes, #1 "Pricing changes all the time and billing needs to follow". About #2 "Your billing system needs to scale with your user base", we had to go from 0 to 3 million customers in 9 months to be viable. We were not typical, we made it. No idea about #3 "Grandfathering causes headaches" but there were probably many headaches, that one and others.

fmajid 3y ago

I've worked on telco billing systems in the past and the generally accepted metric in the industry is 70% of billing systems projects fail. Not fail as in "did not completely meet requirements or expectations", fail as in "did not deliver anything whatsoever".

PeterStuer 3y ago
That is the standard for any type of large enterprise software project though.

oblib 3y ago

I've been working on an simple invoicing app for small businesses for over 20 years. It has become a somewhat complex project and my app does not do much of what's cited in this article. >> Now, when someone asks for advice about their billing system, my answer is clear: DO NOT build it yourself. Moving from using CGI.pm to CouchDB/PouchDB about 10 years ago made it much easier to manage users and their data, and implement the UI. It also moved most of the workload to the user's web browser and made the app much faster for users. CouchDB was practically designed for this. Early on their developers used "invoicing" as an example for using CouchDB. But... if you have a marketing team that keeps moving the goals it wouldn't matter what you built the app with, it's going to take time to build and debug it. Could be that CouchDB/PouchDB could make that easier, but my experience is developers who've only been using SQL may get frustrated with it.

jwsteigerwalt 3y ago

The number backend engineers with competency in recurring revenue accounting is too small. Strange in our world that is more and more subscription driven.

PeterStuer 3y ago

I was waiting for them to discover the rabbit hole that is 'day counters'. (I wrote a lease payments calculator for the financial services industry. Many gotcha's lurk there.)

eternityforest 3y ago

I'm amazed anyone would ever want to DIY something that touches money. People tend to get pissy about money. Almost as much as water plant scada systems and railway safety. Maybe moreso. And if you're dealing with money, you're already inherently doing cloud stuff, because you're talking to the payment services, plus you already by definition have money to pay for a real billing system... It seems as crazy as people on r/wallstreetbets advising their parents on how to put their retirement money into hand picked stocks or something.

Pesky_Bees 3y ago

That's why you hire Product and UX designers so the engineers don't run wild into endless attempts to find an initial solution through the rabbit's hole.