9 ms·

Engineers' billing nightmares

204 points by AnhTho_FR 4y ago

ohmahjong 4y ago
That was very informative - it certainly does not give me the motivation to build my own billing system, but it does shed light on the scope of the task. Side note: Is "I don't want to write/use a billing system" a compelling reason to release software for free?
- czechdeveloper 4y ago
  It sounds compelling, but it's very difficult to convince free users they need to pay now if your software gets popular and you actually want to monetize it.
  - AnhTho_FR 4y ago
    Lago is currently free to use and we already have paying users. I think indeed some users will use us for free and never pay, whereas larger companies actually want to pay to get the hosted product, custom features, SLA, etc. We are still in the early days, but here is how we think about our own pricing : https://www.getlago.com/blog/how-we-think-about-our-own-pricing https://www.getlago.com/blog/how-we-think-about-our-own-pric...
revskill 4y ago
The project should include a high level Domain Model diagram to see how things are connected.
- AnhTho_FR 4y ago
  Thanks for the feedback, fantastic idea, will work on this indeed!
lifeisstillgood 4y ago
I would be very interested in a call-out guide - say I am going to handover billing to this SaaS API, Instill need to know what events I should identify and call out - it seems to me this is 50%+ of the engineering challenge anyway and I have to do it even if I am not doing in house billing Simple things like your user just did X, in this country and at this time. This has a tax implication
- AnhTho_FR 4y ago
  This is a great idea, we’ll work on it!
jiggawatts 4y ago
Some of this complexity is self-imposed. It's flailing around and trying many different things on a slow meandering path to success. Often the trick is to simply copy what a bigger org has done after a decade of experience. As a random example, Azure generally bills for resources on an hourly basis. Suddenly, months, years, calendars, etc... cease to matter. Hours are hours. You can use UTC for everything, ignore time zones and daylight savings, and aggregate however you like, or how the customer likes. Similarly, consumption-based plans are relatively straightforward. Simply multiply unit costs with the per-unit costs. You can even blend this into the hourly model above, where each hour has a "units consumed this hour" and a "unit price per hour". This way you can handle things like changes to the unit price fairly accurately. With cut-overs between plans, the trick is to just bill for them as separate line items instead of one thing going through transformations. E.g.: 220 hours of Standard SKU and 500 hours of Premium SKU this month. It doesn't matter how many times the customer changed their mind this month that way. A nice trick is that data like the above superficially sounds like it'll be huge in a database, but column-based compression works wonders on it. Any analytics database engine, columnar format, or reporting engine should reduce it by at least 30:1 or more. Also, you now have a ready-made reporting system that allows slicing and dicing! In some database engines like SQL Server, this is as simple as making the main billing table use a Clustered ColumnStore index. In more cloud-native land you could use Parquet files in S3 buckets, or whatever. I'm not saying that complexity doesn't remain, it's just that complexity shouldn't be voluntary. Copy a working model that matches your industry, and then only the essential complexity will remain. An example of a hard-to-solve essential complexity is currency conversions and taxation when dealing with international customers. That's an externally imposed set of problems that can't be hand-waved away.
- herodoturtle 4y ago
  Good grief this was informative! Thank you for taking the time to share that.
  - AnhTho_FR 4y ago
    Of course! Thank you for the kind words!
e63f67dd-065b 4y ago
Billing is complicated because getting money in exchange for services is complicated. The company wants to increase pricing complexity, as to better reflect their own costs and to better price discriminate, and this runs directly into the problem of trying to express those constraints in the real world. As noted, trying to fit billing into our stupid calendar system is tricky, because the calendar is stupid. Then, if you want plan upgrades/downgrades, you need to do pro-rata adjustments, and the interactions between that and tax/billing periods/etc makes the entire thing a headache. And then you often need to add some way for the sales team to change things on an ad-hoc basis for sales/special negotiations/etc because of course big customers are going to negotiate something special, and that's all custom logic. After that, now the marketing wants referral credits, and then they want promotional periods that interact with customer loyalty, and on and on it goes.
TomWhitwell 4y ago
This is great, thank you. Makes me think of all those people saying ‘you could run T_____ with 8 people’
hacsky 4y ago
Finally, im glad I’m not alone. When we were building SuperCMMS, we already had a few companies ready to pay us. So we needed a billing system from the get go. While we use Stripe, we still had to wrangle a lot of stuff (like taxes), since we wanted to integrating other payment gateways in future. To save time we decided to have “only a single plan - all features included”. Now comes the unexpected part. People kept asking us why we have only a single plan. Maybe all of us are so used to multiple pricing plans on saas products. So we had to explain ourselves in the pricing FAQ’s. https://supercmms.com/pricing https://supercmms.com/pricing
- ThePowerOfFuet 4y ago
  >USD 50/- Per User Per Month Enterprise Support (Optional) Starts at USD 1000/- Month Are you Indian, perchance? How you write prices in India doesn't work in other places; this looks strange to someone who works in nearly any other currency. Any of the following would look a lot less out of place: $50 50 USD $50 USD 50.00 USD $50.00 Of these, I recommend the first one; dollars default to USD worldwide unless specified otherwise.
  - hacsky 4y ago
    I think currency notation is more of a regional thing. Had we just mentioned "$", our Singapore customers might think its SGD (Singapore Dollars). We'd be better off using "USD $50.00". Thanks for the perspective.
  - deleted 4y ago
    [deleted]
  - nottorp 4y ago
    It was clear to me. Are you in the US perchance? :)
ArcMex 4y ago
Thanks for sharing. This had made me really think about how I can tackle billing in future.
- AnhTho_FR 4y ago
  Awesome, keep us posted!
davedx 4y ago
Eh, I've worked on billing, payments and salary systems, some of these problems are not that awful. It sounds like the source of much of it is just classic underestimation of the amount of work & complexity to implement something. If someone gets away with saying "shouldn't be hard right?" to the point it dictates the size of the team (2 engineers? That's barely a team), then you're going to have problems. Some of it comes down to picking the right tool for the job. The article doesn't touch on technology choice at all, but it matters. When I built a salary system, I deliberately chose a tech stack that had good numeric calculation support (so not JavaScript). These choices matter and have lasting impacts on your project. Same thing for handling dates, for example here .NET and Java have excellent standard library support for everything date related. Other languages, not so much. Another point the article talks about is change: companies change how they bill over time; customers change their plans. Again there's nothing specific to billing systems here. All systems have to deal with change. All systems beyond a trivial size end up with migration scripts, and sometimes those scripts end up being productized and becoming buttons in dashboards. I guess my main point here is don't underestimate what might seem to be an "ancillary" product when it's an essential service that makes sure cash keeps flowing. Treat every important project with the care and respect necessary. Allocate resources properly. Don't underestimate. Oh, and don't underestimate.
- jiggawatts 4y ago
  My boss told me to estimate the effort it would take to make an app that integrates with a bunch of server systems. He estimated 'x' units of time, thinking only of the core functionality of what the app actually does. I told him it's more like 10x, because the core functionality is only a small part of a product. Billing, testing, the dev environment, customer-facing documentation, the product web site, etc... all add up.
  - texasbigdata 4y ago
    What did it end up being? I’m guessing closer to your order of magnitude? Excluding scope creep, of course.
    - 4y ago
doctor_eval 4y ago
I'm a billing engineer with more than 25 years experience, and I used to run a specialist billing company. I think that this article is good in that it provides a significant amount of detail about many of the complex problems of billing. I don't agree that everything in the article is a big deal, but it certainly calls out a bunch of the things you have to think about, and it's a long, long list. And it's also just scratching the surface - some of the questions lead to many more questions, and significant complexity. So, if you don't architect your billing platform in a way that properly separates the high level moving parts, you'll eventually end up with a big pile of spaghetti. Which is bad form for something that's so critically important. That's not to say that you have to implement every single possible feature, or even that separating things is particularly hard. It's possible to build a simple billing system with just a few SQL tables - but they are probably not the exact tables that you are thinking of. But it does means that your billing platform needs to make as few assumptions as possible, and be structured in a way that leaves room for change in the future. That's because billing is critical to a business' success, and it therefore requires lots of flexibility. One way to find product/market fit is to do your billing in a way that better resonates with your customers. I mean ultimately that's what I think AWS did to colocation. So if you can't keep up with the demands of your sales team or your customers, you're going to be in a bit of trouble. As engineers, we don't get to set the rules for billing, because that's not how customers work. The billing system needs to support the requirements of the business; not dictate to them how they need to do things. Anyway I wanted to add a couple more things to TFA, just to illustrate that there is even more to it than they've listed. Policy management - what do you do when a user spends too much before the invoice is raised? Do you send an email / sms / restrict the service? how do you avoid doing this multiple times in a month? how do you integrate the billing policies with other policy systems such as usage policies? This can get really deep... Invoice rendering - very few people seem to understand that invoicing is the one time every month that you have the legal right to send unsolicited emails to customers. But in my experience, invoices almost always look like crap. What a waste! There is heaps more to this: how do you quickly fix billing problems, such as an incorrect plan? how do you backdate changes? what's your pro-rata policy on fixed charges? what do you need to integrate with the finance platform? ... the list goes on. So please forgive my shameless plug - I'm available for consulting on billing systems any time. I really know what I'm doing. See my profile for contact details.
Gordonjcp 4y ago
I worked on billing for a small UK telco about 18 years ago, and mostly it came down to reasonable-sounding business logic actually being a complete shitemare to make sense of. We got in Call Data Records (CDRs) every day from BT, which we then had to calculate billing for based on our business rules, which had to be somewhat aligned with BT's pricing so we'd actually not lose our shirts. It was pretty straightforward with one price between 8am and 1pm, then a cheaper one between 1pm and 6pm, then calls were free from 6pm until 10pm as long as they were less than one hour then billed at an even lower rate, with an hour of free any time minutes that reset at midnight. So, if you never made more than one hour total of calls between 8am and 6pm, and kept your calls to less than one hour the rest of the time, the actual call were free, right? You would genuinely be surprised how many times the billing system they were using would toss its cookies because someone made a phone call at 1740 (billable), continued right through 1800 (not billable) to 1900 (billable after 1 hour but there's still 40 minutes on "anytime" left so logged but free), right past 1940 (billable again) and on past midnight (free for another hour until anytime runs out). No, you can't just kick someone's call off after an hour.
- doctor_eval 4y ago
  Yeah telco billing can be crazy hard, it took me a long time to come up with a reliable model for it. > No, you can't just kick someone's call off after an hour. Agreed... but I worked with a telco that did exactly this!
  - ansgri 4y ago
    I too remember in some 00s that calls would disconnect at exactly 1 hour mark.
noduerme 4y ago
>> These systems were no fun They are indeed no fun. And payment APIs make me want to shoot myself. Whoever writes them seems to enjoy making end developers bleed from the eyeballs. But I've written my own in-house billing software three times over since 2002 when I had to write my own front-end API to a raw CC gateway and guess what the outputs meant ... it's much simpler now to deal with asynchronous payments and it's not the end of the world to manage them. The much, much bigger problem is getting your clients to pay on time. Knowing who usually pays and who's okay being late (versus who is habitually delinquent) and keeping tabs on that before you bill them again so you don't have to suddenly confront them about canceling their services. The smaller the business, the more significant a portion of time this can take up. When I started working for an ad agency in 1995, the owner had me stand outside a guy's factory next to her with a picket signs because he owed her $5000 for an ad that ran but was a month overdue. I have written backdoors into most large pieces of software I've built, in case of nonpayment; but never had to use them. I've pulled the rug out from hosting and left people's websites saying "Joe X doesn't pay his bills" after being treated badly by clients who were 6 months late paying. These things all give me a certain sense of glee but never get me paid. The best strategy I've found is calling your clients on a rotating basis and checking in to see what they need next and gently reminding them that their bill is due. I try to make this round about once a month.
- doctor_eval 4y ago
  Yeah - collections is a whole problem on its own. But calling customers on the regular doesn't scale too well.
inopinatus 4y ago
Whatever your billing requirements may be, use double-entry bookkeeping as the core data structure. Every time I didn’t, I’ve wished I had.
- beesorflies 4y ago
  Interesting. You mean you use double-entry bookkeeping instead of a database? or mixed with it?
  - inopinatus 4y ago
    Something has to be the store of data. The double-entry model is a paradigm, not a serialisation, of accounting and financial data. I’ve written double-entry code using text files, key-value stores, and relational databases as the underlying storage. They all work, at varying levels of scale and manageability.
doctor_eval 4y ago
In the spirit of DDD I'd like to offer a few domain boundaries and vocabulary terms that my team and I used when building our (multiple) billing platforms. "chargeable items" are things we want to charge for. In my experience there are exactly three kinds of chargeable item, which operate largely in their own domain: - "usage charges" are charges based on usage of some resource (eg, bytes, time or phone calls). - "recurring charges" are charges that are billed on a regular basis, e.g. a monthly fee. Recurring charges get complex due to the need for pro-rata, e.g. if your customer is on a monthly plan but signs up in the middle of the month, or changes plan in the middle of a month. (you can get away without this, but that just moves the complexity somewhere else - and computing refunds can become a huge pain at scale, especially if you want to build a system that is 100% self-service). - "one-time charges" are charges that are raised due to actions taken during the billing cycle; you might also call these "event charges". Things like connection/disconnection fees etc that are generated by a process or external system. These are the simplest, just an SKU and a value, maybe a qty and rate if you need it. Arguably, the most challenging domain for billing in telco circles is "rating", which is the process of computing the usage charges for individual charge detail records (CDRs) and aggregating them so that your database doesn't get overwhelmed. You might get a billion CDRs in a month. Although not all billing applications require rating, anything that does usage billing is doing some kind of rating. In Telco we also talk about "mediation" and "guidance" but that's another story... What's important about the above is the knowledge that there are different charging domains which need to be handled differently. There's a huge world of difference between a recurring charge and a usage charge. "billing" is the process of collecting these chargeable items together, to prepare them for sending to a customer. The resulting database rows might include the raw pricing along with tax, discounts etc. The important thing here is that we have a means of separating what we want to bill, from what we actually send to the customer, because the timing of an invoice is different from the timing of the charges. "invoicing" is the collection of the outstanding chargeable items into a final invoice, which can actually be sent to the customer. Interestingly, once you've issued an invoice to a customer, modifying the amount owed is probably illegal in most countries, and most certainly a bad idea. "payments" is the process of receiving money from credit cards. "collections" is the process of asking people to pay (It's also called "dunning", after the famous debt collectors, Dunn and Bradstreet). It arguably also includes policy actions like restricting access for non-payment, raising late fees, etc. Another can of worms... Anyway ... you can see a kind of billing pipeline here, where you have an ingress of chargeable events, which get rated, aggregated, held for invoicing, and then an invoice is raised, payment is requested, and non-payment is processed. This is just based on my experience, the industry seems to vary somewhat on the specifics. And ... it seems easy enough when I write it like this, but there are many devils in the details. Nevertheless, I hope this helps someone.
nier 4y ago
Section #6 seems to have a paragraph or image missing. The first paragraph ends with „your tax decision tree should look like this:“ and the second paragraph starts with „Now, imagine that you sell different types of goods/services to.“
- AnhTho_FR 4y ago
  Thanks! Let us fix that!
  - AnhTho_FR 4y ago
    Fixed, added other images, I posted that quite late last night!
ThePhysicist 4y ago
One should keep in mind that this is an article from a startup that wants to sell you billing software, so take some of the points with a grain of salt. Definitely some truths in there but having built inhouse billing solutions I can say that some of the issues pictured here are quite exaggerated. If you integrate with a payment provider like Stripe they e.g. already provide some support for tax codes, and even if not the rules aren't rocket science, every online store that sells internationally has faced the same issue. In the end, if you let a third-party provider run your billing you'll be dependent on their solution, and if something is missing that they don't care to provide you're just out of luck, or back to building it yourself, which is why these third-party solutions aren't terribly popular. It's very difficult to build a standardized solution that just works, so relying on outside software can make your problems worse.
- ivanmontillam 4y ago
  Sure, until you find out Stripe is killing accounts wrongfully[0], something that has been happening since a long while. Even the first comment there hints it[1]. > It's very difficult to build a standardized solution that just works, so relying on outside software can make your problems worse. I want to see you doing billing without software. Also, Stripe is not available for every country under the Sun because it's U.S. based. Not that Lago isn't, but at least Lago is open source so you just use it locally. -- [0]: https://news.ycombinator.com/item?id=33743750 https://news.ycombinator.com/item?id=33743750 [1]: https://news.ycombinator.com/item?id=33744053 https://news.ycombinator.com/item?id=33744053
  - ThePhysicist 4y ago
    Our payment/billing backend is built so that it can support multiple payment providers, as some larger customers e.g. pay by bank transfer. Stripe is just there to store sensitive billing data like credit card numbers and actually charge customers, everything else happens in our backend. Stripe supports that use case well, they e.g. provide web hooks and an event log which you can replay locally to integrate with them, which makes it easy to run your own billing/invoicing logic on top.
- 4y ago
hardware2win 4y ago
Thats why SAP is worth as much as it is If there are invoices involved then it is going to be fun :)
gizmo 4y ago
Honestly, billing is really not that hard. I've built some billing systems myself, and although it's easy to make mistakes if you read up on the known pitfalls rolling your own billing system is not a big deal. One key thing to remember is that if you want your billing to make sense to your customer you can't get too clever about it. A few rules of thumb: when you have to round anything, round in favor of the customer. Split the logic that calculates what should happen and the billing part of the billing system. For example, if you store a next_billing_date for each customer you can trivially adjust the date as needed, but if you have a crobjob that calculates which customers should get billed and then immediately bills them manual overrides are harder. Giving a user what amounts to a few free days of service when they upgrade/downgrade is no big deal. Especially because you should see more upgrades than downgrades in any given month. In some cases the easiest way to credit people is to move their next_billing_date forward. For usage based pricing (e.g. number of active users in a month) do the easy and fair thing where you take the minimum of active users during the month (store num_active_users each night for each customer, and use that for billing and for user facing billing dashboard). Otherwise you get edge cases where you overcharge customers when they add and remove many users because then the max of unique user ids is >> the number of active users at any given time. The example query in the blogpost gets this wrong. The SQL query that aggregates the number of API calls for a user is wrong for the same reason because it confuses API calls with billable API calls. You should have a cron that counts the API calls for each user and stores them separately in a billing table. Then when a good customer has a junior programmer make a ton of API calls by mistake you can easily go into the billing table for that customer and manually adjust the usage, with total confidence that future invoices will be correct. If you compute the usage at the time of billing then everything gets needlessly complicated.
miiiiiike 4y ago
This is an ad that gets posted around a lot of places. > To solve this problem at scale, we've adopted a radical approach: we're building an open-source billing API for product-led SaaS. Our API and architecture are open, so that you can embed, fork and customize them as much as your pricing and internal processes require.
choeger 4y ago
Do these guys really say "We ran a neobank and figured out that billing is surprisingly complex."? Like, seriously? That's what you think was complex in your domain. If I had an account at that bank, I'd seriously consider closing it ASAP, because that smells like no one had an appreciation of the complexities of finance.
jerf 4y ago
I can add a few more from my own experience: Billing really is complicated, but the people who set the policies don't understand that. They want to be able to say "Hey, let's give people a 10% discount if they sign up for a yearly bill, but also don't have this service over here, and for governments and students let's cut that down by another 15%" and so on, basically "billing by English spec", and because they think this is trivial and easy it gets handed as a spec written in stone to the billing team. Even in engineering-type organizations, treating this as an engineering problem is often a foreign concept to the org, so the idea of having the eng team estimate effort and then go back to the original requestor and make sure they really want it 1000 engineer-hour's worth, and that the rest of the org is willing to dedicate that much time to it, is generally not done. (Or whether or not there's an alternative thing that could be done at 1/10th the price that achieves the original goals, e.g. "we don't have annual discounts built into the system but we have this other fairly-close concept already built in, can we just slightly tweak the ask to use that, rather than having to implement a whole new complicated concept?") Even a company that has learned the hard way that "Let's just change the button to blue and move it down an inch" is more complicated than it looks thinks that management can just write whatever billing policy they want and there are no costs associated with it. On a more pure-engineering front, engineers have the tendency to want to implement billing discounts and such by lying to the usage database. "Ah, we will implement this 10% off policy by telling our record-keeping system that they used 10% less stuff." I've banged on how important it is to never lie to your database a few times, and the billing database is what really brought this home to me. This way lies madness, as you start to stack lies on lies on other lies in the database and then counter the lies with other heuristics and then counter the heuristics with other lies and before you know it your database contains no truth anymore, which is a problem because neither does anything else. The database was supposed to be the source of truth.
- fatnoah 4y ago
  > Billing really is complicated, but the people who set the policies don't understand that. The very first startup I joined in the late '90s tackled this problem. The company tackled billing and all of its complexities by building a SaaS product before SaaS was a widely used term (even the company didn't call it SaaS for a few years). The level of complexity was very high. Creating rules, calculating charges, dealing with taxes, ensuring things were auditable, and making it all performant were all huge challenges. Basic billing of hours x rate is easy, but (as you've noted) it's a rare person that is content with that. I cringe every time someone says "We'll build it ourselves" when it comes to billing. Even for simple things like the 10% discount (new customers only, plans > $9.99/month, discount expires after 3 months) get complicated very quickly.
savrajsingh 4y ago
So this is why we use Stripe, right?
fmajid 4y ago
I worked on telco OSS and billing systems at the beginning of my career. The industry benchmark is that 2/3 of all billing systems fail. Fail as in fail completely to deliver, not as in fail to deliver a 100% satisfactory result. A big part of this is legacy and marketers inability to discipline themselves and KISS, but legal and accounting rules also play their part. Never assume billing is going to be easy.
est 4y ago
Once a wise senior dev told me, never underestimate the complexity of an e-commerce shopping cart.

204 points by AnhTho_FR 4y ago

That was very informative - it certainly does not give me the motivation to build my own billing system, but it does shed light on the scope of the task. Side note: Is "I don't want to write/use a billing system" a compelling reason to release software for free?

czechdeveloper 4y ago
It sounds compelling, but it's very difficult to convince free users they need to pay now if your software gets popular and you actually want to monetize it.
- AnhTho_FR 4y ago
  Lago is currently free to use and we already have paying users. I think indeed some users will use us for free and never pay, whereas larger companies actually want to pay to get the hosted product, custom features, SLA, etc. We are still in the early days, but here is how we think about our own pricing : https://www.getlago.com/blog/how-we-think-about-our-own-pricing https://www.getlago.com/blog/how-we-think-about-our-own-pric...

revskill 4y ago

The project should include a high level Domain Model diagram to see how things are connected.

AnhTho_FR 4y ago
Thanks for the feedback, fantastic idea, will work on this indeed!

lifeisstillgood 4y ago

I would be very interested in a call-out guide - say I am going to handover billing to this SaaS API, Instill need to know what events I should identify and call out - it seems to me this is 50%+ of the engineering challenge anyway and I have to do it even if I am not doing in house billing Simple things like your user just did X, in this country and at this time. This has a tax implication

AnhTho_FR 4y ago
This is a great idea, we’ll work on it!

jiggawatts 4y ago

Some of this complexity is self-imposed. It's flailing around and trying many different things on a slow meandering path to success. Often the trick is to simply copy what a bigger org has done after a decade of experience. As a random example, Azure generally bills for resources on an hourly basis. Suddenly, months, years, calendars, etc... cease to matter. Hours are hours. You can use UTC for everything, ignore time zones and daylight savings, and aggregate however you like, or how the customer likes. Similarly, consumption-based plans are relatively straightforward. Simply multiply unit costs with the per-unit costs. You can even blend this into the hourly model above, where each hour has a "units consumed this hour" and a "unit price per hour". This way you can handle things like changes to the unit price fairly accurately. With cut-overs between plans, the trick is to just bill for them as separate line items instead of one thing going through transformations. E.g.: 220 hours of Standard SKU and 500 hours of Premium SKU this month. It doesn't matter how many times the customer changed their mind this month that way. A nice trick is that data like the above superficially sounds like it'll be huge in a database, but column-based compression works wonders on it. Any analytics database engine, columnar format, or reporting engine should reduce it by at least 30:1 or more. Also, you now have a ready-made reporting system that allows slicing and dicing! In some database engines like SQL Server, this is as simple as making the main billing table use a Clustered ColumnStore index. In more cloud-native land you could use Parquet files in S3 buckets, or whatever. I'm not saying that complexity doesn't remain, it's just that complexity shouldn't be voluntary. Copy a working model that matches your industry, and then only the essential complexity will remain. An example of a hard-to-solve essential complexity is currency conversions and taxation when dealing with international customers. That's an externally imposed set of problems that can't be hand-waved away.

herodoturtle 4y ago
Good grief this was informative! Thank you for taking the time to share that.
- AnhTho_FR 4y ago
  Of course! Thank you for the kind words!

e63f67dd-065b 4y ago

Billing is complicated because getting money in exchange for services is complicated. The company wants to increase pricing complexity, as to better reflect their own costs and to better price discriminate, and this runs directly into the problem of trying to express those constraints in the real world. As noted, trying to fit billing into our stupid calendar system is tricky, because the calendar is stupid. Then, if you want plan upgrades/downgrades, you need to do pro-rata adjustments, and the interactions between that and tax/billing periods/etc makes the entire thing a headache. And then you often need to add some way for the sales team to change things on an ad-hoc basis for sales/special negotiations/etc because of course big customers are going to negotiate something special, and that's all custom logic. After that, now the marketing wants referral credits, and then they want promotional periods that interact with customer loyalty, and on and on it goes.

TomWhitwell 4y ago

This is great, thank you. Makes me think of all those people saying ‘you could run T_____ with 8 people’

hacsky 4y ago

Finally, im glad I’m not alone. When we were building SuperCMMS, we already had a few companies ready to pay us. So we needed a billing system from the get go. While we use Stripe, we still had to wrangle a lot of stuff (like taxes), since we wanted to integrating other payment gateways in future. To save time we decided to have “only a single plan - all features included”. Now comes the unexpected part. People kept asking us why we have only a single plan. Maybe all of us are so used to multiple pricing plans on saas products. So we had to explain ourselves in the pricing FAQ’s. https://supercmms.com/pricing https://supercmms.com/pricing

ThePowerOfFuet 4y ago
>USD 50/- Per User Per Month Enterprise Support (Optional) Starts at USD 1000/- Month Are you Indian, perchance? How you write prices in India doesn't work in other places; this looks strange to someone who works in nearly any other currency. Any of the following would look a lot less out of place: $50 50 USD $50 USD 50.00 USD $50.00 Of these, I recommend the first one; dollars default to USD worldwide unless specified otherwise.
- hacsky 4y ago
  I think currency notation is more of a regional thing. Had we just mentioned "$", our Singapore customers might think its SGD (Singapore Dollars). We'd be better off using "USD $50.00". Thanks for the perspective.
- deleted 4y ago
  [deleted]
- nottorp 4y ago
  It was clear to me. Are you in the US perchance? :)

ArcMex 4y ago

Thanks for sharing. This had made me really think about how I can tackle billing in future.

AnhTho_FR 4y ago
Awesome, keep us posted!

davedx 4y ago

Eh, I've worked on billing, payments and salary systems, some of these problems are not that awful. It sounds like the source of much of it is just classic underestimation of the amount of work & complexity to implement something. If someone gets away with saying "shouldn't be hard right?" to the point it dictates the size of the team (2 engineers? That's barely a team), then you're going to have problems. Some of it comes down to picking the right tool for the job. The article doesn't touch on technology choice at all, but it matters. When I built a salary system, I deliberately chose a tech stack that had good numeric calculation support (so not JavaScript). These choices matter and have lasting impacts on your project. Same thing for handling dates, for example here .NET and Java have excellent standard library support for everything date related. Other languages, not so much. Another point the article talks about is change: companies change how they bill over time; customers change their plans. Again there's nothing specific to billing systems here. All systems have to deal with change. All systems beyond a trivial size end up with migration scripts, and sometimes those scripts end up being productized and becoming buttons in dashboards. I guess my main point here is don't underestimate what might seem to be an "ancillary" product when it's an essential service that makes sure cash keeps flowing. Treat every important project with the care and respect necessary. Allocate resources properly. Don't underestimate. Oh, and don't underestimate.

jiggawatts 4y ago
My boss told me to estimate the effort it would take to make an app that integrates with a bunch of server systems. He estimated 'x' units of time, thinking only of the core functionality of what the app actually does. I told him it's more like 10x, because the core functionality is only a small part of a product. Billing, testing, the dev environment, customer-facing documentation, the product web site, etc... all add up.
- texasbigdata 4y ago
  What did it end up being? I’m guessing closer to your order of magnitude? Excluding scope creep, of course.
  - 4y ago

doctor_eval 4y ago

I'm a billing engineer with more than 25 years experience, and I used to run a specialist billing company. I think that this article is good in that it provides a significant amount of detail about many of the complex problems of billing. I don't agree that everything in the article is a big deal, but it certainly calls out a bunch of the things you have to think about, and it's a long, long list. And it's also just scratching the surface - some of the questions lead to many more questions, and significant complexity. So, if you don't architect your billing platform in a way that properly separates the high level moving parts, you'll eventually end up with a big pile of spaghetti. Which is bad form for something that's so critically important. That's not to say that you have to implement every single possible feature, or even that separating things is particularly hard. It's possible to build a simple billing system with just a few SQL tables - but they are probably not the exact tables that you are thinking of. But it does means that your billing platform needs to make as few assumptions as possible, and be structured in a way that leaves room for change in the future. That's because billing is critical to a business' success, and it therefore requires lots of flexibility. One way to find product/market fit is to do your billing in a way that better resonates with your customers. I mean ultimately that's what I think AWS did to colocation. So if you can't keep up with the demands of your sales team or your customers, you're going to be in a bit of trouble. As engineers, we don't get to set the rules for billing, because that's not how customers work. The billing system needs to support the requirements of the business; not dictate to them how they need to do things. Anyway I wanted to add a couple more things to TFA, just to illustrate that there is even more to it than they've listed. Policy management - what do you do when a user spends too much before the invoice is raised? Do you send an email / sms / restrict the service? how do you avoid doing this multiple times in a month? how do you integrate the billing policies with other policy systems such as usage policies? This can get really deep... Invoice rendering - very few people seem to understand that invoicing is the one time every month that you have the legal right to send unsolicited emails to customers. But in my experience, invoices almost always look like crap. What a waste! There is heaps more to this: how do you quickly fix billing problems, such as an incorrect plan? how do you backdate changes? what's your pro-rata policy on fixed charges? what do you need to integrate with the finance platform? ... the list goes on. So please forgive my shameless plug - I'm available for consulting on billing systems any time. I really know what I'm doing. See my profile for contact details.

Gordonjcp 4y ago

I worked on billing for a small UK telco about 18 years ago, and mostly it came down to reasonable-sounding business logic actually being a complete shitemare to make sense of. We got in Call Data Records (CDRs) every day from BT, which we then had to calculate billing for based on our business rules, which had to be somewhat aligned with BT's pricing so we'd actually not lose our shirts. It was pretty straightforward with one price between 8am and 1pm, then a cheaper one between 1pm and 6pm, then calls were free from 6pm until 10pm as long as they were less than one hour then billed at an even lower rate, with an hour of free any time minutes that reset at midnight. So, if you never made more than one hour total of calls between 8am and 6pm, and kept your calls to less than one hour the rest of the time, the actual call were free, right? You would genuinely be surprised how many times the billing system they were using would toss its cookies because someone made a phone call at 1740 (billable), continued right through 1800 (not billable) to 1900 (billable after 1 hour but there's still 40 minutes on "anytime" left so logged but free), right past 1940 (billable again) and on past midnight (free for another hour until anytime runs out). No, you can't just kick someone's call off after an hour.

doctor_eval 4y ago
Yeah telco billing can be crazy hard, it took me a long time to come up with a reliable model for it. > No, you can't just kick someone's call off after an hour. Agreed... but I worked with a telco that did exactly this!
- ansgri 4y ago
  I too remember in some 00s that calls would disconnect at exactly 1 hour mark.

noduerme 4y ago

>> These systems were no fun They are indeed no fun. And payment APIs make me want to shoot myself. Whoever writes them seems to enjoy making end developers bleed from the eyeballs. But I've written my own in-house billing software three times over since 2002 when I had to write my own front-end API to a raw CC gateway and guess what the outputs meant ... it's much simpler now to deal with asynchronous payments and it's not the end of the world to manage them. The much, much bigger problem is getting your clients to pay on time. Knowing who usually pays and who's okay being late (versus who is habitually delinquent) and keeping tabs on that before you bill them again so you don't have to suddenly confront them about canceling their services. The smaller the business, the more significant a portion of time this can take up. When I started working for an ad agency in 1995, the owner had me stand outside a guy's factory next to her with a picket signs because he owed her $5000 for an ad that ran but was a month overdue. I have written backdoors into most large pieces of software I've built, in case of nonpayment; but never had to use them. I've pulled the rug out from hosting and left people's websites saying "Joe X doesn't pay his bills" after being treated badly by clients who were 6 months late paying. These things all give me a certain sense of glee but never get me paid. The best strategy I've found is calling your clients on a rotating basis and checking in to see what they need next and gently reminding them that their bill is due. I try to make this round about once a month.

doctor_eval 4y ago
Yeah - collections is a whole problem on its own. But calling customers on the regular doesn't scale too well.

inopinatus 4y ago

Whatever your billing requirements may be, use double-entry bookkeeping as the core data structure. Every time I didn’t, I’ve wished I had.

beesorflies 4y ago
Interesting. You mean you use double-entry bookkeeping instead of a database? or mixed with it?
- inopinatus 4y ago
  Something has to be the store of data. The double-entry model is a paradigm, not a serialisation, of accounting and financial data. I’ve written double-entry code using text files, key-value stores, and relational databases as the underlying storage. They all work, at varying levels of scale and manageability.

doctor_eval 4y ago

In the spirit of DDD I'd like to offer a few domain boundaries and vocabulary terms that my team and I used when building our (multiple) billing platforms. "chargeable items" are things we want to charge for. In my experience there are exactly three kinds of chargeable item, which operate largely in their own domain: - "usage charges" are charges based on usage of some resource (eg, bytes, time or phone calls). - "recurring charges" are charges that are billed on a regular basis, e.g. a monthly fee. Recurring charges get complex due to the need for pro-rata, e.g. if your customer is on a monthly plan but signs up in the middle of the month, or changes plan in the middle of a month. (you can get away without this, but that just moves the complexity somewhere else - and computing refunds can become a huge pain at scale, especially if you want to build a system that is 100% self-service). - "one-time charges" are charges that are raised due to actions taken during the billing cycle; you might also call these "event charges". Things like connection/disconnection fees etc that are generated by a process or external system. These are the simplest, just an SKU and a value, maybe a qty and rate if you need it. Arguably, the most challenging domain for billing in telco circles is "rating", which is the process of computing the usage charges for individual charge detail records (CDRs) and aggregating them so that your database doesn't get overwhelmed. You might get a billion CDRs in a month. Although not all billing applications require rating, anything that does usage billing is doing some kind of rating. In Telco we also talk about "mediation" and "guidance" but that's another story... What's important about the above is the knowledge that there are different charging domains which need to be handled differently. There's a huge world of difference between a recurring charge and a usage charge. "billing" is the process of collecting these chargeable items together, to prepare them for sending to a customer. The resulting database rows might include the raw pricing along with tax, discounts etc. The important thing here is that we have a means of separating what we want to bill, from what we actually send to the customer, because the timing of an invoice is different from the timing of the charges. "invoicing" is the collection of the outstanding chargeable items into a final invoice, which can actually be sent to the customer. Interestingly, once you've issued an invoice to a customer, modifying the amount owed is probably illegal in most countries, and most certainly a bad idea. "payments" is the process of receiving money from credit cards. "collections" is the process of asking people to pay (It's also called "dunning", after the famous debt collectors, Dunn and Bradstreet). It arguably also includes policy actions like restricting access for non-payment, raising late fees, etc. Another can of worms... Anyway ... you can see a kind of billing pipeline here, where you have an ingress of chargeable events, which get rated, aggregated, held for invoicing, and then an invoice is raised, payment is requested, and non-payment is processed. This is just based on my experience, the industry seems to vary somewhat on the specifics. And ... it seems easy enough when I write it like this, but there are many devils in the details. Nevertheless, I hope this helps someone.

nier 4y ago

Section #6 seems to have a paragraph or image missing. The first paragraph ends with „your tax decision tree should look like this:“ and the second paragraph starts with „Now, imagine that you sell different types of goods/services to.“

AnhTho_FR 4y ago
Thanks! Let us fix that!
- AnhTho_FR 4y ago
  Fixed, added other images, I posted that quite late last night!

ThePhysicist 4y ago

One should keep in mind that this is an article from a startup that wants to sell you billing software, so take some of the points with a grain of salt. Definitely some truths in there but having built inhouse billing solutions I can say that some of the issues pictured here are quite exaggerated. If you integrate with a payment provider like Stripe they e.g. already provide some support for tax codes, and even if not the rules aren't rocket science, every online store that sells internationally has faced the same issue. In the end, if you let a third-party provider run your billing you'll be dependent on their solution, and if something is missing that they don't care to provide you're just out of luck, or back to building it yourself, which is why these third-party solutions aren't terribly popular. It's very difficult to build a standardized solution that just works, so relying on outside software can make your problems worse.

ivanmontillam 4y ago
Sure, until you find out Stripe is killing accounts wrongfully[0], something that has been happening since a long while. Even the first comment there hints it[1]. > It's very difficult to build a standardized solution that just works, so relying on outside software can make your problems worse. I want to see you doing billing without software. Also, Stripe is not available for every country under the Sun because it's U.S. based. Not that Lago isn't, but at least Lago is open source so you just use it locally. -- [0]: https://news.ycombinator.com/item?id=33743750 https://news.ycombinator.com/item?id=33743750 [1]: https://news.ycombinator.com/item?id=33744053 https://news.ycombinator.com/item?id=33744053
- ThePhysicist 4y ago
  Our payment/billing backend is built so that it can support multiple payment providers, as some larger customers e.g. pay by bank transfer. Stripe is just there to store sensitive billing data like credit card numbers and actually charge customers, everything else happens in our backend. Stripe supports that use case well, they e.g. provide web hooks and an event log which you can replay locally to integrate with them, which makes it easy to run your own billing/invoicing logic on top.
4y ago

hardware2win 4y ago

Thats why SAP is worth as much as it is If there are invoices involved then it is going to be fun :)

gizmo 4y ago

Honestly, billing is really not that hard. I've built some billing systems myself, and although it's easy to make mistakes if you read up on the known pitfalls rolling your own billing system is not a big deal. One key thing to remember is that if you want your billing to make sense to your customer you can't get too clever about it. A few rules of thumb: when you have to round anything, round in favor of the customer. Split the logic that calculates what should happen and the billing part of the billing system. For example, if you store a next_billing_date for each customer you can trivially adjust the date as needed, but if you have a crobjob that calculates which customers should get billed and then immediately bills them manual overrides are harder. Giving a user what amounts to a few free days of service when they upgrade/downgrade is no big deal. Especially because you should see more upgrades than downgrades in any given month. In some cases the easiest way to credit people is to move their next_billing_date forward. For usage based pricing (e.g. number of active users in a month) do the easy and fair thing where you take the minimum of active users during the month (store num_active_users each night for each customer, and use that for billing and for user facing billing dashboard). Otherwise you get edge cases where you overcharge customers when they add and remove many users because then the max of unique user ids is >> the number of active users at any given time. The example query in the blogpost gets this wrong. The SQL query that aggregates the number of API calls for a user is wrong for the same reason because it confuses API calls with billable API calls. You should have a cron that counts the API calls for each user and stores them separately in a billing table. Then when a good customer has a junior programmer make a ton of API calls by mistake you can easily go into the billing table for that customer and manually adjust the usage, with total confidence that future invoices will be correct. If you compute the usage at the time of billing then everything gets needlessly complicated.

miiiiiike 4y ago

This is an ad that gets posted around a lot of places. > To solve this problem at scale, we've adopted a radical approach: we're building an open-source billing API for product-led SaaS. Our API and architecture are open, so that you can embed, fork and customize them as much as your pricing and internal processes require.

choeger 4y ago

Do these guys really say "We ran a neobank and figured out that billing is surprisingly complex."? Like, seriously? That's what you think was complex in your domain. If I had an account at that bank, I'd seriously consider closing it ASAP, because that smells like no one had an appreciation of the complexities of finance.

jerf 4y ago

I can add a few more from my own experience: Billing really is complicated, but the people who set the policies don't understand that. They want to be able to say "Hey, let's give people a 10% discount if they sign up for a yearly bill, but also don't have this service over here, and for governments and students let's cut that down by another 15%" and so on, basically "billing by English spec", and because they think this is trivial and easy it gets handed as a spec written in stone to the billing team. Even in engineering-type organizations, treating this as an engineering problem is often a foreign concept to the org, so the idea of having the eng team estimate effort and then go back to the original requestor and make sure they really want it 1000 engineer-hour's worth, and that the rest of the org is willing to dedicate that much time to it, is generally not done. (Or whether or not there's an alternative thing that could be done at 1/10th the price that achieves the original goals, e.g. "we don't have annual discounts built into the system but we have this other fairly-close concept already built in, can we just slightly tweak the ask to use that, rather than having to implement a whole new complicated concept?") Even a company that has learned the hard way that "Let's just change the button to blue and move it down an inch" is more complicated than it looks thinks that management can just write whatever billing policy they want and there are no costs associated with it. On a more pure-engineering front, engineers have the tendency to want to implement billing discounts and such by lying to the usage database. "Ah, we will implement this 10% off policy by telling our record-keeping system that they used 10% less stuff." I've banged on how important it is to never lie to your database a few times, and the billing database is what really brought this home to me. This way lies madness, as you start to stack lies on lies on other lies in the database and then counter the lies with other heuristics and then counter the heuristics with other lies and before you know it your database contains no truth anymore, which is a problem because neither does anything else. The database was supposed to be the source of truth.

fatnoah 4y ago
> Billing really is complicated, but the people who set the policies don't understand that. The very first startup I joined in the late '90s tackled this problem. The company tackled billing and all of its complexities by building a SaaS product before SaaS was a widely used term (even the company didn't call it SaaS for a few years). The level of complexity was very high. Creating rules, calculating charges, dealing with taxes, ensuring things were auditable, and making it all performant were all huge challenges. Basic billing of hours x rate is easy, but (as you've noted) it's a rare person that is content with that. I cringe every time someone says "We'll build it ourselves" when it comes to billing. Even for simple things like the 10% discount (new customers only, plans > $9.99/month, discount expires after 3 months) get complicated very quickly.

savrajsingh 4y ago

So this is why we use Stripe, right?

fmajid 4y ago

I worked on telco OSS and billing systems at the beginning of my career. The industry benchmark is that 2/3 of all billing systems fail. Fail as in fail completely to deliver, not as in fail to deliver a 100% satisfactory result. A big part of this is legacy and marketers inability to discipline themselves and KISS, but legal and accounting rules also play their part. Never assume billing is going to be easy.

est 4y ago

Once a wise senior dev told me, never underestimate the complexity of an e-commerce shopping cart.