46 ms·

Billing systems are a nightmare for engineers

• deleted 4y ago
  [deleted]
• retcon 4y ago
  >/as Qonto is a 'neobank', we wanted to charge our customers directly in their wallet, through a ledger connected to our internal billing system. The team started from a duo of full-time engineers building a billing system (which is already a considerable investment), to currently a dedicated cross-functional team called 'pricing'. Stored procedures (ca. Oracle v6) suddenly sound a walk in the park.
  • AnhTho_FR 4y ago
    They do indeed!
• eesmith 4y ago
  A classic, from http://www.canonical.org/~kragen/tao-of-programming.html http://www.canonical.org/~kragen/tao-of-programming.html > There was once a programmer who was attached to the court of the warlord of Wu. The warlord asked the programmer: ``Which is easier to design: an accounting package or an operating system?'' > ``An operating system,'' replied the programmer. > The warlord uttered an exclamation of disbelief. ``Surely an accounting package is trivial next to the complexity of an operating system,'' he said. > ``Not so,'' said the programmer, ``when designing an accounting package, the programmer operates as a mediator between people having different ideas: how it must operate, how its reports must appear, and how it must conform to the tax laws. By contrast, an operating system is not limited by outside appearances. When designing an operating system, the programmer seeks the simplest harmony between machine and ideas. This is why an operating system is easier to design.'' > The warlord of Wu nodded and smiled. ``That is all good and well, but which is easier to debug?'' > The programmer made no reply.
  • AnhTho_FR 4y ago
    OMG love the story, thanks for sharing!
  • PaulHoule 4y ago
    Applications programming has a complexity of it's own that's distinct from the problems in systems programming.
  • alex_suzuki 4y ago
    Thoughtful, and made me laugh. Thanks!
  • bombcar 4y ago
    Heh, I'd say the operating system is easier to debug, because the accounting package could literally have "bugs" that are caused by errors in the tax laws, business processes, etc, which cannot be fixed, only worked around.
  • teddyh 4y ago
    The linked page does not show it, but The Tao of Programming is actually a book which you can buy: https://en.wikipedia.org/wiki/The_Tao_of_Programming https://en.wikipedia.org/wiki/The_Tao_of_Programming I own a copy; it’s great.
  • buescher 4y ago
    Which came first? I am pretty sure it was computer accounting systems. Which is older, the oldest continuously supported operating system, or the oldest continuously supported (computer) accounting system?
    • eesmith 4y ago
      The oldest computer accounting system was likely LEO - https://en.wikipedia.org/wiki/LEO_(computer) https://en.wikipedia.org/wiki/LEO_(computer), online in 1951 and doing accounts probably by 1955, if not earlier. I believe it predate operating systems by a few years. (Wikipedia claims the first OS was the GM-NAA I/O, produced in 1956.)
• spchampion2 4y ago
  Just wait until you meet billing's angry roommate: invoicing. In the US, an invoice is just a weird PDF that you might glance at before sending off to your accounts payable team. But in other countries, especially those that use VAT style taxing systems, an invoice can be a legal document that expresses costs and taxes in a legally prescribed way for accounting purposes. Many countries have prescriptive rules over how you invoice, who can write invoices, when you can invoice, what goes on an invoice, etc. And every country does it differently. Are you building a service that charges incrementally many times per period? Or even worse, refunds incrementally? You might be sending a truckload of expensive accounting papers to your customer every month, one per transaction. And each of those pages was printed using government prescribed software from oddball vendors you must use or else.
  • trollied 4y ago
    Yup, it's a nightmare. Invoices needing to be in specific number sequences. Any corrections need to be dealt with by reissuing a special "corrective invoice". Don't even get me started on geocoding for taxes. Urgh. Also, when you've got the invoicing right, you've got to work out how to do the feeds to the accounting systems so that it all posts to the right place.
    • radicalbyte 4y ago
      A credit note. How the ** do you handle it in the US without issuing credit notes in your bookkeeping?
      • indymike 4y ago
        We issue a credit note* * Except (these aren't the right thing to do, but I've seen it in the wild many times): - When someone in sales issues an invoice with a negative amount on line 3. Hey, if I take payments with invoices, why not give back? - When someone agrees to settle up by providing free services and throws it on the next bill.
        • 4y ago
• christophilus 4y ago
  This has been my life for the past few months. Billing and dates / times are the worst part of the job.
  • AnhTho_FR 4y ago
    Have you looked at existing solutions? Why did not they fit?
    • gen220 4y ago
      Not the OP, but anybody who has built a deep integration with any billing provider (inclusive of Stripe) knows there are some bodies buried, although you typically don't discover them until you're late in the game. 90% of your use cases are covered, and with excellent UX and docs to boot. But that sweet 10%! --- For a concrete example, Stripe has a very nice "subscription" abstraction that makes it easier to operate a SaaS business. They also offer a nice "tax" product, that doesn't do filing for you, but does offer tax computation if you code your products appropriately. Sounds great! But if you want to sell 2 subscriptions to the same customer, and those 2 subscriptions are fulfilled in 2 distinct taxable regimes (say, you're shipping widgets to NYC once per month and to a vacation home in NH once per quarter), you're screwed because Stripe only allows 1 shipping address for customer for tax purposes, and each subscription references this address when computing tax. We went through support, which filtered through to engineering, and the eventual answer was "no plan to change this anytime soon", which is difficult to interpret any other way than "good luck". :) There is no clean or even acceptably-dirty solution to this problem, tragically. So as a business we can't allow customers to transact with us this way. Oh well. There's another, similar, story around creating draft invoices to give people quotes before checking out. The dashboard API has a function to modify the line items of a draft invoice, but it's not a public-facing API and there's no short-term intention to make it public-facing, so one has to build an ugly workaround. This effectively means one needs to maintain internal models for subscriptions, products, taxes, invoices, invoice items, products, shipping addresses, on and on. And Stripe becomes a somewhat-dumb processor for these models (although the state machine for subscriptions is admittedly very valuable). I'd love to "offshore" everything to Stripe, status as the source of truth for data included, but it's not feasible in a surprising frequency of scenarios. Caveat that every payment provider is like this, Stripe is much better and actually improves over time.
• lelandbatey 4y ago
  Billing is a nightmare, and if I had one piece of advice for people building a pay-for-what-you-use system like most SaaS, it's this: DO NOT bill against your business logic entities. They change, and doing a COUNT at the end of the month won't catch all things which changed or which cost you money during the month. Instead, figure out what you bill for and and when you do that thing, record a row in a DB or into a steam of that "billable event". Reconciling billable events is much easier to do, and it's tolerant of all the weird edge cases (such as a support person hard deleting data when they should soft delete, which would otherwise throw off your end of month counts). There's a reason AWS (in general) can produce a log of everything you did which cost you money. It's painful, but it's less painful than the alternative.
  • bombcar 4y ago
    And depending on how you do it, it's not as hard as it sounds - for example, the first minute of "cloud server V1" can get a row in the db, and each "period" that row gets updated with the current minutes. Or you can log them into tables that get processed into other invoice tables. This also lets you keep grandfathered customers around as long as you want.
  • bob1029 4y ago
    I think event sourcing / logging is the way to solve this kind of thing. You have an obvious start/stop checkpoint in the log, processing can be done in batch after hours, no information ever gets destroyed, etc. Mutable database rows that are used for accounting purposes should sound dangerous to anyone trusted with these systems.
  • theptip 4y ago
    Sound advice. Anything to do with accounting, you’ll probably want to treat as an append-only log of events. (Note, you can also use event-sourcing and have your domain entities _be_ events, in a billing bounded context that might make sense. Not usually the first approach I’d recommend though.) On a similar note, make sure you think about bitemporality as well. In other words, “effective_at” vs. “”created_at”. You might learn that due to a bug, you didn’t record a billable event, and you need to insert it into the history (say, to put it in the correct billing period). But setting the “created_at” in the past brings a bunch of issues; it’s confusing and you’ll soon hit issues where lying about created-time trips you up. (Migrations, reasoning about what version of the code applied what events, etc). Fowler has good further reading here: https://martinfowler.com/articles/bitemporal-history.html https://martinfowler.com/articles/bitemporal-history.html.
• api 4y ago
  Any kind of business or accounting system tends to be an endless long tail of "can the invoicing system send the invoice in Esperanto using only 16-bit Unicode characters on Tuesdays but only while it is raining and only for customers whose last names end with E and who signed up more than one year ago according to the Chinese calendar?" type features. That's why attempts at "no-code" systems where biz/accounting people can design their own system are a perennial fad, but as these systems are developed they always just end up evolving into weird domain specific programming languages. It's also why any sufficiently large company ends up implementing their own in-house accounting system and/or customizing some byzantine monstrosity like SAP which is really just another way of implementing your own.
  • nikanj 4y ago
    The #1 reason those systems are moderately successful: When your sales person needs to write the logic for billing customer X, they'll try to make the customer contract more reasonable. Normally sales people are deal-driven, and if promising the customer "14% off on the first three transactions on every rainy Tuesday, unless Monday was also rainy" gets the deal closed, then that's going in the contract. No better way to ensure the contracts are reasonable, than making sure the person negotiating the deal feels the pain of billing it
    • Rafsark 4y ago
      Agreed! this triggers also complex behaviors because sales people want to handle this in their env (Salesforce, hubspot or whatever); You also have to build the logic to update subscriptions and pricing directly from it, in a two-way sync. Things getting crazier ;)
• londons_explore 4y ago
  Random idea: Don't build a billing system. Or at least not a precise one. Have a price list as normal, but just guestimate the user's bills based on any easily accessible information. Eg. For a hosting company I might just count how many VM's they have active when the billing script runs . If the user isn't happy with the total, give them a button to correct the total and pay that. Spot check what the user's pay, and warn then ban any users which are clearly deliberately paying substantially less than list price for your services.
  • AnhTho_FR 4y ago
    I love it, sounds like the Swedish shops with no cashier https://mashable.com/article/swedish-store-cashiers https://mashable.com/article/swedish-store-cashiers Has it been tried before? How were disputes managed?
  • CharlesW 4y ago
    How does auditing work if billing is non-deterministic?
  • __alexs 4y ago
    This is a great way to be bankrupted by crypto currency miners.
  • recursive 4y ago
    This sounds like trouble at several levels. Someone's going to reverse engineer your heuristic and min-max it. At that point, you'll need to defend your heuristic against edge cases. And then you may as well just make it official. Also, B2B customers are allergic to "pay what you feel"-type schemes.
  • Hackbraten 4y ago
    You’d still have to apply the right tax rate for every line item. If you allow the user to edit the total, then which of the tax amounts are you going to recalculate?
• fatnoah 4y ago
  My very first startup (wayyyy back in 2000) built a billing SaaS. I fully agree that rating, billing, invoicing, and all of the related things are hard. I learned a lot in that role about dealing with complexity...and date/time.
  • Rafsark 4y ago
    What are the top 3 things you learned that every company should be aware of? Could be interesting to have an experienced view on this :)
    • fatnoah 4y ago
      Wow, that's a tough one. Just spitballing, I'd say: 1). Don't try to build it yourself, no matter how simple your case. Your needs will ALWAYS evolve to be more complex AND keeping up with the thousands (or more) of individual regulatory regimes in the world is a massive task. 2). At any given point in time, you need to know the state of everything required to rate, bill, and invoice at any other point in time. (i.e. bitemporal). As a consequence, critical data effectively becomes immutable, auditing is possible, and allows pre/post views of things after corrections. 3). Once you start to get into tiered/volume discount, combo discounts, providing multiple services, etc. complexity rapidly increases, so observability into how exactly things are calculated is of paramount importance. This one is probably pretty obvious, but missed a lot. How many times have you received a bill with little to now information about how things were calculated? It's possibly a business decision, but just as likely the process is a black box and not readily observable.
• aerovistae 4y ago
  > To solve this problem at scale, we adopted a radical sharing approach. We’ve started building an Open-Source Alternative to Stripe Billing (and Chargebee, and all the equivalents). This is so great. I personally haven't had to deal with this problem, but I've worked at a number of organizations (and heard about many others!) where this sort of business logic had to be implemented. It's just reinventing the wheel. I shudder to think how many companies have implemented a system for managing recurring subscriptions. We have things like Ruby-on-Rails, Django, Laravel, and many others to take the bite out of building web applications. They keep us from having to reinvent the wheel. We need similar open source frameworks for common business use-cases - billing, subscriptions, order/purchase management, and so on. Sometimes it's weird to remember we're in the stone age of technology - all of this is a few decades old at most, and even early predecessors don't go back more than 70 years. Human history goes back tens of thousands of years. There's so much yet to come.
  • ptsneves 4y ago
    This is where solutions like WooCommerce shine. WooCommerce is an amazing piece of software that is many times used for free, but generates thousands of dollars.
  • intuxikated 4y ago
    > We need similar open source frameworks for common business use-cases - billing, subscriptions, order/purchase management, and so on. You mean like an ERP? There's 2 open source ones I can think of: - Odoo, which is the bigger one, mostly Open-source (CRM / sales / subscriptions / invoices / webshop / inventory / purchase) but some enterprise-only modules (Rental, Field Services), I work with this software daily as an Odoo Developer (Customizing Odoo for customers' needs). - Erp-Next, which is completely open source as far as I can tell, through my limited testing it seems to be less advanced than Odoo currently. EDIT: you can even check runbot.odoo.com for some test-environments which are automatically built, where you can test/experiment, login is always admin:admin
    • aerovistae 4y ago
      I hadn't heard of that! Maybe that is what I meant.
• hedgehog 4y ago
  Can corroborate, my experience working on an upgrade to an existing multi-platform (iOS IAP + Strip for web product) subscription service was certainly a bit painful.
  • Rafsark 4y ago
    Yes, upgrades and downgrades were painful for me too. I think it's due to different subscription dates between all our customers, having to create specific code for each edge case. In the end, it works, but so much harder than I thought
• thibo_skabgia 4y ago
  Love this comprehensive article. I’ve seen that pain in many a company I freelanced for. And it’s much deeper than they even know.
• hkon 4y ago
  Depends on your requirements I would say. If your target customers are global, then yes, you'll have to deal with a lot of rules and regulations. But the engineering side of things I don't think is that hard. Perhaps when you first enter the domain it seems like a lot because you are unfamiliar with it, but such is it with all things. Also, these guys have an economical incentive of convincing you it's hard. So keep that in mind.
• jakewins 4y ago
  Pro-rated billing is one of the many places outside of database query planners where Dynamic Programming is useful! Maybe there's a better way to do it, but well, if you give a man a hammer.. I wrote up how we did this at Equipmentshare back in 2016: https://tech.davis-hansson.com/p/price-wars.html https://tech.davis-hansson.com/p/price-wars.html
• Brystephor 4y ago
  I've worked at two companies in their payment systems. This post does a good job of mentioning many of the billing system challenges. I think it understates how difficult and important idempotency is to maintain at scale with multiple teams though. Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)? He also did not mention fraud, which is basically uncollected spend that results in charge backs typically. If you get too many chargebacks, you'll get fined. If you get even more chargebacks, you'll get kicked off the card network and will no longer be able to accept cards within that network. Post pay (customer gets the product and pays afterwards) models are subject to more fraud, uncollected charges, etc. There's also the fun of dealing with downstream payment processors. I haven't worked with a processor or bank yet that was very reliable. Sometimes they'd return 500 http codes and then process a payment, requiring manual intervention. Sometimes that happens with files of payments because most banks deal with files for transactions.
  • tablespoon 4y ago
    > Additionally, it didn't mention financial regulation changes. India had changes sometime in the last few years which required whole new systems built that were specific to India customers. One example of complexity is this: does the system apply to customers who are in India or does it apply to businesses who are in India (the owners might not be)? IIRC, I've read this is one of the reasons that systems like SAP are so popular. They might be an ugly monster from a technical perspective and seem overly complex, but they provide thoroughly tested implementations that handle all that stuff.
    • AnhTho_FR 4y ago
      Hey @tablespoon, I'm one of the cofounders of Lago here. Products like SAP do seem complex because of (i) the complexity of the billing problem and all the edge cases, (ii) they were not built with an API-first or engineer-first mindset We're only at the beginning of the journey, but that's one of the reasons why we thought of open-sourcing our billing API, so that the work could be forked and tweaked to address edge cases if needed (no need to wait for SAP to update the product: and they will never prioritize small edge cases). Also, it seemed to be no middle ground between 'Stripe Billing' (it's great for B2C Subscriptions) and billing systems that could address the millions of nuances of pricing between 'subscription' and 'usage-based' (most companies are a mix of both).
• awillen 4y ago
  Once upon a time I was the first product person at a now-decacorn, and my first task was fixing the billing system. It was quite the monster, and we ended up implementing a combination of Zuora and an internal system, as there were some parts of the billing model Zuora couldn't handle. I came away from this with one big lesson - if you're considering a complex billing model, consider the engineering implications first. With most products, engineering feedback gets taken into account - often product proposes something, engineering breaks it down, product realizes that feature x is vastly more complicated than they thought and not worth the effort, and the requirements are changed to simplify or remove it. The one thing that never seems to be true of is pricing models - that decision gets made at the very top and handed down with no chance for feedback. I think that if it the folks designing the billing system realized the costs, they might simplify things. If the complexity of your billing system means that 3% of your engineering team (plus additional folks in support and finance) is going to be working on it forever, but if you simplify it a bit you could keep 90% of it and only have 1% of engineering working on it, that might be a good tradeoff - after all, that leaves you more engineers to build features, which should drive additional sales. Unfortunately, that analysis never seems to get done up front and the cost is only understood after the billing system is deeply integrated into everything and would take an unpalatable amount of effort to change.
  • PeterisP 4y ago
    Yes, this is a big thing - there needs to be a clear model of how variable pricing and discounts are going to work in this company, with the sales team being able to apply the actual amounts of any prices and discounts, but not arbitrarily changing the model. It doesn't work when the model is too simple or restrictive, which will simply result in the model being violated; you do need at least the ability to customize pricing for individual customers, and for specific invoices, but other than that the decisions (e.g. whether you apply invoice discounts as percentages or specific amount or both) can vary but you just need to pick any reasonable option and stick with it.
    • awillen 4y ago
      Yeah, this is tough because when the VP of Sales knows that one particular change will close the deal, they're going to push for it no matter what the cost. In reality it may be better from a big picture perspective to give a concession that is actually better for the customer and worse for the company than x as it relates to that particular deal, because the larger concession fits within the current billing system with no custom work, so the overall cost to the business is less.
• rsanaie 4y ago
  I'm getting PTSD reading this article
  • Rafsark 4y ago
    Ahah so sorry. Hope this is because of the substance, not about the form ;)
• walnutclosefarm 4y ago
  Always has been. Marketing and sales create complexity on the front end, devising all kinds of products, plans, discount schemes and related complexity; jurisdiction, taxes, and accounting rules add complexity on the backend. In the middle, measuring what you're billing for is the hook around which all that complexity chaotically orbits. Decades ago - the computer involved was an honest to goodness System 360 (which is to say, not a 370, Z, or any of the the code compatible 360 successors) - I spent an evening trying to figure out the service charge on my bank account. I couldn't get at all close to the figure on the statement by applying the bank's schedule of charges. I took it into the bank, and got bumped by the teller to a department manager, and thence to a VP (small town; small bank), whom I told, "I will pay the service charge if you can explain to me what I'm being charged for." 2 hours later he gave up, having not gotten any where near an explanation, grinned, and said, "I can fix this." He set some flag on my account, and I never again paid service charges at that bank. Yep - that flag remained set on my account for nearly 20 years, through multiple software and hardware upgrades, until the bank was acquired by a big, expanding regional bank, and I left for a different, entirely local bank.
  • deleted 4y ago
    [deleted]
  • Rafsark 4y ago
    The marketing team of my previous company created a whole fake price plan that never existed in the code base. Somehow it works to drag acquisition, but it’s a pain in the ** to explain why it’s not as easy as updating webflow ;)
    • AnhTho_FR 4y ago
      Was I really such a pain? :)
• daxaxelrod 4y ago
  Poked around their site a bit, they claim to be open source but I don't see a link to a git repo anywhere. Also searched GitHub a bit but might have missed it.
  • AnhTho_FR 4y ago
    Hey @daxaxelrod! I'm one of the cofounders of Lago, the lib will be opened very soon, we're in the final steps of QA, that's why. I will make sure to ping you back here when it is. We wanted to share this post about our first-hand experience with billing a bit ahead of this. Thanks for your patience!
    • daxaxelrod 4y ago
      Hi! Cool, can't wait to poke around. I've used killbill https://github.com/killbill https://github.com/killbill previously and would be interested in your offering.
      • AnhTho_FR 4y ago
        Will ping you asap!
• giantg2 4y ago
  I work on a team collecting fees at a financial company. It is tedious and boring. There is a lot of complexity. I've often asked the business if they had ever thought about a different fee model that would be less complex. They just want to stick the legacy business model into the new tech...
  • AnhTho_FR 4y ago
    I'm Anh-Tho, one of a co-founders of Lago, thanks for your comment! Financial and cloud infrastructure companies are the ones with the most complexity. I've been on the business side, and sometimes you do want to change the pricing but there's so many implications: - Maybe existing users will churn and/or revenue will decrease as a result - If you change the billing system, there's a risk of bugs/errors and complains - If you spend engineering time on this, then you need to deprioritize other projects So... business teams often end up giving up, and that's a shame because iterating on pricing is a very powerful lever of revenue growth.
• yobbo 4y ago
  One design decision that, for me, seems to simplify things is to consider the "business system" a type of state machine that records all business events and serves as a "source of truth". If the events are not recorded, they have not occurred from the business perspective. A ledger-type architecture can be useful. This means that business events or user operations generate state transitions, which eventually are implemented as database transactions. The event log can be stored and inspected. The end user terms are encoded in the state-transitions. Contract obligations are encoded in the state of the database. As for calendar issues - operations need to be performed over "real calendars". It might be practical to "materialize" the business calendar into discrete units for this purpose.
  • Rafsark 4y ago
    For sure; I do think an event based solution working as a source of truth for the billing is the right solution. However it still creates engineering difficulties (making sure you don't ingest the event twice for instance). The ledger-type architecture can definitely work. When we built the system for a fintech, it was actually an event-based architecture connected to a ledger (taking the money out of a wallet). I think the whole process would be: - Ingesting events for usage based feature - Store these events in a database and make sure you make the idempotency right - Aggregate the usage of these events based on most common aggregate functions - Price this usage inside plans or subscriptions - Assign a subscription to a customer - Trigger a webhook (used for invoice/payment) at the end or beginning of the billing period
    • yobbo 4y ago
      And that events, as much as possible, are formulated as observations or state-changes rather than operations. Events in the form of "set x=1" or "y=0 was observed" are idempotent, where as "do this with that" is problematic. In the framework of state-transitions, actions such as "price this with that" are implicit in the state matrix, and shouldn't be called from APIs. Rather, API calls should be used to produce reports which are print-outs of the "current state". Webhooks in this framework are attempted state-transitions. User interactions generate events, and requests for reports (for example invoices), but as little as possible in the form "do this with that". My opinion currently is that there is little value in sharing code (libraries/frameworks) for these sorts of solutions because the code is trivial, application/language/domain-dependent, and coupled to everything else in the system. But design patterns, templates, and concise examples are hugely valuable because they help illustrate the complexity and solutions up front.
• nico75 4y ago
  Solid job unearthing the numerous billing system challenges (almost triggering PTSD from prior experiences interacting with such systems...). While I haven't fully checked out the details of the solution you offer, I'm excited to see builders embarking on building better solutions :huggingface
  • spookthesunset 4y ago
    I occasionally get PTSD flashbacks from my time maintaining a home brew billing system. That shit is hard. There are so many ways to fuck it up because you didn’t know what you were doing. The list of unknown unknowns for a billing system are huge unless you are an actual expert in billing, which you aren’t and neither is anybody in your team.
• my_usernam3 4y ago
  Reading this article gives me a smidgen of sympathy for Xfinity. They have so many plan options/timed promotional deals, and seem to run into so many issues every seemingly minor change, now it makes sense to me why. But in the same breadth still ... F** Xfinity
  • AnhTho_FR 4y ago
    Anh-Tho, one of the co-founders of Lago, here! yes totally, each time I open a pricing page, I have full sympathy for the engineering team who made it happen! I used to be the one tweaking with pricing plans on a spreadsheet and handing it over to engineers in a 'just make it happen' fashion!
• bcrosby95 4y ago
  I've had to deal with subscription based billing, and in my experience it wasn't a nightmare. There are some corner cases, but, those exist in every domain. And the stakes are definitely raised since you're dealing with people's money. Usage based billing sounds like a nightmare though. Taxes in the USA are a nightmare too. I have a friend that lives on a street, and for one side the tax rate is 7.5%, and the other side its 9.25%. Literally no company we tested got this right. Even Amazon.
  • Rafsark 4y ago
    For simple subscription-based billing, it's somehow not this hard, you are right. It also depends on how much plans you provide to your customers, and you still need to scratch your head for upgrades and downgrades. I truly believe in usage-based billing, so I do think pricing are getting more and more complex over time. Taxes are a nightmare for everyone also (in Europe too...)
    • spookthesunset 4y ago
      Dont forget price changes. Gotta deal with that. Then discounts, trials, weird durations, and a bunch of other shit I forget. My advice for anybody thinking of building their own subscription management system is to stop and go look at something like Zuora. You do NOT want to get stuck maintaining some home grown subscription management system--it will never be better than Zuora and you'll always be playing catchup, leaving your company unable to quickly react to the market. All the time spent adding functionality you can buy from a third party is money that you could have invested in your actual product. And worse, the fact you couldn't quickly react to the market because some key feature was missing in your homebrew thing is literally leaving money on the table.
      • lowercased 4y ago
        Transitioning to any non-homegrown system is not trivial either, as you'll need some process to reconcile info from current system and some external system. However powerful some external systems are, planning a migration to something non-homegrown is itself a monstrous process. Benefits of staying 'in house' can simply be you're not wasting time trying to fit your existing processes and data in to something external. That's really great that Ford uses Zuora. But for someone trying to work with an inhouse system that doesn't map to what a system like Zuora can handle... you now have technical work and a lot of political and non-technical work, and a lot of internal coordination to get all relevant people on board to commit to a change.
• biermic 4y ago
  Absolutely agree with the poster. This is my life at the moment. Just to take money for a small piece of software I built. Multiple currencies, trial periods, when in the process do you ask for the creditcard, country specific taxes,... And the worst: invoices. While paddle.com takes care of many of those things - I was shocked after realizing how much work this is going to be. Please think carefully about those things before you quit your job to work on your "Micro SaaS" or "unicorn startup".
  • dand31 4y ago
    Have you not looked at Stripe? Their product is fairly straight forward to set up
    • LeonM 4y ago
      Stripe is fairly straight to set up if you sell products in the US and only accept cards. For selling a SaaS worldwide, Stripe quickly becomes just as much of a nightmare. Taxes, billing, chargebacks, subscriptions, web hooks, different payment methods, VAT validation, card expiries, legislation, changing laws, SCA flows, and my favourite: SCA iframes that become non-responsive on browsers in a certain language. It is a never ending source of edge cases. Don't ask me how I know... My biggest mistake with my business was that I vastly underestimated how much work it is to accept payments. In my case, it proved to be more work than building the product itself.
      • biermic 4y ago
        Exactly that. While stripe would tick my needs, it's gonna cause some headaches on the long run. They can calculate the taxes meanwhile, but you have to still take care of e.g. MOSS yourself.
  • akrymski 4y ago
    Have also been considering Paddle. Would love to know what you found it's missing?
    • 4y ago
• sandworm101 4y ago
  Do not worry. Blockchain will solve or at lease simplify all of these problems. It is the way.
  • StevenWaterman 4y ago
    Look, Poe's law!
  • Rafsark 4y ago
    Even if with blockchain and crypto you need to define a logic to bill your customer. This logic is pre-transactional and define how much you need to price the transaction.
    • anticensor 4y ago
      Adopt a nothing is fungible, everything is serialised approach and maintain real equivalence classes instead of assuming fungibility as a proxy of replaceability.
• chasd00 4y ago
  I spent a lot of time right out of college working on the equivalent of an invoicing system for a pharmacy chain (rx drug pricing, electronic carrier submission, and reconciliation). Account receivables is also another nightmare. We would get checks that randomly show up at corporate for no reason from insurance carriers but then when the carrier realized their error instead of handling it as a separate process they would deduct the check amount from whatever invoice ( or even across a range of invoices! ) from us they felt like. We literally had a bucket called "magic money" these random deposits went into that we would use to fill in the A/R gaps from insurance carrier insanity. There was no connection between magic money and whatever invoice they decided to short change us on so it was just a hope-for-the-best process.
  • thibo_skabgia 4y ago
    Hey, I feel your pain. The "hope-for-the-best process" definitely is something happening in many companies... Yet, it's not a fatality anymore. I work with Upflow (YC 20) and it is exactly what we offer to our clients : 1. Making sure they have clear visibility on their AR 2. designing systematized workflows for their customers (as many as needed) 3. Customer payment portals to help their customers pay on time, with no friction. That's here at www.upflow.io
• cletus 4y ago
  Pretty much everything is more complicated than people (including engineers) think. Take a simple example: selling clothing in a retail store. Sounds simple right? You have an inventory of items and someone pays for it. Should be a simple transaction, right? Well, now you're dealing with: - How do you determine the price? There might be a retail price but there are sales, discounts, overrides for various reasons (eg defective goods), etc; - You have to handle different payment options eg cash, credit card, debit card, gift cards, store credit. Payment methods can be mixed; - You have to handle returns. These generally have to go back to the original form of payment. Some goods might not be eligible for a return. What if the payment is split between store credit and a credit card? Where does the balance go? - Identity of the person making the sale or handling the return; - Authorization procedures for returns and price overrides. Everything is complicated. As an aside, this is one reason why I'm so bearish on Web3 and smart contracts in general. The edge cases are so complex and possibly unknowable that codifying these, particularly on an immutable blockchain, to remove the need for human intervention seems doomed to failure. Take Web3 identity. If you lose your password there are methods for recovering your account. The first is a reset password option. That may be insufficient and there'll sometimes be a human avenue to recover your account. Now consider a Web3 identity. I've seen various incarnations of these such as authorizing other identities who with consensus can recover your account. Well, those identities can be compromises so you've just added an attack vector. Alterantively that recovery mechanism may become insufficient as people lose identities themselves. If these problems weren't complex they'd be solved problems.
  • AnhTho_FR 4y ago
    Hey Cletus! I'm Anh-Tho, one of the cofounders of Lago. Totally agree with you on web3 identity and its edge cases, it's just billing ^10000 in terms of complexity. I think for 'billing', usually people (especially non-tech people), don't understand the different blocks of the pricing stack: authorization, pricing grid, billing itself, invoicing (they often only see end-result: a receipt or invoice), payment gateways, dunnings, accounting etc). This post barely scratches the surface of it, but we intend to write and deep-dive more on each of these building blocks.
  • deleted 4y ago
    [deleted]
• kache_ 4y ago
  nightmare, or job creator?
  • Rafsark 4y ago
    Job creation comes always because of hard-things to build; When getting easier, it cuts job!
  • bloodyplonker22 4y ago
    nightmare job creator.
• 0xbadcafebee 4y ago
  Don't spend time, effort and money on things that do not bring business value. Any time I see a company with engineers building something they could buy off the shelf [and the thing they're building is not the product] I know their priorities are whack. I've worked for several companies that had more money than brains. They'd spend years, and millions of dollars in salaries, to build shit they could have bought and implemented in two months. And not only that, but multiple iterations of failing to build said thing, by multiple teams. Just because a tractor is used for farming doesn't mean a farmer should build her own tractor.
  • afarrell 4y ago
    At some companies, it takes more effort to go through the procurement process than to build the thing.
  • kdeldycke 4y ago
    The essence of MVP I guess. Now, back to the article's topic, the real question is: is there an off-the-shelf billing product available out there?
    • indymike 4y ago
      > Is there an off-the-shelf billing product available out there? There are lots and lots of OTS billing systems. They might work the way you want or need. Billing is the classic "I wanted wheel, except I wanted a round one" kind of problem. Right now, I'm looking for a system that will let sales invoice whatever they put on paper, and be good a structured, recurring billing that is easy to hook up to my SaaS platform. Oh, and Bob in sales wants to be able to edit the invoices after they are issued...
• dboreham 4y ago
  Note OP makes a billing product (OSS, granted) so is motivated to characterize the field as complex and hard. In reality billing is just like many other processes humans engage in. For sure it's more complex than computing prime numbers, but hey try writing software to capture medical services processes :) In the end this is exactly what engineers are supposed to do: figure out ways to model/capture/represent complex processes and data.
  • Aeolun 4y ago
    If it’s as annoying as translating human processes into code, I can see how they’d want to standardize it so they’d never have to do that particular part again. Then again, if you make it too customizable, you end up with current single sign on solutions.
  • spookthesunset 4y ago
    Or better, buy a billing system instead of inventing one inhouse. Billing is hard to do correctly. For one thing it interconnects with so many different platforms in your company. For another thing, the cost of fucking something up is high--people don't exactly like seeing mistakes in their bill. Hell, think about tax... you wanna deal with that shit? To do it "correct" you've got to have people who truly understand accounting concepts like double entry bookkeeping, financial reporting, cash vs. accrual, financial regulations, auditors, etc. Unless your company's product is billing, good luck hiring an engineer who knows this shit. But trust me from experience. Don't build your own billing system. There are several good ones out there that will grow with your organization. Buy one.
    • piva00 4y ago
      Similar experience in some companies in my CV. Billing systems aren't fun to maintain, develop nor debug. They take a non-trivial amount of time when done wrong and if you are building one in-house you'll inevitably run into some mistake. Buy a billing system, it'll save engineering hours having to deal with it and if you choose right they'll scale with your company.
• welder 4y ago
  I implemented billing at WakaTime with Stripe, Braintree, and Coinbase as swappable payment providers. It's definitely a source of edge case bugs, but overall it works pretty well. It supports prorating subscriptions across providers, monthly & yearly billing, trails, discounts. * We don't remit taxes for customers because that would be impossible to build in-house, but everything else was achievable. * We have tons of logging, and generate transient Quotes on our end, then check that the expected charge matches what the payment provider actually charged. * Each Quote is reproducible, so if any bugs come up we can give the "create quote" code the same input and see what went wrong. * What helped when building was hourly tasks running in the background to make sure customers who should be billed were being charged, and make sure we didn't double charge any customers, and then alerts so we could find and fix those bugs. I would do it again to support both PayPal and Stripe, but it's definitely a source of complexity for the code base.
• epberry 4y ago
  Usage based billing is not only hard to implement but hard to consume. No one really knows what they are spending and how much particular customers or products cost. I did a writeup a week ago on CDNs and ended up spending several hours in spreadsheets. I had felt this pain vaguely before but actually seeing it everyday at my current job makes me realize there are engineers suffering on both sides of software billing.
  • hobo_mark 4y ago
    Speaking of usage-based billing, if I intend to charge users a usage-based bill at the end of the month, as far as I have looked Stripe is the only provider that supports this (they call it "metered billing"), is there really no alternative?
    • epberry 4y ago
      The OG poster website is actually a Stripe alternative. They also call out Chargebee which is another alternative that should support this.
  • Rafsark 4y ago
    Stripe offers simple metering, but in the end, you make the job of calculating metering charges (aggregate all usage and send the value to stripe). It’s becoming harder when you are an API or a cloud company who want to bill a server usage for instance. Calculations are getting heavy, costful and hard to maintain. This is why I do think stripe is not a usage based solution, and also the reason why I decided to create a company trying to solve this pain :)
  • kdeldycke 4y ago
    This. Usage-based billing is incomplete without its suite of reporting, consolidation and visualization tools. No wonders there is a cottage industry of cloud computing consultants and SaaS solutions focused on helping you to make sense of your AWS/GCP/Azure invoices.
• msluyter 4y ago
  I rewrote an invoicing system (originally in .NET) in Python and yeah, I totally feel this. All the system did was generate a CSV file that could be input into another system (Quickbooks, iirc). The original .NET version was nonsensical at times, replete with bugs, and I spent a lot of time trying to figure out what the original intent was (there were no docs, of course, and finance often had conflicting/incorrect ideas about what the system should do.) Version 1 had to simply replicate the csv generated by the original system exactly, which required purposefully writing some bugs that I discovered along the way. But yeah, there were all sorts of nasty complexities -- this was a usage based system, but there were (optional) tiers (pay less for higher tiers), as well as certain time segments that were discounted. And then there were cases where multiple accounts could share a set of data and get a discounted rate and/or only one of the accounts would be billed. And on and on.... One lesson I drew from this experience was that, when discussing things with finance to try to figure out what the requirements were, it really helped to have pictures. I would create these elaborate usage/time graphs, and color in portions to reflect different billing/pricing scenarios. These made it much easier to discuss the current/future behavior of the system. Because saying "really, this is simply the integral of a usage function over time" didn't fly.
• robot 4y ago
  Problem is it still requires an API to integrate. Stripe has all these capabilities but API integrations are needed to use them. If you are looking to build a micro SaaS without needing to deal with billing API integrations check out our software: https://saasbox.net https://saasbox.net. Built for completely eliminating any billing related SW development. It doesn't handle all the corner cases mentioned in the article, but some of them are handled, such as plan upgrade / downgrades with pro-rating, editing plans on the fly, migrating users across plans, notifying your application on those changes.
  • vikeri 4y ago
    My understanding is that Stripe billing can be managed completely no-code from the Stripe dashboard? And I think it includes automatic tax, prorations etc.
  • Rafsark 4y ago
    Why is it a problem to use an API?
• ben7799 4y ago
  I agree with thesis but there are entire aspects of this that he didn't even touch upon. There is a multiverse of different billing nightmares for engineers. I worked on a piece of software that bridged billing data from IBM mainframes out to the web. The hoops that had to be jumped through to get data out of the mainframes and the number of hacks and kludges involved was legendary. Likely everyone here has paid many bills in their life on websites that used that software. At one point I knew the vast majority of my personal bills had gone through it. Credit Cards, Utilities, Gas cards, etc.. Stuff everyone had to pay with paper before the web.
  • colejohnson66 4y ago
    Could you expand on what kind of hacks/kludges were required?
    • ben7799 4y ago
      Basically the mainframes were almost incapable of any data exchange that was meaningful in the 21st century. Instead of IBM doing something same the market just built software to scrape bills off the mainframes printer output, which you could intercept in electronic form. But then every accounts bill files were different. And sometimes the mainframes moved data around on the page, and so on and so forth.
      • koreth1 4y ago
        That gives me flashbacks. I worked on a payment system that had to pull data from third-party service providers that only made it available in the form of downloadable spreadsheets from godawful slow and unreliable JavaScript-heavy web portals. Just getting to the point of downloading the spreadsheets was painful enough, and then the spreadsheets themselves were inconsistent and would sometimes flap back and forth between two different formats for a few weeks as the service provider repeatedly deployed and then rolled back new versions of their software.
• asciimike 4y ago
  Nit: there's a lot of noise being made in comments about it being an open source product, but I haven't found the source (maybe I have to speak to an expert to get it?). github.com/getlago has the docs, which is nice (and gives a lot more info than is available on the website about how the product works), but not quite the same. Also note: > Our open-core version is forever free. We will introduce paying add-ons in the future, with a consumption-based approach.
  • AnhTho_FR 4y ago
    It's coming soon, we're in the final steps of QA, but we could not wait to share this post! Thanks for the comments on the documentation, by the way!
    • asciimike 4y ago
      Awesome, glad to hear, would love to take a look whenever it's available! Looking at the docs it seems like the current state of the product is much more about subscriptions than usage based billing (e.g. no aggregation or rating). Any plans to add those in?
      • AnhTho_FR 4y ago
        We have an event-based architecture and offer 'aggregations' at the moment: https://doc.getlago.com/docs/guide/billable-metrics/aggregation-examples https://doc.getlago.com/docs/guide/billable-metrics/aggregat... We plan to add more aggregation types, gauge metering as well! We definitely want to address usage-based pricings, but also all the pricings that are hybrid between usage and subscriptions (ie the vast majority of software pricings). Feel free to reach out if you have more question: anhtho@getlago.com
  • AnhTho_FR 4y ago
    Hello! As promised, the repo is available here: https://github.com/getlago/lago https://github.com/getlago/lago ! We also opened our roadmap: https://github.com/getlago/lago/projects/2 https://github.com/getlago/lago/projects/2 You can request and vote for features here: https://community.getlago.com/c/feature-requests/11 https://community.getlago.com/c/feature-requests/11
• MarcinMas 4y ago
  Yeah I can agree. I made custom integration with Chartmogul and it was a nightmare. But we did it! And now I feel like real engineer
• mc4ndr3 4y ago
  "Should we build it in house?" No. No, spelled F-U-C-K-N-O. Should you compare and contrast off the shelf solutions, test one or two, and slowly expand its use to cover more of your business? By all means, yes. Should you develop an in-house custom variant after that? Still no. For billing or anything else. Even a pure tech company like Google should not vertically integrate everything. You quickly end up wasting time making horrid systems instead of generating your main business revenue. "Oh, but our needs don't align with the way the industry standard third party systems work." That's a symptom of the disease. You won't cure it by enabling the virus to proliferate even further. People who don't execute I.T. low level changes mistake computers for magic. Yes, computers can do a lot. But even the tiniest feature takes an incredible amount of focus, cost, development time, maintenance time, headaches, and gnashing of teeth to accomplish. The magic is that it looks easy, from the user's point of view. Let's put it bluntly: We can draw a direct line from third party systems with multiple businesses as customers, with cost savings. It's far more likely to be a complete waste of resources to in-house a new software system. Maybe you don't have a name for the kind of system you want or need. That's okay. Keep assuming it's probably already out there, and continually search for it. Googling is cheaper than building. "But we can build it better than them because..." Please, that's the fallacy of exceptionalism. Remember, every company wants to believe they're truly pioneering, talented, and logistically capable of doing all the things. Don't spread yourself so thin. Keep the lights on. Grow your customer base. Everything else is a pipe dream. "But the third party systems suck." Yes, they very much suck. And yet the cheapest path to success is pushing them to un-suck their product. Not starting from scratch to create a second, even more immature product. Frankly, computer systems are already quite flexible. You can often bend an existing one to fit your needs. This happens with Jira, for example. They customize it until it can't function at all anymore. But this sad extension situation is still cheaper and more effective for conducting business than invention. If you truly believe a new kind of system needs to be built, go and start a new software company. Or pay a software vendor to do it. (Software vendors, just take their money, smile, and nod.) But for Warren Buffet's sake, don't spend a dime on a new line of code if at all possible.
• nickdothutton 4y ago
  The reason billing systems are hard is because they tend to just accumulate features/behaviours/requirements. Forever. Although billing is not my specialty I have had to involve myself in it as product owner and business unit manager. Try and imagine the amount of logic, complexity, rules, and exceptions involved in a system that has accumulated perhaps 30-40 years of customer sign-ups. Umpteen contracts, umpteen iterations of each contract, and perhaps 50 or 100 products, each sold in 10 or 100 countries. I now know more about the tax system in South America than I ever wanted to.
  • Rafsark 4y ago
    Yep, totally right. Same at the previous company we worked for and built the billing logic. You start doing it in house because the first version is simple. It’s getting complicated when you keep adding paying features to a product. Often see product managers becoming experts in taxes and accounting :)
  • r00fus 4y ago
    I see building or implementing these kind of systems as "discovering" use cases for given scenarios/customers/markets. That's why "billing" is almost too generic - instead "billing for telco in South American markets" - is more useful because those verticals have specific sets of use-cases and challenges.
• Andrew_nenakhov 4y ago
  As someone, who has built a billing system not once in my life, but twice (one for an internet privider I worked for, which counted the amount of traffic, and another one for a SaaS project(, I fully sympathize with the post. Billing is an unbelievable can of worms even before you get to taxes. Add in all the things the marketing people want from billing (trials, discounts, per-seat per usage pricing, etc), and you have enough tasks to last till retirement, no matter how young you are.
  • Rafsark 4y ago
    Yep, do agree that non tech team created a lot of pricing complexities, sometimes without any reasons. Remember a marketing team updating plans on webflow and telling the product team « see, not that hard » ;) Ended up with a whole squad of engineers called « the pricing cross functional team » …
• bokohut 4y ago
  As a co/founder and core architect of multiple PCI level 1 compliant payment processing companies in the last two decades I will add as others have that billing and payments is exactly like anything else which is "difficult". Once you spend enough time in the trenches dealing with the challenging technical issues your experience and creativity leads to the proper solutions that further enhance the system. Everything is just inputs and outputs yet it is what one does in between those "puts" which matters and thus where experience can reduce such challenges.
• hammock 4y ago
  Double-entry accounting is tough for a lot of people. It's a different domain and trying to fit it into traditional "math" will only cause headaches.
  • tabtab 4y ago
    An alternative to DE is "Assume Balance". See near the bottom of http://wiki.c2.com/?AccountingModeling http://wiki.c2.com/?AccountingModeling
  • kdeldycke 4y ago
    Still, someone tried in the "Algebraic Models for Accounting Systems" book: https://www.amazon.com/Algebraic-Accounting-Systems-Salvador-Rambaud/dp/9814287113 https://www.amazon.com/Algebraic-Accounting-Systems-Salvador... Source: https://github.com/kdeldycke/awesome-billing#finance https://github.com/kdeldycke/awesome-billing#finance
• einpoklum 4y ago
  > Why billing systems are a nightmare for engineers My guess: Because basically nobody wants to develop one of these as a hobby FOSS project. The whole idea is collecting people's money, which when you write your own code for public benefit, you basically don't do :-) > We quickly decided to add plans, and 'pay-as-you-go' features So, the author talks about things getting more complex, beyond their expectations. But - why did they not expect this? I mean, they know that companies pricing changes over time; that payment models change, including rates, cadence, use of internal credit etc. We all know this without ever having worked on a billing system. > Qonto is a 'neobank' ... Fintech unicorn So they were expecting billing to be easy for a bank-like endeavor? Hmm. > #1 - Dates The date problems per se are quite mundane and any non-trivial software system which involves timed activity deals with them. The billion period problems are not really date problems and are a recap of what OP had already written about. They are also problems most of us are well aware of from our personal experience as people in modern society ("Does my bus pass expire a day after I bought it, i.e. 24 hours later, or does it expire at the turn of the day = at midnight?") etc.
• didip 4y ago
  This is why Stripe has a big moat. Not many people want to deal with this problem.
• janci 4y ago
  I was working on a billing system for utility company and it was a nightmare but not for the reasons from the article. Dates are no problem, everything is billed at the end of month, quarter or year. Upgrades are downgrades are simplified, customer can legally change tariff only on predefined dates/events. Usage - yes, but exampled in the article do not even scratch the complexity of this Idempotency is not really needed, the billing process is not continuous, it is a batch job. Cash collection is out of scope of billing software. Taxing was clear. Yes, there were multiple billable items with different tax rates, but not too complex. All customers were local. The nightmare part was: - customers can have multiple consumption locations, location can have multiple meters and customer can request to split the bill on multiple invoices or combine multiple bills to single invoice as they please - meters can be replaced at any time - meter readings are available on dates totally independent from billing cycle. Most of consumption data are mere forecasts. - when the actual consumption data is available, everything must be recalculated AND compensated (on a correction invoice showing the difference) - actual consumption data can be wrong and may be corrected at a later date, even multiple times - consumption points can be added or removed or moved to different customer at any time, but this information is only available after the fact - the prices can change mid-billing cycle for some customers but the consumption data is not available with that granularity - customer legal information (company name, address) can change mid-cycle
  • politician 4y ago
    Did the company use a relational database to deal with the billing and subsequent adjustments? I've been there: it can be more straightforward to use an event streaming approach that recalculates billing as new events arrive.
    • janci 4y ago
      Yes, almost everything ended as temporal or bitemporal model. Pain to work with. Even for simple questions like "What is the customer #1234 company name?" the system needs to know when are you asking.
      • 4y ago
• perlgeek 4y ago
  I work at a b2b company with a 25+ years history, and OH MY GOD... * Sales is payed (partially) off commissions, so they do their very best to sell whatever the customer wants, not what the engineers know how to bill, or that product management has prepared. It's really hard to push back against a contract with revenue 10x your yearly salary "just" because the new features to bill it don't fit together with existing features. * It's very hard to keep track of which customer pays for what, exactly, and is entitled to which service. Easy for standard products, hard for bespoke products, nearly impossible for shared services (it took us >9 months to sunset a shared HTTP proxy that was in use by just 4 customers that didn't even pay for it...) * Legacy contracts: imagine 25 years iterating on product design, and keeping the old contracts running (except when the customer leaves on their own). Some of the old contracts are hard or impossible to find, nobody wants to spend their time rummaging through them to find out what exactly a customer is entitled to. A year ago we still had a customer that still pays for 8€ per GB HTTP traffic (web hosting), because that was the going rate back in the days * Usage-based billing: half of our customers want to be flexible and only pay for what they use, and the other half uses SAP and for them, invoice amounts changing from month to month are the pure horror. So clever sales people invent hybrid models ("pay for what you use, but fixed price until a certain threshold, we'll notify you when you get close to reaching the threshold"). Another source of complexity. * Usage-based billing: it's basically a topic for the whole company, most departments provide services that needs metering, but metering is usually an afterthought to the architecture, hard to verify for somebody on the outside * Usage-based billing: for big customers you cannot put all the details on the invoice (our mailing service provider is limited to 99 pages per invoice...), so you need separate reporting to drill into the details. Fire and brimstone will rain down if reporting and invoice disagree on some numbers... This just scratches the surface. Customers that want to split their bills along cost centers that don't align with access control tenants, different electricity prices based on location, the list is endless.
  • Rafsark 4y ago
    The article just scratches the surface too. I could have written a book to describe it! Reporting for revenue is a hard game, indeed. Spent a lot of time trying to reconcile revenue streams in the same company we ended up building the whole billing system. It’s also not easy because metering is not something that is predictable nor easy to make it fit into a finance report.
• bhawks 4y ago
  'Tax logic' is an oxymoron and a trap for smart technologists to fall into. Tax codes are not logically consistent or clearly defined. They're a bunch of illfitting laws, beauracratic/regulatory guidance and interpretations of judicial decisions. I tell folks working in this area to accept the illogical nature of it all and to be ready for all sorts of arbitrary last minute changes. As the article points out understanding time plays an important role here too - when do different tax treatments take effect is an important question to answer.
  • c3534l 4y ago
    Tax is written by a bunch of people whose background is overwhelmingly from law, not finance, for reasons that have nothing to do with logic, consistency, financial literacy, understanding, sanity, or common sense. Politicians frequently pass laws without understanding what it will do or what its implications are, because they just didn't think that far ahead. It is quite possibly the worst system you could come up with for writing tax law. That said, there is a logic to most of it, but not in a way that allows you to come up with general heuristics or universal abstractions.
• cdkmoose 4y ago
  Just wait until you have to do this under government controlled (tariff) environments. Years ago I worked on a tele-conference platform for large international telecom. We built the software to run all aspects of their platform including billing. Government tariffs covered billing rules for each leg of a call including rates based on caller's local time of day. Converting bridge time to local time via area/country codes(only data available) and time-zones is quite a mess. Add DST calculations to that per locale to reconcile negative duration legs or legs that should be 1.5 hours but show up as .5 hours just makes it cloudier. If your system repeatedly gets it wrong, you could be in trouble for billing fraud or tariff violation. No joy in that job.
• po1nt 4y ago
  Implementing and maintaining billing is my daily job for the last 6 years. I would also mention topics like: - Reporting (various data aggregation and audit reports) - High reliability, nothing hurts company more than unability to bill their customers - Rounding, this can be seen as a subset of "taxes" but it's much more complex - Locating user, also can be seen as a subset of "taxes" but user can have country A set in profile configurations, country B on credit card and country C geolocated from IP address - Timezones, issue everywhere but when we talk accounting it's super important - Talking to moronic payment gateway providers. This is the biggest ones. I would love to just flip Apple and Google like Linus flipped of nVidia. Proprietary, poorly documented, "find out yourself by weeks of experimenting" bullshit with no easy way of getting technical support even when you bring those companies millions per year. Things don't work, deprecate monthly or weekly and expect you to be always ready to make changes. Some implementation make zero sense at all with complete paradigm shift of handling payments like between Google in-app pay and Google subscriptions. But as my SO says "Don't cry, it pays well"
  • AnhTho_FR 4y ago
    Wow, thanks for sharing! We found out 'billing engineers' are very rare and really demanded, as it requires to be very detail oriented, technical, and business processes oriented. But even if there is high demand (and I guess: corresponding high pay, as you mention), very few engineers are up to the challenge!
  • Rafsark 4y ago
    Yep you are right, indeed. We could have mentioned Reporting/Analytics for revenue, which is always a huge pain for companies!
  • patrck 4y ago
    Yeah, Audit for the win. And it's not "billing", it's Compliance + Bill Presentment (marketing!) where we are always trying to find the most profitable local maximum of explainability to Sales, Customers, and Management/Enforcement while also having hard checks to prevent or at least mitigate losses. No one wants a Knight Trading excursion....
• jlg23 4y ago
  I've implemented a few billing systems and "the nightmares" were never related toengineering but always to specification: If bizdev does not know what they want, no outsourced imlementation will be able to help them.
  • AnhTho_FR 4y ago
    From your experience, who should specify the billing system? The Product team? We're building Lago so that the right questions / decisions frameworks are asked during the implementation, so it's like a forcing power embedded in the product. Our experience is, unless the Product/Biz team has been exposed to billing, they will never specify in a way that is precise enough, so the engineers who implement billing will have to think/assess/decide themselves.
    • jlg23 4y ago
      Have someone from accounting/legal work at least 50% in bizdev; don't let them talk to engineering until they have sorted out what they want and, much more important, whether it is legal.. But equally important is to cut back engineering's enthusiasm for abstraction/unification": In my experience it is better to have 2 distinct, tested implementations against one pretty generic data model than trying to coerce both into one. Intellectually one can, but "oh so great billing concept" #3 or #4 will break ones neck. I don't really think anything about billing is hard, it is just incredibly dull if done the pragmatic way.
• pbreit 4y ago
  If you try to keep it simple, it's not that hard.
• awinter-py 4y ago
  billing is a chronological typesystem and is only hard to build because most languages don't offer time-safe types (and because municipalities don't expose their tax rules as types)
• Melatonic 4y ago
  Why XXXXXXX are a nightmare for engineers: PEOPLE On a more serious note though billing can be hugely complicated and I am really glad I do not have to deal with it
  • Rafsark 4y ago
    Do you have a dedicated team in charge of it?
• simonjgreen 4y ago
  There's few things worse in B2B than encountering a system where engineers have reinvented general accounting principles from scratch.
  • kdeldycke 4y ago
    The only worthwhile re-invention of late might have been triple-accounting.
• elietoubi 4y ago
  Hi! I co-led the billing stack re-write at Uber circa 2017 and it was an absolute nightmare. Some of the unexpected hard things that are not mentioned in this article: - Manage promotion was very complex - Modeling money (we used to model it as cents ie 1/100 of the common denomination ... turns out it broke a lot of countries like Japan where the Japanese Yen doesn't have cents) - Timezone is a hellhole One interesting open source project was https://killbill.io/ https://killbill.io/ PS: Since I built a billing and invoicing startup .. check it out: zenbill.com
  • 0des 4y ago
    Hey thanks for showing me this. What lies beyond the starter plan in concrete terms?
• schwinn140 4y ago
  This is some brilliant, next-level, content marketing here. Nicely done.
• oznog 4y ago
  It's missing when you find out that business people don't know how the business must be executed with total precision.
• CSMastermind 4y ago
  I just want to say that this article is a great intro to the topic and I credit a lot of my growth and development as an engineer to working on billing systems in my first job out of college.
• fhrow4484 4y ago
  Since founders are commenting here. getlago.com webpage is strange: "Lago is backed by Y Combinator" "The Open Source Stripe Billing Alternative" Why would YC invest in both Stripe and a competitor of Stripe?
  • beachy 4y ago
    To avoid the innovator's dilemma?
• ab_testing 4y ago
  I think companies should not try to reinvent the wheel with building billing systems. It is better to implement an ERP to take care of billing rather than building out your own. If you ever grow large enough, this homegrown system will never be SOX compliant and will not stand up to audits.
• pizzaknife 4y ago
  first sentence. homie rip
• micromacrofoot 4y ago
  Ok now integrate billing with flight booking and timezones… the devil’s stack
• dekhn 4y ago
  Basically everything about enterprise is a nightmare for engineers. What amazes me (after having worked at some of the most profitable companies in the world) is just how little intelligence the leadership has about their own revenue or costs, beyond "wow huge amounts of money is coming in or going out". And how many critical processes are implemented manually, by individuals, with personal spreadsheets, on their laptops.
  • makeitdouble 4y ago
    > intelligence the leadership has about their own revenue or costs I see the positionning of the CFO right next or below the CEO as a mechanism to let the CEO care about “big pitcure” money flow, while having someone in contact with reality guide decisions and veto stuff that won’t fly in cost/revenue terms.
  • deathanatos 4y ago
    Yep, I'm about ready to start implementing one of those very spreadsheets just to start trying to categorize Azure costs.
  • Rafsark 4y ago
    It's a difficult topic to tackle; The financial leader has different needs than the tech one. One needs a perfect report with exact match day over day, month over month, year over year. The second one needs to do the math to calculate instant consumption of the product. If we add marketing leaders of a company to it, it brings creativity to the extreme (like inventing new prices, just to use it "as a hack"). Same goal (revenue), but definitely not the same path to success
  • BatteryMountain 4y ago
    Yup, I work at a tech company and our entire finance & billing team is using excel exclusively (I think our payroll/leave system is the only thing not on excel), where every other team uses one system or another. All of the higher ups makes decisions based on docs created out of all those excel artifacts. I have to admit, their productivity is quite good and they are kings at aggregating and graphing data in sexy ways. It would be a challenge to build them something that actually works better and can adapt quickly to changes (the software lifecycle too slow for them and now they have to deal with some developer types to get something done, so skip all that noise and stick to excel).
• templaedhel 4y ago
  Great article and excited to see another solution tackling the complex problem of billing. I’m an founding engineer at https://metronome.com https://metronome.com (building usage-based billing infrastructure) and definitely echo the sentiments shared here — building a billing system that can scale is no small feat and anyone who previously built billing in-house can attest to just how painful that is.
• johnzim 4y ago
  I'm the CTO of a company tackling exactly this problem (www.salesbricks.com) and I'd go one further. The complexity in billing, invoicing, payments etc are _not_ the stuff that computers or systems do. The complexity is and always will be in the stuff humans do.
• btilly 4y ago
  Wait until you get into the complexities of taxation. If you think you know them, pay attention to https://twitter.com/aotearoa_ben/status/1526786701750050817 https://twitter.com/aotearoa_ben/status/1526786701750050817. That's a law that changes the tax rate if 1. The purchase happens in Texas. 2. The payment cleared in a specific 2 day period. 3. The item cost < $100. The cost of shipping has to be included in the item's cost. 4. The item falls into specific categories that almost certainly don't match whatever categorization system you are attempting to use. Seriously, calling out just one example, https://comptroller.texas.gov/taxes/publications/98-490/school-supplies.php https://comptroller.texas.gov/taxes/publications/98-490/scho... points out that a kit containing school supplies qualifies as long as the cost of the qualifying school supply items in it exceeds the cost of the non-qualifying items. I guarantee that you don't have a category for that in your accounting system...
  • jef_leppard 4y ago
    I worked on a tax app. Total nightmare. I remember having to adjust tax rate for a couple parishes in Louisiana and a city in Florida for example. Trying to come up with a readable, testable, way to get that done was something I tell ya.
    • lowercased 4y ago
      similar. worked on invoice app having to apply tax, and working with existing foxpro(?) tables that were in postgresql. Trying to reverse engineer without having access to original code, and make things work "like they used to". Someone manually pulled down some free tax thing once per month, but there were so many exceptions that people knew about that were just hardcoded in to the code that I had no access to that I could never make it "just work". We transitioned to using an external tool (taxjar, IIRC, but might have been something else). Was a pain to feed them data to get back tax data, and.. so many exceptions and exemptions I had to learn about. Weeks later I get "all these numbers are wrong". I dug in and went through logs. "no, these are right. taxjar is adding x% on ABC and shipping". "We don't charge tax on shipping". "Well, taxjar is doing it, because it's how you're supposed to do it in this state". "We've never charged tax on shipping" "Well... you've probably been non-compliant for years then". I think when they'd started 'computerizing' this back in the early 90s, shipping costs weren't taxed. Or... (IIRC) the amount of shipping cost taxed was proportional to the amount of taxable goods being shipped in the state (vs out of the state) and when they started, what they were shipping wasn't mostly taxable (like... some of the main items were taxable, but the supplies weren't, or something similar). And... if they were shipping to a non-taxable entity (non-profit) then the in-state shipping charge wasn't taxable, but they were supposed to pro-rate it based on the proportional distance in the target state, if that state charged tax on shipping, then track that and remit to that appropriate county/state authority. These were some of the rules that a variety of people had tried to describe to me both what they thought was supposed to happen and what they thought the original software was doing, but... they could never find 2 examples to line up to confirm or justify those explanations. Another example: certain items (chemicals) were taxable in some counties and not others, and might be taxed differently based on whether they were being 'used' by technicians vs delivered to a customer (again, different counties had different rules - and some of these rules were indeed confirmed by digging through taxjar and their support and state agency websites). When this company started, it was one location in two counties in one state, and as they grew, the specifics grew (and also changed yearly) and what they should do became lost to everyone. Moving things to an external hosted tax-as-a-service system was supposed to 'simplify' things, but it just introduced a lot of turmoil and headache, because routine numbers people were used to for years were suddenly replaced with "new" numbers which no one trusted.
• fmajid 4y ago
  I started my career in Telecoms in a field adjacent to billing systems, and industry lore is that only one in three billing systems project succeeds in delivering anything, and only half of those fail to meet requirements.
• ddingus 4y ago
  (1 of 2) In the 00's, I was part of an effort to get the State of Oregon to consider open source software. A 40 million billing system debacle was part of what drove this effort. The details are not important, save to say the ASA paid big to stall the legislation and that worked. It was stalled, and word got out that future bills would be toxic. So, that's over, but a great idea remains! I am putting it here, because someone, somewhere should do it. Will save us all a lot of money and do a great public good. And it's likely to pay a fellowship of some kind very well over the next couple decades. I'm on a different project and need to finish it, and I'm likely to be paid well on that, so... Instead of paying the likes of Oracle, or some consulting firm to write and support the billing system, the State should do it. So far, I've seen several bad contracts in my State where the State has a right to use the software, does not own the code, must pay for support, and on it goes. This is quite expensive, and frankly, it didn't work well. Instead do this: The State sets up an organization. This org is public, and those who work there are public servants, or some similar arrangement. The key thing is the billing system gets written on open code, uses open data, and the resulting software is owned by the public, through the State essentially. This code gets published on Github under an appropriate open license. Fund this org, paying respectable developer salaries, and get the system written, tested, deployed, whatever is required. This funding comes from public tax dollars, and I would suggest the lottery as a primary source. Once the initial system is complete, and let's say it's for water. Once it's complete, the system can be published and the whole thing becomes an export for the State that does it. Any municipality anywhere can do one of a few basic things: They can build and deploy the system and handle billing themselves. They can hire the Org to help them do this, contact for support, whatever. They can hire private entities to deploy the system. Changes can come from anywhere, but the organization is best suited to implement them. This can be on a contract basis, funded reasonably in any number of ways. The organization continues on to develop other public works tools, each funded by public dollars, each displacing some expensive thing or other, and each saving the people quite a bit over the course of say a decade. Wash, rinse, repeat. The end product is a nice pile of public works software needed by tons of municipalities most all of whom are paying hard for it now and they pay hard for basically everything, and continue to do that. The public could make these investments once and benefit for a very long time. Perhaps more than one State wants to be a part of something like this, or whatever State gets going on it first, might want a few organizations, depending on what is to be done and the scope / scale required. Either way, the Organizations provide jobs and consulting / support and basically own the public works they develop and are funded to maintain those software works. An advantage over contracts and proprietary license is cost, particularly when the likes of Oracle are involved (and no judgement here and no calling out as I'm only saying Oracle as a well known example among many out there and I just feel the public works can be done without Oracle and the cost associated with similar entities). Having the data be open is another advantage in that it can be used in all sorts of ways, and the public records can be made available to the public in simple, effective ways. Mix in smart API's and the whole thing becomes very high value and an attractive base target for all sorts of public software development having to do with the basic machinery of society. None of this is particularly sexy, but it does pack a big punch in terms of modeling how a more modern, digital society could employ open code, open data to get the work we need done, done in a cost effective, efficient way. And that's it basically. Over time, this first, or core organization will accumulate considerable domain knowledge. And it would become the basis for a lot of other work. Ideally, others started up in various places would all garner the knowledge needed by domain: Utilities, water, power, internet (where it's actually municipal, and we know that's not popular with the big telecoms...), garbage, etc. City functions, parking, various permits, anything that has a process that could be automated and or just performed on open code in simple, direct ways that are lean, consistent. Vehicles: Registration, and all that tends to be tied into these systems. In my state, we've got vehicle registration, drivers licenses / ID Cards that link to facial biometrics (I don't like this, but that is another discussion), and it's also voter information central with a signature, party affiliation, address, and everything needed to manage the voting rolls, and execute Vote By Mail nicely, and in a trustworthy fashion. Oregon has done well on that front. Citizenship records are in this pool of software too. Health Care: Right now, again in my State the Medicaid system insures people who are diabled, are poor, perhaps wards of the State. This is currently on Oracle, and has been such a mess it's ended up in court. I don't even know the state of it all right now, but it's a lot of money and a lot of grief.
• wbarber 4y ago
  Anyone know of a company other than Paddle and PayPro Global that act as the merchant of record and remit taxes for you globally? I'm a solo founder not interested in dealing with either the complexities of these billing issues nor with the taxes issue. But my product would easily be of interest globally and the API's I've built have more interenational customers than global customers so I suspect the same may be true for the product I'm building out of those apis...
• tikiman163 4y ago
  I program billing systems, and even account reconciliation systems for payment processes. It's not a nightmare, it just take a greater understand of accounting. If you want to be an engineer that works on financial systems, take the first year of classes that accounts have to take and pay attention.
  • Rafsark 4y ago
    You are totally right. But, somehow, most of the engineers working on those topics don't want to work on financial system. They are just part of a company and take this topic "because someone has to take it". They accumulate the tech knowledge, but not necessarily the whole accounting part
  • datavirtue 4y ago
    This. I'm astounded at the gaps in some people's inability to understand business processes because they have no accounting training. It is not that difficult but it is something that needs to be integrated into your mindset. You might then realize that accountants have been using event sourcing to solve complex problems for hundreds of years. Very senior developers will nearly always default to juggling updates to database records for everything. I cringe because it is often the source of serious gaps in domain language between developers and business operations people, leading to unreliable systems that are very difficult to reason about. Immutable ledgers are your friend.
• makeitdouble 4y ago
  > Another option is to rely on existing billing platforms, built by specialized teams. If you’re considering choosing one or switching, and you think I can help, please reach out! To note, this option only absolve of _some_ of the complexity and burden. You still have to understand: - what problems the billing platforms is solving for you, how they're solving it (does it match the process you need ? -> you also need to know how it should work) and what to provide them to make it work. - what problems they are _not_ solving, and you need to deal with on your end A good service provider will help you at least on the first part, and educate on the parts you need to be aware of, but there's always dark corners that easily get overlooked if the business owner didn't do their homework.
• reilly3000 4y ago
  I am so fucking exhausted with the content marketing on Hacker News. It’s not that it has no value. It’s not that there isn’t a place for ‘developer evangelists’. It’s not about this piece. I’m just sick of it. I come here for a refuge from manipulation- a place where people can share and debate things on their own merits. Content marketing is a not-so-sly way to insert one’s business interests into these types of conversations I enjoy for the purposes of driving metrics, sales, and series A’s. I hear that taxation and billing and collection are complex. Try to implement WA State destination based sales tax for digital only goods. Where is the relevant tax district? It doesn’t align with zip codes or congressional districts. Municipalities move and reshape the landscape like rivers do. Is a billing address representative of the location of the purchasing party for tax purposes? We took this to a tax judge in 2014, and their final response was ‘do your best’. I would love to read a blog post with such a war story- about parsing the tax rate soap API into a relational database that can not slow down a checkout page. About how time zones affect rate calculations. Where rate file updates accidentally modified past transactions and how to model data to prevent that. I don’t want to hear something is hard so you should buy my services. Not here. Not on a call to my cell phone. Not ever. Business leaders need to determine which complexity they want to their developers to tackle- what is the most valuable for profit growth and risk mitigation. If the audience is business leaders, appeal to them directly, and take it somewhere else. If it’s to appeal to software developers, it shouldn’t be attached to ulterior motives. We see through it and we resent it. To the extent that we have influence over purchasing decisions, I’m likely to punish companies who regularly engage in this behavior. Content marketing is different than creating useful content as a marketing strategy. I’ll raise Digital Ocean as an example. Their sysadmin content is top notch, and I’m happy to give them business for it out of gratitude. That is my decision. Nobody is trying to convince me of how to invest my or my company’s resources. We get it. Auth is hard. SMS is hard. Billing is hard. None of it is core to most companies core business. If you really want to make your point, show me some code to prove it. Or spend your content budget on something that helps people. Just don’t think you’re going to win me over with a sales piece on HN. You’re not. I’ve already forgotten your name, and if it does come up I’ll have nothing but a negative memory to recall. Sorry not sorry.
• jwr 4y ago
  B2B SaaS founder here, several years in: this article is right on point. Additionally, using Stripe Billing is not all it's cracked up to be, as it actually doesn't solve all my problems, while taking a chunk of revenue. I've been building parts of a billing system myself over the years and I'm considering removing the remaining external parts, because I'm not happy with their limitations (Braintree's subscriptions are really limited, and absolutely nobody does correct invoicing for my situation (EU/Poland)). Also, in a B2B SaaS you will end up with customers on totally different payment methods. Not every business has or wants to use a credit card. So you get annual billing with proforma invoices and the complexities involved with switching from invoiced billing to credit card billing and vice versa.
• ArdelleF 4y ago
  A relational database scalable for the billing data volume, should be a best answer, in term of realtime process and analysis，say complex query. Suggest an article of a big logistics company case here, not only billing, but also order, messaging, transshipment: https://en.pingcap.com/blog/how-to-reduce-cost-per-order-with-a-distributed-sql-database/ https://en.pingcap.com/blog/how-to-reduce-cost-per-order-wit...
• mtoddsmith 4y ago
  Hour 1: 10 KW used for 0.5 hour = 5 KW (10 x 0.5) Hour 2: 20 KW used for 1 hour = 20 KW (20 x 1) Hour 3: 0 KW used for 1 hour = 0 KW (0 x 1) Hour 4: 30 KW used for 0.5 hour = 15 KW (30 x 0.5) TOTAL = 40 KW used x $10 ⇒ $40 Shouldnt that be 50kw used in total? Now we know who's writing all the bugs :)
  • quantum_magpie 4y ago
    No.
• 6510 4y ago
  Billing is a nightmare for everyone but its not so bad for engineers. In general it's great to keep track of what things cost but in situations where the overhead is more expensive than the product it self we should be considering more unmetered approaches. Billing, like taxes, can be broken down into relatively simple smaller problems. You solve all of those then do a bit of maintenance. In return you get something close enough to all-you-can eat on the cheap! Automation is wonderful!
• dhzhzjsbevs 4y ago
  I'll see you're billing and raise you localization. All in? Here, have some timezones for good measure.
  • AnotherGoodName 4y ago
    Ok for the one-upmanship I raise you billing systems for telecommunications. 1000x the transactions per customer and 1000x the complexity of the business rules. Eg. An SMS on an off peak time is cheaper. If the user makes 100 or more calls a month a special rate. How do you charge data per plan? If the user roams to another country there's an entirely new set of complex billing rules.
• buro9 4y ago
  For our sins we are all destined to work on a billing system or an authentication system at some point in our careers.
• AnhTho_FR 4y ago
  As promised, the repo is available here: https://github.com/getlago/lago https://github.com/getlago/lago ! Public roadmap: https://github.com/getlago/lago/projects/2 https://github.com/getlago/lago/projects/2 Request features here: https://community.getlago.com/c/feature-requests/11 https://community.getlago.com/c/feature-requests/11 We also have a fully hosted version if anyone is interested. Looking forward to receiving your feedback!