5 ms·

We're neck deep in figuring out usage based billing for Fly.io. What we're doing is pretty extreme, and may not be representative of many other people, but it f

by mrkurt 3y ago

We're neck deep in figuring out usage based billing for Fly.io. What we're doing is pretty extreme, and may not be representative of many other people, but it feels like no one's really solved this in a way that makes sense for us. We run millions of tiny VMs. Each gets billed on a number of dimensions: egress, runtime (per cpu / memory combo), storage / io. We also have other metered services: ssl certificates, IP addresses, etc. The thing is, we _already_ have metrics for everything we want to bill. They're in a time series DB (VictoriaMetrics, in this case). Sending a shit ton of events to yet-another-system is complicated and brittle. Your k8s pods example is a good source of my hives. Anything that runs on a timer to fire off events is going to get bogged down and lose data at our scale. And we're not very big! This is a somewhat solved problem for metrics stacks. It's relatively straightforward to scale metrics scrapes. And once we have that infra, it's pretty easy to just start pushing "events" to it when events become useful. We don't end up needing Kafka and its complexity. My dream for billing is: a query layer on top of a time series database that takes data I already have and turns it into invoices. One thing about your post that struck me – the last mile to billing and reporting is the thing we're most interesting in buying. It's less specialized. There also aren't any products out there that have really figured this out, I don't think (we've evaluated pretty much all of them). Usage tracking and reporting is a thing we're ok building, because it's core to our product.

ylere 3y ago
Fixing the missing links in the billing chain is what we have been working on at Wingback (https://wingback.com https://wingback.com). Agree that the actual metering part is not that hard to get (and having a standard here would be great!), but using it to correctly enforce entitlements in your product and accurately bill your customers who might be on all kind of different plans, models prices and discounts over a large number of metered (and umetered) features is.
kshitijgrover 3y ago
At Orb (https://www.withorb.com/ https://www.withorb.com/) we’ve worked with a bunch of infra customers that are in a similar situation to what you’re describing. Generally, usage tracking and usage reporting is not the hard part. We think that giving you a full SQL layer on top of raw events is very powerful, so we provide direct SQL access to your events. Naturally, we use a real-time time series database under the hood to power that. We’re working hard to make it easy to send data to us (e.g. just drop it in S3). Granted, this isn’t ideal per your note of already throwing it into VictoriaMetrics, but we’re finding that often times companies don’t have the optimized in-house infra to run the sort of usage metric queries to go from raw events -> invoice, especially when you get to >100k events/second scale. It often makes sense for us to control the storage (and sharding) format — and we can bring those costs down over time. Given the usage events and the SQL queries over them, we then focus a bunch on making the core billing interactions work well, do invoice delivery for you (example: https://invoices.withorb.com/view?token=IlJwOVRSdWhEQ3NiRUVHbXMi.aidEm0HVom6qoRlsT0_ZoHEur-U https://invoices.withorb.com/view?token=IlJwOVRSdWhEQ3NiRUVH...), and make it easy for you to do financial reporting & rev-rec (honestly, this is its own rabbithole — but we find it’s extremely outside the core competency of eng/product teams)
- hekike 3y ago
  I think it's substantial risk for the core business. To be able to glue usage to invoice, Orb has to become the source of truth and billing engine top of Stripe, while Stripe is still the payment gateway. This creates a fragmented world where the customer has to use both systems simultaneously; this is challenging especially that Stripe ships features fast and frequently, leaving the customer exposed and vendor locked.
tonyhb 3y ago
Hi Kurt! We do the same metrics type thing at Inngest, and we share the exact same thoughts as you. We 'solved' it by, well, using Inngest to charge for Inngest. We track everything as metrics, then hook up functions to read from the TSDB each billing period. How it works: - One Inngest function runs each billing period (scheduled), and "fans out" by creating an event for each user we want to charge. - A billing function reacts to these individual user events, queries metrics, queries our plans, and then generates a stripe invoice - This is then charged by stripe. - We have other functions that respond to other billing events and do dunning like flows. We only have to focus on the small amount of billing code required. We don't have to handle scheduling, state, retries, or the event flows — that's built into Inngest. I'd really like to make this more robust. We had to put in the work on emails, billing, and dunning, even if it is simple function code. It would be nice to have someone "productize" this. I feel your pain. In general, it definitely makes sense to read from our own metrics stores to charge for our products. Very much agree with everything you've said. There's not the most joy in billing.
- hekike 3y ago
  Very cool, thanks for sharing. I like this reporting approach, once you have the usage data aggregated somewhere a state machine is a great way to report usage to Stripe. Do you trigger the Inngest billing fan out end of the month CRON style or when Stripe changes billing period and triggers a webhook to Inngest?
dangoodmanUT 3y ago
Yeah pulsing should always be a last resort, never works at scale
michaelt 3y ago
> Your k8s pods example is a good source of my hives. Anything that runs on a timer to fire off events is going to get bogged down and lose data I'd have said if you want to gracefully deal with a system that loses events, a timer-based system can ensure the errors from lost events will always be in your customer's favour. The last thing you'd want is to start the taximeter with a start-pod event, then lose the stop-pod event and end up billing the customer for nothing, forever.
johndbritton 3y ago
A couple things to check out if you haven't already seen them: * Lago - https://www.getlago.com https://www.getlago.com * Octane - https://www.getoctane.io https://www.getoctane.io * Metronome - https://metronome.com https://metronome.com
- hekike 3y ago
  Those are all solid products for billing. Tough none of them are both open-source and have the scalability of streaming aggregation. Even Lago is AGPL which can be challenging for some.
hekike 3y ago
That's a great insight, thanks for sharing. Would it be the ideal solution code / config driven, something like: SELECT user, SUM(runtime) FROM usage => invoice_template(Z) Wdyt? Btw how do you handle idempotency with time series? I think replays and least-once delivery in distributed systems can be challenging around usage collection if you don't have a key to deduplicate by.
- mrkurt 3y ago
  I may not be interpreting your question right, but most of our metrics are counters. It doesn't matter how often they get scraped, or if they get scraped twice, because we only care about the increase. Almost everything we track is aggregated with `increase()`: https://prometheus.io/docs/prometheus/latest/querying/functions/#increase https://prometheus.io/docs/prometheus/latest/querying/functi... I'm fond of SQL, but I'd be ok with some other way to express "aggregate my usage into an invoice".
  - andrastoth 3y ago
    Have you encountered any challenges related to metric aggregation? To the best of my understanding, such issues often stem from inherent inaccuracies due to sampling (https://www.youtube.com/watch?v=67Ulrq6DxwA&t=175s https://www.youtube.com/watch?v=67Ulrq6DxwA&t=175s). I'm curious if using a smaller window for metric aggregation and routing these events to a stream aggregator could provide a possible solution.
    - mrkurt 3y ago
      Not really, but we might've just implicitly designed around it. Continuously aggregating smaller windows and sending them off is roughly how we send usage to Stripe right now (which, let's be clear, is not working for us AT ALL). We've resigned ourselves to having to do the same thing if we want to use a billing SaaS, but I don't love it.
      - jmacd 3y ago
        We’ve built https://GitHub.com/tierrun/tier https://GitHub.com/tierrun/tier with a local sidecar that can talk directly to stripe to help simplify situations like this. A pure SaaS model for billing doesn’t do much more for you at then end of the day other than to apply billing ratings and generate an invoice (which Stripe can and is doing for you already). I’d love to learn more about what you’d like to do with Stripe and how you are handling entitlements, plan versioning, etc
deleted 3y ago
[deleted]
strawberrysauce 3y ago
We at zenskar.com are building a billing platform that can plug into your metering store (in addition to the traditional api way) and generate invoices for your customers. We have a query layer that can be used to define aggregates that in turn can be used to create invoices. I will reach out to you on email to see if we can solve your problems.

by mrkurt 3y ago

ylere 3y ago

Fixing the missing links in the billing chain is what we have been working on at Wingback (https://wingback.com https://wingback.com). Agree that the actual metering part is not that hard to get (and having a standard here would be great!), but using it to correctly enforce entitlements in your product and accurately bill your customers who might be on all kind of different plans, models prices and discounts over a large number of metered (and umetered) features is.

kshitijgrover 3y ago

At Orb (https://www.withorb.com/ https://www.withorb.com/) we’ve worked with a bunch of infra customers that are in a similar situation to what you’re describing. Generally, usage tracking and usage reporting is not the hard part. We think that giving you a full SQL layer on top of raw events is very powerful, so we provide direct SQL access to your events. Naturally, we use a real-time time series database under the hood to power that. We’re working hard to make it easy to send data to us (e.g. just drop it in S3). Granted, this isn’t ideal per your note of already throwing it into VictoriaMetrics, but we’re finding that often times companies don’t have the optimized in-house infra to run the sort of usage metric queries to go from raw events -> invoice, especially when you get to >100k events/second scale. It often makes sense for us to control the storage (and sharding) format — and we can bring those costs down over time. Given the usage events and the SQL queries over them, we then focus a bunch on making the core billing interactions work well, do invoice delivery for you (example: https://invoices.withorb.com/view?token=IlJwOVRSdWhEQ3NiRUVHbXMi.aidEm0HVom6qoRlsT0_ZoHEur-U https://invoices.withorb.com/view?token=IlJwOVRSdWhEQ3NiRUVH...), and make it easy for you to do financial reporting & rev-rec (honestly, this is its own rabbithole — but we find it’s extremely outside the core competency of eng/product teams)

hekike 3y ago
I think it's substantial risk for the core business. To be able to glue usage to invoice, Orb has to become the source of truth and billing engine top of Stripe, while Stripe is still the payment gateway. This creates a fragmented world where the customer has to use both systems simultaneously; this is challenging especially that Stripe ships features fast and frequently, leaving the customer exposed and vendor locked.

tonyhb 3y ago

Hi Kurt! We do the same metrics type thing at Inngest, and we share the exact same thoughts as you. We 'solved' it by, well, using Inngest to charge for Inngest. We track everything as metrics, then hook up functions to read from the TSDB each billing period. How it works: - One Inngest function runs each billing period (scheduled), and "fans out" by creating an event for each user we want to charge. - A billing function reacts to these individual user events, queries metrics, queries our plans, and then generates a stripe invoice - This is then charged by stripe. - We have other functions that respond to other billing events and do dunning like flows. We only have to focus on the small amount of billing code required. We don't have to handle scheduling, state, retries, or the event flows — that's built into Inngest. I'd really like to make this more robust. We had to put in the work on emails, billing, and dunning, even if it is simple function code. It would be nice to have someone "productize" this. I feel your pain. In general, it definitely makes sense to read from our own metrics stores to charge for our products. Very much agree with everything you've said. There's not the most joy in billing.

hekike 3y ago
Very cool, thanks for sharing. I like this reporting approach, once you have the usage data aggregated somewhere a state machine is a great way to report usage to Stripe. Do you trigger the Inngest billing fan out end of the month CRON style or when Stripe changes billing period and triggers a webhook to Inngest?

dangoodmanUT 3y ago

Yeah pulsing should always be a last resort, never works at scale

michaelt 3y ago

> Your k8s pods example is a good source of my hives. Anything that runs on a timer to fire off events is going to get bogged down and lose data I'd have said if you want to gracefully deal with a system that loses events, a timer-based system can ensure the errors from lost events will always be in your customer's favour. The last thing you'd want is to start the taximeter with a start-pod event, then lose the stop-pod event and end up billing the customer for nothing, forever.

johndbritton 3y ago

A couple things to check out if you haven't already seen them: * Lago - https://www.getlago.com https://www.getlago.com * Octane - https://www.getoctane.io https://www.getoctane.io * Metronome - https://metronome.com https://metronome.com

hekike 3y ago
Those are all solid products for billing. Tough none of them are both open-source and have the scalability of streaming aggregation. Even Lago is AGPL which can be challenging for some.

hekike 3y ago

That's a great insight, thanks for sharing. Would it be the ideal solution code / config driven, something like: SELECT user, SUM(runtime) FROM usage => invoice_template(Z) Wdyt? Btw how do you handle idempotency with time series? I think replays and least-once delivery in distributed systems can be challenging around usage collection if you don't have a key to deduplicate by.

mrkurt 3y ago
I may not be interpreting your question right, but most of our metrics are counters. It doesn't matter how often they get scraped, or if they get scraped twice, because we only care about the increase. Almost everything we track is aggregated with `increase()`: https://prometheus.io/docs/prometheus/latest/querying/functions/#increase https://prometheus.io/docs/prometheus/latest/querying/functi... I'm fond of SQL, but I'd be ok with some other way to express "aggregate my usage into an invoice".
- andrastoth 3y ago
  Have you encountered any challenges related to metric aggregation? To the best of my understanding, such issues often stem from inherent inaccuracies due to sampling (https://www.youtube.com/watch?v=67Ulrq6DxwA&t=175s https://www.youtube.com/watch?v=67Ulrq6DxwA&t=175s). I'm curious if using a smaller window for metric aggregation and routing these events to a stream aggregator could provide a possible solution.
  - mrkurt 3y ago
    Not really, but we might've just implicitly designed around it. Continuously aggregating smaller windows and sending them off is roughly how we send usage to Stripe right now (which, let's be clear, is not working for us AT ALL). We've resigned ourselves to having to do the same thing if we want to use a billing SaaS, but I don't love it.
    - jmacd 3y ago
      We’ve built https://GitHub.com/tierrun/tier https://GitHub.com/tierrun/tier with a local sidecar that can talk directly to stripe to help simplify situations like this. A pure SaaS model for billing doesn’t do much more for you at then end of the day other than to apply billing ratings and generate an invoice (which Stripe can and is doing for you already). I’d love to learn more about what you’d like to do with Stripe and how you are handling entitlements, plan versioning, etc

deleted 3y ago

[deleted]

strawberrysauce 3y ago

We at zenskar.com are building a billing platform that can plug into your metering store (in addition to the traditional api way) and generate invoices for your customers. We have a query layer that can be used to define aggregates that in turn can be used to create invoices. I will reach out to you on email to see if we can solve your problems.