5 ms·

There's another caveat to this, which is that even in places where usage-based pricing may be best, doing too much of it may lose the advantages. An example fr

There's another caveat to this, which is that even in places where usage-based pricing may be best, doing too much of it may lose the advantages. An example from my experience: Datadog. It makes sense that we pay for usage, after all things like logging and metrics scale in wildly different ways from product to product, and charging flat rates or per-user rates doesn't factor this in well enough. The problem is that for a feature-complete Datadog deployment, there may be 10-20 axes of pricing – how much log data, how many unique metrics, how much of your logs do you want to index, and so on. The problem comes with cost estimation. Because there are so many axes, we can't just take our size and figure out a cost estimate, we have to model out different scenarios. Does a traffic spike for us equal a logging spike? or Metrics? How does it impact second order pricing like log indexing or archival? Even when you know exactly how much, e.g. logging you generate, it's still all approximated modelling, and many teams won't have the necessary input numbers at the beginning anyway. Stripe's usage-based pricing works well because it's so direct – it's just X% of revenue. Datadog's is a pain because there are many factors, all of which are quite far removed from revenue. I understand that at some level this is required. They're pricing for compute and storage, essentially, and a cloud provider offering those wouldn't price them at a flat rate or per-user rate, but it always felt much harder than it needed to be. I'd almost have rather paid more, but a known amount. However,

323 4y ago
Why not offer two types of pricing, a flat (expensive) one and a usage-after-the-fact (cheap) one. Like in Electricity markets - you can either get market pricing (cheaper, but very volatile) or fixed price.
- judge2020 4y ago
  That most likely is an option, where DataDog (and others) will happing sign an enterprise contract with you and take more of your money than what you end up using. The issue is that you're still going to be bound to a maximum logging/metrics/etc budget, so you still end up needing to estimate your usage to avoid going over your pre-paid limits.
- jerf 4y ago
  Let's take the simplest example of flat pricing for a logging service: You pay $X and you get all the service you can consume. A salesperson tries to intelligently set $X for you. We'll even spot the salesperson full knowledge of the previous year of usage, and we'll even stipulate that the purchasing company is 100% honest about their intent to consume, but that they do not have a 100% accurate prediction of the future. The problem here is the statistical distribution of the usage will bankrupt you as a company if you try that. People mentally want to model everything as nice, polite distributions through a combination of all the statistics classes they've ever taken using them (where the "uniform" distribution is the default and "gaussian" is if you want to get fancy) and internal cognitive biases around the nice distributions generally encountered in most real life and a general lack of experience with pathological ones, but the real distribution of consumption is grotesquely pathological and has huge spikes in the tail. The probability of one of your 10 biggest customers, which collectively account for ~90% of your business, getting an unexpected 10x or greater spike, isn't negligible like a naive high school statistical analysis might suggest. It is instead virtually certain. I've simplified this problem down just to concentrate on that bare statistical problem, which is something engineers should try to develop an intuition for. However, as we de-simplify the problem by re-introducing back all the real world complications, they all tend to make this problem even worse. For instance, give a company an all-they-can-eat deal, and if they accidentally spike your usage because the intern accidentally committed a debug logging message that spiked their usage by 50x, they have no particular incentive to fix that. Naturally, in the real world you'd put limits in the contract... but if you think about it, flat pricing with limits that you can realistically hit because the limits really ought to be thought of in logarithmic terms, rather than linear ones, is just usage based billing with extra steps. I've also taken advantage of the local topic of conversation being a logging service. Other businesses are not necessarily so pathologically distributed. Email is more reasonable to charge per seat, partially because people are used to the pathological part of the resource consumption distribution being cut off... or to put it in the terms everyone is used to, people are used to being told they can't attached a 3.5 terabyte attachment to their email and that emails to a few thousand people at a time generally cause trouble. Since the number of emails an account can use is bound to some degree by human attention and the size and therefore resource consumption of the email is bound by size limits, it tends to be less pathological. Email also permits throttling, and indeed, if someone is suddenly receiving millions of emails per hour they may be happy that you are throttling their inbox. Although in the end it's probably still more pathological than you may intuit. Even here though, when you return the real world back into the picture it doesn't mean a flat pricing model perfectly works. Give them an $X/month all-you-can-eat, and one of your customers will find a way to provide email addresses to every resident in a country or something. Put limits on that and it's just usage with extra steps. There is a sense in which usage-based-billing is the only real option, it's just a matter of the details of how you price and market it.
stingraycharles 4y ago
> The problem comes with cost estimation. Because there are so many axes, we can't just take our size and figure out a cost estimate, we have to model out different scenarios. Does a traffic spike for us equal a logging spike? In the case of Datadog it’s even worse that they don’t allow you to cap costs. If you have an accidental spike (e.g. some deployment gone haywire and relaunching containers and suddenly you have 100x the amount of containers as usual in a 1 hour time span), they will not refund it. I had to plea a lot for me as a solo developer, and they ended up locking me into a 1 year contract in order to be reimbursed. I’m convinced it’s entirely deliberate and they prey upon these types of things. It was a terrible, hostile experience, because I had to pay a huge sum I did not get any value out of. I was so scared of using their product for the rest of the year that I simply wrote off the yearly subscription as a loss.
- switch007 4y ago
  > I’m convinced it’s entirely deliberate I'm sure it is too. My colleagues are investigating Datadog as a vendor and I think they're off their heads. We're meant to be tightening our belts and I think we'll waste a metric fuck ton of money on Datadog for a gram of value. Hopefully we don't go with them.
  - danpalmer 4y ago
    In my experience, Datadog requires non-trivial active account management internally to ensure a cost effective usage, rather like a cloud infrastructure provider. If you've got 20+ people, it's a no-brainer, the value is immense. Fewer than that and it's a tricky call. It's a great product though, and it's in my list of "things I'd sign up for immediately" if starting a new company.
  - pranay01 4y ago
    You might want to have a look at SigNoz, https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz
- danpalmer 4y ago
  > e.g. some deployment gone haywire and relaunching containers and suddenly you have 100x the amount of containers as usual in a 1 hour time span To be fair to them, I believe they bill at something like a 98% high water mark for the month, so you can spike briefly like this. I completely agree with the point though.
philstephenson 4y ago
It's true that with Datadog, a spike in your usage can have a dramatic impact on your bill. This feels true though of any usage model. We've all read stories about out of control AWS bills. Datadog does at least provide some guardrails to prevent things from getting out of control: https://docs.datadoghq.com/account_management/billing/usage_metrics/ https://docs.datadoghq.com/account_management/billing/usage_... With these usage metrics and a little bit of automation (for example the Webhooks integration: https://docs.datadoghq.com/integrations/webhooks/ https://docs.datadoghq.com/integrations/webhooks/), you could stop shipping telemetry at a certain threshold. Disclosure: I work at Datadog.
- danpalmer 4y ago
  My issue isn't necessarily that a usage spike impacts the bill, that's fine! The problem is that because of the billing complexity, it's hard to predict how billing will scale with usage. There are just too many axes. Even steady-state billing is hard to price out before signing up, so much so that the answer we received from sales was to just try it and see what the bill is. This becomes a road block. "I'd like to move us to Datadog! – Oh yeah, how much does it cost? – I won't know until after we've moved." – these conversations are hard to have internally. The product is fantastic, I'm a big fan. And the total price isn't necessarily bad, it's just opaque, and required a lot of work on my side to model out the pricing, present the business case, get sign-off, and manage the contract over the years. Not something that I, as one of ~8 engineers, wanted to spend my time doing.
drewda 4y ago
> Stripe's usage-based pricing works well because it's so direct – it's just X% of revenue. For what it's worth, that's how Stripe Payments works. But if you want to also use billing, invoicing, other additional functionality, then it's X% + Y% + $Z/month.
- danpalmer 4y ago
  You're right, but they're all typically fractions of the revenue figure, so really very easy to price out. They may not be worth it for some businesses, but my point isn't about price, it's about the complexity in figuring out that price.
  - drewda 4y ago
    Yeah, agreed that it's all along the "dimension" of revenue. But the details of how Stripe prices its add-on functionality is different than competitors like Chargebee or Recurly (for subscription management) or QuickBooks Online (for invoicing), so it does make it hard to make "apples to apples" price comparisons.
indymike 4y ago
As a rule, if I can't understand the pricing in two minutes, I do not do business with a vendor. Also, if there is not a stop-loss, I view the vendor as predatory.

323 4y ago

Why not offer two types of pricing, a flat (expensive) one and a usage-after-the-fact (cheap) one. Like in Electricity markets - you can either get market pricing (cheaper, but very volatile) or fixed price.

judge2020 4y ago
That most likely is an option, where DataDog (and others) will happing sign an enterprise contract with you and take more of your money than what you end up using. The issue is that you're still going to be bound to a maximum logging/metrics/etc budget, so you still end up needing to estimate your usage to avoid going over your pre-paid limits.
jerf 4y ago
Let's take the simplest example of flat pricing for a logging service: You pay $X and you get all the service you can consume. A salesperson tries to intelligently set $X for you. We'll even spot the salesperson full knowledge of the previous year of usage, and we'll even stipulate that the purchasing company is 100% honest about their intent to consume, but that they do not have a 100% accurate prediction of the future. The problem here is the statistical distribution of the usage will bankrupt you as a company if you try that. People mentally want to model everything as nice, polite distributions through a combination of all the statistics classes they've ever taken using them (where the "uniform" distribution is the default and "gaussian" is if you want to get fancy) and internal cognitive biases around the nice distributions generally encountered in most real life and a general lack of experience with pathological ones, but the real distribution of consumption is grotesquely pathological and has huge spikes in the tail. The probability of one of your 10 biggest customers, which collectively account for ~90% of your business, getting an unexpected 10x or greater spike, isn't negligible like a naive high school statistical analysis might suggest. It is instead virtually certain. I've simplified this problem down just to concentrate on that bare statistical problem, which is something engineers should try to develop an intuition for. However, as we de-simplify the problem by re-introducing back all the real world complications, they all tend to make this problem even worse. For instance, give a company an all-they-can-eat deal, and if they accidentally spike your usage because the intern accidentally committed a debug logging message that spiked their usage by 50x, they have no particular incentive to fix that. Naturally, in the real world you'd put limits in the contract... but if you think about it, flat pricing with limits that you can realistically hit because the limits really ought to be thought of in logarithmic terms, rather than linear ones, is just usage based billing with extra steps. I've also taken advantage of the local topic of conversation being a logging service. Other businesses are not necessarily so pathologically distributed. Email is more reasonable to charge per seat, partially because people are used to the pathological part of the resource consumption distribution being cut off... or to put it in the terms everyone is used to, people are used to being told they can't attached a 3.5 terabyte attachment to their email and that emails to a few thousand people at a time generally cause trouble. Since the number of emails an account can use is bound to some degree by human attention and the size and therefore resource consumption of the email is bound by size limits, it tends to be less pathological. Email also permits throttling, and indeed, if someone is suddenly receiving millions of emails per hour they may be happy that you are throttling their inbox. Although in the end it's probably still more pathological than you may intuit. Even here though, when you return the real world back into the picture it doesn't mean a flat pricing model perfectly works. Give them an $X/month all-you-can-eat, and one of your customers will find a way to provide email addresses to every resident in a country or something. Put limits on that and it's just usage with extra steps. There is a sense in which usage-based-billing is the only real option, it's just a matter of the details of how you price and market it.

stingraycharles 4y ago

> The problem comes with cost estimation. Because there are so many axes, we can't just take our size and figure out a cost estimate, we have to model out different scenarios. Does a traffic spike for us equal a logging spike? In the case of Datadog it’s even worse that they don’t allow you to cap costs. If you have an accidental spike (e.g. some deployment gone haywire and relaunching containers and suddenly you have 100x the amount of containers as usual in a 1 hour time span), they will not refund it. I had to plea a lot for me as a solo developer, and they ended up locking me into a 1 year contract in order to be reimbursed. I’m convinced it’s entirely deliberate and they prey upon these types of things. It was a terrible, hostile experience, because I had to pay a huge sum I did not get any value out of. I was so scared of using their product for the rest of the year that I simply wrote off the yearly subscription as a loss.

switch007 4y ago
> I’m convinced it’s entirely deliberate I'm sure it is too. My colleagues are investigating Datadog as a vendor and I think they're off their heads. We're meant to be tightening our belts and I think we'll waste a metric fuck ton of money on Datadog for a gram of value. Hopefully we don't go with them.
- danpalmer 4y ago
  In my experience, Datadog requires non-trivial active account management internally to ensure a cost effective usage, rather like a cloud infrastructure provider. If you've got 20+ people, it's a no-brainer, the value is immense. Fewer than that and it's a tricky call. It's a great product though, and it's in my list of "things I'd sign up for immediately" if starting a new company.
- pranay01 4y ago
  You might want to have a look at SigNoz, https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz
danpalmer 4y ago
> e.g. some deployment gone haywire and relaunching containers and suddenly you have 100x the amount of containers as usual in a 1 hour time span To be fair to them, I believe they bill at something like a 98% high water mark for the month, so you can spike briefly like this. I completely agree with the point though.

philstephenson 4y ago

It's true that with Datadog, a spike in your usage can have a dramatic impact on your bill. This feels true though of any usage model. We've all read stories about out of control AWS bills. Datadog does at least provide some guardrails to prevent things from getting out of control: https://docs.datadoghq.com/account_management/billing/usage_metrics/ https://docs.datadoghq.com/account_management/billing/usage_... With these usage metrics and a little bit of automation (for example the Webhooks integration: https://docs.datadoghq.com/integrations/webhooks/ https://docs.datadoghq.com/integrations/webhooks/), you could stop shipping telemetry at a certain threshold. Disclosure: I work at Datadog.

danpalmer 4y ago
My issue isn't necessarily that a usage spike impacts the bill, that's fine! The problem is that because of the billing complexity, it's hard to predict how billing will scale with usage. There are just too many axes. Even steady-state billing is hard to price out before signing up, so much so that the answer we received from sales was to just try it and see what the bill is. This becomes a road block. "I'd like to move us to Datadog! – Oh yeah, how much does it cost? – I won't know until after we've moved." – these conversations are hard to have internally. The product is fantastic, I'm a big fan. And the total price isn't necessarily bad, it's just opaque, and required a lot of work on my side to model out the pricing, present the business case, get sign-off, and manage the contract over the years. Not something that I, as one of ~8 engineers, wanted to spend my time doing.

drewda 4y ago

> Stripe's usage-based pricing works well because it's so direct – it's just X% of revenue. For what it's worth, that's how Stripe Payments works. But if you want to also use billing, invoicing, other additional functionality, then it's X% + Y% + $Z/month.

danpalmer 4y ago
You're right, but they're all typically fractions of the revenue figure, so really very easy to price out. They may not be worth it for some businesses, but my point isn't about price, it's about the complexity in figuring out that price.
- drewda 4y ago
  Yeah, agreed that it's all along the "dimension" of revenue. But the details of how Stripe prices its add-on functionality is different than competitors like Chargebee or Recurly (for subscription management) or QuickBooks Online (for invoicing), so it does make it hard to make "apples to apples" price comparisons.

indymike 4y ago

As a rule, if I can't understand the pricing in two minutes, I do not do business with a vendor. Also, if there is not a stop-loss, I view the vendor as predatory.