4 ms·

In your article or in the posts here you have not said why you think its a crazy idea.

by ustad 2y ago

tptacek 2y ago
I've said it over and over again: if you have a serious app, and someone somehow steals a credential from you and uses it to light up a bunch of crypto miners, you don't want us shutting down your main app in response. We perceive it mostly as a feature that will blow people up. We agree about the underlying problem! You don't want to spend $5000 in a month for services you never wanted. We don't want you spending that either. We'd rather just improve our billing so that you can fix this after the fact without trading off availability. But I have just planted a flag: we are prepared to be wrong about this. I don't think we are, but like, I'm the only one. :)
- ustad 2y ago
  Crypto miners is the reason for not having a “crazy” option for “cap on/off” at sign up? You know what you can’t fix after the fact of getting a huge bill? a heart attack.
  - tptacek 2y ago
    This post is about us not giving you that heart attack. I'm pushing back, and I'm going to keep pushing back; if we do this feature, it is going to be kicking and screaming. :)
- bee_rider 2y ago
  I think your position is essentially that you mostly want to serve people who have serious apps, for whom the cap idea doesn’t work. And you definitely know more about the general trajectory of your customers than all of us sitting here randomly speculating. But, is it really so uncommon for an application to transition from unserious, I want to cap, to serious? I guess I’m slightly confused because I thought one of the nice things about cloud services is that they give you the ability to fit your infrastructure to your size while you are trying things. If I’m still trying things, I might not even know if I’m serious yet, right?
  - tptacek 2y ago
    Sure, but bear in mind that across the entire industry of public clouds, which has been around for something like 2 decades now, nobody really has this feature. In fact, didn't Google have this feature and then pull it?
    - bee_rider 2y ago
      Yes, clearly is it not the convention to provide this feature. But the chain of thought seems so straightforward and obvious to me: cloud infrastructure is great for exploring what you’ll be doing, and guardrails make you more willing to explore fearlessly. I’m obviously missing something, though!
      - Aeolun 2y ago
        It’s also easy to see why the convention would be to not have it. It earns the provider more money. After all, not every customer will ask for forgiveness for a surprise bill.
        tptacek 2y ago
        This is not why we don't have caps. Maybe it's why AWS doesn't. This simply isn't how we make our money. Our top line is dominated by companies growing businesses with us. Getting a hobbyist to cough up an extra $100 doesn't move a single dial for us.
        Aeolun 2y ago
        I didn’t mean to imply you did (or even AWS for that matter). Just that from the consumers perspective it’s impossible to see/know the difference.
    - myaccountonhn 2y ago
      I've def. wanted to have spending caps on a lot of my projects. The ones I've tried: AWS and GCP made this ridiculously difficult. It might be that they will waiver the bill, but the general feeling that they get to decide if I go bankrupt or not is nightmare fuel. In the end, I've just set up a 5$ vps where I self-host all my apps. That removes all the stress.
- belthesar 2y ago
  I'll be honest, I think you've got an uphill battle to convince so many of us that are concerned about accidental pricing that this is exactly the thing we actually want, vs. what we feel we want. This position parrots what Vercel's leadership said when someone had a massive surprise bill, and the story made the rounds both here and elsewhere. To be super honest, you might be right too. You might go down a huge engineering effort to build this in only for 5% of your customers to ever engage it. I think the real question is what percentage of your customers will feel better knowing that the have the choice to set those limits, and how will that comfort actually improve their trust with Fly, and cause them to choose it over another cloud provider. It may end up being a lot like this feature we had in a platform that I used to support. It had a just-in-time analytics pipeline that at one point required tens of thousands of dollars in compute, storage and network hardware alone to function. Based on our analytics, it was barely used compared to the usage of the rest of our app, which made the zounds of resources and fairly frequent support attention it needed feel silly in comparison, so I advocated to sunset it. Product assured me that, regardless of how silly it might be to continue supporting this feature, it was a dealmaker, and losing it would be a dealbreaker. So yeah, y'all might be right in that the majority of your customers don't actually want it. But maybe what they do need is to know that it's there, ready for them if they ever need to engage with it.
  - dlisboa 2y ago
    > You might go down a huge engineering effort to build this This is an overlooked issue: billing caps are hard to implement and will likely incur losses for the cloud company that does. Take an object storage service as an example. Imagine Company X has a hard cap at US$ 1000 but some bug makes their software upload millions of files to a bucket and rack up their bill. Since objects are charged in GB/month they will not reach the cap until some time later that month. Then, when they do, what does termination of service mean? Does the cloud provider destroy every last resource associated with the account the second the hard cap is reached? If they don't, and they still have to store those files somewhere in their infra, then they'll start taking a loss while Company X just says "oops, sorry". That's what tptacek is talking about: you want to NOT destroy the customers' resources because they can quickly figure out that something went wrong and then adjust while still maintaining service. But the longer you keep the resources the more you're paying out of pocket as a cloud provider. If you can't bill the overages to the customer, which a hard cap would imply, then you're at a loss. Reclaiming every resource associated to an account the moment a cap is reached is an extreme measure no one wants. A hard cap then becomes only a "soft" cap, a mere suggestion, and cloud providers would then say "you hit the cap, but we had to keep your resources on the books for 12 hours, so here's the supplemental overages charges". Which would lead to probably just as many charges disputes we have today.
- lilyball 2y ago
  I'm pretty sure I'm fairly far from your target customer, given that I'm not building products, and right now my only usage of fly.io is to host a tiny app that expects to be used a few times per year by my spouse and me. Having said that, the reason why I personally would be happier with the ability to set a hard cap is that if I'm going to put a project on fly.io, I'm spending my own personal money on it. If I'm spending my own personal money on it, I want a guarantee that it cannot possibly cost more than a given amount. When it's my own money on the line, I absolutely want the service to shut down rather than have even 0.01% chance of costing me a lot of money. The moment I'm actually building something for a business, the moment it's company money instead of personal money, then priorities change and everything you're saying makes sense. But as long as I'm just a single developer playing with stuff and billing it to my personal credit card, I want that guarantee that I won't accidentally make myself go broke.
- School-Cotton 2y ago
  Giving people the ability to use caps as an optional feature doesn’t force anyone to use it. Also, you could turn things off in the inverse order they were turned on until the cap is satisfied. So all the crypto miner instances would be turned off before database backups being deleted.
  - Aeolun 2y ago
    When you go over the cap everything turns off, otherwise, what’s the point? Should you turn things off until you reach a certain spend rate? Then you can set your cap high enough for your server +1, but no more.
    - School-Cotton 2y ago
      I was imagining the cap would be a rate, not a total.

In your article or in the posts here you have not said why you think its a crazy idea.

by ustad 2y ago

In your article or in the posts here you have not said why you think its a crazy idea.

I've said it over and over again: if you have a serious app, and someone somehow steals a credential from you and uses it to light up a bunch of crypto miners, you don't want us shutting down your main app in response. We perceive it mostly as a feature that will blow people up. We agree about the underlying problem! You don't want to spend $5000 in a month for services you never wanted. We don't want you spending that either. We'd rather just improve our billing so that you can fix this after the fact without trading off availability. But I have just planted a flag: we are prepared to be wrong about this. I don't think we are, but like, I'm the only one. :)

ustad 2y ago
Crypto miners is the reason for not having a “crazy” option for “cap on/off” at sign up? You know what you can’t fix after the fact of getting a huge bill? a heart attack.
- tptacek 2y ago
  This post is about us not giving you that heart attack. I'm pushing back, and I'm going to keep pushing back; if we do this feature, it is going to be kicking and screaming. :)
bee_rider 2y ago
I think your position is essentially that you mostly want to serve people who have serious apps, for whom the cap idea doesn’t work. And you definitely know more about the general trajectory of your customers than all of us sitting here randomly speculating. But, is it really so uncommon for an application to transition from unserious, I want to cap, to serious? I guess I’m slightly confused because I thought one of the nice things about cloud services is that they give you the ability to fit your infrastructure to your size while you are trying things. If I’m still trying things, I might not even know if I’m serious yet, right?
- tptacek 2y ago
  Sure, but bear in mind that across the entire industry of public clouds, which has been around for something like 2 decades now, nobody really has this feature. In fact, didn't Google have this feature and then pull it?
  - bee_rider 2y ago
    Yes, clearly is it not the convention to provide this feature. But the chain of thought seems so straightforward and obvious to me: cloud infrastructure is great for exploring what you’ll be doing, and guardrails make you more willing to explore fearlessly. I’m obviously missing something, though!
    - Aeolun 2y ago
      It’s also easy to see why the convention would be to not have it. It earns the provider more money. After all, not every customer will ask for forgiveness for a surprise bill.
      - tptacek 2y ago
        This is not why we don't have caps. Maybe it's why AWS doesn't. This simply isn't how we make our money. Our top line is dominated by companies growing businesses with us. Getting a hobbyist to cough up an extra $100 doesn't move a single dial for us.
        Aeolun 2y ago
        I didn’t mean to imply you did (or even AWS for that matter). Just that from the consumers perspective it’s impossible to see/know the difference.
  - myaccountonhn 2y ago
    I've def. wanted to have spending caps on a lot of my projects. The ones I've tried: AWS and GCP made this ridiculously difficult. It might be that they will waiver the bill, but the general feeling that they get to decide if I go bankrupt or not is nightmare fuel. In the end, I've just set up a 5$ vps where I self-host all my apps. That removes all the stress.
belthesar 2y ago
I'll be honest, I think you've got an uphill battle to convince so many of us that are concerned about accidental pricing that this is exactly the thing we actually want, vs. what we feel we want. This position parrots what Vercel's leadership said when someone had a massive surprise bill, and the story made the rounds both here and elsewhere. To be super honest, you might be right too. You might go down a huge engineering effort to build this in only for 5% of your customers to ever engage it. I think the real question is what percentage of your customers will feel better knowing that the have the choice to set those limits, and how will that comfort actually improve their trust with Fly, and cause them to choose it over another cloud provider. It may end up being a lot like this feature we had in a platform that I used to support. It had a just-in-time analytics pipeline that at one point required tens of thousands of dollars in compute, storage and network hardware alone to function. Based on our analytics, it was barely used compared to the usage of the rest of our app, which made the zounds of resources and fairly frequent support attention it needed feel silly in comparison, so I advocated to sunset it. Product assured me that, regardless of how silly it might be to continue supporting this feature, it was a dealmaker, and losing it would be a dealbreaker. So yeah, y'all might be right in that the majority of your customers don't actually want it. But maybe what they do need is to know that it's there, ready for them if they ever need to engage with it.
- dlisboa 2y ago
  > You might go down a huge engineering effort to build this This is an overlooked issue: billing caps are hard to implement and will likely incur losses for the cloud company that does. Take an object storage service as an example. Imagine Company X has a hard cap at US$ 1000 but some bug makes their software upload millions of files to a bucket and rack up their bill. Since objects are charged in GB/month they will not reach the cap until some time later that month. Then, when they do, what does termination of service mean? Does the cloud provider destroy every last resource associated with the account the second the hard cap is reached? If they don't, and they still have to store those files somewhere in their infra, then they'll start taking a loss while Company X just says "oops, sorry". That's what tptacek is talking about: you want to NOT destroy the customers' resources because they can quickly figure out that something went wrong and then adjust while still maintaining service. But the longer you keep the resources the more you're paying out of pocket as a cloud provider. If you can't bill the overages to the customer, which a hard cap would imply, then you're at a loss. Reclaiming every resource associated to an account the moment a cap is reached is an extreme measure no one wants. A hard cap then becomes only a "soft" cap, a mere suggestion, and cloud providers would then say "you hit the cap, but we had to keep your resources on the books for 12 hours, so here's the supplemental overages charges". Which would lead to probably just as many charges disputes we have today.
lilyball 2y ago
I'm pretty sure I'm fairly far from your target customer, given that I'm not building products, and right now my only usage of fly.io is to host a tiny app that expects to be used a few times per year by my spouse and me. Having said that, the reason why I personally would be happier with the ability to set a hard cap is that if I'm going to put a project on fly.io, I'm spending my own personal money on it. If I'm spending my own personal money on it, I want a guarantee that it cannot possibly cost more than a given amount. When it's my own money on the line, I absolutely want the service to shut down rather than have even 0.01% chance of costing me a lot of money. The moment I'm actually building something for a business, the moment it's company money instead of personal money, then priorities change and everything you're saying makes sense. But as long as I'm just a single developer playing with stuff and billing it to my personal credit card, I want that guarantee that I won't accidentally make myself go broke.
School-Cotton 2y ago
Giving people the ability to use caps as an optional feature doesn’t force anyone to use it. Also, you could turn things off in the inverse order they were turned on until the cap is satisfied. So all the crypto miner instances would be turned off before database backups being deleted.
- Aeolun 2y ago
  When you go over the cap everything turns off, otherwise, what’s the point? Should you turn things off until you reach a certain spend rate? Then you can set your cap high enough for your server +1, but no more.
  - School-Cotton 2y ago
    I was imagining the cap would be a rate, not a total.