10 ms·

1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral 2. While I've always favored erring towards keeping people's si

1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages

CogitoCogito 3y ago
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral This isn't what you said in your first post, you said: > It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact. So forgiving "lots and lots" doesn't move the needle. Do you or do you not forgive _all_ such cases where your DDOS protection doesn't take down the site? What was your employee referring to when saying that the usual discount is 20%? Are you saying that you _never_ discount 20% and instead always discount 100% i.e. "forgive"?
- meh___ 3y ago
  I don't think this will get answered, unfortunately.
Sephr 3y ago
Do the changes you are working on that will cause "the default behavior to never let free sites incur overages" involve providing users with spending limit controls? Solving this only for the free site use case doesn't address the core problem that people are bringing up about a lack of spending limit controls.
noir_lord 3y ago
Any cloud platform should have a spend-stop amount built in. i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline". Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you. This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.
- c120 3y ago
  Yes, any price that's not hard capped is unacceptable. One reason why I quit Amazon cloud after the trial year. No because they were too expensive, but there was no way to guarantee they wouldn't charge me more than planned...
- raiyu 3y ago
  We did this at DigitalOcean for similar reasons, wasn't a feature that was commonly used. Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down. What Netlify is doing here is really the best approach for both parties. And typically speaking a $104k bill would be hard to get paid up regardless if the customer's typical transaction balance was $5/mo and their credit card limit wouldn't be that high. Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
  - ohnoitsahuman 3y ago
    A debt is still a debt even charged back. There would be many attorneys interested in collecting a $105k debt.
    - quickthrower2 3y ago
      Yes if a corporation says I owe them $100k I will lose sleep. Even if they don’t have my card details.
      - dwighttk 3y ago
        The secret is to bump that debt up to $100B
        quickthrower2 3y ago
        There is a trope about how much you owe and thus whose problem it is.
oneeyedpigeon 3y ago
2. is obviously what should have always been the case, but it's good news to hear you've now gotten there. Every single hobbyist website would always choose downtime over a hundred thousand dollar charge.
- rjzzleep 3y ago
  With a properly configured nginx, you can easily serve 10's of thousands of requests a second on vserver type hardware. Netlify just offers these build pipeline kind of static site with cms UI. But this is a good reminder why my gut feeling always made me avoid these overengineered solutions.
  - hirako2000 3y ago
    They aren't engineered, they subsidize (more) enginnering effort , and (are meant to) cost less as a result. They do. But of course maximizing profit is the sole true prerogative of capitalist enterprises. And the market is not totally competitive. So yes your intuition was correct, to be cautious against over enginnered pricing to get y'a. I mean those companies cater to hobbyist. Then ... Render seems more fair-play. Until a change of mgt occurs of course.
  - mcapodici 3y ago
    It wasn't the engineering that did it, it was the egress charges. But I think the IaaS tend to charge more for this than the clouds, who in turn charge more than the "servers for rent" people.
Hamuko 3y ago
If forgiving bills for this kind of a thing is a standard practice, how come this was the customer support's first reaction: >We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225. 20% and 5% are quite a bit higher than forgiven.
- op00to 3y ago
  This needs to be much higher up.
- CogitoCogito 3y ago
  Given this has been asked here multiple times without response from /u/bobfunk, it’s hard to conclude anything except that he is lying.
  - hirako2000 3y ago
    Lying but a good talker for sure. I wouldn't want to be CEO these days. A lot are trained and paid to do damage control.
  - Jgrubb 3y ago
    Or that usage and billing is a difficult space and this particular thread has been dogpiled by a bunch of folks that have never actually worked in it.
    - CogitoCogito 3y ago
      His claims are directly contradicted by his employee's actions. When asked about this, he provides no clarification. I just don't understand why you'd consider him even remotely believable.
  - hirako2000 3y ago
    I will put there the other obvious offender: Vercel . Not sure about bandwidth, but dark patterns, keeping serious RBAC procedures we'll hidden until asking a fortune even for startups, to provide not things like SSO, just reasonable RBAC. With all that money they then can finance the free tier until they get too far and become platform locked-in. Surge.sh Im not sure. But shows all the sign of some greedy acquisition, regular long outages , as if I have been sitting as a free tier for too long, quick nudge to pay. For barely accessed sites even behind CDNs, steep. I even worry they one day just wipe all my buckets (they did for a few already) and support would recommend me to be a "normal" paying user . Nothing is free. And nothing too good to be true is true .
SXX 3y ago
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral No offence, but this sounds like "trust me bro" billing and it is not good enough. Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially. > 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages I hope you understand that chance someone who used to pay you $20 / month unlikely want to ever get $10,000 bill. Yeah people might dislike that their website went down due high traffic, but it's not gonna bring this much negative PR as incidents like this. There should be some sanity check at least.
- hnbad 3y ago
  > Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially. Not a heart attack but there has literally been at least one suicide over an unexpected life-ruining bill like that: https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-year-old-robinhood-customer-dies-by-suicide-after-seeing-a-730000-negative-balance/ https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-y...
FlyingSnake 3y ago
Regarding #2: I would rather have my hobbyist website go down rather than facing the daunting task to raise a query on HN and hope the bill goes away.
hirako2000 3y ago
1. Forgiven many, is Netlify forgiving all obvious anomalies? Is the question, which if so but you said many so it is a no, it would make you reconsider the next point 2. Favoring keeping people site up ? Would you go as far as keeping them up if they stopped paying for the meter? If not you simply should not let that meter go overboard. Hey I'm a taxi driver. Hailer fell asleep on the back, so I kept driving all night, once he woke up I dropped him to his place and asked for my monthly wage. I "forgive" many, but just a few are juicy income so I adopted the policy to never wake any customer up. If people ask I say it would be impolite, principles prime.
awill 3y ago
I’m sorry. You are working on changing things so FREE sites don’t get charged??? That’s the elephant in the room here. I understand an enterprise plan where you state billing is $xx per GB, but billing someone with a free site?? Give me a break.
charles_f 3y ago
> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral Sequence of events doesn't support this answer: 1. User gets charged 100k 2. User complains to support 3. User receives discount to 20k, then 5k. Support states policy is normally 20k 4. User discloses to the world. Goes viral. 5. Invoice is forgiven While you might forgive "lots and lots", fact is that you still presented the invoice to a free tier customer, and when they complained you gave them a discount, but still charge. Only when it went viral did you forgive it.
- ponector 3y ago
  They had forgiven lots and lots of bills, but forced to pay lots and lots more people.
- sanitycheck 3y ago
  Quite... It does seem that either the story we're getting isn't completely accurate or the support people who handled this need a little reminder of what's supposed to happen. I'm a paranoid person by nature so "It's free... just... give us your card details" is always suspicious.
  - krab 3y ago
    The issue is that you don't have to give the credit card details in this case. That charge wouldn't probably have gone through anyway.
pierat 3y ago
Give that there are free stressers/booters , and reasonable prices to rent a DDoS cloud.... https://stresser.su/#pricing https://stresser.su/#pricing 1. What are you doing to prevent DDoS's from hitting your network? 2. Why do customers have to allow an unlimited credit burden to use services? 3. Why arent there cost controls to "if $$ exceeds X, shut acct down"? Azure can do this. Long story short, why are you by default (except for social media escalation) passing fraud costs to customers?
deleted 3y ago
[deleted]
EasyMark 3y ago
You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries
HeatrayEnjoyer 3y ago
Do we have anything more binding than your word to rely on? From what I see you could change this policy tomorrow unilaterally and we would have no recourse.
- hunter2_ 3y ago
  > this policy I wouldn't think it's a binding policy at all, because the billing procedure (automatic full bill, manually discounted bill, etc.) would follow it if it were. More of a procedure.
deleted 3y ago
[deleted]
paultopia 3y ago
This seems like a really good idea to me. Or at least cap overages at a specified amount, like 2x the free level (a $55 surprise bill is a totally different universe from a $100,000 surprise bill, obviously). Honestly, this terrifies me---I run a bunch of different sites off netlify, and I would have never imagined that a site could jump from 0 to six figures of bills a month without something hitting a tripwire somewhere and cutting it off or at least communicating with the account owner. At least users should have the capacity to self-impose bandwidth caps to prevent this sort of thing.
Salgat 3y ago
Why did this happen in this case if you said it doesn't? Netlify fought to bill OP repeatedly until it went viral.
ilaksh 3y ago
But you realize that a small business or startup can't rely on "generosity" to avoid going bankrupt? It seems that significant bills appearing without warning or cut-offs is clearly intentional. I am embarrassed that I recommended Netlify before.
boeing767 3y ago
"wOrKiNg oN"

CogitoCogito 3y ago

> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral This isn't what you said in your first post, you said: > It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact. So forgiving "lots and lots" doesn't move the needle. Do you or do you not forgive _all_ such cases where your DDOS protection doesn't take down the site? What was your employee referring to when saying that the usual discount is 20%? Are you saying that you _never_ discount 20% and instead always discount 100% i.e. "forgive"?

meh___ 3y ago
I don't think this will get answered, unfortunately.

Sephr 3y ago

Do the changes you are working on that will cause "the default behavior to never let free sites incur overages" involve providing users with spending limit controls? Solving this only for the free site use case doesn't address the core problem that people are bringing up about a lack of spending limit controls.

noir_lord 3y ago

Any cloud platform should have a spend-stop amount built in. i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline". Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you. This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.

c120 3y ago
Yes, any price that's not hard capped is unacceptable. One reason why I quit Amazon cloud after the trial year. No because they were too expensive, but there was no way to guarantee they wouldn't charge me more than planned...
raiyu 3y ago
We did this at DigitalOcean for similar reasons, wasn't a feature that was commonly used. Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down. What Netlify is doing here is really the best approach for both parties. And typically speaking a $104k bill would be hard to get paid up regardless if the customer's typical transaction balance was $5/mo and their credit card limit wouldn't be that high. Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.
- ohnoitsahuman 3y ago
  A debt is still a debt even charged back. There would be many attorneys interested in collecting a $105k debt.
  - quickthrower2 3y ago
    Yes if a corporation says I owe them $100k I will lose sleep. Even if they don’t have my card details.
    - dwighttk 3y ago
      The secret is to bump that debt up to $100B
      - quickthrower2 3y ago
        There is a trope about how much you owe and thus whose problem it is.

oneeyedpigeon 3y ago

2. is obviously what should have always been the case, but it's good news to hear you've now gotten there. Every single hobbyist website would always choose downtime over a hundred thousand dollar charge.

rjzzleep 3y ago
With a properly configured nginx, you can easily serve 10's of thousands of requests a second on vserver type hardware. Netlify just offers these build pipeline kind of static site with cms UI. But this is a good reminder why my gut feeling always made me avoid these overengineered solutions.
- hirako2000 3y ago
  They aren't engineered, they subsidize (more) enginnering effort , and (are meant to) cost less as a result. They do. But of course maximizing profit is the sole true prerogative of capitalist enterprises. And the market is not totally competitive. So yes your intuition was correct, to be cautious against over enginnered pricing to get y'a. I mean those companies cater to hobbyist. Then ... Render seems more fair-play. Until a change of mgt occurs of course.
- mcapodici 3y ago
  It wasn't the engineering that did it, it was the egress charges. But I think the IaaS tend to charge more for this than the clouds, who in turn charge more than the "servers for rent" people.

Hamuko 3y ago

If forgiving bills for this kind of a thing is a standard practice, how come this was the customer support's first reaction: >We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225. 20% and 5% are quite a bit higher than forgiven.

op00to 3y ago
This needs to be much higher up.
CogitoCogito 3y ago
Given this has been asked here multiple times without response from /u/bobfunk, it’s hard to conclude anything except that he is lying.
- hirako2000 3y ago
  Lying but a good talker for sure. I wouldn't want to be CEO these days. A lot are trained and paid to do damage control.
- Jgrubb 3y ago
  Or that usage and billing is a difficult space and this particular thread has been dogpiled by a bunch of folks that have never actually worked in it.
  - CogitoCogito 3y ago
    His claims are directly contradicted by his employee's actions. When asked about this, he provides no clarification. I just don't understand why you'd consider him even remotely believable.
- hirako2000 3y ago
  I will put there the other obvious offender: Vercel . Not sure about bandwidth, but dark patterns, keeping serious RBAC procedures we'll hidden until asking a fortune even for startups, to provide not things like SSO, just reasonable RBAC. With all that money they then can finance the free tier until they get too far and become platform locked-in. Surge.sh Im not sure. But shows all the sign of some greedy acquisition, regular long outages , as if I have been sitting as a free tier for too long, quick nudge to pay. For barely accessed sites even behind CDNs, steep. I even worry they one day just wipe all my buckets (they did for a few already) and support would recommend me to be a "normal" paying user . Nothing is free. And nothing too good to be true is true .

SXX 3y ago

> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral No offence, but this sounds like "trust me bro" billing and it is not good enough. Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially. > 2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages I hope you understand that chance someone who used to pay you $20 / month unlikely want to ever get $10,000 bill. Yeah people might dislike that their website went down due high traffic, but it's not gonna bring this much negative PR as incidents like this. There should be some sanity check at least.

hnbad 3y ago
> Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially. Not a heart attack but there has literally been at least one suicide over an unexpected life-ruining bill like that: https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-year-old-robinhood-customer-dies-by-suicide-after-seeing-a-730000-negative-balance/ https://www.forbes.com/sites/sergeiklebnikov/2020/06/17/20-y...

FlyingSnake 3y ago

Regarding #2: I would rather have my hobbyist website go down rather than facing the daunting task to raise a query on HN and hope the bill goes away.

hirako2000 3y ago

1. Forgiven many, is Netlify forgiving all obvious anomalies? Is the question, which if so but you said many so it is a no, it would make you reconsider the next point 2. Favoring keeping people site up ? Would you go as far as keeping them up if they stopped paying for the meter? If not you simply should not let that meter go overboard. Hey I'm a taxi driver. Hailer fell asleep on the back, so I kept driving all night, once he woke up I dropped him to his place and asked for my monthly wage. I "forgive" many, but just a few are juicy income so I adopted the policy to never wake any customer up. If people ask I say it would be impolite, principles prime.

awill 3y ago

I’m sorry. You are working on changing things so FREE sites don’t get charged??? That’s the elephant in the room here. I understand an enterprise plan where you state billing is $xx per GB, but billing someone with a free site?? Give me a break.

charles_f 3y ago

> 1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral Sequence of events doesn't support this answer: 1. User gets charged 100k 2. User complains to support 3. User receives discount to 20k, then 5k. Support states policy is normally 20k 4. User discloses to the world. Goes viral. 5. Invoice is forgiven While you might forgive "lots and lots", fact is that you still presented the invoice to a free tier customer, and when they complained you gave them a discount, but still charge. Only when it went viral did you forgive it.

ponector 3y ago
They had forgiven lots and lots of bills, but forced to pay lots and lots more people.
sanitycheck 3y ago
Quite... It does seem that either the story we're getting isn't completely accurate or the support people who handled this need a little reminder of what's supposed to happen. I'm a paranoid person by nature so "It's free... just... give us your card details" is always suspicious.
- krab 3y ago
  The issue is that you don't have to give the credit card details in this case. That charge wouldn't probably have gone through anyway.

pierat 3y ago

Give that there are free stressers/booters , and reasonable prices to rent a DDoS cloud.... https://stresser.su/#pricing https://stresser.su/#pricing 1. What are you doing to prevent DDoS's from hitting your network? 2. Why do customers have to allow an unlimited credit burden to use services? 3. Why arent there cost controls to "if $$ exceeds X, shut acct down"? Azure can do this. Long story short, why are you by default (except for social media escalation) passing fraud costs to customers?

deleted 3y ago

[deleted]

EasyMark 3y ago

You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries

HeatrayEnjoyer 3y ago

Do we have anything more binding than your word to rely on? From what I see you could change this policy tomorrow unilaterally and we would have no recourse.

hunter2_ 3y ago
> this policy I wouldn't think it's a binding policy at all, because the billing procedure (automatic full bill, manually discounted bill, etc.) would follow it if it were. More of a procedure.

deleted 3y ago

[deleted]

paultopia 3y ago

This seems like a really good idea to me. Or at least cap overages at a specified amount, like 2x the free level (a $55 surprise bill is a totally different universe from a $100,000 surprise bill, obviously). Honestly, this terrifies me---I run a bunch of different sites off netlify, and I would have never imagined that a site could jump from 0 to six figures of bills a month without something hitting a tripwire somewhere and cutting it off or at least communicating with the account owner. At least users should have the capacity to self-impose bandwidth caps to prevent this sort of thing.

Salgat 3y ago

Why did this happen in this case if you said it doesn't? Netlify fought to bill OP repeatedly until it went viral.

ilaksh 3y ago

But you realize that a small business or startup can't rely on "generosity" to avoid going bankrupt? It seems that significant bills appearing without warning or cut-offs is clearly intentional. I am embarrassed that I recommended Netlify before.

boeing767 3y ago

"wOrKiNg oN"