8 ms·
Tell HN: I accidentally ran up a $1000 Heroku bill
This is pretty typical. Every few years something slips my mind and I end up stuck with a "Erik, you're a moron tax".
I was experiencing a spike of traffic when my site was on national news a few weeks ago, and I turned my dynos and workers way up on Heroku to handle the traffic. The spike went down, but I forgot to turn off the dynos.
Flash forward a few weeks, and I am stuck with $1300 bill on a site that I'm not making any money on. I asked Heroku if they could help me out a little... I'm a good customer, and even though I'm a cash-strapped in-the-red bootstrapper, I gladly pay them hundreds of dollars a month to run my sites, bills payed on time and in full. I was hoping they'd maybe give me a little discount... not pay for my servers, but maybe not take a profit off of my stupid mistake.
They shot me down. No refunds available, even though these were virtual servers, and didn't contribute any real load to any of their machines. Although obviously they provisioned machines for me, so they had real costs.
Part of me wants to be upset, but obviously I was the one who screwed up. Still, it leaves a bad taste in my mouth. I guess a smaller host might offer more personal care, but I wouldn't get Heroku's world-class engineering baked into their platform, which I really, really appreciate.
What do other folks think? Am I just a dumbass who lost a chunk of paycheck, or is this poor customer care?
- collint 16y agoHmm, you might be out of luck, but it does bring up the question of why Heroku doesn't have load-triggered scaling, or some sort of dead-man's switch option.
- jmonegro 16y agoThere are third-party solutions for that, like this one: http://github.com/ddollar/heroku-autoscale http://github.com/ddollar/heroku-autoscale
- aaronbrethorst 16y agoHave you been able to get that to work with Rails 3? I've tried and failed miserably.
- jmonegro 16y agoNot really. If/when I do I'll see if I patch it up if the owner hasn't already.
- andrewacove 16y agoHaving been a Heroku user for a while, and having looked into autoscaling for dynos and workers a few times (but not having really needed it yet), I really think they could do a better job of publicizing that these solutions exist. I imagine that tons of their user base (especially for small hobby sites that only occasionally, if ever, see high traffic or run background tasks) look for this, and yet there's nothing pointing to it other than old posts on the mailing list. At least put information and resources for autoscaling in the Docs section of the site. Acquiring resources only as long as you need them, and then releasing them when you're done, should be the standard service - complete user control should be the advanced feature set. They already do this for websites on a single dyno: they spin them down when they're idle for a while. But they don't do it with workers (what effect does the latency of starting a backup job up really have? it's already running in the background).
- tfe 16y agoIt's because they need to keep a process running all the time to poll the queue for new jobs. So basically, they keep an instance of your app booted at all times for job processing (if I understand how DJ workers work). And that's pretty much like a dyno, except it doesn't serve traffic. And there are no web requests to trigger it to spin up like there are for dynos.
- aaronbrethorst 16y agoOne of Heroku's employees has a fork of delayed_job that handles autoscaling workers in the background. I use it in conjunction with several hourly cron jobs and it works like a charm. http://github.com/pedro/delayed_job/tree/autoscaling http://github.com/pedro/delayed_job/tree/autoscaling
- andrewacove 16y agoYes, I was aware of this, though I haven't had the need to use it yet. But my question stands: why isn't this stuff very visible - i.e., on Heroku's DJ doc page: http://docs.heroku.com/delayed-job http://docs.heroku.com/delayed-job , or at least linked from there? Or published as an official piece of code on the Heroku github page?
- spooneybarger 16y agoI'm going to go with dumbass. Nothing personal there, but you turned up the level of service and continued to use it even though you didn't want/need it anymore. The fault is entirely on you. Think of it this way, they provisioned load to you which in turn even if you didn't use, couldnt be provisioned to others w/o potentially over provisioning.
- jw84 16y agoThat's not how clouds work.
- spooneybarger 16y agowell heroku runs on ec2 and amazon charges them for it because amazon does actually have a finite number of resources ( large but finite ) which means eventually, when you provision to someone, it means you cant to someone else without over provisioning. just because it is the cloud doesn't make it the IT version of big rock candy mountain.
- chc 16y agoYou're making the assumption that they don't overprovision at all.
- deno 16y agoMaking money on people's mistakes seems like a bad business model. Give him at least discount and you'll have an evangelist.
- sr3d 16y agoYou wrote as if Heroku were taking advantage of their customers. They don't, you pay for what you use. The poster should not even post such a self-centered post on HN to tell people to sympathize with him. He screwed up and he is a whiner. I don't want someone like that becomes an "evangelist" because all he'd try to do is to get everything his way.
- 16y ago
- patio11 16y agoHeroku runs on top of EC2, and they get charged the same amount of money by Amazon whether your instances are idle or on fire. That doesn't matter, though, because businesses do not do cost-based pricing. There comes a time in a young man's life when "haha, whoopsie, I'm a kid who is not responsible for his own decisions" is no longer sufficient for a do-over. I am not sure when that is, but if you're running a business, well, welcome to being an adult.
- jacquesm 16y agoThere comes a time in a business owners life when he realizes that he may be able to take the money from this particular customer but that the PR damage done by that customer may be larger than being reasonable and understanding that mistakes happen. And if there is some egg on the face of the business owner for not alerting the user to their sudden suspicious consumption pattern then there are valuable lessons to be learned for all parties. So, the adult business owner would likely split the bill with the customer to cover his costs, fore go his profits on this particular occasion, have a very happy customer (who might even write about that!) and would apply the lessons learned to their business.
- ErrantX 16y agoHow do you judge the PR damage of a customer? Unless they bring up who they are elsewhere on the net (assuming it is not obvious) you simply can't judge such things. So what do you do; lose money each time this happens? Or take a hard line "we're nice guys and all but, sorry, it's costing us too". And the happy customer might write about you; but the risk is it sets a precedent. So the next month when another customer screws up they think to try the same tactic - and if you refuse them it's even worse press ("oh, apparently I'm not good enough to be treated that way", "2 tier customer service" etc.). It's a minefield all round.
- patio11 16y agoAs an empirical question, I think you are likely right that charging off some money owed might buy Heroku disproportionate amounts of goodwill. (I do it all the time.) That is, properly, their call. As a normative question, though, for-profit businesses do not have a generalized right to make their mistakes into others' obligations. More broadly, one of my periodic crusades is trying to get HN people to understand that norms in the business world are very unlike the norms of cash-strapped twenty-somethings regarding things like a) the appropriateness of charging money (charge more) b) how much money $1,000 really is (a lot to you, a lot to me, rat spit to businesses with an HR department) and c) expectations for professional behavior (you're doing taxes like a boss, reading contracts like a boss, capacity planning like a boss, paying your invoices like a boss, etc).
- webandrank 16y agoNeed to read before and some time ask about
- chr15 16y agoI am not surprised that they didn't give a refund; most businesses wouldn't. However, I think businesses these days should start integrating a one-time "whoopsie" allowance as part of their customer service (within reason, of course. It may or may not be feasible depending on your business model). You're probably going to have people that try to take advantage of this, but there are some people who make honest mistakes, and I would give them the benefit of the doubt. Just treat people well. I believe if you take care of your customers, they'll take care of you. In general, the one thing that people will remember for life is how you make them feel. The benefits of providing awesome customer service outweighs the cost IMO (I haven't done the math, but that's my gut feeling). I remember reading a story on reddit yesterday where a gamer experienced the almost exact same situation with Steam: http://www.reddit.com/r/gaming/comments/dcnp7/one_of_many_reasons_i_love_steam_customer_support/ http://www.reddit.com/r/gaming/comments/dcnp7/one_of_many_re... that provides a good example of how news of awesome customer service can be spread, and possibly grow sales (Zappos). For example, I have several sites hosted with WebFaction.com and I'm unbelievably happy with their customer service. They respond within 10-30 minutes on any support tickets. I'll continue to use them in the future. (see how that was a good example of what awesome customer service can do?)
- steve19 16y agoApple apparently has a one-time whoopsie allowance. According to many reports on the 'net, they will one-time refund an app purchase.
- benatkin 16y agoExpedia has this. I once booked a flight & hotel for the wrong month. I was told they could cancel the hotel free of charge, but for the flight they would be able to take care of it just this time and if it happened again it would be $100.
- nandemo 16y agoCanceling ahead of time is one thing. "Canceling" later because you didn't use your reservation is another. The example in question is more like: you make a reservation for September instead of October. In October you realized your mistake. Then you ask for your money back, after all you didn't use either the tickets or the hotel. And because the airplane and hotel weren't fully booked, you claim they didn't have any additional costs because of you. Well, good luck getting your refund.
- alextgordon 16y agoI'm going to go against the grain here and say it's partly Heroku's fault. If a customer is paying hundreds of dollars a week for a feature they're not using, that's the time to send a friendly automated email saying "Is everything okay?". This is especially important with a service such as Heroku, where users will be changing these settings all the time and so there's a high probability of human error.
- dotBen 16y agoI can think of loads of reasons why you might be legitimately in that situation... Perhaps you intentionally provisioned a large amount of headroom for a forthcoming expected/possible traffic spike. In that example it would be annoying to get "are you sure you want to have this extra resource running?" emails. (sure, you should be building the system to auto-provision but that would have required a better programmer and you just got the interns to build it, etc)
- alextgordon 16y agoI get tons of emails saying "Would you like to buy viagra?". Getting an occasional email saying "You might be paying us too much money" is low on my list of annoyances.
- sliverstorm 16y agoNot only that, but I will almost never be annoyed by someone who says "wait, are you sure you really want to pay us that much?"
- X-Istence 16y agoMy bank emails me when I withdraw over a certain amount, they send me a text message and sometimes will give me a call. Those are not annoyances, those are safety guards I want in place.
- novum 16y agoI think it'd be a useful feature for a good number of their users. It should be able to extrapolate based on previous usage, and notify me if I'm extremely outside my average historical utilization rate. They ought to allow you to set a sensitivity or threshold, and they also ought to let you opt out altogether.
- StavrosK 16y agoThis reminds me of something my friend did. We had an assignment to try out EC2 with Hadoop and we needed 8 machines, and we were warned in class to turn them off or they'd keep charging. Guess who didn't turn them off. So he casually mentions it to me and tells me that he racked up a 500 pound bill, but his CC didn't have any money in it so it was fine. I asked "so did you turn them off?" He said "no, not really, but it's okay because they can't charge me". So he racks up another 500 pounds, and only turns the instances off when he finds out that he didn't use the inactive CC but another, active one. Long story short, he's 500 pounds out (Amazon were kind enough to refund the first 500, which disappeared as soon as it hit his account, since he owed another 500).
- dotBen 16y agoreminds me of a friend who was studying for his MSc Physics at Warwick... the students had to do a lab experiment with a radioactive isotope and were warned to only have the box containing the isotope open for a few minutes in order to reduce exposure to safe limits. Guess who forgot to close this box and after 2 hrs of exposure now has to worry about cancer and/or infertility in later life. True story. Look at it this way: at least your friend took CS and not Physics and thus is only £500 in the hole.
- StavrosK 16y agoJesus, that's horrible. Yeah, I bet he wishes he could pay 500 quid to make that go away :/
- ams6110 16y agoBorderline negligent, though, on the part of the instructor. Lab assistant or supervisor should have been double-checking safe handling of something that dangerous.
- sigstoat 16y agohe can't be expected to keep track of something for 2 minutes without someone watching over him, but we can let the fellow operate a car, have mains power in his home, purchase cleaning chemicals, etc? loads of graduate students in the physical sciences are regularly given access to stuff that dangerous or worse. if we've got to have somebody watching over each and every one of them, graduate education will grind to a complete halt.
- sad 16y agoIt's tough to say. You really sound like a reasonable person and if you are the model customer you appear to be then I would probably side with you. I like a very simple rule: Customer retention is far more important than customer acquisition. It may be a flawed rule, I'm not sure since I'm not a successful business person. But it sounds good to me. It seems vitally important to retain your good customers.
- ams6110 16y agoYes, the very fact the the OP is now considering (however causally) the idea of finding a smaller host with more personal care is something I'd be concerned about if I were his current host. Who knows, maybe he'll go out and find an awesome small host he never knew existed. Customer gone, and he'll probably start recommending the new host to all his friends and associates too. If it were me, and a good customer made a mistake, I'd eat the cost to keep the goodwill.
- audyyy 16y agoIf you could get away with turning up your dynos and workers in anticipation of a traffic spike and then not have to pay; it wouldn't be fun any more.
- jacquesm 16y agoWhy? It could be fun for everybody, just set a baseline and a peak level and use as much as required with a nice little graph of how close you came to your maximum allowance so you can plan your future hosting expenses.
- 100k 16y agoI wish Heroku officially supported autoscaling. I've been using Google App Engine lately and the autoscaling and pay for what you _really_ use is very compelling.
- arfrank 16y agoThey do for their background workers, but not the dynos. http://github.com/pedro/delayed_job/tree/autoscaling http://github.com/pedro/delayed_job/tree/autoscaling
- gfodor 16y agoHappened to me on EC2 once. They cut me a break and gave me 20-30% discount. They can though, because the idle machines were not costing them any money.
- Emore 16y agoThis didn't happen to me, but that's only because I set up dozens of post-its and alarms reminding me to quit those tests I was running on EC2. That's perhaps the one web-service that has made me the most paranoid.
- amock 16y agoThey weren't costing Amazon any money if they had excess capacity, but if those machines being in use prevented them from being sold to someone else then they did cost it.
- japherwocky 16y agothis happened to me on ec2 this summer- dozen idle instances running for two months because of a bug in a script. I didn't get any sort of break though. :( June was a lot of hotdogs and Budweiser till I could dig out of that hole.
- sliverstorm 16y agoAt this point, I'd probably just be thanking my lucky stars it was only $1.3k. You'll make it back.
- akshayubhat 16y agoSomething like this wont happen to Google App Engine, thankfully.
- theDoug 16y agoIn my experience a smaller host would have likely been sunk by your lack of attention requiring them to allocate more services (used or not), not be able to provide more personal care. You're lucky you're on a big host.
- bfung 16y agodoes Heroku have an api? if so, think of it as investment towards writing a traffic spike script/cron job =)
- checoivan 16y agoIt's human error. Simple as that. Happens to absolutly everyone even the brightest individuals. I've seen in "the livesite" that this kind of spike problems are more common than you think, and the ideal solution is to automate an alarm that tells you both, when a spike is happening, and when X throughput passes over a threshold. This kind of automation is easy and spares lots of trouble.Could be as simple as scheduled task parsing some logs. Some more fancy option is always good though :)
- JJMalina 16y agoReading this makes me laugh because today I got a $1400 bill from National Grid for gas and heating. The normal bill is around $40 usually. An interesting phone call awaits tomorrow. In all seriousness though Heroku should strike a deal since this is your first offense, just like banks will waive fees for you when you overdraft the first time.
- fierarul 16y agoWhat you are doing is distasteful because you made a mistake and now you are using bad publicity against Heroku to force them to lower your bill (or just out of spite). Heroku might optimize for this usecase by detecting spikes somehow and notifying you that you still have machines running, but other than that, I don't see how any of this is their fault.
- pclark 16y agoNot sure why anyone but yourself is to blame, and I think it would have been nice for Heroku to give you a discount then, after this post they should go quiet and let you pay. I think this post is bad form.
- thibaut_barrere 16y agoI don't find this poor customer care, really: I consider this normal. You're talking about "a few weeks": did you check your expenses in the meantime ? I would put some reminders (google calendar etc) or habits in place to regularly check your expenses.
- thrashr888 16y agoI got stuck with a large, unexpected bill at my auto dealer. They comped the next large bill. They're awesome and I'll be a customer for life. They're also the highest (Yelp) rated shop in the area so I bet I'm not the only one. I suppose it depends on what sort of shop Heroku wants to run. It just seems like customer service is not a big deal to them.
- niico 16y agoI happened something similar to me. I forgot I was running a recurring billing adwords campaign. I realized I had that campaign when the bill came with a lovely 3.700$ total. The worst part is I didnt got to sell anything.
- djb_hackernews 16y agoThis happened to me, but not an such a large scale with an EC2 instance. I emailed amazon and they severely discounted the bill, with a warning to be more careful because they might not be able to do it in the future.
- dh 16y agoI can see the problem from both sides but really Heroku had some more more to gain by just crediting it or most of it in what you would say about them. Clearly by the rules, you should have to pay, but that does not mean that a business should not do the right thing and work with a customer. Imagine how big a raving fan you would have been if they had credited the entire thing. That is a tiny cost compared to acquiring a new customer and turning them into a raving fan.
- sr3d 16y agoI'd go with "Dumbass" and add "whiner" to the list. Why is it poor customer care when all Heroku did was saying no to an unjust request? You can move your site to another hosting service, or host it yourself. "I am stuck with $1300 bill on a site that I'm not making any money on." So that means if you make money on the site, then you'd pay? What's the difference here? You're saying that because you're poor now, you don't have enough integrity to say "oops I did something wrong" and take the responsibility? I'm myself a boostrapper burning thru my savings and to tell you the truth, I do everything myself to cut on expenses. I'd love to use Heroku, but they are too expensive for me to run. So if you don't want to make another mistake, then start to accept responsibility and learn to take care of problems yourself.
- thepumpkin1979 16y agoFirst: If I were Heroku I won't give you any discount, not after this. Secondly: Heroku is high class hosting, they charge you for being able to host yours "spikes of traffic". Third: I'm pretty sure you enjoyed their free hosting plan for a long time, you should be thankful.
- bdmac97 16y agoI got burned by this on Heroku recently as well but to a much lesser extent $$$-wise. My app typically runs on 1 dyno because it's basically not supported anymore. At some point I must have been messing around in the interface and accidently bumped my dynos. Oops. I personally think they should have a "WTF your load is like zero, you normally have X dynos, but you're using Y dynos for no real reason... dumbass" alert email... I certainly would have appreciated it!
- daemin 16y agoWell the lesson that I've learnt here is that when adjusting something like this up temporarily it is a wise idea to add an event to your calendar to switch the thing back down to normal levels.
- jschuur 16y agoI'm kind of disappointed that Heroku didn't at least chime in to comment on the situation in this thread. It's one thing to play hardball and stick to their guns, but when there's an obvious, popular thread about it, they should at least own up to the decision and chime in.
- jschuur 16y agoI'm kind of disappointed that Heroku didn't at least chime in to comment on the situation in this thread. It's one thing to play hardball and stick to their guns, but when there's an obvious, popular thread about it, they should at least own up to the decision and chime in.