50 ms·

I got pwned by my cloud costs

• intricatedetail 5y ago
  If you are not a VC backed corporation you must be insane to run anything on a "cloud". Why not rent a dedicated server from OVH or others where you can actually control costs and pay 10-100 times less?
  • Nextgrid 5y ago
    Because experience getting shit done using boring tools doesn't translate well to a future career in a VC-backend company wrangling Terraform & YAML files.
• llampx 5y ago
  Very nice writeup, thanks to the author for writing it so clearly for someone who is not familiar with the nitty-gritty to be able to follow it.
• cyberCleve 5y ago
  Ouch. If Troy Hunt of all people can make this mistake, it can happen to anybody. HIBP is an awesome service funded totally by donations, so it's too bad this happened. Of course Microsoft is happy to hide behind their confusing pricing model and let customers overpay for Azure without alerting them.
  • tpetry 5y ago
    Every cloud makes this mistake easy! You have to manually activate billing alerts for everyone because they want you to spend more snd more each month. I am still waiting for a cloud without these dark patterns. But that will never happen because it‘s leaving a big amount of money on the table by not being hostile.
    • TedDoesntTalk 5y ago
      > I am still waiting for a cloud without these dark patterns. This is how mobile and landline phone companies made enormous fortunes before flat rate billing. It’s called post-paid vs pre-paid billing.
    • iso1631 5y ago
      Get a VPS from linode for $5 a month and it costs $5 a month.
      • nix23 5y ago
        Or hetzner, or Vultr.
      • gkhartman 5y ago
        They have transfer limits and an associated overage fee iirc. I can still see this sort of thing happening if that is the case.
        • iso1631 5y ago
          My understanding is you hit your bandwidth cap and that's it, no more bandwidth. (edit) looks like that's not the case, I'm sure I used to have to buy a second instance a few years ago if I did want to use more bandwidth that was allocated
        • BenjiWiebe 5y ago
          I barely even realized that, since my hobby stuff doesn't come anywhere near the limit. For those curious, the overage rate is $10/TB ($0.01/GB) after the transfer included in the plan. The smallest amount of included transfer is 1TB for the $5/mo VPS.
          • Ostrogodsky 5y ago
            Or just go with Hetzner and have a limit 20X as big with cheaper prices https://www.hetzner.com/cloud https://www.hetzner.com/cloud
  • moritonal 5y ago
    So I guess one method would be to set spending-limits when you setup your account. But that'd lead to constant moments of having to bump your budget (or worse, get approval to do so from Accounting) when you're trying to work. There are both spending limits and alerting that you could use, but would be impossible to predetermine from Azure's perspective, so they rightly ask you to.
  • Grollicus 5y ago
    Do you have any substance to your allegation of Microsoft hiding behind their pricing model? This is very straight forward from their view, before: almost no traffic = almost no costs, now: huge traffic = $$$. On the other hand, it doesn't seem that Troy did try to talk to them about this and seems to want to eat the costs himself. As it was his mistake. I think that's commendable. I also think with the amount of free advertisement Troy has done for them they'd be open to this and I can imagine we might see a followup post like "MS was so nice they waived my costs".
    • NicoJuicy 5y ago
      He's an Azure MVP. He already has 13 k in credits/yr, which could absorb the costs ( just guessing here)
      • mdoms 5y ago
        He's also independently very wealthy. Dude drives a GT-R and AMG C-Class (as at 2017, probably upgraded by now). He got a generous payout when he was laid off from Pfizer.
  • baybal2 5y ago
    Who is Hunt Troy?
    • jacquesm 5y ago
      The guy behind 'Have I been pwned', a website where you can check if your login credentials to some website have been leaked. https://haveibeenpwned.com/ https://haveibeenpwned.com/
  • j1elo 5y ago
    > If Troy Hunt of all people can make this mistake, it can happen to anybody. Exactly this. As a low-level / embedded / non-cloud stuff dev, I've been getting up to speed through all the cloud-ification of the industry, but I'm still scared (not literally ofc) of running most things on my own on any big cloud provider (smaller ones seem more manageable). I'm reading this and seems like being a customer of cloud services is like walking a dangeous path filled with gotchas and caveats, just jumping from cover to cover while hiding from danger, and hoping you're safe and didn't mess it up so far, "fingers crossed". Like this tiny detail that he didn't realize was critical, so I would fall on it too plus on another 500s small papercuts: "oh I set cache up, so I hope all is well". "Yeah, no you aren't, I guess you didn't think of this detail about maximum cached file size! Gotcha, Game Over!" Yeah cloud providers should have clearer communitacions and etc etc... but the fact of today is that they don't. So I'd never sleep well feelin 100% confident that I had covered and taken into account every minuscule detail and possible scenario that could end up being a disaster.
    • qw 5y ago
      > I reached out to a friend at Cloudflare and shortly thereafter, the penny dropped Another advantage is his big network that he can ask for help. There's also a chance that his blog post will reach the right person in Azure and he'll get a reduced bill. As someone who doesn't have the same network or the "fame", I am concerned about what would have happened to me in that situation.
    • chasd00 5y ago
      Remember when no-sql came out and everyone was rushing to it because "rdbms don't scale"? I'm beginning to feel the same way towards "cloud" in the Azure or AWS sense. You can go really really far with standard issue VMs from linode or digital ocean and so on. I wonder how many are overpaying for Cloud services so far above and beyond what their actual needs are.
    • sydthrowaway 5y ago
      How are you teaching yourself?
  • Nextgrid 5y ago
    Putting anything internet-facing on the cloud is as irresponsible as posting your credit card number publicly. Anyone can essentially charge you an infinite bill and you can't do anything about it until it's too late. Maybe it's not a problem when you're dealing with millions of VC money, but there's no way in hell I would host anything in a bandwidth-metered cloud service when my or my own company's money is involved.
  • capableweb 5y ago
    Correct me if I'm wrong, but Troy Hunt is a person focusing on security, not infrastructure, deployments or development even. If anyone is near making that mistake, it's people like Troy Hunt. Operators would of course see the problem easily (paying for bandwidth like that would be the first warning sign), while they are sometimes blind to other issues, like security.
    • Closi 5y ago
      > Correct me if I'm wrong, but Troy Hunt is a person focusing on security, not infrastructure, deployments or development even. Eh, I don't know - either way he is a Microsoft Regional Director and MVP for Cloud (as well as security), runs courses on cloud deployment on Pluralsight, and has done speeches on Azure and reducing cloud bills, so if a he can get stung it doesn't say a whole lot good about my chances.
      • capableweb 5y ago
        Thanks for the correction :) I had no idea he did all of those things in addition to the security work. Certainly doesn't paint Azure or him in a good light then.
  • brimble 5y ago
    There's an entire surprisingly-large industry built around providing better UI to the major cloud providers, so you can actually tell WTF is going on with billing, access control, networking, et c. They're so hostile that it has to be intentional.
    • scapecast 5y ago
      The underlying issue is that the cloud console is owned by a single product team, and THEY decide what gets exposed - not the underlying product teams for the individual services. At least that's the case for AWS. The result is that you get a lowest common denominator type of dashboard. And hence a whole industry of providing just a prettier dashboard on top of AWS / GCP / Azure metrics. Datadog started with a prettier dashboard for Cloudwatch data. Cloudability started with a prettier dashboard for the Cost and Usage Report. And also works the other way around. The individual product teams buy development environments to circumvent the console restrictions. For example, a few years ago, the Redshift team purchased "DataRow".
• cdmckay 5y ago
  It would be really classy if MS forgave that debt, especially considering the service is a public benefit.
  • kelsolaar 5y ago
    I would go as far as saying that the hosting for such a service should be entirely sponsored by Microsoft.
    • lodovic 5y ago
      He's a "Microsoft Regional Director and MVP" so Microsoft pays the bill one way or another. I expect that he has reduced Azure rates as well.
      • akoeplinger 5y ago
        Regional Directors and MVPs aren't employed by Microsoft: https://rd.microsoft.com/en-us/about/ https://rd.microsoft.com/en-us/about/
        • alkonaut 5y ago
          I have a monthly azure credit of $150 and some reduced pricing simply by having a ms developer subscription. I'm guessing Microsoft MVP's (in general, and Azure MVPs perhaps in particular) have extremely generous azure credits so hopefully he isn't on the line for the full amount here.
  • anothernewdude 5y ago
    Would be even classier if the major cloud providers responded to customers calling out for budget limits for the past decade. Not many people want to risk potentially infinite costs.
• stevehind 5y ago
  Have you contacted Azure? On one hand you owe the money “fair and square”, but on the other if I were them I’d waive an unexpected $10k bill to a good faith actor that was incurred without any proactive notification by Azure.
  • jacquesm 5y ago
    As opposed to all those other customers who are not good faith actors?
    • scrollaway 5y ago
      You’re trying to be snarky to GP, why exactly? Yes there are bad faith actors that might try to get some free cash out of cloud refunds. And other customers can also be good faith actors and included in the assertion. The post applies to everyone and I’d second it. Ask nicely for a refund in these situations, the worst that can happen is they say no. Where did they say that “only Troy Hunt shall receive a refund, for only Troy Hunt is a good faith actor, so say we all”?
      • jacquesm 5y ago
        No snark intended. It's just that I would assume that all customers of such services are in principle good faith actors, not just Troy. The one thing that is special about Troy is that he is providing a service for the public good but that has nothing to do with being 'good faith' or not.
      • fastball 5y ago
        The point is that if they give refunds to all the good actors they won't make any money.
        • LeonB 5y ago
          Only if their regular services run at a loss and their business model relies on people making mistakes. But Azure’s regular prices are definitely high enough that they’re not a loss leader.
• bawolff 5y ago
  Seems at least a little unethical that cloud companies do pay as you go up to infinity, instead of some model where you transfer money in and if you use it all up your service gets cut.
  • XorNot 5y ago
    There'd be value in a model which allowed you to pay up to some limit then switch into a user-pays model if the user wanted the service right now.
• dtx1 5y ago
  If Microsoft doesn't show the decency to forgive that bill, i'd be happy to chip in!
• polote 5y ago
  As I spent a few hours to successfully get cf cache b2 files. I'm curious about the part of support Cloudflare requests due to caching issues. It's time for cf to work a bit on its UX
• oneepic 5y ago
  It is worth mentioning that the alert itself costs money. So if you're evaluating the alert every 5 minutes on the past 24h of data it can burn a small but surprising amount of money. From TFA it looks like that would be 10 cents per "time series". Or what I translate it to, is 10 cents every 5 minutes (*I think, but I havent used Azure in some time*). $1.20/hour, $28.80/day, almost $900/month. Not too hard to drop that by making the alert less frequent. (edit: I think I saw AU$ there, so maybe it is AU$900.)
  • manarth 5y ago
    A time-series represents a "thing you're monitoring" – in this instance, it's aggregate egress, so $0.10 per month, regardless of the evaluation period. Monitoring CPU? Another $0.10 per month. Memory? Another $0.10. Thankfully, not $900.
    • oneepic 5y ago
      I meant to emphasize frequency, not eval period. Apologies. That said I took a look at the pricing docs and didnt see frequency mentioned, so hopefully I am in the wrong about the price. As an aside, their (Azure's) pricing docs are written in the same fishy way their technical docs are written (my opinion only)...
      • giaour 5y ago
        The service being used here is Azure Monitor, which charges $0.10/month for each metric you monitor with alert rules: https://azure.microsoft.com/en-us/pricing/details/monitor/ https://azure.microsoft.com/en-us/pricing/details/monitor/ They do charge by the MB for ingesting custom metric data, but storage outbound bandwidth is one of the free "standard metrics" built into the platform. So the alert's full cost is $0.10 per month. Since they charge by the "time series," I think that means the OP could configure different alerting thresholds on storage outbound bandwidth, and they'd still only be on the hook for $0.10/month.
  • mnahkies 5y ago
    This is something to be mindful of when using datadog synthetics monitors as well - if you have a short interval, or many locations being tested from they can become expensive quickly
• jskrablin 5y ago
  First thing one should always set on any cloud account is billing alerts. Set > 1 and set first to ~ 80% of what you think will be your normal cost then add extra alerts all the way up to 100%. That way you'll usually get an early warning with some time to act before it becomes really expensive.
• zzt123 5y ago
  Interestingly, Troy says that egress is expensive on Azure at $0.014 AUD/gB (~$0.010 USD/gB), but that is the same price as additional egress for Linode and DO, and Linode egress has never struck me as expensive. In fact, I’m kind of shocked (as an AWS user) that Azure egress is the same price as Linode. Actually, wow it seems AWS is also the same price as Linode and DO for egress. While Linodes and DO do come with decent free bandwidth, this is a surprise to me.
  • deleted 5y ago
    [deleted]
  • patrec 5y ago
    How can $10 per TB not strike you as expensive? You can easily download that much a day on consumer broadband that will cost you far less than $10/day.
    • fabian2k 5y ago
      If you download the data twice at that price point, you could buy an HDD to store it for the same price (the bigger HDDs seem to be at ~ 18 EUR per TB here).
    • CodesInChaos 5y ago
      $10/TB is between availability zones in the same region. Egress to the internet costs $50-$90. So it's much more expensive than the already expensive $10.
  • coder543 5y ago
    You’ve interpreted the numbers wrong. Yes, Linode, DigitalOcean, and most of this class of providers charge $0.01/GB. Almost literally an order of magnitude less than Azure or AWS. The megaclouds massively overcharge for bandwidth. It’s not even close. AWS charges $0.09/GB, and Azure charges $0.0875/GB. Maybe Troy Hunt gets a discount for being a Microsoft Regional Director and MVP. (Neither of which make him an employee of Microsoft, confusingly enough.) https://docs.digitalocean.com/products/billing/bandwidth/ https://docs.digitalocean.com/products/billing/bandwidth/ https://www.linode.com/docs/guides/network-transfer/ https://www.linode.com/docs/guides/network-transfer/ https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/ https://azure.microsoft.com/en-us/pricing/details/bandwidth/ https://azure.microsoft.com/en-us/pricing/details/bandwidth/
• usr1106 5y ago
  That's the typical story. Something goes wrong and it costs you (typically a small company) a lot of money. At that time just nobody is looking at metrics. Even alarms don't help absolutely because they can also be missed. The only thing that would really help were a hard spending limit that stops all services except storage. If your site is important there will be such an amount of user feedback that it is impossible to miss it for a long time.
  • dspillett 5y ago
    Alerts can also fail to be timely due to mail/SMS/other delivery issues, or the right people being in the middle of something else. This delay means it is still possible to rack up and unexpected cost. Or they can fail completely. And the alerts themselves cost if you want something reliable so you have to weight that against the danger. Pay as you go cloud can be a maze of costing concerns.. > The only thing that would really help were a hard spending limit that stops all services except storage. Yep. Though that is small comfort if you need to guarantee more than a couple if 9s of uptime, hopefully those with that requirement can soak up the unexpected billing blips.
  • alfiedotwtf 5y ago
    > The only thing that would really help were a hard spending limit that stops all services except storage. Sadly, I haven't found a way to do that with AWS
    • dx034 5y ago
      It's funny that even Hetzner can do that and AWS can't. Shows that there's no interest from AWS to prevent these things from happening.
      • bencollier49 5y ago
        IMO this is something which ought to be written into law. It'd be easy to implement a kill switch, and would actually encourage innovation, as people would feel more empowered to experiment with the technology.
• OtomotO 5y ago
  Well, the cloud is just a convenient way of accessing someone else's server. Convenience always costs money, there is no (big) cloud provider doing it out of their own pocket or rather not optimizing for huge profits. It's the same as with any other service, really. So I don't understand, why some people assume it would be different here. (Note: I am not saying that Troy Hunt assumed this, but I know people who go to the cloud because "It's cheaper". It was never cheaper, on no project I worked on. It was more convenient, but in the end it was more expensive mostly)
• deleted 5y ago
  [deleted]
• superphil0 5y ago
  First thing i do is set an alert when costs go over 10$ for any new project. Highly recommend
  • onion2k 5y ago
    Do you also make sure you never go on vacation, never go anywhere that doesn't have a phone signal, never turn off your phone, that your alerts have multiple levels of redundancy, and that you always have access to a computer to modify settings?
• suction 5y ago
  I wonder if before cloud computing, has there ever been a successful product / service where it was accepted with just a shrug that the volatility of monthly costs means it could bankrupt you with next month's bill, because of complexities and opaqueness of the cost structure make it virtually impossible to predict and protect against extreme peaks in all parts of the setup. Even if you run a relatively opaque cost structure business like a restaurant, you can still calculate the maximum cost of ingredients for one month, the salaries, energy, etc. if you simply use the "best case scenario" of having every seat at every table booked for all opening hours, with people ordering your most sold dishes. Cloud computing is still leagues above that in terms of cost predictability. I once worked for small, non-startup software company who pondered moving servers to Azure. The Azure partner shop analysed the needs and came up with a monthly cost "between 30k and 120k per month". They were really surprised the company stuck with their non-cloud setup because "everybody is moving into the cloud!!"
  • avrionov 5y ago
    I worked on both cloud computing and on premise project. Before cloud computing the risks were different: - much harder to scale. It was much more common to over provision and have machines and bandwidth being unused for years. - when we were hit with very high traffic due to a bug or something else, most of the time it would lead to customer outages. Based on the contract some times it requires to pay back because SLAs were not reached. Also an outage could lead to customers canceling the subscription. We swapped one type of problems with another.
    • suction 5y ago
      If you have a bug that renders your product unusable and refunds are in order, the flexibility of handling traffic peaks which a cloud provider offers won't solve that problem for you. It could even aggravate it. If a show-stopping bug is introduced, it would probably be preferable to fail quickly.
      • philliphaydon 5y ago
        If there was an outage in early 2000, we just went outside to play, or watch tv. Now if Facebook is down for 15 seconds everyone has heart failure like their life is over.
• ksec 5y ago
  >What we're talking about here is egress bandwidth for data being sent out of Microsoft's Azure infrastructure (priced at AU$0.014 per GB). AUD $0.014 is roughly USD $0.01. Which I thought was reasonable. But on [1] only "Data transfer between Availability Zones(Egress and Ingress)" cost $0.01. Do transferring from Azure to CF count as that? Other Internet egress (routed via Routing preference transit ISP network) starts at $0.08 I hope someone from Azure CS could give him a custom discount. It is also worth thinking, the cost HIBP saved on Cloud / Serverless over the years could have wiped out ( if not more ) by this single incident. [1] https://azure.microsoft.com/en-au/pricing/details/bandwidth/#faq https://azure.microsoft.com/en-au/pricing/details/bandwidth/...
  • nbevans 5y ago
    Cloudflare and Azure have a "Bandwidth Alliance" peering which - if you correctly set up your Azure resources to use "Internet Routing" - will result in a modest discount. It is a bit of a scam though as it is marketed as though you'll get 100% discount but in reality it is more like 15% off. I think GCP is 100% though.
    • gcbirzan 5y ago
      Definitely not 100%, more like 66% off: https://cloud.google.com/network-connectivity/docs/cdn-interconnect https://cloud.google.com/network-connectivity/docs/cdn-inter...
• knorker 5y ago
  As soon as I saw "17GB file" i thought "that's what torrents are for". Otherwise one mistake and... Well this happens. Or someone maliciously bypasses CF cache e.g. by parameters. Cloud just is not suitable for any kind of volume egress. It's a death trap. Like going on vacation with data roaming enabled.
  • Aissen 5y ago
    Yeah, HIBP is using torrents: > I removed the direct download links from the HIBP website and just left the torrents which had plenty of seeds so it was still easy to get the data. Since then, Cloudflare upped that 15GB limit and I've restored the links for folks that aren't in a position to pull down a torrent. Crisis over.
    • knorker 5y ago
      I know, I read the article. But I feel like Dr Strangelove here. Of course, the whole point of a torrent on a cloud service is lost if you also provide a raw download link. Also providing a download link is tempting, but can easily cost (for a 17GB file and growing) up to US $3 per click. Even off of their premium global network it's over $2 per click. The cheapest in Microsofts entire egress table would be $0.68 per click. (but that only kicks in after you've spent way more than $9400 in cheaper tiers in a given month) Egress kills you, in cloud. "Oh, cloudflare probably caches most of this" is not something I'd recommend.
    • dx034 5y ago
      And then Cloudflare will not cache it at some locations for random reasons and the cloud bill is back. Anyone with technical knowledge should have no problem routing static files via machines at OVH/Hetzner and the like, no reason to enter such risks for maybe an hour of setup time saved.
  • dx034 5y ago
    Or Hetzner server auction to get a cheap 20/30€ machine with unlimited traffic at 1Gbps. Setup time is max 1h even if you do it manually, with cloudflare Tunnel it's also really easy to lock down everything with a firewall and have minimal exposure to threats.
• kuu 5y ago
  One thing I hate about the cloud providers is that there isn't an option to set a maximum cost. I would prefer to plug the cable of my side project than just receive an email saying me that next bill is going to be over my cost. I understand not everyone would like to do that, but I would like to have that option.
  • herodoturtle 5y ago
    I assume you meant “pull” or “unplug” the cable :)
    • kuu 5y ago
      Yes ;)
  • cma 5y ago
    They'd rather refund small guys for mistakes than give big guys an easy limit to set.
    • kuu 5y ago
      I guess big guys don't want they service to suddenly stop, so they probably would not use this... But it's just a guess
      • electroly 5y ago
        Absolutely that. Storage costs money, so in order to absolutely cap your spending they would have to delete all your stored data, too. Deleting S3 buckets and EBS volumes on a spending blip is absolutely the last thing any company with any budget at all wants to happen, ever. It would be preferable for that not to even be possible in any situation. This is the sort of thing that only extremely small casual users want, and it isn't worth it to AWS to cater to those users. For everyone else, more complexity than a "kill everything at $X" switch is needed, and that's exactly what we do have. We don't get to absolutely cap our spending to the penny but we also don't risk having our data vanish because of a billing issue.
• joking 5y ago
  outbound transfer cost is one of the most expensive things in cloud computing, it's much better when you can pay for allocated bandwith.
• lpcvoid 5y ago
  Can somebody explain to me why I wouldn't just rent a 40 EUR dedicated server from Hetzner with unlimited traffic and gigabit uplink? His 600GB/day is way less than what you get over a gigabit link within a day. Sure, sudden bursts would perhaps "throttle" at a gigabit, but according to his article that was only the cloudflare proxy anyhow, so no pain in having that take a few seconds longer. As far as I am concerned, I just don't understand why people use cloud services.
  • unixhero 5y ago
    You are not wrong. Hetzner would be a good choice instead.
  • technion 5y ago
    He is a Microsoft MVP. A title that is given for being a "community evangelist" of Microsoft. You wouldn't get that throwing it on a Heztner machine. Edit: Consider this article, and Geoff's statement about Azure credits. https://www.theregister.com/2021/04/21/microsoft_revokes_mvp_status/ https://www.theregister.com/2021/04/21/microsoft_revokes_mvp...
    • fs111 5y ago
      Sounds like a pretty expensive privilege. How is using cloudflare okay in this then? Cloudflare is also not Azure
      • rcarmo 5y ago
        There are plenty of people using both, it's a good combination.
        • dewey 5y ago
          The point is about Cloudflare not being from Microsoft / Azure which is the company he's evangelising for.
          • 5y ago
• unixhero 5y ago
  It would be good if he contacts Microsoft about this. Sometimes they will give credits for situations such as this.
  • goodguyamericun 5y ago
    He is Troy hunt and an ms MVP, as soon as ms gets wind, they'd be the one to contact him
• rcarmo 5y ago
  This prompted me to go and check my custom static site generator (which renders my blog onto an Azure storage account exposed via HTTP and Cloudflare). Turns out I wasn't setting x-ms-cache-control when writing all the blobs, so that's a win right there. (interestingly, it appears that rclone, which I was in the process of moving to, doesn't do that, so I might have to keep my custom Azure storage library around)
• hdjjhhvvhga 5y ago
  It is very good these things are getting publicized. More and more people realize these payment schemes for what they are: a scam. Every cloud provider that refuse to put a hard spending limit participates in this. It is important to remember that not all cloud providers participate in it. For example, in Hetzner Cloud, they explicitly provide the maximum amount you are going to pay for a given instance or service in a given month. You are guaranteed not to pay more. Everybody knows why Amazon etc. refuses to do it this way.
  • mawalu 5y ago
    Hetzner Cloud(!) only has 20TB/Month included in the monthly costs and states that you have to pay for any additional traffic. I never reached that on one of their cloud boxes so I don't know how it looks like but it definitely isn't all up front. But yes the dedicated machines come with no additional traffic charges whatsoever
    • CodesInChaos 5y ago
      Additional traffic costs 1 EUR/TB (plus VAT, depending on where you live). So it's about 50 times cheaper than the big clouds.
  • zekica 5y ago
    On Hetzner and with their €1.00 per TB after 20TB included, you can pay up to €324 per vps as you are limited to 1Gbps if you fully saturate the link all month.
    • dx034 5y ago
      I doubt you'll manage to get the exact 1Gbps per VPS out all month. On dedicated that's more likely. But luckily they have a very easy setting for billing alerts and maximum in the settings page.
• scanr 5y ago
  I wonder how much of the cloud provider revenue comes from situations like this. I suspect quite a lot. I think that the cloud provider business model that allows for uncapped maximum costs is a bit of a commercial dark pattern. What makes it somewhat more nefarious is that it is relatively easy to blame the customer. I’m not surprised that the cloud providers are quick to refund users as it’s likely that they only do it in a fraction of cases and it buys a lot of goodwill. It would be interesting to try and design a cloud that supports OutOfMoneyException’s with gradual degradation and capped liability for costs built in.
  • Nextgrid 5y ago
    > I suspect quite a lot. I don't actually believe so. Cloud providers are known to refund bills incurred by mistake. They make so much margins on legitimate usage by big companies & startups that it's just not worth burning developer goodwill & potentially waste efforts trying to collect a bill the customer legitimately can't pay (and will guarantee he will never use nor advocate for your service again).
• Dave3of5 5y ago
  Ah the old cloud provider switcheroo. Yip this is the way they make money. They make it easy to setup some gigantic hugely scalable website then hit you with a gigantic scaled up bill. AWS would do this as well. Team I'm in at the moment is in the early stages of cloud adoption but the company in total has fell hook line and sinker for AWS. When I mentioned the cost there is always an excuse. The main one being that you don't have to hire sysadmins anymore as that's taken care now by AWS. Ah yes but they have actually been replaced with a "DevOps" team plus just our department now spend > £1 million per year to AWS in hosting costs. A 20% reduction in those fees could pay for a few sysadmin(s). The next one is that no other vendor would be able to supply the kit. You know StackOverflow is able to run on a single webserver (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...). Plus many of the other providers have loads of instances available. I mean I'm not against cloud it's just not the cheapest option if you choose one of the big 3 providers. I use a company called scaleway (https://www.scaleway.com/en/ https://www.scaleway.com/en/) they have all the essential cloud services you need and everything else you can run yourself in docker or k8s.
  • mcbain 5y ago
    That stackoverflow infra blog post is out of date. They use more than a single webserver now. For example: https://stackexchange.com/performance https://stackexchange.com/performance
    • andrewxdiamond 5y ago
      Most importantly, SO is extremely read-heavy, write-lite, and cache-friendly. A similar “scale” e-commerce site would be significantly more load, have more dynamic data, and just be overall harder to run.
    • dijit 5y ago
      Now they have 9. They still serve a lot more traffic than I do and I have hundreds of instances; thousands of containers.
• YetAnotherNick 5y ago
  I don't understand it. Does a cloudflare edge server sit inside Azure?
  • mstrem 5y ago
    No. Cloudflare is configured as a reverse proxy in front of the site. So traffic reaches the Cloudflare edge first, then it is proxied to the origin on Azure unless the file is served directly from the Cloudflare cache.
• rkwasny 5y ago
  I guess all Microsoft PR and Marketing departments are now on the phone trying to get this guy a refund and take down this post :)
  • throwawayffffas 5y ago
    This guy is a Microsoft Regional Director he is part of the Microsoft PR engine.
• taubek 5y ago
  It is good thing to know that this could happen to anyone. I guess that setting limits and alters should be one of the first things that one should do. What would happen if a credit card limit was exceeded, a site would just stop working?
• 3pt14159 5y ago
  Happily donated to Troy. He's done more than most to help everyday folks weather these data breaches.
  • dx034 5y ago
    My issue with this is that the donation is basically to Microsoft for their dark patterns. There's no way this traffic cost much to Microsoft, so it all is added profit for their shareholders. Other providers would've provided the same service and bandwidth for a much lower price. I really appreciate the work that Troy is doing, but seeing much needed money ending up and Microsoft or Amazon leaves a bitter taste. I hope at some point it will become cool again to just rent a VM or dedicated server for small projects and stop throwing so much money at the already richest people in the world.
    • jimmydorry 5y ago
      Unfortunately, data in Aus really costs this much (more actually), from my experience colocating in a few data centres (I was typically paying $0.3/GB). It’s certainly possible it cost them less, but very doubtful on it being close to free. EDIT: Apparently it was hosted out of US West, so I agree that the actual data cost would probably be a lot less.
• buro9 5y ago
  Don't put Cloudflare in front of a Cloud egress bill. i.e. don't do this: Azure|Amazon > Cloudflare Always use your own proxy where the egress is well within your free tier, i.e. do this: Azure|Amazon > Hetzner|Linode > Cloudflare Why? Because Cloudflare cache is a massively multi-tenant LRU cache and whilst hot files will be cached well (and with Cloudflare Tiered Cache even better - but this itself is a cost) anything else is still going to expose you to some degree of egress cost. When I exposed AWS to the web I paid $3k per month to AWS. With Cloudflare in front of AWS I paid $300 per month to AWS. With Linode in front of AWS and behind Cloudflare I paid $20 per month to Linode and about $12 per month to AWS. A Linode, Hetzner instance... or any other dumb cheap web server that comes with a healthy free tier of bandwidth is all you need to set up a simple nginx reverse proxy and have it cache things to disk https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/ https://docs.nginx.com/nginx/admin-guide/content-cache/conte...
  • sascha_sl 5y ago
    Or simply use a proper CDN that doesn't pretend to eat all the cost for a flat fee but then sometimes does not. BunnyCDN has an amazing volume tier at half a cent per GB.
    • buro9 5y ago
      Oh exactly that. Or if caching is your biggest priority then Fastly or Akamai will shine too. But if you're balancing all considerations and want the cheap "good enough" caching with the DDoS protection, free TLS certs, and unmetered (assuming you aren't imgur or something)... then Cloudflare does a great job at being good enough. And for those sharp edges... drop in a proxy of your own, or layer your CDNs.
      • _xnmw 5y ago
        I don't understand, what is the advantage of Cloudflare over Fastly or Akamai if caching is not your biggest priority? Does Cloudflare have better DDoS protection, or something else?
• Havoc 5y ago
  These things really should have a AI like alert that is basically “cost is departing dramatically from historical pattern” without the need to set thresholds and the like
• csours 5y ago
  Cloud seems like a pet tiger - really cool and fun, until it turns on you.
• jve 5y ago
  > I, uh, have a bill I need to pay Kind of sad that service we are accustomed to using, various software integrates it (whether using HIBP API or downloaded pwned passwords archive) - is on a shoulder of single guy that now has to pay for his mistake. Great that Cloudflare helps him with the service, otherwise who knows if we had access to HIBP at this scale?
• nbevans 5y ago
  One wonders how Cloudflare can essentially absorb all bandwidth costs. But AWS and Azure are using them as a profit center.
  • uncertainrhymes 5y ago
    On the cloud providers, you are paying for your usage (yes, marked up, but they have costs too). Cloudflare has the same model, but they distribute the costs. The vast majority of people never use anywhere close to their share, so they subsidize the outliers and the free tier.
  • tyingq 5y ago
    Lots of peering. They pay $0 for roughly half of their egress. https://blog.cloudflare.com/the-relative-cost-of-bandwidth-around-the-world/ https://blog.cloudflare.com/the-relative-cost-of-bandwidth-a...
• fleddr 5y ago
  Cloud providers should really start protecting customers from these spikes. Alerts are not enough, there should also be hard caps (stop serving) and soft caps (serve at reduced speed/capacity) based on configured max budgets.
• progx 5y ago
  Clouds are good for quick start and fast grow. But after this phase, you should think about "classic" hosting solutions (multiserver, load balancer, etc.), they could be much cheaper. as long as your human admin costs are lower then cloud services
• lysecret 5y ago
  This is a big trap to fall in to. I dont understand why network trafficking is so expencive also in AWS. I once had a 2k monthly bill purely from networking because i accidentally routed a lot of requests through a NAT. That hurt haha. Now i stay away from those things :D
• alkonaut 5y ago
  Cloud providers should always have a max spend and it should be a standard feature. The cap shouldn't even be some optional feature or notification service. It should be a hard cap that you can move - at your own risk.
  • manquer 5y ago
    SMB or indie developers are not the first/primary customers for Azure/AWS that they design their application for. Any enterprise will not want any limits because of spends, they would be lot more pissed if service was pulled because spending cap set by someone sometime in the past is now exceeded. Likely is why such feature is optional not mandatory. Excess/unexpected billing would be negotiated in typical sales cycle discussions. Making a default hard cap however would result in a lot of senior people are going midnight calls for emergency budget approvals, management would get annoyed by that.
    • foobiekr 5y ago
      I 100% work for a large enterprise and we would absolutely like spending policies in place. After all, we have fixed OPEX budgets planned in advance of the quarter.
      • manquer 5y ago
        Having a policy is not same as a hard block, you would want spending alerts and escalations and reports, but do you really want a hard block on crossing a limit ? Admittedly I don't have experience working with very many devops /SRE teams, however I have never seen any enterprise vendor relationships were spend is hard capped for b2b sales with centrally managed procurement.
        • foobiekr 5y ago
          You know, there are alternatives between a sudden total cut off of service and nothing at all. Some mechanism to prevent sudden, large unexpected bills, yes, that’s desired. Notifications are not enough. A lot of people have provided tons of examples of follow ups by vendors in this thread where novel large orders are double checked. AWS could provide any manner of preventative policies and cost controls but they choose not to. Given the choice between blowing 2x past the planned opex and an outage, yeah, there are plenty of applications where the latter is preferable.
• sudhirj 5y ago
  This particular problem basically boils down to "CDN providers don't like caching large files", which is a very common problem. Everything else was configured and setup exactly right to not have a large bill. Most CDN providers have a lot of machines out on the edges of their networks, and it's understandable that they don't stuff these machines with large disks, likely preferring smaller faster SSDs. But this is a very common pitfall of CDNs that needs more attention, along with messaging on the dashboards and settings pages. I've had problems with no warning on Cloudfront, Cloudflare, Bunny.net all from not realising that my files were beyond the CDN's cache size limit, but none of them seem to do a good job at surfacing this other than "talk to customer support". Cloudfront does list the max size clearly in the limits and quotas page, though, and if you front your S3 bucket with Cloudfront, you could turn caching off and still get the discounted bandwidth out rates (S3 -> Cloudfront is always free, even if the file is fetched every time).
  • jrochkind1 5y ago
    Cloudfront isn't much discounted bandwidth out compared to S3 though, is it? I see S3 is initial $0.09/GB, going down to $0.07 after 50TB or $0.05 after 150TB. Cloudfront North America is $0.085 for first 10TB; but $0.110 and up for other regions. going down to $0.060 north america after 100TB, and okay $0.025 after 1PB. (but $0.050 and up in other regions even after 1PB). So okay, Cloudfront gets cheaper egress at large scale, I guess. By about 50% though, not an order of magnitude, and could be much less depending on region.
    • sudhirj 5y ago
      The reserved capacity pricing is lower, in a business setting your account manager will usually suggest this pretty quickly if you have a steady and/or increasing Cloudfront bill.
      • jrochkind1 5y ago
        Oh I didn't even know about that, thanks! Something else for me to look into.
• throwawayffffas 5y ago
  Question, is the 0.014AUD per GB quoted here correct? Looking at the linked page[1] I would think the cost would be 0.1102AUD per GB as is quoted in the Internet egress section. https://azure.microsoft.com/en-au/pricing/details/bandwidth/ https://azure.microsoft.com/en-au/pricing/details/bandwidth/
  • throwawayffffas 5y ago
    Also (3200 GB per day * 30 days) * 0.014 AUD per GB is 1344 AUD. While (3200 GB per day * 30 days * 0.1102 AUD per GB) is 10579.2 AUD much closer to the final bill. My conclusion Troy still doesn't know how much he is paying.
  • CodesInChaos 5y ago
    It clearly isn't. It looks like he confusing transfers between availability zones in one region with egress to the internet. A factor 10 mistake like that should be obvious, but he didn't fix it, even after I pointed it out in the comments on his blog (he responded that the price for me might be different due to region/currency settings).
• philliphaydon 5y ago
  It seems like everyone is blaming azure when this was an issue with CloudFlare… I get that everyone has an obsession with dirt cheap providers instead of cloud solutions like aws/azure. But that doesn’t mean it’s better. Everything has pros and cons.
• pibefision 5y ago
  Most of the clouds have functionalities to manage this. In AWS for example you can create an alarm with AWS Budget to monitor costs by tools/service/etc. Using a complex cloud without using this is not good practice.
• DigitalSea 5y ago
  I would be surprised if Azure doesn't waive or reduce this bill dramatically. Something similar happened to me with AWS. I had a simple file upload service where files would expire if they hadn't been accessed in 24 hours. Someone started using it to upload music and videos. I ended up with a high bandwidth bill on Amazon S3. I reached out and explained what happened, they waived the costs entirely (to the tune of $5000).
• hkh 5y ago
  We've been thinking about this for a while, and if there is any way we can catch these types of cost spikes before they happen. We've managed to do it for Terraform resources using an estimation approach, and using a usage file, you can model expected usage-based resources (https://github.com/infracost/infracost/blob/master/infracost-usage-example.yml https://github.com/infracost/infracost/blob/master/infracost...), but this one has got us thinking more about policies. To be clear - we would not have been able to catch this one right now :'( Would love to hear thoughts / brainstorm ideas - is there any way we can proactively catch these types of cost spikes?
  • Olreich 5y ago
    I think this is fundamental to on-demand services. Anything outside terraform or another configuration file system is hard to reason about. If cloudflare is in your config system, then you could put up a warning that files bigger than whatever won’t get cached, but that still assumes a level of knowledge about the system that you don’t generally have. Setting up limits and alerts as part of the system creation is usually the best strategy.
    • hkh 5y ago
      I like that, maybe we have to build up a knowledge base of wisdom (probably learnt through the hard way), and warn if the conditions are met or at least a list of the things to note. Then the cloud cost alert being a fallback safety net.
• pontifier 5y ago
  Everything can be going fine for a long time, and then cloud costs kill your business. This happened to Murfie a couple of years ago, and that's why I had to step in to try to fix things. I'm still trying, and there are still challenges, but I won't allow landlords and cloud costs to disrupt things again.
• faebi 5y ago
  I have 10gbits internet at home. Sometimes I wonder how many services/people I could bankrupt by using it harder. Not that I want this, but more like, why is it even possible?
• rob_c 5y ago
  close account, cancel card and move on with life before they charge you.
• bluedino 5y ago
  Reminds me of a time, we had a new site that was going to run on GCP, we had been using a couple co-located servers for years. When everything was moved to production, URL went live, nobody ever did any kind of bandwidth checking, caching, no CDN, no cost tracking. $10,000 in our first week. That's about 1/4 what our total spend on the co-located servers was for the whole year. Boss flipped his lid and wanted to kill the new guy who was on the project. After about 2 years we got rid of all the co-located stuff and were spending about 1.5x, but we had more apps, they served heavier pages, etc.
  • dijit 5y ago
    1.5x is pretty good. We overspent quite heavily on our on-prem stuff for a game I helped launch, for political reasons the next game ended up running on the cloud. The price was roughly 10x before discounts. With our heavy discounts and a wide amount of slimming down/cost optimisation (easily 3 months of work) we got it to 2.3x There will always be a need for sysadmins/cloudops/devops for that environment, so we didn't save any headcount either. I can't imagine getting anywhere close to parity in costs, Functions-as-a-service ended up costing more than compute instances too so we went back to compute instances in places where we thought we'd get away from it. That said, it was a lot nicer to use!
  • hogrider 5y ago
    Awful toxic boss.
• floor_ 5y ago
  This guy needs to clean up his bio. There seems to be a lot of confusion on whether or not he works for Microsoft when it appears that he is a uhh... reverse pay midlevel manager inter?
• lkxijlewlf 5y ago
  I'm sure some cloud providers have it, but they all should have a global, "If my account hits $XXX shut it all down immediately and email me" flag. And yes, that's kind of what he did here, I get that.
• Abishek_Muthian 5y ago
  Valuable investigation steps to find the erring cloud resource, But as Troy concludes 'Budget Alerts' would have saved him from this issue. No matter what the traffic is, The first thing to do with any cloud service provider is to set the budget alerts according to our wallet, be it one with credits or otherwise. At this point, I don't even try any new cloud service provider who doesn't offer credible budget alerts. Another key takeaway is, > Huh, no "CacheControl" value. But there wasn't one on any of the previous zip files either and the Cloudflare page rule above should be overriding anything here by virtue of the edge cache TTL setting anyway. Even this could blow up. All cloud service providers set the "CacheControl" to "No" and if we would want to cache something which is not cached by CF by default e.g. *html using Page Rules then we need to set CacheControl (e.g. max-age) at the cloud service provider end too. P.S. I've written about these recently on my blog titled 'Saving Cloud Costs'[1] from a frugal solopreneur PoV. [1] https://hitstartup.com/saving-cloud-costs/ https://hitstartup.com/saving-cloud-costs/
• joantune 5y ago
  Donated! Hope it helps
• TacticalCoder 5y ago
  Are there cloud services that allow to easily put a maximum budget, to make sure you have no surprise costs like that?
  • snovv_crash 5y ago
    Google App Engine allows you to set up hard spending caps, after which your application will start returning 503s
  • napolux 5y ago
    In my experience you can only setup billing alerts, which are fair, if you ask me. I took a good course on pluralsight about AWS and the first lesson was to setup a billing alert. What will hard limits will do to your infra? You can't take down / suspend DBs, EC2s, etc... Just because you set a 1k USD limit and that's it. Alerts are the 1st thing you should setup IMHO
    • notreallyserio 5y ago
      > You can't take down / suspend DBs, EC2s, etc... Just because you set a 1k USD limit and that's it. You (the cloud provider) can shut down VMs, block access to all services, and just retain the content in storage until the bill is resolved or the account is permanently closed. The cost would be trivial as storage is dirt cheap.
      • napolux 5y ago
        Sure, but will they do that? It's easier to just charge people. :P AFAIK Heroku shuts down your stuff if your Dynos are overspending :P
• kidsil 5y ago
  Shameless plug - the core of my work is about ensuring these unexpected costs never happen. We have some recent case studies where we've successfully reduced cloud costs by 95% https://www.cloudexpat.com/case-studies/ https://www.cloudexpat.com/case-studies/ hi(at)cloudexpat.com - happy to help!
  • Nextgrid 5y ago
    Out of curiosity, do you merely optimize existing cloud usage or do you help your clients move to hybrid/bare-metal?
• therealbilly 5y ago
  Yeah the problem with Cloud vendors is that if they make a mistake, it will usually disadvantage the customer...not them. I'm a little biased as I don't completely buy into the whole Cloud paradigm.
• cgtyoder 5y ago
  It's unconscionable that MS doesn't have warning notifications in place BY DEFAULT, so when you start incurring charges e.g. 10x of normal, you get notified immediately. One shouldn't have to set these up manually ever.
• jrochkind1 5y ago
  > But these would always cache at the Cloudflare edge node, that's why I could provide the service for free, and I'd done a bunch of work with the folks there to make sure the bandwidth from the origin service was negligible. If you're not Troy Hunt or another celebrity with special access to Cloudflare -- I don't think you really have access to Cloudflare to do a lot of work with you to ensure that your data gets cached and your egress is minimal, for large files on a very cheap cloudflare plan. (Based on the costs reported by Hunt as catastrophic, I don't think he's paying cloudflare for a large enterprise plan) (Also, it's unclear if caching large data like this is even within the ToS of Cloudflare?) I don't think Cloudflare promises to cache any particular URLs for any particular amounts of time (except no greater than cache headers etc; but they don't promise never to evict from cache sooner; they evict LRU according to their own policies). Cloudflare's marketed purposes include globally distributed performance, and security. I don't think they include "saving egress charges by long-term caching your data". I have a much smaller project, but egress charges for data are an increasingly large part of my budget. I've been trying to figure out what if anything can be done about it. I wish I had a guaranteed way to get ultra-long-cache promise-to-be-within-ToS for very large data files from Cloudflare for a affordable fixed-rate price. (Maybe I do? But just haven't reassured myself of it yet?) > In desperation, I reached out to a friend at Cloudflare… I recalled a discussion years earlier where Cloudflare had upped the cacheable size… Since then, Cloudflare upped that 15GB limit… Since I'm looking for solutions for this same problem (delivering lots of data at very cheap prices), I am finding myself a bit annoyed that Hunt is talking about how he solved it, using tools/price-levels not available to most of us who don't have his level of access due to position. Interestingly, MSN/Azure is part of the "Bandwidth Alliance" with cloudflare, which initially one thinks means there are no egress charges when delivering to cloudflare. (That is what it means for some other alliance members like backblaze). But that's clearly not the case or this story wouldn't happen, right? Turns out Azure gives you a fairly small egress discount when delivering to cloudflare, and only if you set things up in a non-standard way.
• pdimitar 5y ago
  Enjoyed the article. But still, couldn't help to get the following lasting impression after reading it: these days being able to click around the UIs of the cloud providers should be a billable skill by itself.
• commandlinefan 5y ago
  > I always knew bandwidth on Azure was expensive and I should have been monitoring it better It's suspicious that cloud providers STILL don't have any sort of "circuit breaker" infrastructure for this sort of thing - yes, you can set up alerts, but you can't say, "shut the whole thing down before the costs go above a certain threshold".
• hogrider 5y ago
  I wonder if people will start to make shell companies to just go brankrupt when this happens and start afresh with another company. The cloud vendor doesn't look too closely ehat you are running right? So this could work.
• mathattack 5y ago
  Think about how many big companies struggle with his. Most don’t have one person who can think through the cost of the cloud, as well as the activities to manage the costs. Many even say “Let engineers be engineers, and business people own the costs.” And all of a sudden you get a ton of surprises…
• BonoboIO 5y ago
  Well ... it's not like it was the first time this happened to a software developer. He should have known better that there is a risk, that you don't know some detail that costs you a lot of money. Cloud Bandwidth is soooooooooo expensive. If there is a risk that you have to pay this, please us a provider like Hetzner with fixed costs. If you like your serverless things, just host the big files at Hetzner.
• godot 5y ago
  These stories almost always boil down to this fundamental conflict of what you want for a personal project vs a business. (though in this case yes, Troy Hunt's HIBP is larger than a lot of startup businesses) In a business setting, you want your service to stay up, at the cost of spike in costs if accidents or mistakes happen. In a personal project, you want there to be hard limit on cost, and your service to go down if spikes call for it. (I'm relatively sure that no one wants their personal projects to incur a bill of thousands of dollars by accident.)
  • benbristow 5y ago
    Azure (and I'm sure other cloud providers do) allow you to set email notifications for when your bill goes over a set amount so you can stop it before it happens. If you're using a cloud provider I'd highly recommend setting one of those up. In Azure it's under your Subscription and then Budgets
    • skeeter2020 5y ago
      AWS can send you alerts when it looks like you will go over your budget for the period, before you're way over your projections, which is a nice feature. I truly believe they want you to use a lot of their resources on a consistent, long-term basis; they don't get long-term value from people having short, one-off anomalies, so budgets and monitoring are aligned with their customers - just not total cost of ownership calculations :)
  • bushbaba 5y ago
    Isn’t that why services such as AWS lightsail and digital ocean exist?
  • ghaff 5y ago
    Certainly the cloud providers probably make money by not having hard limits. But it's also the case that if they did implement hard limits of some sort, you'd be reading blog posts about how AWS destroyed my project just when it was going big because someone stuck a circuit breaker foot gun in some corner and everything stopped working properly when usage spiked. I do think there should probably be a hard circuit breaker. It should be simple and therefore inflexible. And it should come with a big warning sign. Still people will get burned because someone will set it, a project grows, and one day it goes off.
• _8j50 5y ago
  Didn't Troy sell HIBP to Verizon?
• mrb 5y ago
  Most worrying is that even an expert like Troy Hunt was UNABLE to figure out the cause of the issue by himself. He "reached out to a friend at Cloudflare" who investigated and found the cause.
• 2ion 5y ago
  This is why I use fixed price offerings for personal projects. A large bill is probably chump change for someone like Troy, for others it's a year or two of savings. The risk is not worth it.
  • schemescape 5y ago
    Would you mind sharing the services you’ve found that have fixed prices? I haven’t had much luck finding services like that (although I’m looking in the < $20/month range).
    • manquer 5y ago
      For fixed price and fixed performance you can use bare metal providers with unmetered bandwidth generally tier 2 vendors offer that. At $20 bare metal is not easily possible, the lowest prices I have seen are usually 40-50 and above. Howveve you can get a VPS with unmetered bandwidth and no other costs at your price range [1]. The price is still fixed some performance variances may be there, at $20 minor variances are unavoidable. [1] https://us.ovhcloud.com/vps/compare/ https://us.ovhcloud.com/vps/compare/
      • schemescape 5y ago
        Thanks! I had not heard of OVHcloud before. That looks like what I want.
        • manquer 5y ago
          Of tier 2 vendors hetzner , OVH, linode and scaleway are all pretty reliable
• Mave83 5y ago
  Just avoid cloud and choose dedicated infrastructure
• emptybottle 5y ago
  This is why I personally won't run projects on infrastructure with what roughly equates to unlimited risk billing. It's my opinion that it's better to work with known limitations and optimize for them. In the case of bandwidth, work with a fixed pipe size, or do the math and set up a QoS that implements a throttle to avoid exceeding your bandwidth allotment.
• ccbccccbbcccbb 5y ago
  > I have been, and still remain, a massive proponent of "the cloud". Mice cried and stung themselves, but kept eating the cactus.
• razzio 5y ago
  Hope it is okay and not too much off-topic. I just donated. He deserves it for this service! Fact is that stuff like this can happen. Consider how many variables are in play to determine the final cost of a cloud service it is very much a double-edged sword. Sometimes you cut yourself unintentionally. So now we all learn from this, I suggest we help him out.
• sergiotapia 5y ago
  >This was about AU$350 a day for a month. It really hurt, and it shouldn't have happened. I should have picked up on it earlier and had safeguards in place to ensure it didn't happen. It's on me. Uh no - it's on cloudflare and azure. Why don't they have a global setting that says Max Charges Per Month: $X and it just shuts down when it hits that number? This is why I don't really like using big cloud services like this.
• queuebert 5y ago
  Looking forward for the followup post in early 2033 when he forgets to extend the cost alert expiration.
• parentheses 5y ago
  TL;DR: I got a big bill from my cloud provider, so I used more cloud provider features, to make sure I know before I get the bill; isn't my cloud provider great?
• lom 5y ago
  If anything, this shows the insane scalability of the cloud
• Seattle3503 5y ago
  My (naive) solution. Every new account by default has an SMS alert that trips at $100. It says "Your account has exceed $100 spend. Reply 'SHUTDOWN' to shutdown all services, 'STOP ALERTS' to never see this alert again, or 'DOUBLE TRIGGER' to double the alert trigger value to $200." $100 is arbitrary, it could be any nominal sum. The idea being that the user can double the alert each time they get it just from SMS. I bet 95% of users would double their alert limit to a comfortable point. The other ~5% will be power users who customize their alerts. The idea that these companies couldn't know what limits customers want is kinda silly. We can use the same techniques for alerts that we use in algorithms for expanding vector storage, for example. We can "amortize" alerts, so to speak.
  • Nextgrid 5y ago
    The problem is that metering these services at such granularity is difficult: https://news.ycombinator.com/item?id=30066538 https://news.ycombinator.com/item?id=30066538
    • Seattle3503 5y ago
      It doesn't need to be very accurate. As long as the values are the same order of magnitude it is probably okay.
• mro_name 5y ago
  Shouldn't lookups be where cdb shines? Hold my beer: $ shard="$(echo "${sha1}" | cut -c 1)" $ cdb -q pwned-passwords-v8-sha1-${shard}.cdb "${sha1}" But as a cloud evangelist at Microsoft, you may sing the corporate IT gospel anyway. ¹https://mro.name/agakdfa https://mro.name/agakdfa