11 ms·
Render nuked my entire account with no notice
Recently I decided to migrate one of my side projects from AWS to Render for convenience and to manage some of my high ALB costs on AWS.
This weekend I began the process to move some of my infra to Render, including a static website, Postgres database and Redis. I switched the static website over but had some issues restoring my database from SQL backups (Render's Postgres appears to timeout connections after a certain amount of time).
After spending a few hours attempting to import all my data, I was logged out from dashboard and all my services were abruptly taken offline with no notice.
When trying to login again, it just throws me back to the login screen and sleuthing the network requests I can see I am now getting "Unauthorized" responses.
My production website was abruptly taken offline with the message "This service has been suspended by its owner." (I didn't suspend it)
For clarity, I was on a paid plan, had a credit card added, and all of these services were paid. I have received no communication, now I have to migrate back to AWS.
I can only assume they saw I was using the database heavily (normal during a restore) and decided I was abusing something? I absolutely did not violate any of their terms nor was I doing anything shady. It's just a static website and a Node service. Nothing crazy.
I just wanted to provide a warning to others because this seems egregious.
- tehlike 3y agoHetzner might be a good choice, cheap and simple to use. You manage your own infra, which may be a positive for some.
- hestefisk 3y ago+1 for Hetzner. I run my own server infra using FreeBSD. Much cheaper than GCP where I was before.
- thenickdude 3y agoHetzner deleted my dedicated server a couple of days after signup, and banned my account. A friend had the same experience. At least I didn't get charged anything for it (IIRC billing happens at the end-of-month). I even went through their live-video face verification and passport scan and they didn't budge on this.
- initplus 3y agoAnd yet in spite of this the majority of abusive traffic we receive is from Hetzner. What’s the point of these draconian identity measures if criminals are abusing others through your service anyway?
- abigail95 3y agoI hope this is sarcastic, it's causal. They have identity checks because people use it for abuse. The less strict they make their identity checks - would abuse increase, decrease or stay the same do you think?
- voyagerfan5761 3y agoThey might be preventing new malicious actors from signing up. Maybe. Catching a lot (anecdotally, from this thread) of legit users along the way. Meanwhile, what have they done to catch and ban malicious actors who passed verification long ago and have been abusing other services on the internet from the safety of Hetzner's network for months or years?
- _a9 3y agoFrom what I've experienced in the few years using them is that they're pretty strict on abuse, new account or not. I think initially they would null route all ips for the server when getting copyright reports but the past few times they would just email me the report and I had to reply with the fix or else they would null route (or maybe suspend) the server after a day or so of silence. While I do run a mail server with them I've never had any reports so I can't say anything on that but I'd assume theyre equally or even more strict with email abuse reports they get.
- alwa 3y agoWe are, of course, tacitly assuming that all the users on this thread were in fact doing legit things from Hetzner’s perspective when they were caught up (and, given the ID verification, that they have unremarkable pasts from a risk perspective). Although I’d be sore too if I were turned away in the signup process, I’d just as soon learn early that they don’t want my business, before I invest more time and effort getting moved in.
- andersa 3y agoAlso running everything on Hetzner that can possibly go there. Then I regularly see startups proudly explaining how they are wasting incredible sums on AWS when the same thing could be done 100 times cheaper with Hetzner and Cloudflare. Makes one think, dude, why??
- dist-epoch 3y agoBragging rights. "My AWS bill was $100k last month" is the new "I'm a manager with 10 direct reports".
- mnd999 3y agoMythic Beasts also good.
- bandergirl 3y agoCould it be that some keyword triggered the spam filters? It still doesn’t justify silent deletion, but spam fighting is so painful.
- soultrees 3y agoThat’s a business problem though, not a customer problem. And even more so now that this customer tried to move all their services, failed and has to move back to their old provider. This business will never ever get this customer back.
- gostsamo 3y agoWe need a new sub category [Support HN] when we need high level intervention from the CEO or at least a product manager. YC startups get an immediate notification if their name shows up there.
- andersa 3y agoFeels like making this an official thing would immediately lead to it being abused so much that people will stop paying attention, like they do on other social media.
- mcherm 3y agoI have long had an idea that the best way to handle this is a community supported prioritization service. In other words, a community (hacker news, some subreddit, or perhaps an independent site) would invite people to post problems that ought to be easily solved by customer support except that the company is failing at customer support. Because problems tend to repeat themselves, the people who spend time on this forum would be able to identify (and sometimes provide workarounds for) the majority of cases -- all at no cost to the company that was failing at customer support. The remaining cases could be ranked by the community according to how egregious they were. Then the company that was failing at customer support could assign a tiny amount of actual attention to look at just the top few issues sorted by the community.
- krger 3y ago> The remaining cases could be ranked by the community according to how egregious they were. It’s bad enough companies started outsourcing deciding which features to implement. I don’t want them also outsourcing deciding which cases to support.
- mcherm 3y agoI think it would be better to outsource it than to simply NOT fix anything for any customers OR only fixing issues for customers who manage to get a lot of attention on Hacker News.
- intothemild 3y agoI wanted to say that this isn't the first time I've heard of something similar to this with Render on HN.
- dabber 3y agoSo why didn't you?
- taneq 3y agoThey... did? Presumably because they wanted to?
- mcny 3y agoI think there is some miscommunication here. When I read I wanted to, that made me think that they knew about this before OP's problem. If they had made a warning post like warning - don't use render. It is a piece of poop, maybe OP would have saved a weekend.
- drekipus 3y agoKeen to hear from render about this. I just joined render and so I want to know what might cause them to do this. Always happy to hear alternative suggestions but I really just love being able to git push and leave it to CI to deploy easily.
- js4ever 3y agoYou might be interested by my similar service: https://elest.io/cicd https://elest.io/cicd We offer a similar experience to Render/Heroku but we do it on dedicated VM instead of a giant cluster with all the customers. So this means full isolation of your workloads. And you can deploy anywhere (112 regions supported and also BYOVM) We support 17 managed DB (including Postgres of course) and also 233 open source software.
- cpursley 3y agoThis looks nice. Can it handle distributed Elixir/Erlang like Render and Fly?
- js4ever 3y agoTBH I'm not aware about their specific offering related to distributed Elexir. But we do provide load balancers that can be easily configured from the UI, from our api or from our Terraform provider.
- cpursley 3y agoThat’s not the same thing. In the Erlang world, the machines are literally distributed - meaning they are meshed and talk to each other (share state, etc). It’s one of the big selling points of Render or Fly over say Heroku.
- beebeepka 3y ago> Full isolation. What hardware are you using? Asking since Intel just got hit real hard with Downfall - another round of speculative execution.
- js4ever 3y agoWe don't have our own hardware, we use cloud providers like AWS, Digital Ocean, Hetzner, Linode, Scaleway, Vultr to provide Intel, AMD and Ampere Altra (ARM64) So to mitigate Downfall, we rely on cloud providers infra. It's still way more isolated than a giant cluster hosting all the apps and routing becoming a single point if failure.
- prhrb 3y agoI had the same experience, I tried to deploy a repository of telegram userbot in python, they immediately banned without any notice
- dbcloudart-com 3y agoFor anything business critical I usually assume that any "cloud" / "provider" / "registry" may go dark on me any second. Therefore, your own DNS VMs, multicloud, your own database, no cloud vendor locking (agnostic microservices) etc etc. And if you want full control - the on-prem / colo solution is still the king.
- quickthrower2 3y agoYour own DNS VMs? Probably good enough to use registrar for most people for this job. Just pick a good one.
- ranting-moth 3y agoIt's 100% good enough most of the time.
- satvikpendem 3y agoSorry to hear that. For a rock solid alternative, I use Hetzner with Coolify (https://coolify.io https://coolify.io) installed, it's an open source Heroku alternative that uses Docker. I deploy all my apps on a 10 dollar Hetzner VPS that can handle millions of hits, and that 10 dollars gets you a much beefier machine than anything you'd get on other paid services like Render, Railway, Fly etc. They just aren't worth using in my experience when something like Coolify does the job for a fraction of the price.
- devjab 3y agoHetzner is quickly positioning itself to be the Azure/AWS replacement for a lot of shops here in the EU.
- moonchrome 3y agoDo they even offer managed database services ? If you don't even have rudimentary stuff like message broker/database you're not an AWS/Azure alternative.
- arghwhat 3y agoDoesn't really matter when using AWS, Azure and GCP is not an option. They only have to compete and compare with other "EU native" solutions, not the US cloud giants.
- hectormalot 3y agoThey don’t. But I’ve explored launching a EU-based business that focuses on large businesses and Hetzner was one of the logical players. Good managed services without US exposure are really rare. I run some smaller stuff on Hetzner. Have for the last 5-10 years and never had any issues. If Hetzner added managed Postgres, something like Cloud Run, and maybe PubSub I’d be moving over a bunch more stuff.
- devjab 3y agoIn my experience in non-SWE related enterprise this isn't really an issue. To put it as bluntly as possible, most EU organisations don't have enough work to staff an team for their IT operations. Maybe you could have one or two people on staff, but that is realistically too few people to run reliable operations, even in Azure or AWS. So what happens is that we outsource all the parts that aren't directly related to development or IT support and operations to 3rd party companies. Most of those 3rd party companies will happily sell you managed database service that are run on Hetzner. So this part of it is sort of a non-issue for a lot of organisations if the cost are low enough. Heck, even if you have your own inhouse operations department going for you, why wouldn't you be able to run your own stuff on Hetzner? That's what people did in the transition from having your iron in the basement and buying it from Microsoft/Amazon. The only reason we left the "own cloud" options was cost, and if Azure exceeds the alternative. Or the EU makes it too cumbersome to use Azure, then there won't be much of a reason to stay.
- deleted 3y ago[deleted]
- madsbuch 3y agoI have assessed Render a couple of times for work. Both. times they were just too unstable for production (in 2020 and in 2023)
- D13Fd 3y agoI’ve been using it since 2020, very happy with it. This post worries me though.
- hankmh 3y ago(Aptible employee) I help run another PaaS, and I’m very curious about your production evaluation criteria. Would you be willing to share? For context, our PaaS has been focused on running reliable and scalable production workloads for about a decade. We don’t really do hobbyist use cases, sadly. But instead we prioritize actual businesses with production use cases. If you share your evaluation criteria, I would love to test Aptible against it. I can share the results with you if you’d find it interesting, but frankly I just want the results for my team and me. I’d be so grateful, and happy to return the favor in any way I can. If you prefer, you can email me at henry AT aptible.com
- anurag 3y agoWhat did you find unstable exactly?
- madsbuch 3y agotheir IaaS system. it hangs. this gives very bad CI pipelines that you have to surveil. also it makes, eg. Github actions, insanely expensive as they are billed. per. minute, and render can hang upwards of hours or unbound if not stopped.
- anurag 3y agoOuch. We'll investigate and prioritize a fix.
- Justsignedup 3y agoI feel like this is a message for the render support team, not hn. Idk why people are going here first as if that's the shortcut to getting a response. Hit up here if render told you to stick it. But I also often see these end up with "I fucked up but want to yell at someone"
- EspressoGPT 3y agoEspecially because they should get an extra-fast support as a paid customer.
- mittermayr 3y agoI mean, I am with you mostly, but I also feel sometimes these "public support requests" help me be a tad more cautious when considering the companies mentioned. There's a few similar mentions of the exact same thing happening to other users in this thread, so I've definitely taken away a mental note to be weary when considering Render. Doesn't mean I won't use it because of a single bad comment, but it just helps addding one layer of extra diligence before switching.
- voyagerfan5761 3y agoWeary of Render is what OP will be if they don't take care of the problem. Wary of Render is what you (and I) should be.
- 3y ago
- EspressoGPT 3y agoAlthough the Render CEO is here on HN (@anurag), you should reach out to their support and not to HN.
- ranting-moth 3y agoI think a healthy name and shame is a good thing.
- arrowsmith 3y agoPresumably they already reached out to support, but getting your support query on the front page of HN is sure to increase the resolution speed.
- criddell 3y agoDo both. Render should have notified Oxy before suspending service to let them know what the problem is and how it can be rectified. They messed up first. By posting here, at least one other person has said they had the same problem and others are suggesting competitors. It’s useful to those of us who are looking for a hosting service. And like you say, anurag shows up here occasionally to say things like “the company is doubling down on making the cloud delightful”. If they want to use this forum for promotion, then it seems only fair for people who have been hurt by the shitty behavior of the company to come here to complain. A day ago, in response to another failure at Render, anurag was on HN saying “it's helpful to overshare in these situations”. Well, now they have another opportunity to overshare. What looks like a pattern of bad behavior may have a more reasonable explanation.
- hluska 3y agoI make a number of infrastructure choices and it’s very helpful for me to know that Render is trigger happy with bans, shows misleading error messages and then doesn’t communicate.
- andrewstuart 3y agoPeople get smacked by the cloud and their solution is to move to…….. the cloud.
- minajevs 3y agoI had exactly the same experience with Render. I was researching where it would be cheaper and easier to setup my infrastructure. So, I deployed bunch of services to Render, and boom, "Unauthorized" and "This service has been suspended by its owner". Sent an email to support, got first reply after 4 days, and then silence for a month, until I angrily asked them what's up. No need to say that I will probably never go with Render.
- jpmoral 3y agoWhat was their response to your angry question?
- minajevs 3y agoThey replied the same day saying that they unblocked my account and I should be able to login. No explanation why it was blocked or anything.
- anurag 3y agoThis is unacceptable. You'd be well within your rights to ignore this request, but I'd appreciate an email with your details so we can fix the underlying problem.
- anurag 3y agoReally sorry about this. Can you email me (address in profile) so we can investigate?
- anurag 3y agoWe've resumed all services for OP. The tl;dr is this was a new account with no billing history and on track for a relatively high monthly spend. We attempted to authorize the card on file for the projected amount, and the authorization failed, leading to automatic suspension. This is obviously overzealous abuse management, and we will fix our systems so this kind of thing never happens to legitimate users. To OP and others who may have been affected, I'm deeply sorry.
- spiralganglion 3y agoAs someone considering migrating from Heroku to Render, the suggestion that you'd suspend an account — paid or not — without notice is concerning. You said you'd fix your systems, but didn't specify how. • Will you pause the resource utilization, but still allow log-in? • Will you give notice?
- anurag 3y agoYes. We will find/build a way for customers to verify their account in these cases before pausing compute.
- rsync 3y ago"We attempted to authorize the card on file for the projected amount, and the authorization failed, leading to automatic suspension." Does "render" (whatever they are) own their own infra ? If so, this is a weird response since, really, who cares what the retail monthly spend is projected to be when, on your end, it's just some ineffable, tiny portion of your electricity and transit bill. Right ? Unless, of course, "render" (whatever they are) is itself hosted on an underlying cloud. Let me guess: you don't own your own infra ?
- anurag 3y agoWe've chosen to run on AWS/GCP for now so we can focus on building higher-level abstractions that help devs move faster, and we've always been very public about it.
- figassis 3y agoI understand the value of all the PaaS and SaaS, but I could never get 100% behind them, even when they started. The highest I ever went was DigitalOcean (so IaaS). I tried some layers on top, lime serverpilot (automated php hosting on your DO droplets), but the entire thing always felt too risky, like giving the keys to your kingdom so someone whose pricing, policies and features you had no control over. I decided to always stay on a layer that is so commoditized that behavior (business, pricing, features, failure modes) is basically uniform across providers. I am atrações to killer pricing and stay away from killer (read lock-in) features. If I need a platform, I setup an open source one, like kubernetes, coolify, etc. This would have been fine if these startups grew on the merits (like DO). But they want to run infrastructure like they run a social network. No.
- 0xy 3y agoUPDATE: I got in contact with Anurag Goel, the CEO at Render and he was very apologetic and restored my account and explained why it was caught up in some anti-fraud processes. I'll be giving them another shot.
- sunguroku 3y agoTry porter.run for full control over your infrastructure with Render style experience on AWS