5 ms·
8 days downtime: Cloudflare r2 subscription bug ruins my business
Our website has been severely impacted for 8+ days because I can't add the R2 Subscription back to my account.
The error is: "You cannot add or modify subscriptions or services until all invoices are paid. Download each "Unpaid" invoice on your Billing page and click the "Pay online" link within each invoice to fix your account"
All my invoices are paid, I contacted Cloudflare support but literally no one is answering. I don't know what to do anymore.
They closed my old ticket then opened a "report" where I had to verify my identity with Stripe. After I did that they didn't respond anymore.
Ticket number: #17419451
- dvfjsdhgfv 2y agoI keep my fingers crossed.
- cpach 2y agoCan you restore your data from backups and move to another storage provider?
- hriis 2y agounfortunately not, i dont have access to the storage at all, also backups wouldn't be a good idea (storage size is 800gb, we have a 100k+ user website where users frequently upload files)
- trollied 2y ago800gb is nothing. You should be backing it up.
- hriis 2y agomight be a little too late now.. also idk if cloudflare even supports backups, i'd have to make my own
- Nullabillity 2y agoCloudflare-managed backups wouldn't help you against Cloudflare revoking access.
- ccakes 2y agoIt's a hard lesson to learn like this, but it's a learning moment nonetheless. You don't rely on your provider to make backups for business-critical stuff like this, you need to manage it (and test!) it yourself. rclone[1] supports tons of storage providers. If you get access back and recover your business, start making backups onto other providers. [1] https://rclone.org/ https://rclone.org/
- kikimora 2y agoI would say the lesson is just use AWS which has outstanding support. Building backups yourself, going multi cloud or else just erase the benefit of CF. If you do this you can just run on a bare metal and save on cloud cost.
- cpach 2y agoAWS is very good, but you could get locked out there too. With no backups in another place, the ability to do disaster recovery is at risk.
- FaridIO 2y agoIf your business depends on it, not backing it up in some other place on a consistent basis is a bad idea. The cost of this should be factored into the price you charge people. 800GB would be less than 20 dollars on S3, all you have to figure out is the replication process, but that shouldn't be rocket science either.
- RKearney 2y agoI’m currently ~6 weeks into a billing issue with CloudFlare. They typically take 5-6 days to respond, ask you for verification to do things, wait another 5-6 days, then they ask you to do it because it turns out they don’t have permission. It’s a mess.
- koollman 2y agoif your business depends on someone else's entirely to work, you have to factor that into your risks and maybe plan for it. No alternatives, no disaster recovery plan, nothing you could try for more than a week with some other provider ?
- hriis 2y agowell, i never knew this could be an issue. the problem is that i dont have access to the r2 storage at all
- koollman 2y agoexternal/independant backups are a thing. They would be part of a disaster recovery plan (and you should test the restore process and result from time to time, else they might be worthless when you actually need them). I understand this is not helpful right now, but there are best practices and many ways to plan for this kind of stuff for good reasons. Some things can and will go wrong, and sometimes that may cost your business a lot, which means you have to anticipate (or accept) the possible costs and risks. It is part of the job
- vorador 2y agoThis is very unhelpful. There's a difference between "someone fatfingered a command and brought this mysql machine down" and "our cloud provider shut down a core feature for our account". When you design a new system do you plan for S3 to go down for more than a day? Do you have a fleet of offsite machines to smoothly transition to? If not, why not?
- misiek08 2y agoIf data on S3 is crucial for business to be alive? Of course! I would even think about writing data to 2 separate storages in real time to be able to switch easily. 3-2-1 backup is a no joke, especially when it's your business, not a hobby project. Having DR too. And it's not a legacy, 20 years old system, so it's much easier to do such plans and test them.
- trollied 2y agoYou appear to be lost, anyway. This isn't a random support site for cloud providers. People will no doubt downvote me for saying this, but if everyone posted their woes on here this site would be next to useless. It's also quite normal for such pleas to leave out lots of vital information, and just present a good looking view, rather than the actual history.
- ceejayoz 2y ago> You appear to be lost, anyway. This isn't a random support site for cloud providers. It often functions this way. Quite a few times with CloudFlare. It's not the first place to go, but it has historically been a helpful last-resort option. For example: https://news.ycombinator.com/item?id=34639212 https://news.ycombinator.com/item?id=34639212
- trollied 2y agoI am well aware of that. It doesn't change the fact that this isn't a support forum, and others being successful should not encourage the behaviour.
- ceejayoz 2y agoEasier to say when one’s business isn’t at risk, I suspect.
- Brian_K_White 2y agoThe old couple who lived across the street from me for the entire 20 years I've been here both died within a few weeks of each other only a few weeks ago. I'm really sad because I liked them. I know this is relevant to you because to say otherwise would be easy to say when one's own friends didn't just die.
- ceejayoz 2y agoIf posting about loved ones dying to HN had a long history of successfully resurrecting people, this would indeed be a great analogy.
- 1over137 2y agoHonest question: why do you need CloudFlare, why not self-host?
- hriis 2y agowell, r2 was cheap and "good" so i went for it, but i also use cloudflare to proxy my backend ip.
- ivanvanderbyl 2y agoI have a similar issue with R2 billing, but related to enabling bucket events. Says I need a subscription, which I already have. I opened a ticket in ~March while they were having a huge billing outage. After a month of pinging them every 3 days for an update they closed the ticket without replying marking it as resolved. I still don’t have access to bucket events. I want to assume it must be an absolute mess internally where they can’t even reply to a ticket.
- LinuxBender 2y agoThe CTO's email is sortof in his profile [2]. I assume he would want to know if there is a business impact on someone assuming in good faith everything stated in the submission is accurate. CEO & co-founder of CloudFlare. [1] Cloudflare CTO [2] Not listing their individual contributors. [1] - https://news.ycombinator.com/user?id=eastdakota https://news.ycombinator.com/user?id=eastdakota [2] - https://news.ycombinator.com/user?id=jgrahamc https://news.ycombinator.com/user?id=jgrahamc https://jgc.org/ https://jgc.org/
- hriis 2y agoyou really think he would respond if i email him?
- LinuxBender 2y agoAll I can say for certain is that he will not reply if you do not email him. Just keep it professional and state only the facts, is the only thing I can offer.
- jgrahamc 2y agoYes, I would. My email is jgc@cloudflare.com.
- spaceheater 2y agoWhy is this flagged? When cloudflare announces R2 there are hundreds comments praising their announcement-ad. Now that we get an actual review of what may happen, it gets flagged?