4 ms·

Literally just sent an email to my devops guys to move off cloudflare asap. This cavalier lack of respect is a diservice and insult to all the people who rely o

Literally just sent an email to my devops guys to move off cloudflare asap. This cavalier lack of respect is a diservice and insult to all the people who rely on my product for their livelihood.

cybrox 4y ago
Are there any good alternatives that you (or him) already looked into?
- ignoramous 4y ago
  Not OP, but we run two environments each of our service on Cloudflare Workers, Deno Deploy, and Fly.io (a small service albeit, 200 to 300 requests per second). In the event one is down, we switch to the other (via DNS).
  - sparrish 4y ago
    This, my system architect friends, is the proper response. Do NOT put all your eggs in 1 basket. Build redundancies and failovers so no 1 vendor can shutdown your business.
- cultofmetatron 4y ago
  we had already been discussing amazon s3 since we are on aws. As clunky as aws can be, you can get in touch with a human if you need to.
kkielhofner 4y ago
You’re changing your arch because you saw a one-sided completely unverified post on HN? At this point @jgrahamc has the worst of it - people show up here time after time hoping they can make enough of a stink to get him involved.
- jgrahamc 4y ago
  When they could just email me (jgc@cloudflare.com)
  - tcldr 4y ago
    Funny. That's what OP said: > I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely?
  - iamacyborg 4y ago
    The fact that stuff needs to be raised at all is the problem. Clearly something has gone wrong if customers get treated this way.
    - mdip 4y ago
      Before piling on too much more, here ... ... Cloudflare has a lot of customers[0]. They have to balance the cost of providing (a lot of) human support against the cost they can reasonably charge for their products. It's a balancing act, and one that has worked out well for me, personally. It sounds like this issue is happening related to R2, which is quite new. You're not likely to see a post hit the front page with the title "I've integrated Cloudflare's products with 30 or so customers and never had an issue" (or even be written). But experience an issue this large and you're going to do everything -- make calls, post things to social media, reach out on HN where you know the CTO is an active participant -- and a lot of those are going to get attention from the small percentage of customers who felt wronged by CF but hadn't spoken up. It's a crappy situation because it gives the impression that things are a mess when -- I'm willing to bet -- it's something along the lines of a problem in a quota checker and a failure of internal process to escalate the problem appropriately. That happens at every big company in various places all the time. Really, the only major difference here is that unlike every other big company, their CTO actively watches Hacker News. When a problem pops up, he willingly chooses to be Customer Service and from the sounds of it, that escalation to address "problems like this" is now happening. There's going to be gaps like this at every company. When I worked at "BigCo", if something like this hit the front page of HN, you could expect a mess of people to have their phones ring. Work would be done to respond to the customer (variations on "acknowledge/minimize/suppress" communications -- on official company hosts). Staff would be forbidden from interacting in the ongoing discussion. The CTO might have had to have explained to him how to get to the web site containing the complaint. [0] I don't work for them; I'm just a happy customer so everything here is my view from the outside.
vertis 4y ago
This is the height of knee-jerk reactions, worse still it's largely pointless. Unless you're big enough to negotiate a specific contract with a cloud provider you're always going to be at the mercy of their catch-all policy. The only way to actually be protected in this case is to run a multi-cloud strategy. Even then it's only going to protect you so far if you piss off the powers-that-be / community (see the hosting trouble Parler had as an example, not that I'm fond of Parler or anything).
- dicknuckle 4y ago
  If the redundancy is already in place to not fully rely on cloudflare's product (whatever it is, DNS, R2, etc) the it's not a kneejerk reaction. It's an "I don't want to wake up to all our stuff running only on the backup provider because cloudflare shut us down for seemingly no reason with no warning". It's avoiding unnecessary alerts and triage for the ops team by snipping an apparent liability from the stack. I've already done the same after seeing a few of these kinds of interactions with cloudflare in the R2 discord. When I see a blog post detailing why this has been happening so often, and what they've done to fix it, I'll happily pull that infra code out of the mothballs.
  - vertis 4y ago
    So instead you're running only on the backup provider now. Congratulations on invalidating your multi-cloud strategy. You've failed to understand that the core reason you had that strategy is that any of them can let you down. Every single one of the cloud providers has had instances of this kind of problem. It's somewhat an inevitability of the way they all work. Eventually someone triggers an automated system somewhere and gets taken down. Or has outages that they shouldn't have had. Better cloudflare where the CTO hangs out on HN, than Google where both the ban and the appeal are not even humans with empathy.
- deleted 4y ago
  [deleted]

Literally just sent an email to my devops guys to move off cloudflare asap. This cavalier lack of respect is a diservice and insult to all the people who rely on my product for their livelihood.

cybrox 4y ago

Are there any good alternatives that you (or him) already looked into?

ignoramous 4y ago
Not OP, but we run two environments each of our service on Cloudflare Workers, Deno Deploy, and Fly.io (a small service albeit, 200 to 300 requests per second). In the event one is down, we switch to the other (via DNS).
- sparrish 4y ago
  This, my system architect friends, is the proper response. Do NOT put all your eggs in 1 basket. Build redundancies and failovers so no 1 vendor can shutdown your business.
cultofmetatron 4y ago
we had already been discussing amazon s3 since we are on aws. As clunky as aws can be, you can get in touch with a human if you need to.

kkielhofner 4y ago

You’re changing your arch because you saw a one-sided completely unverified post on HN? At this point @jgrahamc has the worst of it - people show up here time after time hoping they can make enough of a stink to get him involved.

jgrahamc 4y ago
When they could just email me (jgc@cloudflare.com)
- tcldr 4y ago
  Funny. That's what OP said: > I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely?
- iamacyborg 4y ago
  The fact that stuff needs to be raised at all is the problem. Clearly something has gone wrong if customers get treated this way.
  - mdip 4y ago
    Before piling on too much more, here ... ... Cloudflare has a lot of customers[0]. They have to balance the cost of providing (a lot of) human support against the cost they can reasonably charge for their products. It's a balancing act, and one that has worked out well for me, personally. It sounds like this issue is happening related to R2, which is quite new. You're not likely to see a post hit the front page with the title "I've integrated Cloudflare's products with 30 or so customers and never had an issue" (or even be written). But experience an issue this large and you're going to do everything -- make calls, post things to social media, reach out on HN where you know the CTO is an active participant -- and a lot of those are going to get attention from the small percentage of customers who felt wronged by CF but hadn't spoken up. It's a crappy situation because it gives the impression that things are a mess when -- I'm willing to bet -- it's something along the lines of a problem in a quota checker and a failure of internal process to escalate the problem appropriately. That happens at every big company in various places all the time. Really, the only major difference here is that unlike every other big company, their CTO actively watches Hacker News. When a problem pops up, he willingly chooses to be Customer Service and from the sounds of it, that escalation to address "problems like this" is now happening. There's going to be gaps like this at every company. When I worked at "BigCo", if something like this hit the front page of HN, you could expect a mess of people to have their phones ring. Work would be done to respond to the customer (variations on "acknowledge/minimize/suppress" communications -- on official company hosts). Staff would be forbidden from interacting in the ongoing discussion. The CTO might have had to have explained to him how to get to the web site containing the complaint. [0] I don't work for them; I'm just a happy customer so everything here is my view from the outside.

vertis 4y ago

This is the height of knee-jerk reactions, worse still it's largely pointless. Unless you're big enough to negotiate a specific contract with a cloud provider you're always going to be at the mercy of their catch-all policy. The only way to actually be protected in this case is to run a multi-cloud strategy. Even then it's only going to protect you so far if you piss off the powers-that-be / community (see the hosting trouble Parler had as an example, not that I'm fond of Parler or anything).

dicknuckle 4y ago
If the redundancy is already in place to not fully rely on cloudflare's product (whatever it is, DNS, R2, etc) the it's not a kneejerk reaction. It's an "I don't want to wake up to all our stuff running only on the backup provider because cloudflare shut us down for seemingly no reason with no warning". It's avoiding unnecessary alerts and triage for the ops team by snipping an apparent liability from the stack. I've already done the same after seeing a few of these kinds of interactions with cloudflare in the R2 discord. When I see a blog post detailing why this has been happening so often, and what they've done to fix it, I'll happily pull that infra code out of the mothballs.
- vertis 4y ago
  So instead you're running only on the backup provider now. Congratulations on invalidating your multi-cloud strategy. You've failed to understand that the core reason you had that strategy is that any of them can let you down. Every single one of the cloud providers has had instances of this kind of problem. It's somewhat an inevitability of the way they all work. Eventually someone triggers an automated system somewhere and gets taken down. Or has outages that they shouldn't have had. Better cloudflare where the CTO hangs out on HN, than Google where both the ban and the appeal are not even humans with empathy.
deleted 4y ago
[deleted]