16 ms·

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If

This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees). BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.

diegoperini 7y ago
First, you are awesome, really :) Second, a bunch of honest questions: Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the industry? I'd like to put my skepticism on hold and blindly believe that your post is a reflection of pure concern and not just a PR stunt for damage control.
- wila 7y ago
  The CTO at Cloudflare has to ask his supervisor before trying to help someone over at hacker news? :)
  - diegoperini 7y ago
    I didn't know which probably slightly proves some point other people made.
- stedaniels 7y ago
  jgrahamc is the CTO of Cloudflare. His only supervisor is eastdakota/ Matthew Prince, the CEO of Cloudflare.
  - diegoperini 7y ago
    Then I'd like to know the CEO's opinion then.
    - dboreham 7y ago
      CTO of any company doesn't need the CEO's permission to make a customer relations post.
      - diegoperini 7y ago
        It's not about permission but cooperation of two disciplines.
    - gjs278 7y ago
      he doesn’t care at all. you are asking really bizarre questions.
    - diegoperini 7y ago
      It's hard to understand criticism when it is not put into words. From my perspective, a developer from the company the article is about arrives in the comment section. I ask an honest question about the communication process internal to the company. I am told this developer is the CTO and is supervised by the CEO. Then I state the fact that I would appreciate if we also knew what the CEO thinks about this situation. I don't literally expect answers from the people I addressed and that's OK, they are probably too busy working on the subject matter. If they take time and write one, that is really admirable and I'd be thankful. What I don't understand is why it is considered inappropriate to be curious about these topics, as judged by the amount of downvotes I received. Maybe the tone I intended for my post got lost in translation, since I'm not a native English speaker.
      - manigandham 7y ago
        This reply might be what you're looking for: https://news.ycombinator.com/item?id=22407399 https://news.ycombinator.com/item?id=22407399 Cloudflare is a solid company with a good reputation and has done a lot for general welfare of the internet. The staff there generally care about customers and it seems like they're trying to figure out what went wrong here, and of course preventing negative PR is always a bonus. There's really not much of a process when you're a top officer at a company, you have the freedom and responsibility to exercise your judgement and do whatever it takes for the good of the organization.
paulddraper 7y ago
Please do update if possible. It's likely a good learning for all.
- jgrahamc 7y ago
  Of course, I just don’t want to promise something when it might be revealing information about an individual account.
rationalfaith 7y ago
You better add redundancies here on untracked transactions on your DNS record ledger.
iudqnolq 7y ago
OP here. You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first. I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.) Edit2: I see that the domain is back in my account and listed as "Pending Nameserver Update". I don't think that's because of something I did.
- jgrahamc 7y ago
  Thanks. Appreciate it. I’ll let the team look into it and communicate with you first.
  - iudqnolq 7y ago
    I appreciate you're looking in to this.
    - jgrahamc 7y ago
      I’ve worked for Cloudflare since it was 24 people. I care a hell of a lot about our customers. I know eastdakota does also.
      - iudqnolq 7y ago
        I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.
        melq 7y ago
        This is probably obvious but you're getting this kind of support because they're getting terrible PR by virtue of your post being at the top of HN. Not because he's such a great guy that cares so much about the customers. Not saying he isn't a great guy, or that anyone at cloudflare actually wanted this to be your experience.
andrewstuart 7y ago
I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved. HN constantly has front page posts from people for whom big tech companies have support processes have failed but there is simply no other recourse unless you have "a friend in the business". It just doesn't work to have some random Cloudflare person offer their email address as some post disaster issue resolution process on social media. Formalise it with an official Ombudsman and maybe then companies like Cloudflare might avoid HN front page bad publicity. I had an issue at "one of the biggest tech companies" that went on for days and days in which tech support kept telling me I had set up something wrong, until eventually I emailed one of the top managers who I happen to "know" at that company - it was fixed within hours. That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
- subhro 7y ago
  This is a really nice idea. This already exists for some companies in the form of Twitter accounts. I doubt the people doing the typee-typee actually has any authority over business processes, or demand a change, but I think they at least have the business owners on their speed dial, vs normal support tickets. But having an email is far better.
  - andrewstuart 7y ago
    Yes it's a good point that companies currently have some sort of Twitter presence trying to address bad publicity posts, often they seem to be able to get things done. The Ombudsman role is there to get things fixed when all else has failed, and before the angry customer posts to social media.
  - p0sixlang 7y ago
gist 7y ago
> BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people. I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare. The reason is people will tend to (as a rule) do as little as they can themselves but then use as a hammer the court of public opinion to get something resolved. You say 'ever find yourself so frustrated with Cloudflare' but you know that in itself is different for different people. What will happen is you will get people using you as a help desk and then after you don't help them as quickly as they think you should they will then follow up with a post, comment or story about how you did nothing. Separately if someone is posting publicly about an issue (as this person is) and if you can verify that it's actually coming from the customer (I mean who says it is actually?) I don't think you need them to say it's ok to resolve online. In fact to me it's the opposite. You take the time to reach out publicly and you take what follows good or bad even calling you out (the customer yes you can do that by the way) if you think they didn't put the appropriate effort into finding an answer.
- jgrahamc 7y ago
  I disagree with you. It matters enormously that people like me are available. Sure there are time wasters who’ll send me email. But I don’t care. Dealing with a small number of real customers doesn’t take a lot of my time and matters. Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct emails from customers or others.
  - dkersten 7y ago
    Thank you, this is a good attitude to have and I wish more company exec’s thought like this. We might have fewer post’s like OP’s story.
    - 7y ago
martin1975 7y ago
You guys are the worst censors even on your own blog. Any criticism toward your CEO or the way things have been done, completely out of integrity with your own policies in the past (such as cutting out providers because your CEO woke up self-righteous on the wrong foot that day) gets moderated away or not even admitted to the CF blog. You've screwed up so many times, I am surprised by now more people aren't onto your tired antics. Thankfully, you cannot delete this post - perhaps many fanboys will downgrade it, but at least I can tell you how I feel.
- scrollaway 7y ago
  Not a fanboy, still downvoted your post because it's incredibly whiny and does not contribute anything. All you're doing is accuse of censorship without evidence, you say they screwed up "so many times" yet fail to show one example, and then you preemptively accuse people who would dare downvote you to be fanboys. Sorry, but that comment is noise. Mine is too, but hopefully it helps you see things more clearly rather than let you pat yourself on the back thinking "HN is full of fanboys anyway".
  - martin1975 7y ago
    He already knows what I'm talking about. And the evidence is out there, just search "Matthew Prince censorship". Abusing 1A rights because he feels like it, hiding under the corporate veil. It's about as un-American as it gets.
    - tristor 7y ago
      Please explain to me how it's un-American for a company to choose who they want as their customers? The typical Conservative position would be that businesses have a right to choose who they wish to serve, and denying service should not be an illegal act as their prospective customers can go elsewhere to find another business to serve them, which is exactly what those disgusting hatemongers did. You are of course referring to this: https://blog.cloudflare.com/why-we-terminated-daily-stormer/ https://blog.cloudflare.com/why-we-terminated-daily-stormer/ Why try to hide behind your words? Just straight out say you are angry at CloudFlare for ceasing to host violent neo-Nazi propaganda.
p1necone 7y ago
The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve. Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."
- p1necone 7y ago
  I posted this under the wrong parent comment, moved but cant delete.
rattray 7y ago
Context for the lazy, jgrahamc is the longtime CTO of Cloudflare.
jiggawatts 7y ago
Please explain something to me. For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario. So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process. Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor having a handgun as one of his medical instruments.
- mc3 7y ago
  Why can you delete DNS records? Well it's pretty common to want to do so (usually followed by adding some new ones). Your doctor has a syringe as a medical instrument. They can kill if not used correctly.
  - jiggawatts 7y ago
    This is not the user manually deleting records. This is the doctor seeing a patient that hasn't moved in a few hours and executing them: https://community.cloudflare.com/t/does-cloudflare-automatically-purge-dns-zones/140898/3 https://community.cloudflare.com/t/does-cloudflare-automatic... Oh wait, he was just sleeping? Ah, well... "oops". There is exactly one reason you'd delete customer data without their active involvement: They're overdue on their bill and have ignored all warnings and communications. That's it. There are no other reasons. Feel free to disable things if they're a security risk, but Do. Not. Delete. Customer. Data. Ever.
    - mc3 7y ago
      Ugh! I may have interpreted the comment I responded to wrong, and replied to my own strawman. I think you need the power for the user to delete records (obviously) but this could mean a bug in the system could cause the records to be deleted. But it sounds from what you are saying that they sort of "intentionally" deleted the records (not sure if a human had the intent or an algorithm?) but if intentionally deleted yes that sucks, there is no reason for that I can think of, given what they do is about handling DDOS. Unless the mechanism exists for legal reasons, i.e. they are forced to take some sites down by the government or law suits.
ajonit 7y ago
Now that the OP has given a go ahead to go public, We will eagerly wait for your update jgc
AndrewWarner 7y ago
He gave me an incredibly detailed and fast response to an issue I was fact-checking.
rswail 7y ago
Having the CTO offer up support in this way is heartening. Especially because infrastructure suppliers are getting more and more centralized, which leads to possible SPOFs even with individual company distributed environments.

diegoperini 7y ago

First, you are awesome, really :) Second, a bunch of honest questions: Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the industry? I'd like to put my skepticism on hold and blindly believe that your post is a reflection of pure concern and not just a PR stunt for damage control.

wila 7y ago
The CTO at Cloudflare has to ask his supervisor before trying to help someone over at hacker news? :)
- diegoperini 7y ago
  I didn't know which probably slightly proves some point other people made.
stedaniels 7y ago
jgrahamc is the CTO of Cloudflare. His only supervisor is eastdakota/ Matthew Prince, the CEO of Cloudflare.
- diegoperini 7y ago
  Then I'd like to know the CEO's opinion then.
  - dboreham 7y ago
    CTO of any company doesn't need the CEO's permission to make a customer relations post.
    - diegoperini 7y ago
      It's not about permission but cooperation of two disciplines.
  - gjs278 7y ago
    he doesn’t care at all. you are asking really bizarre questions.
  - diegoperini 7y ago
    It's hard to understand criticism when it is not put into words. From my perspective, a developer from the company the article is about arrives in the comment section. I ask an honest question about the communication process internal to the company. I am told this developer is the CTO and is supervised by the CEO. Then I state the fact that I would appreciate if we also knew what the CEO thinks about this situation. I don't literally expect answers from the people I addressed and that's OK, they are probably too busy working on the subject matter. If they take time and write one, that is really admirable and I'd be thankful. What I don't understand is why it is considered inappropriate to be curious about these topics, as judged by the amount of downvotes I received. Maybe the tone I intended for my post got lost in translation, since I'm not a native English speaker.
    - manigandham 7y ago
      This reply might be what you're looking for: https://news.ycombinator.com/item?id=22407399 https://news.ycombinator.com/item?id=22407399 Cloudflare is a solid company with a good reputation and has done a lot for general welfare of the internet. The staff there generally care about customers and it seems like they're trying to figure out what went wrong here, and of course preventing negative PR is always a bonus. There's really not much of a process when you're a top officer at a company, you have the freedom and responsibility to exercise your judgement and do whatever it takes for the good of the organization.

paulddraper 7y ago

Please do update if possible. It's likely a good learning for all.

jgrahamc 7y ago
Of course, I just don’t want to promise something when it might be revealing information about an individual account.

rationalfaith 7y ago

You better add redundancies here on untracked transactions on your DNS record ledger.

iudqnolq 7y ago

OP here. You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first. I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.) Edit2: I see that the domain is back in my account and listed as "Pending Nameserver Update". I don't think that's because of something I did.

jgrahamc 7y ago
Thanks. Appreciate it. I’ll let the team look into it and communicate with you first.
- iudqnolq 7y ago
  I appreciate you're looking in to this.
  - jgrahamc 7y ago
    I’ve worked for Cloudflare since it was 24 people. I care a hell of a lot about our customers. I know eastdakota does also.
    - iudqnolq 7y ago
      I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.
      - melq 7y ago
        This is probably obvious but you're getting this kind of support because they're getting terrible PR by virtue of your post being at the top of HN. Not because he's such a great guy that cares so much about the customers. Not saying he isn't a great guy, or that anyone at cloudflare actually wanted this to be your experience.

andrewstuart 7y ago

I've put this idea forward a number of times here on HN in regards to other big tech companies. Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed. The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company. To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved. HN constantly has front page posts from people for whom big tech companies have support processes have failed but there is simply no other recourse unless you have "a friend in the business". It just doesn't work to have some random Cloudflare person offer their email address as some post disaster issue resolution process on social media. Formalise it with an official Ombudsman and maybe then companies like Cloudflare might avoid HN front page bad publicity. I had an issue at "one of the biggest tech companies" that went on for days and days in which tech support kept telling me I had set up something wrong, until eventually I emailed one of the top managers who I happen to "know" at that company - it was fixed within hours. That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.

subhro 7y ago
This is a really nice idea. This already exists for some companies in the form of Twitter accounts. I doubt the people doing the typee-typee actually has any authority over business processes, or demand a change, but I think they at least have the business owners on their speed dial, vs normal support tickets. But having an email is far better.
- andrewstuart 7y ago
  Yes it's a good point that companies currently have some sort of Twitter presence trying to address bad publicity posts, often they seem to be able to get things done. The Ombudsman role is there to get things fixed when all else has failed, and before the angry customer posts to social media.
- p0sixlang 7y ago

gist 7y ago

> BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people. I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare. The reason is people will tend to (as a rule) do as little as they can themselves but then use as a hammer the court of public opinion to get something resolved. You say 'ever find yourself so frustrated with Cloudflare' but you know that in itself is different for different people. What will happen is you will get people using you as a help desk and then after you don't help them as quickly as they think you should they will then follow up with a post, comment or story about how you did nothing. Separately if someone is posting publicly about an issue (as this person is) and if you can verify that it's actually coming from the customer (I mean who says it is actually?) I don't think you need them to say it's ok to resolve online. In fact to me it's the opposite. You take the time to reach out publicly and you take what follows good or bad even calling you out (the customer yes you can do that by the way) if you think they didn't put the appropriate effort into finding an answer.

jgrahamc 7y ago
I disagree with you. It matters enormously that people like me are available. Sure there are time wasters who’ll send me email. But I don’t care. Dealing with a small number of real customers doesn’t take a lot of my time and matters. Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct emails from customers or others.
- dkersten 7y ago
  Thank you, this is a good attitude to have and I wish more company exec’s thought like this. We might have fewer post’s like OP’s story.
  - 7y ago

martin1975 7y ago

You guys are the worst censors even on your own blog. Any criticism toward your CEO or the way things have been done, completely out of integrity with your own policies in the past (such as cutting out providers because your CEO woke up self-righteous on the wrong foot that day) gets moderated away or not even admitted to the CF blog. You've screwed up so many times, I am surprised by now more people aren't onto your tired antics. Thankfully, you cannot delete this post - perhaps many fanboys will downgrade it, but at least I can tell you how I feel.

scrollaway 7y ago
Not a fanboy, still downvoted your post because it's incredibly whiny and does not contribute anything. All you're doing is accuse of censorship without evidence, you say they screwed up "so many times" yet fail to show one example, and then you preemptively accuse people who would dare downvote you to be fanboys. Sorry, but that comment is noise. Mine is too, but hopefully it helps you see things more clearly rather than let you pat yourself on the back thinking "HN is full of fanboys anyway".
- martin1975 7y ago
  He already knows what I'm talking about. And the evidence is out there, just search "Matthew Prince censorship". Abusing 1A rights because he feels like it, hiding under the corporate veil. It's about as un-American as it gets.
  - tristor 7y ago
    Please explain to me how it's un-American for a company to choose who they want as their customers? The typical Conservative position would be that businesses have a right to choose who they wish to serve, and denying service should not be an illegal act as their prospective customers can go elsewhere to find another business to serve them, which is exactly what those disgusting hatemongers did. You are of course referring to this: https://blog.cloudflare.com/why-we-terminated-daily-stormer/ https://blog.cloudflare.com/why-we-terminated-daily-stormer/ Why try to hide behind your words? Just straight out say you are angry at CloudFlare for ceasing to host violent neo-Nazi propaganda.

p1necone 7y ago

The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve. Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."

p1necone 7y ago
I posted this under the wrong parent comment, moved but cant delete.

rattray 7y ago

Context for the lazy, jgrahamc is the longtime CTO of Cloudflare.

jiggawatts 7y ago

Please explain something to me. For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario. So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process. Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor having a handgun as one of his medical instruments.

mc3 7y ago
Why can you delete DNS records? Well it's pretty common to want to do so (usually followed by adding some new ones). Your doctor has a syringe as a medical instrument. They can kill if not used correctly.
- jiggawatts 7y ago
  This is not the user manually deleting records. This is the doctor seeing a patient that hasn't moved in a few hours and executing them: https://community.cloudflare.com/t/does-cloudflare-automatically-purge-dns-zones/140898/3 https://community.cloudflare.com/t/does-cloudflare-automatic... Oh wait, he was just sleeping? Ah, well... "oops". There is exactly one reason you'd delete customer data without their active involvement: They're overdue on their bill and have ignored all warnings and communications. That's it. There are no other reasons. Feel free to disable things if they're a security risk, but Do. Not. Delete. Customer. Data. Ever.
  - mc3 7y ago
    Ugh! I may have interpreted the comment I responded to wrong, and replied to my own strawman. I think you need the power for the user to delete records (obviously) but this could mean a bug in the system could cause the records to be deleted. But it sounds from what you are saying that they sort of "intentionally" deleted the records (not sure if a human had the intent or an algorithm?) but if intentionally deleted yes that sucks, there is no reason for that I can think of, given what they do is about handling DDOS. Unless the mechanism exists for legal reasons, i.e. they are forced to take some sites down by the government or law suits.

ajonit 7y ago

Now that the OP has given a go ahead to go public, We will eagerly wait for your update jgc

AndrewWarner 7y ago

He gave me an incredibly detailed and fast response to an issue I was fact-checking.

rswail 7y ago

Having the CTO offer up support in this way is heartening. Especially because infrastructure suppliers are getting more and more centralized, which leads to possible SPOFs even with individual company distributed environments.