5 ms·

Your customer doesn’t care, as they don’t have business with cloudflare. Your fault is in not having plan b.

mrkurt 6y ago
They're right, though. Some companies are large enough to give you credibility cover during an outage. It's the same as the old phrase about no one ever getting fired for buying IBM.
- ksec 6y ago
  Yes and these sort of excuses applies to ALL industries. Any Sales people or Support Staff should already know this. So I am often surprised people are arguing about it. At the end, your customer, assuming it is a SaaS, just need an excuse for "their" customers or the End User. And there is nothing better than a big household name's fault so no one get the blame. So the whole thing is written off, everybody is in the clear, and everyone can get on with their other business. :)
  - syshum 6y ago
    Well many people have the ability to see through marketing BS, and these types of excuses are exactly that, marketing BS I often feel sad for people that have not developed the critical thinking skills needed to look past marketing to where the realm of logic and reason resides
    - untog 6y ago
      Patronising much? The simple reality is that if your site is down because AWS is down then it means a lot of other sites are down too. Which means you don’t look anywhere near as bad as you would if you were the only site not working. Logic and reason is all well and good but human perception is a very real thing that businesses need to keep in mind. It isn’t always entirely logical.
      - syshum 6y ago
        I would like to know of a time where all of AWS was down, every region, every data-center, everything down At most one region was impacted, and it is easy to be multi-regional in AWS, in fact that is kinda of the point Further this comment is in service of the moronic axiom of "No one ever got fired for buying <<insert large company>>" my response to that has always been and will always be "sure they have and they should" It is simply not true that buying AWS, IBM, Cisco, or any other large vendor is complete insulator from all responsibility to maintain reliable systems nor should it be Any administrator or developer that is making buying choices based on that is not an person I would like to ever work or do business with
        untog 6y ago
        “AWS is down” is phrasing shorthand, it doesn’t mean every single service they offer is nonfunctional. It means that for the vast majority of people it is not doing what it is supposed to do. Again, it’s not a technically correct statement but it is how humans talk. To extrapolate out to the original point: if multiple top ten traffic web sites are having issues (and this has happened once or twice in the last few years due to AWS or other cloud issues) then your site being down is less notable in customers minds.
        syshum 6y ago
        again that depends on who your "customers" are The grandparent was talking about a SaaS service so then I would expect the customers to be technically minded people. If you are selling to masses then sure, but if you are selling me a Line of Business SaaS service then no I do not care if facebook and reddit was down, that is not relevant to how the SaaS product should be running
        ksec 6y ago
        I think that is a narrow definition of SaaS. But I understand your point. For instance Basecamp could be down due to AWS. And Basecamp ( and in this case Hey as well ), but are SaaS. And their customer may not always be technically minded. Given the usage of these tools, while inside tech circle, are not only used by technical people. And At the end of the day it is all about trade offs. I also dont think anyone ever make a purchase decision purely on brand or not fired for X. For example, despite AMD offering lower price and offer more core and performance. Server Vendors hasn't all switched to AMD at once. In fact, Intel Server still has months of backlog order to fill. This isn't simply because Intel is better connected with Vendors, it is the fact most of those End user / customers are still demanding Intel CPU. Because it is well tested, with more specific libraries, tools, guarantees and support. Many of these factors cant be quantitatively measured, and therefore would only be judged when the final price difference are shown. In this case No one gets fire for using X is another phase for if it aren't broke, dont fix it.
zaroth 6y ago
Wish it were so simple, but alas, fair or not, this is a real and prevalent bias in the world. I’ve had colo equipment that ran with 5-nine uptime for years eventually get unlucky and be down for an hour and it was “all my fault”. Switch the service to Amazon which achieves much worse uptime, but now it’s “well if Amazon is down, what can you do?” Frankly when something seen as core internet infrastructure goes down, the measly SaaS companies pretty reasonably don’t take any blame. To be clear, often these are services where there isn’t the engineering budget, nor honestly the need, for multi-cloud and geographically distributed multi-master services.
- LaserToy 6y ago
  Agree, but it comes across as we don’t care. I worked for a major gaming network, and believe me, gamers don’t care why they can’t play. They want to play. Now. And I can’t imagine anyone on the team relaxing when something like that happened.
  - katzgrau 6y ago
    Yeah, luckily our customers are a little easier to deal with than that, despite the very real impacts to their businesses. You have to know/understand your customers in order to know when you can relax. But the plan B in this case is switching nameservers, which we could certainly have done (and briefly considered), but it could be error prone and would take longer for those changes to propagate than it would for CF to fix the issue, most likely. The best option was simply inaction, if that makes sense. There are times when not doing something is a better idea than doing something.
peteretep 6y ago
I was CTO at a SaaS company with a whole host of companies you've heard of relying on us as a core part of their business. They absolutely would and did accept "AWS was down, soz" as an excuse, where they were much spikier about fuckups where we were to blame. Tangential: nobody got fired for buying IBM
- LaserToy 6y ago
  I was in eng leadership at largest gaming network. Our customers (gamers) absolutely didn’t care about reasons they couldn’t play. It was always our fault as we were responsible for tech choices, not them. And I believe it is a way. If I’m selling you a service it is my responsibility to make sure it is reliable and not to blame it on something that is under my control (like cloud providers). Of course I can’t fix local internet going down, but I can make sure I’m not married to 1 vendor
  - katzgrau 6y ago
    I'd argue that you're probably holding yourself to a standard that is more/less unachievable in such an interdependent world. It's idealistic and idealism is a square peg in the funny shaped hole of reality. Taking accountability and having backup plans are extremely important, but you simply can't remove every last shred of dependence. You eventually have to accept that there are things that are out of your control and may take you by surprise despite best efforts.
    - zzzcpan 6y ago
      In web and online infrastructure pretty much nothing is out of your control except for two things: ISPs people use and domain name registrar you use for your domain name. And even domain name registrar centralization can be mitigated against by having multiple domains from multiple registrars and promoting different domains to different users and having backup communication channels to inform users about new domains in case something happens. Other than that it's your choice whether to make your infrastructure dependent on a bunch of unreliable centralized SPOFs from big corporations or build highly available infrastructure relying on servers from many different providers running your own DNS servers with DNS routing, failover, etc. You will definitely beat Cloudflare's availability this way many times over.
creatonez 6y ago
The point is that the customer is likely using a dozen other sites that have also been hit by cloudflare outage, so they care less that your one specific site is knocked down. The customer immediately knows the blame is not on you when they notice how widespread the outage is.

They're right, though. Some companies are large enough to give you credibility cover during an outage. It's the same as the old phrase about no one ever getting fired for buying IBM.

ksec 6y ago
Yes and these sort of excuses applies to ALL industries. Any Sales people or Support Staff should already know this. So I am often surprised people are arguing about it. At the end, your customer, assuming it is a SaaS, just need an excuse for "their" customers or the End User. And there is nothing better than a big household name's fault so no one get the blame. So the whole thing is written off, everybody is in the clear, and everyone can get on with their other business. :)
- syshum 6y ago
  Well many people have the ability to see through marketing BS, and these types of excuses are exactly that, marketing BS I often feel sad for people that have not developed the critical thinking skills needed to look past marketing to where the realm of logic and reason resides
  - untog 6y ago
    Patronising much? The simple reality is that if your site is down because AWS is down then it means a lot of other sites are down too. Which means you don’t look anywhere near as bad as you would if you were the only site not working. Logic and reason is all well and good but human perception is a very real thing that businesses need to keep in mind. It isn’t always entirely logical.
    - syshum 6y ago
      I would like to know of a time where all of AWS was down, every region, every data-center, everything down At most one region was impacted, and it is easy to be multi-regional in AWS, in fact that is kinda of the point Further this comment is in service of the moronic axiom of "No one ever got fired for buying <<insert large company>>" my response to that has always been and will always be "sure they have and they should" It is simply not true that buying AWS, IBM, Cisco, or any other large vendor is complete insulator from all responsibility to maintain reliable systems nor should it be Any administrator or developer that is making buying choices based on that is not an person I would like to ever work or do business with
      - untog 6y ago
        “AWS is down” is phrasing shorthand, it doesn’t mean every single service they offer is nonfunctional. It means that for the vast majority of people it is not doing what it is supposed to do. Again, it’s not a technically correct statement but it is how humans talk. To extrapolate out to the original point: if multiple top ten traffic web sites are having issues (and this has happened once or twice in the last few years due to AWS or other cloud issues) then your site being down is less notable in customers minds.
        syshum 6y ago
        again that depends on who your "customers" are The grandparent was talking about a SaaS service so then I would expect the customers to be technically minded people. If you are selling to masses then sure, but if you are selling me a Line of Business SaaS service then no I do not care if facebook and reddit was down, that is not relevant to how the SaaS product should be running
        ksec 6y ago
        I think that is a narrow definition of SaaS. But I understand your point. For instance Basecamp could be down due to AWS. And Basecamp ( and in this case Hey as well ), but are SaaS. And their customer may not always be technically minded. Given the usage of these tools, while inside tech circle, are not only used by technical people. And At the end of the day it is all about trade offs. I also dont think anyone ever make a purchase decision purely on brand or not fired for X. For example, despite AMD offering lower price and offer more core and performance. Server Vendors hasn't all switched to AMD at once. In fact, Intel Server still has months of backlog order to fill. This isn't simply because Intel is better connected with Vendors, it is the fact most of those End user / customers are still demanding Intel CPU. Because it is well tested, with more specific libraries, tools, guarantees and support. Many of these factors cant be quantitatively measured, and therefore would only be judged when the final price difference are shown. In this case No one gets fire for using X is another phase for if it aren't broke, dont fix it.

zaroth 6y ago

Wish it were so simple, but alas, fair or not, this is a real and prevalent bias in the world. I’ve had colo equipment that ran with 5-nine uptime for years eventually get unlucky and be down for an hour and it was “all my fault”. Switch the service to Amazon which achieves much worse uptime, but now it’s “well if Amazon is down, what can you do?” Frankly when something seen as core internet infrastructure goes down, the measly SaaS companies pretty reasonably don’t take any blame. To be clear, often these are services where there isn’t the engineering budget, nor honestly the need, for multi-cloud and geographically distributed multi-master services.

LaserToy 6y ago
Agree, but it comes across as we don’t care. I worked for a major gaming network, and believe me, gamers don’t care why they can’t play. They want to play. Now. And I can’t imagine anyone on the team relaxing when something like that happened.
- katzgrau 6y ago
  Yeah, luckily our customers are a little easier to deal with than that, despite the very real impacts to their businesses. You have to know/understand your customers in order to know when you can relax. But the plan B in this case is switching nameservers, which we could certainly have done (and briefly considered), but it could be error prone and would take longer for those changes to propagate than it would for CF to fix the issue, most likely. The best option was simply inaction, if that makes sense. There are times when not doing something is a better idea than doing something.

peteretep 6y ago

I was CTO at a SaaS company with a whole host of companies you've heard of relying on us as a core part of their business. They absolutely would and did accept "AWS was down, soz" as an excuse, where they were much spikier about fuckups where we were to blame. Tangential: nobody got fired for buying IBM

LaserToy 6y ago
I was in eng leadership at largest gaming network. Our customers (gamers) absolutely didn’t care about reasons they couldn’t play. It was always our fault as we were responsible for tech choices, not them. And I believe it is a way. If I’m selling you a service it is my responsibility to make sure it is reliable and not to blame it on something that is under my control (like cloud providers). Of course I can’t fix local internet going down, but I can make sure I’m not married to 1 vendor
- katzgrau 6y ago
  I'd argue that you're probably holding yourself to a standard that is more/less unachievable in such an interdependent world. It's idealistic and idealism is a square peg in the funny shaped hole of reality. Taking accountability and having backup plans are extremely important, but you simply can't remove every last shred of dependence. You eventually have to accept that there are things that are out of your control and may take you by surprise despite best efforts.
  - zzzcpan 6y ago
    In web and online infrastructure pretty much nothing is out of your control except for two things: ISPs people use and domain name registrar you use for your domain name. And even domain name registrar centralization can be mitigated against by having multiple domains from multiple registrars and promoting different domains to different users and having backup communication channels to inform users about new domains in case something happens. Other than that it's your choice whether to make your infrastructure dependent on a bunch of unreliable centralized SPOFs from big corporations or build highly available infrastructure relying on servers from many different providers running your own DNS servers with DNS routing, failover, etc. You will definitely beat Cloudflare's availability this way many times over.

creatonez 6y ago

The point is that the customer is likely using a dozen other sites that have also been hit by cloudflare outage, so they care less that your one specific site is knocked down. The customer immediately knows the blame is not on you when they notice how widespread the outage is.