3 ms·

My company uses 3 CDNs, although not cloudflare. If one (say Aakami) goes down it gets removed from the pool and life continues.

inferiorhuman 4y ago
Well that makes your company more responsible than my credit union.
- iso1631 4y ago
  Our key customer facing services are a 99.995% uptime (and a total of 2 or fewer incidents per year of "any length"), which means once you start concatenating services with 99.995% SLAs you aren't there. How that SLA measures a 2 second outage for some customers is a separate thing, and sort of shows how meaningless these things can be on the internet (if you lose service for 10% of your potential customers is that an outage? How about 90%? How do you know how many were lost).
  - inferiorhuman 4y ago
    Measuring outages doesn't seem so meaningless as long as you money seems inaccessible. Their main site went down for about 20 hours a couple weeks ago because their hosting provider went down. They deployed an HTTPS only static site in its stead, so at first blush it looked like they deployed nothing. Great when you're trying to find contact information hosted on that site. Their online banking site leveraged Cloudflare, so obviously they just rode that outage out with no notifications, etc.
    - iso1631 4y ago
      Sure but that's a total outage for a long time. What if for some reason a single /24 was unreachable from the site (say an errant route for 12.85.25.0/24 somehow got in the path). How would you even know that was a problem - how many customers are on that /24, how would I measure their failed attempts to connect? I have a remote office in India on Tata. The other day it had access to much of the internet, but due to a fibre break in the Mederteranian it didn't have access to end points in Europe for a good 20 seconds. However the other link on a different ISP remained working at that time. Does that count as an outage? If I wasn't actively monitoring that link with a high resolution would I even know about it?
      - inferiorhuman 4y ago
        I'd argue you're starting from a few orders of magnitude more competency than the credit union was. Their non-banking site was hosted by some podunk company in Texas with no sense of redundancy anywhere. Their provider had a near total networking outage and the credit union had no plan to recover from that. Insofar as proactively monitoring a single /24, you (probably) don't. I don't think it's (usually) a company's job to monitor their customer's ISPs. The failures that "my" credit union had were due to their own choice in infra (Armor, Cloudflare). When Sonic nuked my config on their DSLAM after some maintenance I raised an issue with Sonic not with whatever other companies became inaccessible as a result. > Does that count as an outage? My POV may very well differ from whatever contracts and SLAs you have in place, but yeah maybe. If you can't fail over to the alternative ISP then yes that's an outage. Of course a trans-atlantic fiber break would also likely be a lot more noticeable than fat fingering a route for a /24. And sure, I've been stuck at megacorp when the VPN started handing out addresses in a new subnet but our department's networking team hadn't caught up. That's why you listen to your customers instead of throwing out a "someone else screwed up there's nothing we can do" response. Me personally I don't think that a 20 minute banking outage is a massive problem (I've long since moved my money elsewhere), even the 20 hour outage was relatively minor. It just speaks to the unwillingness of the credit union to be highly available. They knew of the Armor outage and didn't actually test the remediation. I assume they didn't know about the Cloudflare outage. Both worry me. What happens when they're faced with a total failure of their online banking system?
        iso1631 4y ago
        But it isn't an outage. My monitoring point in Singapore could reach both ends, they just couldn't talk to each over, due to a routing issue on a third party network over the internet. On my own network which I control I accept that if a circuit breaks I'll have a 1, maybe 2 second outage while traffic reroutes. For some of my services that's would be a problem, for others it's not. If facebook loads 2 seconds later, nobody cares. If the winning penalty in the world cup final blacks out, that's a big problem.
acmecorps 4y ago
I'm new to this whole thing. Can u point me on how I don't depend 100% on CF, if its DNS is down? Is there such a service? (kinda like load balancing, but with DNS?)
- selcuka 4y ago
  DNS is easy. You can (and must) have multiple nameservers for your domain. Just use different companies (and different regions) and if one goes down the others will still resolve.
maidh91 4y ago
what are those 3 CDN companies? Cloudflare, Aakami and what else?
- selcuka 4y ago
  > what are those 3 CDN companies? Cloudflare, Aakami and what else? The OP says "not Cloudflare"; so probably Akamai, Fastly, CloudFront?

inferiorhuman 4y ago

Well that makes your company more responsible than my credit union.

iso1631 4y ago
Our key customer facing services are a 99.995% uptime (and a total of 2 or fewer incidents per year of "any length"), which means once you start concatenating services with 99.995% SLAs you aren't there. How that SLA measures a 2 second outage for some customers is a separate thing, and sort of shows how meaningless these things can be on the internet (if you lose service for 10% of your potential customers is that an outage? How about 90%? How do you know how many were lost).
- inferiorhuman 4y ago
  Measuring outages doesn't seem so meaningless as long as you money seems inaccessible. Their main site went down for about 20 hours a couple weeks ago because their hosting provider went down. They deployed an HTTPS only static site in its stead, so at first blush it looked like they deployed nothing. Great when you're trying to find contact information hosted on that site. Their online banking site leveraged Cloudflare, so obviously they just rode that outage out with no notifications, etc.
  - iso1631 4y ago
    Sure but that's a total outage for a long time. What if for some reason a single /24 was unreachable from the site (say an errant route for 12.85.25.0/24 somehow got in the path). How would you even know that was a problem - how many customers are on that /24, how would I measure their failed attempts to connect? I have a remote office in India on Tata. The other day it had access to much of the internet, but due to a fibre break in the Mederteranian it didn't have access to end points in Europe for a good 20 seconds. However the other link on a different ISP remained working at that time. Does that count as an outage? If I wasn't actively monitoring that link with a high resolution would I even know about it?
    - inferiorhuman 4y ago
      I'd argue you're starting from a few orders of magnitude more competency than the credit union was. Their non-banking site was hosted by some podunk company in Texas with no sense of redundancy anywhere. Their provider had a near total networking outage and the credit union had no plan to recover from that. Insofar as proactively monitoring a single /24, you (probably) don't. I don't think it's (usually) a company's job to monitor their customer's ISPs. The failures that "my" credit union had were due to their own choice in infra (Armor, Cloudflare). When Sonic nuked my config on their DSLAM after some maintenance I raised an issue with Sonic not with whatever other companies became inaccessible as a result. > Does that count as an outage? My POV may very well differ from whatever contracts and SLAs you have in place, but yeah maybe. If you can't fail over to the alternative ISP then yes that's an outage. Of course a trans-atlantic fiber break would also likely be a lot more noticeable than fat fingering a route for a /24. And sure, I've been stuck at megacorp when the VPN started handing out addresses in a new subnet but our department's networking team hadn't caught up. That's why you listen to your customers instead of throwing out a "someone else screwed up there's nothing we can do" response. Me personally I don't think that a 20 minute banking outage is a massive problem (I've long since moved my money elsewhere), even the 20 hour outage was relatively minor. It just speaks to the unwillingness of the credit union to be highly available. They knew of the Armor outage and didn't actually test the remediation. I assume they didn't know about the Cloudflare outage. Both worry me. What happens when they're faced with a total failure of their online banking system?
      - iso1631 4y ago
        But it isn't an outage. My monitoring point in Singapore could reach both ends, they just couldn't talk to each over, due to a routing issue on a third party network over the internet. On my own network which I control I accept that if a circuit breaks I'll have a 1, maybe 2 second outage while traffic reroutes. For some of my services that's would be a problem, for others it's not. If facebook loads 2 seconds later, nobody cares. If the winning penalty in the world cup final blacks out, that's a big problem.

acmecorps 4y ago

I'm new to this whole thing. Can u point me on how I don't depend 100% on CF, if its DNS is down? Is there such a service? (kinda like load balancing, but with DNS?)

selcuka 4y ago
DNS is easy. You can (and must) have multiple nameservers for your domain. Just use different companies (and different regions) and if one goes down the others will still resolve.

maidh91 4y ago

what are those 3 CDN companies? Cloudflare, Aakami and what else?

selcuka 4y ago
> what are those 3 CDN companies? Cloudflare, Aakami and what else? The OP says "not Cloudflare"; so probably Akamai, Fastly, CloudFront?