5 ms·

It's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security.

It's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security. I'd be fine with them changing DNS to point to the origin (or any other user defined IPs) in case of issues (even if it would take hours to return to normal). This is also important for countries with limited connectivity to the Internet, if the PoP in that country looses it's connection back to CF it shuts everything down, so even if the origin is in the next rack over from the PoP, it's un-reachable.

booi 4y ago
Wouldn't this expose the origin IPs to attack?
- phillipseamore 4y ago
  I'm talking about this as an option for users like me, that don't have an attack surface, but need the global performance gains of CF.
  - 1123581321 4y ago
    Probably not many users who need the performance and can handle unexpected failover. There would also be the issue of setting the policy defaults effectively. Most users wouldn’t benefit from this footgun. If you’re serious, you could probably automate this right now with your DNS provider and uptime monitoring.
    - phillipseamore 4y ago
      I'm so serious that I already have failover after 1 hour at the registrar level, but those changes are not immediate and can take up to 24h to roll-in and roll-back due to DNS propagation and caching.
      - 1123581321 4y ago
        There is no immediate option with DNS changes. CF can’t immediately remove their IP from the route. Sounds like you’ve solved your problem in the sense that you have an automatic failover, though, which is good.
      - phillipseamore 4y ago
        We seem to have hit the nesting maximum 1123581321, but to your point "There is no immediate option with DNS changes." There's a huge difference in changing nameservers for a domain and simply changing host records.
        efreak 4y ago
        This is a 7th level comment. Synu below[0] has a 9th level comment, and I've seen nested comments go quite a bit further. [0] https://news.ycombinator.com/item?id=31821497 https://news.ycombinator.com/item?id=31821497
        pxeger1 4y ago
        I think there is no nesting maximum (or if there is, it's much bigger than this). There's a limit which stops you replying to a comment immediately, to prevent super long quick-fire arguments.
        1123581321 4y ago
        Ah, sorry, misunderstood you. You can’t rely on them to change their host records when they’re down. If you want CDN-independent automatic failover, look into anycast with two providers. If one of them is Cloudflare, use the tier that lets you manage your DNS elsewhere.
    - chrisweekly 4y ago
      Typically CF _is_ the DNS provider though, right?
      - 1123581321 4y ago
        Only up until the $200/mo tier. This kind of feature would be locked at that level anyway. As I said to the OP elsewhere, he should be doing something like anycast to multiple CDNs if this is critical.
- hatthew 4y ago
  Yes, but I'd wager that most sites experience cloudflare outages more often than they experience bona fide attacks.
  - Cthulhu_ 4y ago
    You say that, but there's tons of automated attempts doing the rounds on everything directly connected to the internet; centralized providers like Cloudflare can detect and prevent these patterns, whereas you need to be on the ball yourself if you have a service directly open to the internet. Exploits are exploited quickly, and while I make no assumptions about your particular website / application, a lot cannot push an update on short notice.
  - scrollaway 4y ago
    You would lose your wager. DDoS attacks are extremely common at just about any scale, even if you only have a few thousand users. Cloudflare going down like this? Actually first time I remember it happening. There’s been downtime before but nothing so major.
- mccorrinall 4y ago
  Yes, but he says in his second sentence that he doesn’t mind and mainly uses CF for performance.
  - phillipseamore 4y ago
    Plenty of commenters also seem to miss the most important word in the first sentence: "option".
- londons_explore 4y ago
  The origin IP's are already open to attack for most users. It's trivial to scan the whole IPv4 internet to find out which IP you are hosting your site on.
  - renonce 4y ago
    You need to block 443 for any other IP than Cloudflare. The IP list can be found at https://www.cloudflare.com/ips/ https://www.cloudflare.com/ips/.
    - dingdongthe 4y ago
      and/or use Authenticated Origin Pulls with a TLS client certificate.
      - renonce 4y ago
        That would leak your IP nevertheless. People can figure out that you're serving a specific website by inspecting your certificate on handshake without actually connecting.
bluejekyll 4y ago
You’d need to have TLS certs on origin ready to go for this scenario to work. Additionally, you’d need to make sure to test it and ensure that there’s nothing wrong in this event. On top of that, depending on your scale, can you take all the traffic on origin that Cloudflare currently offloads?
- phillipseamore 4y ago
  No issue for me. This is obviously a power-user option. It's kind-of implemented for Enterprise users were you don't have to let CF have full control over the domain.
renonce 4y ago
You can implement your own DNS server that CNAMEs to Cloudflare and falls back to origin IP when there is a problem with Cloudflare. I think a downstream Cloudflare provider could provide such services if they desire.
- phillipseamore 4y ago
  Last time I checked that was limited to the enterprise plan.
  - eins1234 4y ago
    Of course it is... Typical Cloudflare.
    - Cthulhu_ 4y ago
      Imagine having to pay for a service
      - mpol 4y ago
        Imagine having to pay for a service... at enterprise prices. This is not really an extra or a nice-to-have, it might be more a hostage situation.
      - adwww 4y ago
        Cloudflare's pricing has issues. My company was paying $20 a month. We were heavily depended on CF, we'd have been happy to pay more. But... the one feature we wanted was for our accounts team to have their own login so the ops team didn't have to download invoices every month. Nope, that one feature required an enterprise plan which they quoted $4,000 a month for.
        hizanberg 4y ago
        I've been impressed with Cloudflare's (non enterprise) value thus far, what Bandwidth & Users did the $4k quote cover?
yunohn 4y ago
> I'd be fine with them changing This would be best implemented by you. If the point is to avoid CF as a PoF, why would you rely on their infra to fail safe when something breaks?

booi 4y ago

Wouldn't this expose the origin IPs to attack?

phillipseamore 4y ago
I'm talking about this as an option for users like me, that don't have an attack surface, but need the global performance gains of CF.
- 1123581321 4y ago
  Probably not many users who need the performance and can handle unexpected failover. There would also be the issue of setting the policy defaults effectively. Most users wouldn’t benefit from this footgun. If you’re serious, you could probably automate this right now with your DNS provider and uptime monitoring.
  - phillipseamore 4y ago
    I'm so serious that I already have failover after 1 hour at the registrar level, but those changes are not immediate and can take up to 24h to roll-in and roll-back due to DNS propagation and caching.
    - 1123581321 4y ago
      There is no immediate option with DNS changes. CF can’t immediately remove their IP from the route. Sounds like you’ve solved your problem in the sense that you have an automatic failover, though, which is good.
    - phillipseamore 4y ago
      We seem to have hit the nesting maximum 1123581321, but to your point "There is no immediate option with DNS changes." There's a huge difference in changing nameservers for a domain and simply changing host records.
      - efreak 4y ago
        This is a 7th level comment. Synu below[0] has a 9th level comment, and I've seen nested comments go quite a bit further. [0] https://news.ycombinator.com/item?id=31821497 https://news.ycombinator.com/item?id=31821497
      - pxeger1 4y ago
        I think there is no nesting maximum (or if there is, it's much bigger than this). There's a limit which stops you replying to a comment immediately, to prevent super long quick-fire arguments.
      - 1123581321 4y ago
        Ah, sorry, misunderstood you. You can’t rely on them to change their host records when they’re down. If you want CDN-independent automatic failover, look into anycast with two providers. If one of them is Cloudflare, use the tier that lets you manage your DNS elsewhere.
  - chrisweekly 4y ago
    Typically CF _is_ the DNS provider though, right?
    - 1123581321 4y ago
      Only up until the $200/mo tier. This kind of feature would be locked at that level anyway. As I said to the OP elsewhere, he should be doing something like anycast to multiple CDNs if this is critical.
hatthew 4y ago
Yes, but I'd wager that most sites experience cloudflare outages more often than they experience bona fide attacks.
- Cthulhu_ 4y ago
  You say that, but there's tons of automated attempts doing the rounds on everything directly connected to the internet; centralized providers like Cloudflare can detect and prevent these patterns, whereas you need to be on the ball yourself if you have a service directly open to the internet. Exploits are exploited quickly, and while I make no assumptions about your particular website / application, a lot cannot push an update on short notice.
- scrollaway 4y ago
  You would lose your wager. DDoS attacks are extremely common at just about any scale, even if you only have a few thousand users. Cloudflare going down like this? Actually first time I remember it happening. There’s been downtime before but nothing so major.
mccorrinall 4y ago
Yes, but he says in his second sentence that he doesn’t mind and mainly uses CF for performance.
- phillipseamore 4y ago
  Plenty of commenters also seem to miss the most important word in the first sentence: "option".
londons_explore 4y ago
The origin IP's are already open to attack for most users. It's trivial to scan the whole IPv4 internet to find out which IP you are hosting your site on.
- renonce 4y ago
  You need to block 443 for any other IP than Cloudflare. The IP list can be found at https://www.cloudflare.com/ips/ https://www.cloudflare.com/ips/.
  - dingdongthe 4y ago
    and/or use Authenticated Origin Pulls with a TLS client certificate.
    - renonce 4y ago
      That would leak your IP nevertheless. People can figure out that you're serving a specific website by inspecting your certificate on handshake without actually connecting.

bluejekyll 4y ago

You’d need to have TLS certs on origin ready to go for this scenario to work. Additionally, you’d need to make sure to test it and ensure that there’s nothing wrong in this event. On top of that, depending on your scale, can you take all the traffic on origin that Cloudflare currently offloads?

phillipseamore 4y ago
No issue for me. This is obviously a power-user option. It's kind-of implemented for Enterprise users were you don't have to let CF have full control over the domain.

renonce 4y ago

You can implement your own DNS server that CNAMEs to Cloudflare and falls back to origin IP when there is a problem with Cloudflare. I think a downstream Cloudflare provider could provide such services if they desire.

phillipseamore 4y ago
Last time I checked that was limited to the enterprise plan.
- eins1234 4y ago
  Of course it is... Typical Cloudflare.
  - Cthulhu_ 4y ago
    Imagine having to pay for a service
    - mpol 4y ago
      Imagine having to pay for a service... at enterprise prices. This is not really an extra or a nice-to-have, it might be more a hostage situation.
    - adwww 4y ago
      Cloudflare's pricing has issues. My company was paying $20 a month. We were heavily depended on CF, we'd have been happy to pay more. But... the one feature we wanted was for our accounts team to have their own login so the ops team didn't have to download invoices every month. Nope, that one feature required an enterprise plan which they quoted $4,000 a month for.
      - hizanberg 4y ago
        I've been impressed with Cloudflare's (non enterprise) value thus far, what Bandwidth & Users did the $4k quote cover?

yunohn 4y ago

> I'd be fine with them changing This would be best implemented by you. If the point is to avoid CF as a PoF, why would you rely on their infra to fail safe when something breaks?