6 ms·

There are multiple places where he uses a feature only available to enterprise users (read: Not me) to resolve this. I guess the "normal" thing here would be to

There are multiple places where he uses a feature only available to enterprise users (read: Not me) to resolve this. I guess the "normal" thing here would be to disable the WAF and be done with it.

raverbashing 4y ago
Yeah, while I would have tried to investigate this a bit, this seems to be an issue on Cloudflare Bypassing the rules is a workaround, but not a fix
e1g 4y ago
This jumped out at me too. Cloudflare keeps many diagnostic and debugging tools available only on the Enterprise plan (typically >$3k per month). When something like this happens to most small teams and startups, they are working in the dark even on the Business plan at $200/mo.
- tills13 4y ago
  FWIW CloudFlare has worked with us time and time again to get pricing in line with our expectations. We have numerous "Business Plus" or "Free Plus" plans across various accounts and zones that include features that are, as listed in the marketing, only available to Enterprise. Just reach out to someone at Cloudflare. I'm sure they'd love your business.
a2tech 4y ago
And if you have to disable WAF, what exactly is Cloudflare doing for you?
- mpalfrey 4y ago
  Would it still do DDOS even without the WAF?
  - mstrem 4y ago
    Our DDoS mitigation is separate yes and will still work.
- danielheath 4y ago
  Edge Cache with unmetered egress?
  - perfecto_maduro 4y ago
    unmetered up to a point, but yeah. Also network-level DDoS protection
    - tills13 4y ago
      If you are reaching the point where they are sending you emails asking about what you are doing, you should be paying for it.
- perfecto_maduro 4y ago
  yeah, but sadly there aren’t many CDNs that offer WAF, even the most basic one. I literally begged bunnyCDN to build one so we can switch from CF. It’s on their roadmap for like forever.
- Tijdreiziger 4y ago
  In the article, the author disables the WAF only for Stripe outbound IPs, which can be presumed to be safe (unless Stripe's machines/IP space gets hacked). The WAF still works for traffic from all other IPs
- r1ch 4y ago
  Even with the WAF disabled (at least as much as I can disable it without the Enterprise plan, i.e. "Essentially Off"), I've found it will still block legitimate requests. Tainted CGNAT or dynamic IPs are my guess. The WAF doesn't really matter for my use case as the route is handled by a CF worker, in fact I'd prefer it doesn't get in the way.
gz5 4y ago
And WAF isn't the only design option for webhooks. WAF = define what to block. Another options is to define what to accept (block all else by default). OpenZiti approach (1) for example: a. enroll each side of the webhook w/ X.509 identity b. X.509 gates a network overlay between the servers c. each server initiates outbound sessions to the overlay d. block everything else (deny-all inbound on both servers) (1) disclosure: i am a maintainer of the openziti foss, and you can only (fully) use the technique above if you have enough control of both sides, e.g. use a Lambda function: https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda https://blog.openziti.io/my-intern-assignment-call-a-dark-we...
- jeroenhd 4y ago
  Mutual TLS auth seems like such a no-brainer for APIs like these. I do wonder, though, how does OpenZiti deal with client certificates expiring? Do you not set/enforce an expiration date or do you have some kind of automated renewal mechanism?
  - robertlagrant 4y ago
    Cert renewal is why it's not used everywhere, I think. While OAuth2 / client credentials is complicated, at least .jwks endpoints make cert rotation easy.
    - gz5 4y ago
      yes, can be tricky, as you state. see above for the openziti approach.
  - gz5 4y ago
    your overlay network fabric infrastructure - ziti edge routers - leverage the ziti edge client API to submit CSRs for new certs x days before expiry. for client endpoints, there are currently 2 choices and at least 3 in the future: 1. admins chooses to enable client endpoints to continue to auth with expired client certs. the admin revokes access (when necessary) at an endpoint level or at a service level (least privileged access) via those constructs, rather than cert constructs. 2. admins don't permit expired certs to be used. the admin is managing the certs. 3. as a third choice for client endpoints, ziti has plans to enable admins to use the API used by the fabric infra mentioned above. this is future, with timing mainly dependent on openziti community priorities and related work.

There are multiple places where he uses a feature only available to enterprise users (read: Not me) to resolve this. I guess the "normal" thing here would be to disable the WAF and be done with it.

raverbashing 4y ago

Yeah, while I would have tried to investigate this a bit, this seems to be an issue on Cloudflare Bypassing the rules is a workaround, but not a fix

e1g 4y ago

This jumped out at me too. Cloudflare keeps many diagnostic and debugging tools available only on the Enterprise plan (typically >$3k per month). When something like this happens to most small teams and startups, they are working in the dark even on the Business plan at $200/mo.

tills13 4y ago
FWIW CloudFlare has worked with us time and time again to get pricing in line with our expectations. We have numerous "Business Plus" or "Free Plus" plans across various accounts and zones that include features that are, as listed in the marketing, only available to Enterprise. Just reach out to someone at Cloudflare. I'm sure they'd love your business.

a2tech 4y ago

And if you have to disable WAF, what exactly is Cloudflare doing for you?

mpalfrey 4y ago
Would it still do DDOS even without the WAF?
- mstrem 4y ago
  Our DDoS mitigation is separate yes and will still work.
danielheath 4y ago
Edge Cache with unmetered egress?
- perfecto_maduro 4y ago
  unmetered up to a point, but yeah. Also network-level DDoS protection
  - tills13 4y ago
    If you are reaching the point where they are sending you emails asking about what you are doing, you should be paying for it.
perfecto_maduro 4y ago
yeah, but sadly there aren’t many CDNs that offer WAF, even the most basic one. I literally begged bunnyCDN to build one so we can switch from CF. It’s on their roadmap for like forever.
Tijdreiziger 4y ago
In the article, the author disables the WAF only for Stripe outbound IPs, which can be presumed to be safe (unless Stripe's machines/IP space gets hacked). The WAF still works for traffic from all other IPs
r1ch 4y ago
Even with the WAF disabled (at least as much as I can disable it without the Enterprise plan, i.e. "Essentially Off"), I've found it will still block legitimate requests. Tainted CGNAT or dynamic IPs are my guess. The WAF doesn't really matter for my use case as the route is handled by a CF worker, in fact I'd prefer it doesn't get in the way.

gz5 4y ago

And WAF isn't the only design option for webhooks. WAF = define what to block. Another options is to define what to accept (block all else by default). OpenZiti approach (1) for example: a. enroll each side of the webhook w/ X.509 identity b. X.509 gates a network overlay between the servers c. each server initiates outbound sessions to the overlay d. block everything else (deny-all inbound on both servers) (1) disclosure: i am a maintainer of the openziti foss, and you can only (fully) use the technique above if you have enough control of both sides, e.g. use a Lambda function: https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda https://blog.openziti.io/my-intern-assignment-call-a-dark-we...

jeroenhd 4y ago
Mutual TLS auth seems like such a no-brainer for APIs like these. I do wonder, though, how does OpenZiti deal with client certificates expiring? Do you not set/enforce an expiration date or do you have some kind of automated renewal mechanism?
- robertlagrant 4y ago
  Cert renewal is why it's not used everywhere, I think. While OAuth2 / client credentials is complicated, at least .jwks endpoints make cert rotation easy.
  - gz5 4y ago
    yes, can be tricky, as you state. see above for the openziti approach.
- gz5 4y ago
  your overlay network fabric infrastructure - ziti edge routers - leverage the ziti edge client API to submit CSRs for new certs x days before expiry. for client endpoints, there are currently 2 choices and at least 3 in the future: 1. admins chooses to enable client endpoints to continue to auth with expired client certs. the admin revokes access (when necessary) at an endpoint level or at a service level (least privileged access) via those constructs, rather than cert constructs. 2. admins don't permit expired certs to be used. the admin is managing the certs. 3. as a third choice for client endpoints, ziti has plans to enable admins to use the API used by the fabric infra mentioned above. this is future, with timing mainly dependent on openziti community priorities and related work.