4 ms·

> For Workers, I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Currently, they recommend outputting debu

by eis 6y ago

> For Workers, I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Currently, they recommend outputting debug as JSON in a header. That is a general problem I have had with Cloudflare over the years. It's hard to know exactly what happened when things fail or even the fact that things failed! They always branded themselves as that super scalable and stable CDN or DDoS shield that will keep your service up when it would normally break down. And those absolutely horrible "You <-> Cloudflare OK, host down" error pages that suggest it's the host that has the problem and at the same time advertise Cloudflare to your visitors. Well turns out the vast majority of issues that clients reported (automated or not) were caused by Cloudflare themselves. And what really makes it painful is that it's 1. sometimes hard to debug 2. they wont make this visible to you unless I guess you pay for the highest Enterprise plan which offers raw logs. They just don't tell you even how many requests failed. There's no simple error.log. An example: at some point Cloudflare started blocking a lot of requests from Turkey to an API. The only way to solve it was to whitelist Turkey as a country. We've moved off of Cloudflare and had a lot fewer issues. PS: every feature since the simple CDN stuff seems to be now about lock-in. It's all Cloudflare specific. This is a dangerous path that moves away from the commoditization of "hosting" where people can easy move from one hoster to another. I hope it does not succeed. For the sake of the open web.

rattray 6y ago
Who did you move to from Cloudflare?
- eis 6y ago
  For CDN we just put servers in different geological locations, some using cloud service and some dedicated. In fact the system itself was made as distributed wherever possible. For the anti DDoS: it was not a big/common issue and in the rare cases we can work with providers to mitigate.
  - snissn 6y ago
    how do you distribute between your servers? DNS? what DNS service?
    - eis 6y ago
      We started with GeoDNS (e.g. AWS Route53) and later on added Anycast into the mix. The combination of these two is very powerful but I wouldn't recommend Anycast for projects without good network staff. For most, I think GeoDNS is good enough and very simple. The number of users using nameservers that break it (like Cloudflare's 1.1.1.1)* is still small. *: The reason why it breaks is because they don't send along the EDNS client subnet to the upstream nameserver for "privacy reasons". I disagree with that notion because 1. it's just the subnet 2. the website/service will see the full client IP anyways when it gets the http request. I mean how many times does one resolve a hostname and then not connect to it? And we are talking about the authorative nameserver, not the ISP nameserver.
neurostimulant 6y ago
I also noticed random increase in latency when you put a rest api server behind cloudflare. Since then, I don't put api servers behind cloudflare anymore and limit cloudflare use on traditional websites only. I suspects cloudflare might flag api clients as bots (especially those with non-browser user agents) and do additional check on those requests, thus increasing the latency.
- kentonv 6y ago
  I don't think we do anything that would increase latency for API clients specifically. But if your API is not implemented on the edge (e.g. in a Worker), and the responses are not cacheable at the edge, and you aren't using Argo, then there's not much else Cloudflare can do to make the request faster. Cloudflare is just proxying requests at that point, and the presence of a proxy will indeed probably add a few milliseconds of latency vs. the client talking directly to the origin. (Of course, you do get other benefits like DDoS protection.)
  - neurostimulant 6y ago
    Is it possible that the timeouts was due to cloudflare blocking some user agent, maybe the browser integrity check/bad browser detection? When the api is used in a web browser, there was no reported issue with latency/timeouts from users and logging. the latency is not much different than without proxying on my end. but when used in an android and ios app with custom user agent I did see occasional large latency and the some users reported random timeouts. Unfortunately I didn't test updating the app to use chrome user agent and simply turned off proxying for that api server (and other api servers that used by non-browser clients). It was happening last year though, I haven't try it again yet.
    - kentonv 6y ago
      Ah sorry, I didn't realize you meant that things were outright timing out, rather than just being slower than when not proxied. That seems like something went wrong. I don't think we have anti-bot measures that would cause API clients to time out. If anything they would get an error back or maybe get served a captcha or JavaScript challenge.

by eis 6y ago

rattray 6y ago

Who did you move to from Cloudflare?

eis 6y ago
For CDN we just put servers in different geological locations, some using cloud service and some dedicated. In fact the system itself was made as distributed wherever possible. For the anti DDoS: it was not a big/common issue and in the rare cases we can work with providers to mitigate.
- snissn 6y ago
  how do you distribute between your servers? DNS? what DNS service?
  - eis 6y ago
    We started with GeoDNS (e.g. AWS Route53) and later on added Anycast into the mix. The combination of these two is very powerful but I wouldn't recommend Anycast for projects without good network staff. For most, I think GeoDNS is good enough and very simple. The number of users using nameservers that break it (like Cloudflare's 1.1.1.1)* is still small. *: The reason why it breaks is because they don't send along the EDNS client subnet to the upstream nameserver for "privacy reasons". I disagree with that notion because 1. it's just the subnet 2. the website/service will see the full client IP anyways when it gets the http request. I mean how many times does one resolve a hostname and then not connect to it? And we are talking about the authorative nameserver, not the ISP nameserver.

neurostimulant 6y ago

I also noticed random increase in latency when you put a rest api server behind cloudflare. Since then, I don't put api servers behind cloudflare anymore and limit cloudflare use on traditional websites only. I suspects cloudflare might flag api clients as bots (especially those with non-browser user agents) and do additional check on those requests, thus increasing the latency.

kentonv 6y ago
I don't think we do anything that would increase latency for API clients specifically. But if your API is not implemented on the edge (e.g. in a Worker), and the responses are not cacheable at the edge, and you aren't using Argo, then there's not much else Cloudflare can do to make the request faster. Cloudflare is just proxying requests at that point, and the presence of a proxy will indeed probably add a few milliseconds of latency vs. the client talking directly to the origin. (Of course, you do get other benefits like DDoS protection.)
- neurostimulant 6y ago
  Is it possible that the timeouts was due to cloudflare blocking some user agent, maybe the browser integrity check/bad browser detection? When the api is used in a web browser, there was no reported issue with latency/timeouts from users and logging. the latency is not much different than without proxying on my end. but when used in an android and ios app with custom user agent I did see occasional large latency and the some users reported random timeouts. Unfortunately I didn't test updating the app to use chrome user agent and simply turned off proxying for that api server (and other api servers that used by non-browser clients). It was happening last year though, I haven't try it again yet.
  - kentonv 6y ago
    Ah sorry, I didn't realize you meant that things were outright timing out, rather than just being slower than when not proxied. That seems like something went wrong. I don't think we have anti-bot measures that would cause API clients to time out. If anything they would get an error back or maybe get served a captcha or JavaScript challenge.