5 ms·

Note: If I come across as a hater, in my opinion, I am not. I'm a fan, a fan who is frustrated! We moved to Cloudflare from a larger CDN and used workers to du

by redm 6y ago

Note: If I come across as a hater, in my opinion, I am not. I'm a fan, a fan who is frustrated! We moved to Cloudflare from a larger CDN and used workers to duplicate the functionality otherwise not provided by Cloudflare. The initial results were great. As our workers became more complicated, especially when interacting with CloudFlares internal stack, the cracks started to show. Specifically, when you run into a problem, the documentation being too light to resolve corner case issues yourself, and support draws a big zero when it comes to Workers. Even as an Enterprise customer, it can take months to get a clear answer/resolution, and that's if you nag them. As a platform, workers have great promise, and Cloudflare is making strides. As with most things CloudFlare, they could spend more time improving documentation, and exposing details on their internal stacks so you can self-service and resolve your issues. What I find is that Cloudflare is on to the next thing before fully completing, polishing, documenting, the previous three things they started. For Workers, I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Currently, they recommend outputting debug as JSON in a header. The linked article is worthless. See the blog post, as this poster mentioned: https://news.ycombinator.com/item?id=23965764 https://news.ycombinator.com/item?id=23965764

kentonv 6y ago
> I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Does `wrangler tail` help with this? https://developers.cloudflare.com/workers/about/tips/debugging/#tail-production-logs-with-wrangler-tail https://developers.cloudflare.com/workers/about/tips/debuggi...
- redm 6y ago
  What I'm looking for is a full stack trace of a request, and how it progressed through their platform, through pops, Argo, Caches, etc. Akamai and others have tools like this, and it makes it much easier to debug gremlins on the inside. From the docs: "If your Workers application is returning an 1101 error code, your code is throwing an exception. You can catch the exception in your code using a try/catch block:" That's great, but you end up just returning those details in a header or sending them somewhere else. We have to catch errors and send them to our backend for logging so we can debug them at a later point. This doesn't help you debug things like sub-requests from the Worker or how they interact with other Cloudflare features like Cache.
  - prdonahue 6y ago
    We're working on this tracing tool as we move things like Page Rules and Rate Limiting into the same engine that runs Firewall Rules. Lots of advantages of using the same Rust-based engine/syntax to let customers express their configuration rules, e.g., adding rule-based alerting and combining different types of filters and actions. Focus so far in the design process has been how requests get evaluated against a list of Custom and Managed Rules but if you imagine a world in which everything is a rule (incl. routing to/firing a Worker) it becomes tractable to do what you're asking about. We'll be in discussions with the Workers team about how to best illustrate the handoff to Workers and what happens downstream: cache, origin fetch, etc.
- garganzol 6y ago
  Kenton, not exactly topic-related, by I am deeply grateful for what you did in Sandstorm. Cloudflare or not, you a great guy! And I am sure you still have a bunch of magic tricks in your pocket. You sowed the seeds for future crops, literally.
eis 6y ago
> For Workers, I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Currently, they recommend outputting debug as JSON in a header. That is a general problem I have had with Cloudflare over the years. It's hard to know exactly what happened when things fail or even the fact that things failed! They always branded themselves as that super scalable and stable CDN or DDoS shield that will keep your service up when it would normally break down. And those absolutely horrible "You <-> Cloudflare OK, host down" error pages that suggest it's the host that has the problem and at the same time advertise Cloudflare to your visitors. Well turns out the vast majority of issues that clients reported (automated or not) were caused by Cloudflare themselves. And what really makes it painful is that it's 1. sometimes hard to debug 2. they wont make this visible to you unless I guess you pay for the highest Enterprise plan which offers raw logs. They just don't tell you even how many requests failed. There's no simple error.log. An example: at some point Cloudflare started blocking a lot of requests from Turkey to an API. The only way to solve it was to whitelist Turkey as a country. We've moved off of Cloudflare and had a lot fewer issues. PS: every feature since the simple CDN stuff seems to be now about lock-in. It's all Cloudflare specific. This is a dangerous path that moves away from the commoditization of "hosting" where people can easy move from one hoster to another. I hope it does not succeed. For the sake of the open web.
- rattray 6y ago
  Who did you move to from Cloudflare?
  - eis 6y ago
    For CDN we just put servers in different geological locations, some using cloud service and some dedicated. In fact the system itself was made as distributed wherever possible. For the anti DDoS: it was not a big/common issue and in the rare cases we can work with providers to mitigate.
    - snissn 6y ago
      how do you distribute between your servers? DNS? what DNS service?
nix23 6y ago
DON'T USE PROPRIETARY PLATFORMS! Is that so hard to understand??
- KaoruAoiShiho 6y ago
  Let's be real, if they're using it it's because it's cheaper than doing it themselves. If you were to offer them millions of dollars I'm sure they'll use a custom system.
  - nix23 6y ago
    > because it's cheaper than doing it themselves Bullshit!!
  - Ericson2314 6y ago
    Yeah a lot of evidence points to buying one beefy machine being simpler and faster than a gazillion shit machines/slices in the cloud.
- kjeetgill 6y ago
  It's like buying a car. Sure once in a while you're stuck dealing with the dealership waiting on a part (and criticism is warranted) but that's hardly a reason all on its own to stick to walking everywhere.
vegannet 6y ago
This is the same problem GitLab suffer from. I don’t know if it’s necessarily a bad thing: perhaps their success is because of the focus on breadth of features rather than the quality of each individual feature. I don’t have an answer to that but it’s worth considering if the cost of using the value created by CloudFlare _is_ the lack of polish when features are launched. GitLab is a good example of this phenomenon because it can be compared to GitHub: GitHub is high quality slowly whereas GitLab is lower quality faster... maybe CloudFlare is the GitLab to Akamai’s GitHub, both with their valuable place in the market.
- gav 6y ago
  It's a hard problem to solve: getting to some useful amount of "done" where it solves actual problems for customers is a long way from delivering a truly polished, quality product. There's diminishing returns from 80% to 100% done, plus there's the human condition--it's a lot more fun to solve the hard problems than do the polishing at the end. I think that teams can get too enamored of the features they create, it's hard to get perspective when you've been staring at something for a while. I've often wondered if the answer is to have multiple teams (pods) so that one team can get a feature to MVP, then hand it over. You're then not invested in "your" product, there's space to be objective and revisit the path from MVP onward.
rita3ko 6y ago
Hi! I'm a product manager working on Workers — didn't come across hater-y to me at all. We will actually be rolling out a new version of our docs soon — hopefully make them easier to navigate, and expose deeper information. If there are any specific subjects or corner cases you've run into that you'd like to see better documented — our docs are open source (so issues are welcome!). We definitely want to make it as easy as possible for you to self-service. Observability and debugging is also a heavy area of investment for us (and I think we've made significant progress here since the launch of Workers). As Kenton mentioned below, wrangler tail is a great way to get a lot of information out of the Worker (and its subrequests) — you can basically log anything. We'd also like to make it easier to log to existing logging platforms from a Worker. If there's any missing information you'd like to see, please do sent me a note ^
- dkersten 6y ago
  Will there ever be a way to put workers behind the cache (eg maybe the worker can set in the response “yes, please cache this”, the way you already can for the origin, except have it then serve that response rather than run the worker)? Or will workers have an API to interact with firewall rules? Mainly I’m thinking that a major use case for using CF is stuff like DDoS protection, but workers have a per request cost and according to [1] CF’s DDoS protection won’t automatically protect against a layer 7 attack. It seems the options are to either use CF’s rate limiting product (which charges per good request 10x the price of a worker request. Maybe that’s worth it, but it sure makes workers look less affordable than at first), or set firewall rules to block the traffic. For that second option, it would be great if it could be done from inside the worker (although I’m not sure yet how the worker would actually go about detecting that a request aught to be blocked. [1] suggested running fail2ban on the origin and using the firewall API to dynamically update rules — I guess I’m windering if there is a simpler way). Rate limiting seems ideal, I’m just not sold on paying 10x when I don’t know if I’ll even ever need it: maybe simply being able to set optional limits for worker requests after which further requests don’t get serviced, rather than getting billed for an attack? Is that possible? I guess I could count requests somehow myself. and disable the worker. At least then I can make a call between whether uptime is worth the cost of rate limiting or if I can just accept some downtime. [1] https://community.cloudflare.com/t/how-to-protect-cloudflare-worker-from-ddos/179017/3 https://community.cloudflare.com/t/how-to-protect-cloudflare... and https://community.cloudflare.com/t/workers-have-a-critical-issue-or-maybe-im-dumb/173491 https://community.cloudflare.com/t/workers-have-a-critical-i...

by redm 6y ago

kentonv 6y ago

> I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Does `wrangler tail` help with this? https://developers.cloudflare.com/workers/about/tips/debugging/#tail-production-logs-with-wrangler-tail https://developers.cloudflare.com/workers/about/tips/debuggi...

redm 6y ago
What I'm looking for is a full stack trace of a request, and how it progressed through their platform, through pops, Argo, Caches, etc. Akamai and others have tools like this, and it makes it much easier to debug gremlins on the inside. From the docs: "If your Workers application is returning an 1101 error code, your code is throwing an exception. You can catch the exception in your code using a try/catch block:" That's great, but you end up just returning those details in a header or sending them somewhere else. We have to catch errors and send them to our backend for logging so we can debug them at a later point. This doesn't help you debug things like sub-requests from the Worker or how they interact with other Cloudflare features like Cache.
- prdonahue 6y ago
  We're working on this tracing tool as we move things like Page Rules and Rate Limiting into the same engine that runs Firewall Rules. Lots of advantages of using the same Rust-based engine/syntax to let customers express their configuration rules, e.g., adding rule-based alerting and combining different types of filters and actions. Focus so far in the design process has been how requests get evaluated against a list of Custom and Managed Rules but if you imagine a world in which everything is a rule (incl. routing to/firing a Worker) it becomes tractable to do what you're asking about. We'll be in discussions with the Workers team about how to best illustrate the handoff to Workers and what happens downstream: cache, origin fetch, etc.
garganzol 6y ago
Kenton, not exactly topic-related, by I am deeply grateful for what you did in Sandstorm. Cloudflare or not, you a great guy! And I am sure you still have a bunch of magic tricks in your pocket. You sowed the seeds for future crops, literally.

eis 6y ago

> For Workers, I'd like to see a trace tool built for Workers where you can debug what happened in a Live environment. Currently, they recommend outputting debug as JSON in a header. That is a general problem I have had with Cloudflare over the years. It's hard to know exactly what happened when things fail or even the fact that things failed! They always branded themselves as that super scalable and stable CDN or DDoS shield that will keep your service up when it would normally break down. And those absolutely horrible "You <-> Cloudflare OK, host down" error pages that suggest it's the host that has the problem and at the same time advertise Cloudflare to your visitors. Well turns out the vast majority of issues that clients reported (automated or not) were caused by Cloudflare themselves. And what really makes it painful is that it's 1. sometimes hard to debug 2. they wont make this visible to you unless I guess you pay for the highest Enterprise plan which offers raw logs. They just don't tell you even how many requests failed. There's no simple error.log. An example: at some point Cloudflare started blocking a lot of requests from Turkey to an API. The only way to solve it was to whitelist Turkey as a country. We've moved off of Cloudflare and had a lot fewer issues. PS: every feature since the simple CDN stuff seems to be now about lock-in. It's all Cloudflare specific. This is a dangerous path that moves away from the commoditization of "hosting" where people can easy move from one hoster to another. I hope it does not succeed. For the sake of the open web.

rattray 6y ago
Who did you move to from Cloudflare?
- eis 6y ago
  For CDN we just put servers in different geological locations, some using cloud service and some dedicated. In fact the system itself was made as distributed wherever possible. For the anti DDoS: it was not a big/common issue and in the rare cases we can work with providers to mitigate.
  - snissn 6y ago
    how do you distribute between your servers? DNS? what DNS service?

nix23 6y ago

DON'T USE PROPRIETARY PLATFORMS! Is that so hard to understand??

KaoruAoiShiho 6y ago
Let's be real, if they're using it it's because it's cheaper than doing it themselves. If you were to offer them millions of dollars I'm sure they'll use a custom system.
- nix23 6y ago
  > because it's cheaper than doing it themselves Bullshit!!
- Ericson2314 6y ago
  Yeah a lot of evidence points to buying one beefy machine being simpler and faster than a gazillion shit machines/slices in the cloud.
kjeetgill 6y ago
It's like buying a car. Sure once in a while you're stuck dealing with the dealership waiting on a part (and criticism is warranted) but that's hardly a reason all on its own to stick to walking everywhere.

vegannet 6y ago

This is the same problem GitLab suffer from. I don’t know if it’s necessarily a bad thing: perhaps their success is because of the focus on breadth of features rather than the quality of each individual feature. I don’t have an answer to that but it’s worth considering if the cost of using the value created by CloudFlare _is_ the lack of polish when features are launched. GitLab is a good example of this phenomenon because it can be compared to GitHub: GitHub is high quality slowly whereas GitLab is lower quality faster... maybe CloudFlare is the GitLab to Akamai’s GitHub, both with their valuable place in the market.

gav 6y ago
It's a hard problem to solve: getting to some useful amount of "done" where it solves actual problems for customers is a long way from delivering a truly polished, quality product. There's diminishing returns from 80% to 100% done, plus there's the human condition--it's a lot more fun to solve the hard problems than do the polishing at the end. I think that teams can get too enamored of the features they create, it's hard to get perspective when you've been staring at something for a while. I've often wondered if the answer is to have multiple teams (pods) so that one team can get a feature to MVP, then hand it over. You're then not invested in "your" product, there's space to be objective and revisit the path from MVP onward.

rita3ko 6y ago

Hi! I'm a product manager working on Workers — didn't come across hater-y to me at all. We will actually be rolling out a new version of our docs soon — hopefully make them easier to navigate, and expose deeper information. If there are any specific subjects or corner cases you've run into that you'd like to see better documented — our docs are open source (so issues are welcome!). We definitely want to make it as easy as possible for you to self-service. Observability and debugging is also a heavy area of investment for us (and I think we've made significant progress here since the launch of Workers). As Kenton mentioned below, wrangler tail is a great way to get a lot of information out of the Worker (and its subrequests) — you can basically log anything. We'd also like to make it easier to log to existing logging platforms from a Worker. If there's any missing information you'd like to see, please do sent me a note ^

dkersten 6y ago
Will there ever be a way to put workers behind the cache (eg maybe the worker can set in the response “yes, please cache this”, the way you already can for the origin, except have it then serve that response rather than run the worker)? Or will workers have an API to interact with firewall rules? Mainly I’m thinking that a major use case for using CF is stuff like DDoS protection, but workers have a per request cost and according to [1] CF’s DDoS protection won’t automatically protect against a layer 7 attack. It seems the options are to either use CF’s rate limiting product (which charges per good request 10x the price of a worker request. Maybe that’s worth it, but it sure makes workers look less affordable than at first), or set firewall rules to block the traffic. For that second option, it would be great if it could be done from inside the worker (although I’m not sure yet how the worker would actually go about detecting that a request aught to be blocked. [1] suggested running fail2ban on the origin and using the firewall API to dynamically update rules — I guess I’m windering if there is a simpler way). Rate limiting seems ideal, I’m just not sold on paying 10x when I don’t know if I’ll even ever need it: maybe simply being able to set optional limits for worker requests after which further requests don’t get serviced, rather than getting billed for an attack? Is that possible? I guess I could count requests somehow myself. and disable the worker. At least then I can make a call between whether uptime is worth the cost of rate limiting or if I can just accept some downtime. [1] https://community.cloudflare.com/t/how-to-protect-cloudflare-worker-from-ddos/179017/3 https://community.cloudflare.com/t/how-to-protect-cloudflare... and https://community.cloudflare.com/t/workers-have-a-critical-issue-or-maybe-im-dumb/173491 https://community.cloudflare.com/t/workers-have-a-critical-i...