8 ms·

I want to love CloudFlare, I really do. We currently use them, but sadly the number of times that CloudFlare has been the cause of a service interruption is som

by verelo 11y ago

I want to love CloudFlare, I really do. We currently use them, but sadly the number of times that CloudFlare has been the cause of a service interruption is somewhere around 50% mark. They are no longer in use on any critical/important end points, I just don't need PagerDuty waking us up over an issue I have no control over. This is not a problem I expect to improve. As they start to cover all of the web, i imagine that it'll actually only get worse. Curious about how they see this progressing.

jgrahamc 11y ago
I would love to talk to you about this. jgc @ guess the domain.
- hueving 11y ago
  jgc@akamai.com?
  - TheHippo 11y ago
    He works at Cloudflare.
    - nulltype 11y ago
      So you're saying he's some sort of double agent.
- verelo 11y ago
  Happy to chat, feel free to drop me a note. andrew at checkout51.com
saurik 11y ago
I wrote an article a while back digging into the details of an outage in a website I work with caused by CloudFlare, and while at some level the issue here was "CloudFlare didn't care enough to test their stack against popular browsers and then didn't have the in-house expertise to debug and correct issues that were later found", I frankly feel like this problem is pretty engrained into their business :/. If nothing else, the article might be interesting to some as it looks at how to even go about debugging complex interactions between web browser internals and overly clever JavaScript. http://www.saurik.com/id/14 http://www.saurik.com/id/14 "When 'Dumb Pipes' Get Too Smart"
chdir 11y ago
> no longer in use on any critical/important end points What's your recommendation as an alternative. Cloudflare is tempting if you are tight on resources due to free SSl, CDN, etc. and it is super easy to setup. I'd love to try out alternatives though, just as a backup.
- verelo 11y ago
  We're in the process of moving to Akamai
MichaelApproved 11y ago
Would you mind posting the types of problems you've had? I use CF for many critical websites I manage and haven't noticed any problems.
- verelo 11y ago
  Timeouts are the majority of the issues, when our service is running perfectly fine otherwise. They have often been short, but sometimes long enough to cause me to sit there and flip DNS for various end points away from them until the issue is resolved.
- phonon 11y ago
  I'm in financial services...as part of our checkout process, the customer downloads a 1 MB pdf with sensitive and unique data to them. In testing ( a test site with cloudflare set up), everything was fine. In production, apparently cloudflare cached these pdfs after some sort of traffic threshold was reached (I guess they are all very similar in size...though they are probably not exactly the same size.) Now customers are calling that they are getting other people's data! Apparently cloudflare caches files without even comparing hashes of the download first. Needless to say, we entirely shut off any and all caching after that.
  - pipeep 11y ago
    I feel like any CDN or caching proxy (which might even happen at the ISP level if you aren't using HTTPS!) would have that same issue. You should serve confidential files from unique URLs or at least set the right HTTP headers to disable caching.
  - pmtarantino 11y ago
    I am almost sure you did set up this, but worth asking anyway. Was it "Cache-Control: private no-store", right?
  - MichaelApproved 11y ago
    If CF download the file to check the hash, it wouldn't really be helpful as a caching service. That would mean CF would hit your server for each request before it served up the asset to the customer. Your server wouldn't benefit from caching. Maybe you'd get a small speed increase from their worldwide endpoints but that would reduce the advantage of having CF manage your site. One of the biggest advantages is that CF serves hits that your server wouldn't have to. That allows you to need less infrastructure. Further, it might be better to never serve sensitive financial data via a 3rd party server. That kind of data would be best served directly to the customer's browser from your server.
kevingadd 11y ago
I used CloudFlare for a while and it consistently caused more problems than it solved. Really hard to debug, too. Injecting broken JS into pages, serving up corrupted cached assets, etc. Better to just pay for Amazon CloudFront or something and set it all up myself...
- billyhoffman 11y ago
  Those errors are usually caused by CloudFlare automatic Front-end Optimization (FEO) stuff, like trying to losslessly optimize images, add cache control, gzip, SPDY/HTTP2, etc. It used to be a primary marketing point, and while its still there (and still causing problems) they have moved away from promoting it so much. Its best to use of them as a CDN, DDoS sink, and SSL. Their other stuff like FEO or their web app firewall stuff really isn't that impressive enough to be considered a reason for switching. Disclaimer: I work in the web performance space, and used to work in the web security space. As a whole I'm not a fan of the "stick a magic appliance in front of a website to automatically stop [web security issues | web performance problems]" approach, whether from CloudFlare or otherwise.
- manigandham 11y ago
  Just disable all the optimization features and get the benefits of the SSL + DNS + CDN + DDOS protection.
api 11y ago
What problem does CloudFlare really solve except for DDOS?
- nly 11y ago
  Latency, although you can solve that through any number of CDN solutions.
  - api 11y ago
    That can be solved by using a cloud service with global POPs, spinning up nodes everywhere, and using a common core config built with Chef, Puppet, or Docker. Amazon, Digital Ocean, Vultr, Linode, Joyent/Triton, and probably a half dozen others will provide global presence with instant provisioning. Slightly more work but you control it. Cost is probably similar. DDOS mitigation is harder and is definitely something CloudFlare does well enough to earn some market share, but wouldn't it be nice if we had a more global solution to this problem that didn't involve third party firewalls?
    - manigandham 11y ago
      Cost is definitely not similar. Building your own CDN network is not cheap and CloudFlare provides DNS + SSL + lots of locations + better routing + free bandwidth. The servers, ssl and bandwidth costs alone would be more than their fees for any big site.
- foxylad 11y ago
  SSL. SSL is easy on a single server, but gets complex/expensive very fast on cloud platforms. Cloudflare have an incredibly easy solution (branded "flexible SSL") where they handle the SSL between the client and their CDN, and the CDN does un-encrypted requests to your platform. Yes, this is a security shambles - Cloudflare is officially MITM-ing your traffic. But given the HTTPS-only movement this is a perfectly valid solution for public websites. And this announcement actually makes a big difference because if your platform is Google Cloud, then the un-encrypted portion is now over a private Cloudflare-Google interconnect. FWIW, I've been using Cloudflare for 6 months and haven't seen any drop-outs. It's improved latency, and the only issue I've had was it mangling email addresses in transit, to prevent scrapers. It took 3 minutes to find the setting and switch it off, so I'm OK with that. Although I would prefer things that modify your HTML should be off by default.
chrissnell 11y ago
I tried their enterprise tier out about 9 months ago and was entirely unimpressed. Moving a site from Akamai -> CF increased page load times dramatically. We ended up quickly switching back. CloudFlare's complicity with ISIS, however, was what turned me off permanently. For those unaware, CloudFlare was providing proxy shielding of ISIS's propaganda websites. The CF CEO publicly refused to discontinue their service, taking an anti-censorship, pro-free-speech stance. His absolutist views didn't sit well with me and I consider their servicing of these domains to be aiding and abetting this criminal organization.
- mtgx 11y ago
  That's doesn't sound much different than UK asking Facebook and Google to censor "extremist speech" on their platforms.
- bduerst 11y ago
  I'm curious if you think ISPs should block ISIS websites, domains, and IP addresses.
  - chrissnell 11y ago
    It should be entirely up to the ISP. Consequently, the ISP's customers can judge (with their spending) whether they approve of such blocking.
    - chipperyman573 11y ago
      No they can't, at least not in America.
      - erkkie 11y ago
        Actually they could, but they would lose their common carrier (see a sibling comment) protections.
        11y ago
deleted 11y ago
[deleted]
codexon 11y ago
I have a non-critical website with them for free and it randomly becomes unreachable with an error 522 in certain locations. Which seems to indicate it is a problem on their end. When I talked to support about it, they keep telling me I needed to whitelist their IPs when I didn't have any firewall rules in place. All I had was iptables and it had no rules.

by verelo 11y ago

jgrahamc 11y ago

I would love to talk to you about this. jgc @ guess the domain.

hueving 11y ago
jgc@akamai.com?
- TheHippo 11y ago
  He works at Cloudflare.
  - nulltype 11y ago
    So you're saying he's some sort of double agent.
verelo 11y ago
Happy to chat, feel free to drop me a note. andrew at checkout51.com

saurik 11y ago

I wrote an article a while back digging into the details of an outage in a website I work with caused by CloudFlare, and while at some level the issue here was "CloudFlare didn't care enough to test their stack against popular browsers and then didn't have the in-house expertise to debug and correct issues that were later found", I frankly feel like this problem is pretty engrained into their business :/. If nothing else, the article might be interesting to some as it looks at how to even go about debugging complex interactions between web browser internals and overly clever JavaScript. http://www.saurik.com/id/14 http://www.saurik.com/id/14 "When 'Dumb Pipes' Get Too Smart"

chdir 11y ago

> no longer in use on any critical/important end points What's your recommendation as an alternative. Cloudflare is tempting if you are tight on resources due to free SSl, CDN, etc. and it is super easy to setup. I'd love to try out alternatives though, just as a backup.

verelo 11y ago
We're in the process of moving to Akamai

MichaelApproved 11y ago

Would you mind posting the types of problems you've had? I use CF for many critical websites I manage and haven't noticed any problems.

verelo 11y ago
Timeouts are the majority of the issues, when our service is running perfectly fine otherwise. They have often been short, but sometimes long enough to cause me to sit there and flip DNS for various end points away from them until the issue is resolved.
phonon 11y ago
I'm in financial services...as part of our checkout process, the customer downloads a 1 MB pdf with sensitive and unique data to them. In testing ( a test site with cloudflare set up), everything was fine. In production, apparently cloudflare cached these pdfs after some sort of traffic threshold was reached (I guess they are all very similar in size...though they are probably not exactly the same size.) Now customers are calling that they are getting other people's data! Apparently cloudflare caches files without even comparing hashes of the download first. Needless to say, we entirely shut off any and all caching after that.
- pipeep 11y ago
  I feel like any CDN or caching proxy (which might even happen at the ISP level if you aren't using HTTPS!) would have that same issue. You should serve confidential files from unique URLs or at least set the right HTTP headers to disable caching.
- pmtarantino 11y ago
  I am almost sure you did set up this, but worth asking anyway. Was it "Cache-Control: private no-store", right?
- MichaelApproved 11y ago
  If CF download the file to check the hash, it wouldn't really be helpful as a caching service. That would mean CF would hit your server for each request before it served up the asset to the customer. Your server wouldn't benefit from caching. Maybe you'd get a small speed increase from their worldwide endpoints but that would reduce the advantage of having CF manage your site. One of the biggest advantages is that CF serves hits that your server wouldn't have to. That allows you to need less infrastructure. Further, it might be better to never serve sensitive financial data via a 3rd party server. That kind of data would be best served directly to the customer's browser from your server.

kevingadd 11y ago

I used CloudFlare for a while and it consistently caused more problems than it solved. Really hard to debug, too. Injecting broken JS into pages, serving up corrupted cached assets, etc. Better to just pay for Amazon CloudFront or something and set it all up myself...

billyhoffman 11y ago
Those errors are usually caused by CloudFlare automatic Front-end Optimization (FEO) stuff, like trying to losslessly optimize images, add cache control, gzip, SPDY/HTTP2, etc. It used to be a primary marketing point, and while its still there (and still causing problems) they have moved away from promoting it so much. Its best to use of them as a CDN, DDoS sink, and SSL. Their other stuff like FEO or their web app firewall stuff really isn't that impressive enough to be considered a reason for switching. Disclaimer: I work in the web performance space, and used to work in the web security space. As a whole I'm not a fan of the "stick a magic appliance in front of a website to automatically stop [web security issues | web performance problems]" approach, whether from CloudFlare or otherwise.
manigandham 11y ago
Just disable all the optimization features and get the benefits of the SSL + DNS + CDN + DDOS protection.

api 11y ago

What problem does CloudFlare really solve except for DDOS?

nly 11y ago
Latency, although you can solve that through any number of CDN solutions.
- api 11y ago
  That can be solved by using a cloud service with global POPs, spinning up nodes everywhere, and using a common core config built with Chef, Puppet, or Docker. Amazon, Digital Ocean, Vultr, Linode, Joyent/Triton, and probably a half dozen others will provide global presence with instant provisioning. Slightly more work but you control it. Cost is probably similar. DDOS mitigation is harder and is definitely something CloudFlare does well enough to earn some market share, but wouldn't it be nice if we had a more global solution to this problem that didn't involve third party firewalls?
  - manigandham 11y ago
    Cost is definitely not similar. Building your own CDN network is not cheap and CloudFlare provides DNS + SSL + lots of locations + better routing + free bandwidth. The servers, ssl and bandwidth costs alone would be more than their fees for any big site.
foxylad 11y ago
SSL. SSL is easy on a single server, but gets complex/expensive very fast on cloud platforms. Cloudflare have an incredibly easy solution (branded "flexible SSL") where they handle the SSL between the client and their CDN, and the CDN does un-encrypted requests to your platform. Yes, this is a security shambles - Cloudflare is officially MITM-ing your traffic. But given the HTTPS-only movement this is a perfectly valid solution for public websites. And this announcement actually makes a big difference because if your platform is Google Cloud, then the un-encrypted portion is now over a private Cloudflare-Google interconnect. FWIW, I've been using Cloudflare for 6 months and haven't seen any drop-outs. It's improved latency, and the only issue I've had was it mangling email addresses in transit, to prevent scrapers. It took 3 minutes to find the setting and switch it off, so I'm OK with that. Although I would prefer things that modify your HTML should be off by default.

chrissnell 11y ago

I tried their enterprise tier out about 9 months ago and was entirely unimpressed. Moving a site from Akamai -> CF increased page load times dramatically. We ended up quickly switching back. CloudFlare's complicity with ISIS, however, was what turned me off permanently. For those unaware, CloudFlare was providing proxy shielding of ISIS's propaganda websites. The CF CEO publicly refused to discontinue their service, taking an anti-censorship, pro-free-speech stance. His absolutist views didn't sit well with me and I consider their servicing of these domains to be aiding and abetting this criminal organization.

mtgx 11y ago
That's doesn't sound much different than UK asking Facebook and Google to censor "extremist speech" on their platforms.
bduerst 11y ago
I'm curious if you think ISPs should block ISIS websites, domains, and IP addresses.
- chrissnell 11y ago
  It should be entirely up to the ISP. Consequently, the ISP's customers can judge (with their spending) whether they approve of such blocking.
  - chipperyman573 11y ago
    No they can't, at least not in America.
    - erkkie 11y ago
      Actually they could, but they would lose their common carrier (see a sibling comment) protections.
      - 11y ago

deleted 11y ago

[deleted]

codexon 11y ago

I have a non-critical website with them for free and it randomly becomes unreachable with an error 522 in certain locations. Which seems to indicate it is a problem on their end. When I talked to support about it, they keep telling me I needed to whitelist their IPs when I didn't have any firewall rules in place. All I had was iptables and it had no rules.