6 ms·

CloudFlare was down

129 points by zhoutong 14y ago

xPaw 14y ago
Yep, any website using CloudFlare's DNS is not resolving.
- Ateoto 14y ago
  For those of us using cdnjs.com, hopefully you have local fallbacks.
zhoutong 14y ago
It seems that CloudFlare's DNS is down, and affecting NameTerrific as we have a CNAME record pointing to them. I had to change the CNAME record to get our site working again. EDIT: Based on Twitter search, all CloudFlare sites seem to be down.
tonyjin 14y ago
Confirmed, several of my sites are down.
EdisonW 14y ago
Was just about to post this. All of my sites are down. :( Edit: 5:01 EST ..seems to be up and down according to mass pingdom messages.
jgrahamc 14y ago
https://twitter.com/CloudFlareSys/status/308154786316963841 https://twitter.com/CloudFlareSys/status/308154786316963841 - There is a global problem that affects the CloudFlare proxy and DNS services. - The problem appears to be due to bad routing. - We are working to restore correct routes in order to bring both DNS and proxy services back online. - The operations and networking team are all online and treating this as an emergency. - We do not have an ETA on the response time but will continue to post updates via Twitter as we learn more. UPDATE. Sites are being restored now. DNS is operating. I don't know all the details as bugging the network team while they were fixing wasn't going to help. We'll get a postmortem blog post up.
- shanelja 14y ago
  Just discovered this problem on my company website, thanks for the update. -- Seems to be getting better now - intermittent 502's and 504's, the occasional load (speed is choppy though) -- Back to near-instant load times, great job guys! I look forward to the blog entry on this (P.S. It might be worth hosting your status page elsewhere in the future, whilst this might not happen very often, it's when it does that your site needs to be working, the lack of redundancy here is startling.)
endijs 14y ago
Even their status page is down. And sure - all my sites too. And funny part is that for most of my sites i have stopped Clouflare features and use just their DNS. Never thought that I will fail because of DNS not being available.
- mmahemoff 14y ago
  Cloudflare user here. Can I ask why you turned CloudFlare services off?
  - endijs 14y ago
    btw - sites are back. I hope they will not go down. Why I have switched off services? Because once i enabled them sites got slower, not faster. Sure - I doubt that most users noticed that, but for example, if i checked Pingdom or Google Crawl Stats - 'Time spent downloading a page' situation was very clear. With Cloudflare it took Google 2x more time to download page than without. I had no time to investigate why thats so, but after switching Clouflare off i was again back to 500ms. Edit: Sites are not back. I guess that work the ones for which i have DNS cached. Lets wait...
    - mmahemoff 14y ago
      When I briefly did that test, I didn't find much difference on Pingdom; but I like some of their other services and I assume they can handle burst activity at least as well as my own server. So I'm a big fan overall and keep using them.
aytekin 14y ago
This is down as well: http://www.cloudflare.com/system-status http://www.cloudflare.com/system-status They should host this page on a third party provider.
- MichaelApproved 14y ago
  In that case, it should be http://status.cloudflare.com http://status.cloudflare.com
  - tuananh 14y ago
    down as well
  - dbuxton 14y ago
    Except that I think they are hosting their own DNS... I'm not able to retrieve their SOA information even at the moment.
    - mh- 14y ago
      ouch. been awhile since i've seen that failure mode.
  - bromley 14y ago
    Better still cloudflare-status.com or similar: separate domain, separate registrar, separate DNS, separate hosting.
    - rdl 14y ago
      Or just use twitter :) I wish there were a business model for a third party site-monitor and site-uptime service, which let the site owner do more than just post updates, but also prevented the site owner from lying about historical data. Basically New Relic (that actually worked) + Pingdom + Internet Archive + Twitter + status.example.com.
      - 14y ago
jelled 14y ago
imgur seems to be down for me as well.
- mappu 14y ago
  A workaround: try replacing imgur.com in the URL with filmot.com.
DigitalSea 14y ago
Cloudfare may be there when your sites go down, but who is there for Cloudfare when they go down? Nobody it would seem, ha.
- mh- 14y ago
  including their status page
  - mmahemoff 14y ago
    That part surprised me. I thought it was common for ISPs to not use their own services for critical web presence, i.e. places users might visit when it's down.
ddaeo5 14y ago
Really bad.
Hengjie 14y ago
Yeah mine is also down.
vini 14y ago
down here too and I can't change the dns of my site cause my registrar is down too, oh boy.
- tuananh 14y ago
  Olps, major fuck up
- Neso 14y ago
  You are really lucky :p
tuananh 14y ago
all of my sites are down as well. Should I change name server or this issue should be resolved quick (enough)?
jtchang 14y ago
Whatever happened to primary and backup DNS servers?
ryancl 14y ago
And who exactly didn't expect this?
ddaeo5 14y ago
Still down. When they get back up, I will cancel my subscription and gladly DDOS them. :D
- Neso 14y ago
  Our fault, we should have back NS ...
  - ensky 14y ago
    we all expected cloudflare to be more stable than our server ...
    - dbaupp 14y ago
      Presumably they are. I have no idea what is wrong, or how long they will take to fix it, but I'd imagine that CloudFlare has significantly better network engineers than the average company, and so they will fix it in far less time than the average company would fix the same problem.
      - saurik 14y ago
        But, for the amount of money many paid customers are paying them (in essence, anyone at that $3k/mo level that includes the critical 24/7 phone support), you can actually get an account with a company like CDNetworks or Akamai (if nothing else, with a reasonable CDN like EdgeCast) and have still-better network engineers than CloudFlare. Also, even if you are using them for free: they aren't replacing people you have in house... they are an additional component that can independently fail, in addition to any of the things that would have caused your average company's network engineers to fail. They don't promise to cache enough content to replace much of your infrastructure.
      - Igalze 14y ago
        This is what Juniper had to say: “While we have not completed our investigation, we believe this incident was triggered by a product issue that Juniper identified last October, when a patch was also made available" Good network engineers tend to apply newly release patches. This vulnerability was documented for almost half a year...
overshard 14y ago
As someone who hosts hundreds of PAID sites with CloudFlare this is pretty unacceptable. I'm giving them thousands of dollars so that this doesn't happen. Will probably be moving off unless they have some very good reasoning behind a world-wide shutdown of a geo-redundant service...
- robotkad 14y ago
  Have you called the enterprise support number? What was their response? We are in the same boat, but guess where I have the phone number for enterprise support? Webmail on our domain. Lesson learned.
  - overshard 14y ago
    The response is "sorry we are looking into it, we'll call you back when we know what's wrong"... pretty generic.
    - deleted 14y ago
      [deleted]
    - rurounijones 14y ago
      Well to be fair, what do you expect them to say before they have diagnosed the root issue?
      - alan_cx 14y ago
        Fair enough. But then what is the point of paying for enterprise support if all you get is the same generic response a normal punter gets? If they cant offer better, then enterprise support seems poor value at best. Are we sure that if CloudFlare wasn't so popular here, the word scam wouldn't be used?
        mef 14y ago
        So you're saying you want them to say the same thing but reword it since you're enterprise? If they don't know the root cause, they don't know the root cause.
grose 14y ago
Seems like half of the internet is broken just because of this...
oellegaard 14y ago
I'm so happy we didn't go on that wave. Redirecting your DNS to someone else seems like a bad idea in any case. In any case, what do they do, that I could not have done with Varnish?
- bvdbijl 14y ago
  A content delivery network?
  - saurik 14y ago
    They are only sort of a CDN; in addition to DNS, they specialize in site optimization via content transformation (a la mod_pagespeed in the cloud) and "DDoS protection" (which is pretty much them replacing your website for new users with some JavaScript that tries to determine if you are a legitimate client). They don't promise to cache much, and they in fact don't: even on very simple single-page static information websites, such as evasion.com, they have an abysmally poor 81% cache hit ratio. They don't help at all with dynamic content due to having poorly located nodes and lots of heavyweight code running in their proxy. Their lack of many nodes in good positions (compared to something like CDNetworks or Akamai, they are one or two orders of magnitude smaller) also means they can't provide very good latency even for the times when they actually happen to have something in cache. (Note: if someone is now going to say CDNs don't generally do well with dynamic content, they are wrong: normal CDNs actually improve the performance of dynamic content incredibly by maintaining large-window pre-connected HTTP sessions to customer origin servers, often over private networks that already provide better bandwidth: you can easily see 2x latency improvements with a normal CDN even for fully dynamic content). So, they really shouldn't be compared with a "CDN": they have an interesting service that actually provides something valuable for many key use cases (4chan comes to my mind: in essence, something that is actually likely to experience a true DDoS attack sufficiently often and with sufficiently little provocation that it makes sense to add an external system to your infrastructure), but if you need a "CDN" there are many more reasonable alternatives that don't have as many moving parts and are thereby going to break much less often (and, if they actually do, should break only in localized regions).
- richo 14y ago
ck2 14y ago
https://twitter.com/cloudflaresys https://twitter.com/cloudflaresys
hahainternet 14y ago
It's not just a case of DNS being down, I can't see any BGP either. Pretty major failure.
noveltysystems 14y ago
What a let down... Very disappoint.
andyhmltn 14y ago
I use a pretty major forum that has a huge amount of traffic. The owner migrated it to CloudFlare. For the past 5-6 weeks the site has 50% of its request go to a 'Sorry xyz is not available right now'
- earless1 14y ago
  You should make sure that the site is not actually returning 500's for those request. We had some similar issues when we first started using the service
fiendsan 14y ago
its sunday! seems like they pushed another faulty update (like last time)! yep confirmed, all is down including their own site, thats pretty fucked up, when they dont even have offsite status! good thing i dont use cloudflare in all my sites...
tuananh 14y ago
Seems like the main website is back up
- joe5150 14y ago
  I'm getting alternating 504 and 502 errors for the main website. The server signature is "cloudflare-nginx", so it's definitely getting through, though.
- rorrr 14y ago
  504 Gateway Time-out for me.
  - tuananh 14y ago
    it's back up for me http://monosnap.com/image/pYrdbObsYUhKwAWla5gDzzsg0 http://monosnap.com/image/pYrdbObsYUhKwAWla5gDzzsg0
ultimoo 14y ago
My feels go out to the ops folk at Cloudflare. Mistakes happen no matter how many years of experience people bring in, or how much they're paid. We're all humans after all. It must be a pressurizing task to be responsible for potentially millions of dollars of losses during this downtime. I hope the issue is resolved soon and if a person caused it, they're not in too much trouble.
- moot 14y ago
  Amen.
  - shanelja 14y ago
    I guess you're here because 4chan is down? Something I always wondered - do you use Cloudflare simply for the CDN in respects to photos, or does 4chan also frequently become the target of DDoS and other external attacks?
    - moot 14y ago
      We use CloudFlare for everything -- CDN, DDoS mitigation, Railgun (see below), et cetera. They serve ~1.5 petabytes of bandwidth on our behalf every month and proxy billions upon billions of requests. I'm a huge fan of the product and team, despite hiccups like this one. More on Railgun: http://arstechnica.com/information-technology/2013/02/cloudflare-blows-hole-in-laws-of-web-physics-with-go-and-railgun/ http://arstechnica.com/information-technology/2013/02/cloudf... Edit -- I am officially faster than carrier pigeon: me: was i the first human to notify you guys? | me: i caught it within the first 30-60 seconds i think | me: because i have no life and never sleep | CloudFlare Ops pal: yeah. you did. If anyone wants to hire me to check their site instead of Pingdom, feel free to ping me!
      - shanelja 14y ago
        It's expected that eventually things will go down, I've never once found a piece of software, server, person or traffic light which has performed at 100% for their entire lifetime, the best thing we can take from this however is response times. Barely an hour from outage to being live again - I barely had time to fire an email to my colleagues and update our Twitter before it came back online - however, what's worrying here is that there was no fallback in place whatsoever, we will be hearing from the Cloudflare team soon hopefully to let us know what went wrong, but if they are half the company they purport to be, they will learn a lot from whatever went wrong today.
endijs 14y ago
All my sites are now back online! ~ 40min downtime. Edit: Looks like DNS is back. However if you use CloudFlare services then you might still have problems. Like: 504 Gateway Time-out cloudflare-nginx
ippa 14y ago
dns back, now I get bad gateway instead, progress :P
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
nodesocket 14y ago
Seems to be coming back, though intermittent http 502 bad gateways errors.
SchizoDuckie 14y ago
I'm going to say this on the posibility of this being seen as a flamebait... But You should have chosen Akamai over Cloudflare. It's so funny how everybody jumps on top of new companies that say they can proxy all of the interwebz for a low price. (Cloudflare, Blackberry) And then they fail...
- rdl 14y ago
  Akamai won't even talk to people for $0, $25, or $200/mo. I doubt they'd do so for $3000/mo, directly, unless they thought you'd grow.
  - SchizoDuckie 14y ago
    So if you spent just $200/mo on a CDN, Did you really need that CDN at all?
    - rdl 14y ago
      That's irrelevant. A free hobby site might be worth spending $200/mo on CDN for, but I'm not going to drop the $10-30k/mo to get an $AKAM sales rep to call me back. If you're doing mostly static data which rarely changes, you can probably get very high hit rates on CloudFlare, and it's cheaper than even crappy $750/Gbps/mo colo bandwidth then.
      - LemonDrizzle 14y ago
        Rackspace Cloud Files is powered by Akamai.
        rdl 14y ago
        I thought it was Limelight by default, and you had to pay extra for Akamai. (EDIT: I guess they switched; I haven't kept up with Rackspace)
  - saurik 14y ago
    At that $3k/mo level, you can definitely talk to CDNetworks (the CDN I currently use), which is sandwiched between two orders of magnitude of scale, CloudFlare on one side and Akamai on the other. (That said, CDNetworks seems to be much better positioned with regards to China than Akamai.) (That said, I'm actually pretty certain that Akamai would talk to you at the $3k/mo level: have you even tried calling them?)
oron 14y ago
Going back to go daddy :-(
sairamkunala 14y ago
Its back as of 10:51 GMT . CloudFlare and my sites on CloudFlare seem to work again.
Neso 14y ago
They are back online, all my sites active
ensky 14y ago
UP now!!! https://twitter.com/CloudFlare https://twitter.com/CloudFlare
deleted 14y ago
[deleted]
zobzu 14y ago
In my experience, cloudflare has been little more than a scam for anyone with half decent traffic. Not really surprised. Funny how the status page shows all green (are those just static button?) while they acknowledge there is an issue and that they don't know what's going on. http://www.cloudflare.com/system-status http://www.cloudflare.com/system-status
stenehall 14y ago
And it's up again! https://twitter.com/CloudFlareSys/status/308170566760792064 https://twitter.com/CloudFlareSys/status/308170566760792064
rdl 14y ago
I'm starting to think it's questionable to depend on CloudFlare more than necessary, but they're still the best option for some things. (I'm a customer, but probably going to stop being a customer this week; I was mostly curious to test it out. Not really decided, though.) 1) The CloudFlare security model for SSL basically lets them MITM all your traffic. Probably not a big deal for SSLizing a normal website, or even for accepting credit cards), since they're a decent-sized US company with legal liability, although I'd be concerned about their internal security vs. your own internal security (since you're still fully exposed on your side, too -- it doesn't improve security, and can at best not be a source of new vulnerability). 2) Their DNS doesn't appear particularly redundant; it's just anycast in one big block. Using CloudFlare for DNS seems to be bad practice; you should use something else and cname to CF. Ideally something with multiple DNS servers either individually anycast or in at least two independent (probably anycast) netblocks. 3) Performance of the proxy service seems adequate in my experience but for sites with large amounts of overseas-source traffic, I've heard of people getting lots of suspected-bad-guy path. For a free forum like 4chan that's probably fine; for an e-commerce site, probably not.
- saurik 14y ago
  Can you say more about "lots of suspected-bad-guy path"? (I do not use CloudFlare currently and am not intending to do so, but I do run an e-commerce site, and have not heard this term used before: the idea of an entire axis along which I have not been evaluating my infrastructure intrigues me.)
  - rdl 14y ago
    If they think you (as an end user accessing a CF site) are a source of DDoS or other abusive traffic, they push your traffic into a sandbox. You then have to answer a captcha before moving on; I assume the bounce rate is incredibly high among most users at that point.
  - benologist 14y ago
    I saw their 'bad guy' page many times living in Costa Rica then Nicaragua. I didn't realize what it was until a friend started testing out CF and I got it on his site and then we realized what was going on. That was over a year ago though, I assume things have improved as they got more and better data (and resources). I was in Costa Rica in January/February then again in August/September last year and didn't come across it I don't think. I'm in Turkey at the moment and for the last couple weeks and haven't seen one at all, and that's including using an EC2 proxy because of the censorship here.
UnoriginalGuy 14y ago
This took out Imgur.com (Reddit's favourite image hosting site) for several hours. So it was definitely felt.
wyuenho 14y ago
It took a CloudFlare total wipeout to discover how useless our browsers are against domain name lookups that take a ridiculously long time to timeout. News flash: CDN fallback like the one below is next to useless unless the first request times out reasonably quickly. http://css-tricks.com/snippets/jquery/fallback-for-cdn-hosted-jquery/ http://css-tricks.com/snippets/jquery/fallback-for-cdn-hoste...
brador 14y ago
From the cloudfare business page: "2500% guarantee This extended Service Level Agreement guarantees 100% uptime, and adds a multiplier to owed service credits resulting from any lapse: 5 times any downtime minutes and 5 times customers affected = 2500% guarantee."
- TazeTSchnitzel 14y ago
  Ouch.
- hu_me 14y ago
  well they have 785000 sites were down for 60mins so that adds up to... a lot of service credits
  - robotkad 14y ago
    Most of them would be on the free tier
- ksec 14y ago
  So anyone know what is the services credit for 1 Min of downtime?
packetlss 14y ago
Looks like they dropped off the internet: http://www.youtube.com/watch?v=wMRaKtydILI http://www.youtube.com/watch?v=wMRaKtydILI AS13335 = Cloudflare
ksec 14y ago
Somebody pointed out about the CNAME available on Cloudflare. I never knew that and i checked out the article. The First paragraph: "CNAME setup is a manual process generally available to paid CloudFlare plans only. If you are interested in testing CNAME setup, please contact CloudFlare first with the domain you would like to test CNAME with. Please specifically mention CNAME Setup in the subject field for faster review. Allowing for CNAME setup is entirely at the discretion of CloudFlare." So NO: This isn't even a features at all. They made it as hard as possible to set this up and will grant you the use of it as they like.

CloudFlare was down

129 points by zhoutong 14y ago

xPaw 14y ago

Yep, any website using CloudFlare's DNS is not resolving.

Ateoto 14y ago
For those of us using cdnjs.com, hopefully you have local fallbacks.

zhoutong 14y ago

It seems that CloudFlare's DNS is down, and affecting NameTerrific as we have a CNAME record pointing to them. I had to change the CNAME record to get our site working again. EDIT: Based on Twitter search, all CloudFlare sites seem to be down.

tonyjin 14y ago

Confirmed, several of my sites are down.

EdisonW 14y ago

Was just about to post this. All of my sites are down. :( Edit: 5:01 EST ..seems to be up and down according to mass pingdom messages.

jgrahamc 14y ago

https://twitter.com/CloudFlareSys/status/308154786316963841 https://twitter.com/CloudFlareSys/status/308154786316963841 - There is a global problem that affects the CloudFlare proxy and DNS services. - The problem appears to be due to bad routing. - We are working to restore correct routes in order to bring both DNS and proxy services back online. - The operations and networking team are all online and treating this as an emergency. - We do not have an ETA on the response time but will continue to post updates via Twitter as we learn more. UPDATE. Sites are being restored now. DNS is operating. I don't know all the details as bugging the network team while they were fixing wasn't going to help. We'll get a postmortem blog post up.

shanelja 14y ago
Just discovered this problem on my company website, thanks for the update. -- Seems to be getting better now - intermittent 502's and 504's, the occasional load (speed is choppy though) -- Back to near-instant load times, great job guys! I look forward to the blog entry on this (P.S. It might be worth hosting your status page elsewhere in the future, whilst this might not happen very often, it's when it does that your site needs to be working, the lack of redundancy here is startling.)

endijs 14y ago

Even their status page is down. And sure - all my sites too. And funny part is that for most of my sites i have stopped Clouflare features and use just their DNS. Never thought that I will fail because of DNS not being available.

mmahemoff 14y ago
Cloudflare user here. Can I ask why you turned CloudFlare services off?
- endijs 14y ago
  btw - sites are back. I hope they will not go down. Why I have switched off services? Because once i enabled them sites got slower, not faster. Sure - I doubt that most users noticed that, but for example, if i checked Pingdom or Google Crawl Stats - 'Time spent downloading a page' situation was very clear. With Cloudflare it took Google 2x more time to download page than without. I had no time to investigate why thats so, but after switching Clouflare off i was again back to 500ms. Edit: Sites are not back. I guess that work the ones for which i have DNS cached. Lets wait...
  - mmahemoff 14y ago
    When I briefly did that test, I didn't find much difference on Pingdom; but I like some of their other services and I assume they can handle burst activity at least as well as my own server. So I'm a big fan overall and keep using them.

aytekin 14y ago

This is down as well: http://www.cloudflare.com/system-status http://www.cloudflare.com/system-status They should host this page on a third party provider.

MichaelApproved 14y ago
In that case, it should be http://status.cloudflare.com http://status.cloudflare.com
- tuananh 14y ago
  down as well
- dbuxton 14y ago
  Except that I think they are hosting their own DNS... I'm not able to retrieve their SOA information even at the moment.
  - mh- 14y ago
    ouch. been awhile since i've seen that failure mode.
- bromley 14y ago
  Better still cloudflare-status.com or similar: separate domain, separate registrar, separate DNS, separate hosting.
  - rdl 14y ago
    Or just use twitter :) I wish there were a business model for a third party site-monitor and site-uptime service, which let the site owner do more than just post updates, but also prevented the site owner from lying about historical data. Basically New Relic (that actually worked) + Pingdom + Internet Archive + Twitter + status.example.com.
    - 14y ago

jelled 14y ago

imgur seems to be down for me as well.

mappu 14y ago
A workaround: try replacing imgur.com in the URL with filmot.com.

DigitalSea 14y ago

Cloudfare may be there when your sites go down, but who is there for Cloudfare when they go down? Nobody it would seem, ha.

mh- 14y ago
including their status page
- mmahemoff 14y ago
  That part surprised me. I thought it was common for ISPs to not use their own services for critical web presence, i.e. places users might visit when it's down.

ddaeo5 14y ago

Really bad.

Hengjie 14y ago

Yeah mine is also down.

vini 14y ago

down here too and I can't change the dns of my site cause my registrar is down too, oh boy.

tuananh 14y ago
Olps, major fuck up
Neso 14y ago
You are really lucky :p

tuananh 14y ago

all of my sites are down as well. Should I change name server or this issue should be resolved quick (enough)?

jtchang 14y ago

Whatever happened to primary and backup DNS servers?

ryancl 14y ago

And who exactly didn't expect this?

ddaeo5 14y ago

Still down. When they get back up, I will cancel my subscription and gladly DDOS them. :D

Neso 14y ago
Our fault, we should have back NS ...
- ensky 14y ago
  we all expected cloudflare to be more stable than our server ...
  - dbaupp 14y ago
    Presumably they are. I have no idea what is wrong, or how long they will take to fix it, but I'd imagine that CloudFlare has significantly better network engineers than the average company, and so they will fix it in far less time than the average company would fix the same problem.
    - saurik 14y ago
      But, for the amount of money many paid customers are paying them (in essence, anyone at that $3k/mo level that includes the critical 24/7 phone support), you can actually get an account with a company like CDNetworks or Akamai (if nothing else, with a reasonable CDN like EdgeCast) and have still-better network engineers than CloudFlare. Also, even if you are using them for free: they aren't replacing people you have in house... they are an additional component that can independently fail, in addition to any of the things that would have caused your average company's network engineers to fail. They don't promise to cache enough content to replace much of your infrastructure.
    - Igalze 14y ago
      This is what Juniper had to say: “While we have not completed our investigation, we believe this incident was triggered by a product issue that Juniper identified last October, when a patch was also made available" Good network engineers tend to apply newly release patches. This vulnerability was documented for almost half a year...

overshard 14y ago

As someone who hosts hundreds of PAID sites with CloudFlare this is pretty unacceptable. I'm giving them thousands of dollars so that this doesn't happen. Will probably be moving off unless they have some very good reasoning behind a world-wide shutdown of a geo-redundant service...

robotkad 14y ago
Have you called the enterprise support number? What was their response? We are in the same boat, but guess where I have the phone number for enterprise support? Webmail on our domain. Lesson learned.
- overshard 14y ago
  The response is "sorry we are looking into it, we'll call you back when we know what's wrong"... pretty generic.
  - deleted 14y ago
    [deleted]
  - rurounijones 14y ago
    Well to be fair, what do you expect them to say before they have diagnosed the root issue?
    - alan_cx 14y ago
      Fair enough. But then what is the point of paying for enterprise support if all you get is the same generic response a normal punter gets? If they cant offer better, then enterprise support seems poor value at best. Are we sure that if CloudFlare wasn't so popular here, the word scam wouldn't be used?
      - mef 14y ago
        So you're saying you want them to say the same thing but reword it since you're enterprise? If they don't know the root cause, they don't know the root cause.

grose 14y ago

Seems like half of the internet is broken just because of this...

oellegaard 14y ago

I'm so happy we didn't go on that wave. Redirecting your DNS to someone else seems like a bad idea in any case. In any case, what do they do, that I could not have done with Varnish?

bvdbijl 14y ago
A content delivery network?
- saurik 14y ago
  They are only sort of a CDN; in addition to DNS, they specialize in site optimization via content transformation (a la mod_pagespeed in the cloud) and "DDoS protection" (which is pretty much them replacing your website for new users with some JavaScript that tries to determine if you are a legitimate client). They don't promise to cache much, and they in fact don't: even on very simple single-page static information websites, such as evasion.com, they have an abysmally poor 81% cache hit ratio. They don't help at all with dynamic content due to having poorly located nodes and lots of heavyweight code running in their proxy. Their lack of many nodes in good positions (compared to something like CDNetworks or Akamai, they are one or two orders of magnitude smaller) also means they can't provide very good latency even for the times when they actually happen to have something in cache. (Note: if someone is now going to say CDNs don't generally do well with dynamic content, they are wrong: normal CDNs actually improve the performance of dynamic content incredibly by maintaining large-window pre-connected HTTP sessions to customer origin servers, often over private networks that already provide better bandwidth: you can easily see 2x latency improvements with a normal CDN even for fully dynamic content). So, they really shouldn't be compared with a "CDN": they have an interesting service that actually provides something valuable for many key use cases (4chan comes to my mind: in essence, something that is actually likely to experience a true DDoS attack sufficiently often and with sufficiently little provocation that it makes sense to add an external system to your infrastructure), but if you need a "CDN" there are many more reasonable alternatives that don't have as many moving parts and are thereby going to break much less often (and, if they actually do, should break only in localized regions).
richo 14y ago

ck2 14y ago

https://twitter.com/cloudflaresys https://twitter.com/cloudflaresys

hahainternet 14y ago

It's not just a case of DNS being down, I can't see any BGP either. Pretty major failure.

noveltysystems 14y ago

What a let down... Very disappoint.

andyhmltn 14y ago

I use a pretty major forum that has a huge amount of traffic. The owner migrated it to CloudFlare. For the past 5-6 weeks the site has 50% of its request go to a 'Sorry xyz is not available right now'

earless1 14y ago
You should make sure that the site is not actually returning 500's for those request. We had some similar issues when we first started using the service

fiendsan 14y ago

its sunday! seems like they pushed another faulty update (like last time)! yep confirmed, all is down including their own site, thats pretty fucked up, when they dont even have offsite status! good thing i dont use cloudflare in all my sites...

tuananh 14y ago

Seems like the main website is back up

joe5150 14y ago
I'm getting alternating 504 and 502 errors for the main website. The server signature is "cloudflare-nginx", so it's definitely getting through, though.
rorrr 14y ago
504 Gateway Time-out for me.
- tuananh 14y ago
  it's back up for me http://monosnap.com/image/pYrdbObsYUhKwAWla5gDzzsg0 http://monosnap.com/image/pYrdbObsYUhKwAWla5gDzzsg0

ultimoo 14y ago

My feels go out to the ops folk at Cloudflare. Mistakes happen no matter how many years of experience people bring in, or how much they're paid. We're all humans after all. It must be a pressurizing task to be responsible for potentially millions of dollars of losses during this downtime. I hope the issue is resolved soon and if a person caused it, they're not in too much trouble.

moot 14y ago
Amen.
- shanelja 14y ago
  I guess you're here because 4chan is down? Something I always wondered - do you use Cloudflare simply for the CDN in respects to photos, or does 4chan also frequently become the target of DDoS and other external attacks?
  - moot 14y ago
    We use CloudFlare for everything -- CDN, DDoS mitigation, Railgun (see below), et cetera. They serve ~1.5 petabytes of bandwidth on our behalf every month and proxy billions upon billions of requests. I'm a huge fan of the product and team, despite hiccups like this one. More on Railgun: http://arstechnica.com/information-technology/2013/02/cloudflare-blows-hole-in-laws-of-web-physics-with-go-and-railgun/ http://arstechnica.com/information-technology/2013/02/cloudf... Edit -- I am officially faster than carrier pigeon: me: was i the first human to notify you guys? | me: i caught it within the first 30-60 seconds i think | me: because i have no life and never sleep | CloudFlare Ops pal: yeah. you did. If anyone wants to hire me to check their site instead of Pingdom, feel free to ping me!
    - shanelja 14y ago
      It's expected that eventually things will go down, I've never once found a piece of software, server, person or traffic light which has performed at 100% for their entire lifetime, the best thing we can take from this however is response times. Barely an hour from outage to being live again - I barely had time to fire an email to my colleagues and update our Twitter before it came back online - however, what's worrying here is that there was no fallback in place whatsoever, we will be hearing from the Cloudflare team soon hopefully to let us know what went wrong, but if they are half the company they purport to be, they will learn a lot from whatever went wrong today.

endijs 14y ago

All my sites are now back online! ~ 40min downtime. Edit: Looks like DNS is back. However if you use CloudFlare services then you might still have problems. Like: 504 Gateway Time-out cloudflare-nginx

ippa 14y ago

dns back, now I get bad gateway instead, progress :P

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

nodesocket 14y ago

Seems to be coming back, though intermittent http 502 bad gateways errors.

SchizoDuckie 14y ago

I'm going to say this on the posibility of this being seen as a flamebait... But You should have chosen Akamai over Cloudflare. It's so funny how everybody jumps on top of new companies that say they can proxy all of the interwebz for a low price. (Cloudflare, Blackberry) And then they fail...

rdl 14y ago
Akamai won't even talk to people for $0, $25, or $200/mo. I doubt they'd do so for $3000/mo, directly, unless they thought you'd grow.
- SchizoDuckie 14y ago
  So if you spent just $200/mo on a CDN, Did you really need that CDN at all?
  - rdl 14y ago
    That's irrelevant. A free hobby site might be worth spending $200/mo on CDN for, but I'm not going to drop the $10-30k/mo to get an $AKAM sales rep to call me back. If you're doing mostly static data which rarely changes, you can probably get very high hit rates on CloudFlare, and it's cheaper than even crappy $750/Gbps/mo colo bandwidth then.
    - LemonDrizzle 14y ago
      Rackspace Cloud Files is powered by Akamai.
      - rdl 14y ago
        I thought it was Limelight by default, and you had to pay extra for Akamai. (EDIT: I guess they switched; I haven't kept up with Rackspace)
- saurik 14y ago
  At that $3k/mo level, you can definitely talk to CDNetworks (the CDN I currently use), which is sandwiched between two orders of magnitude of scale, CloudFlare on one side and Akamai on the other. (That said, CDNetworks seems to be much better positioned with regards to China than Akamai.) (That said, I'm actually pretty certain that Akamai would talk to you at the $3k/mo level: have you even tried calling them?)

oron 14y ago

Going back to go daddy :-(

sairamkunala 14y ago

Its back as of 10:51 GMT . CloudFlare and my sites on CloudFlare seem to work again.

Neso 14y ago

They are back online, all my sites active

ensky 14y ago

UP now!!! https://twitter.com/CloudFlare https://twitter.com/CloudFlare

deleted 14y ago

[deleted]

zobzu 14y ago

In my experience, cloudflare has been little more than a scam for anyone with half decent traffic. Not really surprised. Funny how the status page shows all green (are those just static button?) while they acknowledge there is an issue and that they don't know what's going on. http://www.cloudflare.com/system-status http://www.cloudflare.com/system-status

stenehall 14y ago

And it's up again! https://twitter.com/CloudFlareSys/status/308170566760792064 https://twitter.com/CloudFlareSys/status/308170566760792064

rdl 14y ago

I'm starting to think it's questionable to depend on CloudFlare more than necessary, but they're still the best option for some things. (I'm a customer, but probably going to stop being a customer this week; I was mostly curious to test it out. Not really decided, though.) 1) The CloudFlare security model for SSL basically lets them MITM all your traffic. Probably not a big deal for SSLizing a normal website, or even for accepting credit cards), since they're a decent-sized US company with legal liability, although I'd be concerned about their internal security vs. your own internal security (since you're still fully exposed on your side, too -- it doesn't improve security, and can at best not be a source of new vulnerability). 2) Their DNS doesn't appear particularly redundant; it's just anycast in one big block. Using CloudFlare for DNS seems to be bad practice; you should use something else and cname to CF. Ideally something with multiple DNS servers either individually anycast or in at least two independent (probably anycast) netblocks. 3) Performance of the proxy service seems adequate in my experience but for sites with large amounts of overseas-source traffic, I've heard of people getting lots of suspected-bad-guy path. For a free forum like 4chan that's probably fine; for an e-commerce site, probably not.

saurik 14y ago
Can you say more about "lots of suspected-bad-guy path"? (I do not use CloudFlare currently and am not intending to do so, but I do run an e-commerce site, and have not heard this term used before: the idea of an entire axis along which I have not been evaluating my infrastructure intrigues me.)
- rdl 14y ago
  If they think you (as an end user accessing a CF site) are a source of DDoS or other abusive traffic, they push your traffic into a sandbox. You then have to answer a captcha before moving on; I assume the bounce rate is incredibly high among most users at that point.
- benologist 14y ago
  I saw their 'bad guy' page many times living in Costa Rica then Nicaragua. I didn't realize what it was until a friend started testing out CF and I got it on his site and then we realized what was going on. That was over a year ago though, I assume things have improved as they got more and better data (and resources). I was in Costa Rica in January/February then again in August/September last year and didn't come across it I don't think. I'm in Turkey at the moment and for the last couple weeks and haven't seen one at all, and that's including using an EC2 proxy because of the censorship here.

UnoriginalGuy 14y ago

This took out Imgur.com (Reddit's favourite image hosting site) for several hours. So it was definitely felt.

wyuenho 14y ago

It took a CloudFlare total wipeout to discover how useless our browsers are against domain name lookups that take a ridiculously long time to timeout. News flash: CDN fallback like the one below is next to useless unless the first request times out reasonably quickly. http://css-tricks.com/snippets/jquery/fallback-for-cdn-hosted-jquery/ http://css-tricks.com/snippets/jquery/fallback-for-cdn-hoste...

brador 14y ago

From the cloudfare business page: "2500% guarantee This extended Service Level Agreement guarantees 100% uptime, and adds a multiplier to owed service credits resulting from any lapse: 5 times any downtime minutes and 5 times customers affected = 2500% guarantee."

TazeTSchnitzel 14y ago
Ouch.
hu_me 14y ago
well they have 785000 sites were down for 60mins so that adds up to... a lot of service credits
- robotkad 14y ago
  Most of them would be on the free tier
ksec 14y ago
So anyone know what is the services credit for 1 Min of downtime?

packetlss 14y ago

Looks like they dropped off the internet: http://www.youtube.com/watch?v=wMRaKtydILI http://www.youtube.com/watch?v=wMRaKtydILI AS13335 = Cloudflare

ksec 14y ago

Somebody pointed out about the CNAME available on Cloudflare. I never knew that and i checked out the article. The First paragraph: "CNAME setup is a manual process generally available to paid CloudFlare plans only. If you are interested in testing CNAME setup, please contact CloudFlare first with the domain you would like to test CNAME with. Please specifically mention CNAME Setup in the subject field for faster review. Allowing for CNAME setup is entirely at the discretion of CloudFlare." So NO: This isn't even a features at all. They made it as hard as possible to set this up and will grant you the use of it as they like.