25 ms·

Facebook, Instagram go down around the world in an apparent outage

610 points by ajiang 8y ago

letientai299 8y ago
This is the first time I experience this. Also note that current session on messenger.com still work, we can still send/receive message, but can't upload any image or send sticker. Looking forward for a post mortem analysis on this.
allannienhuis 8y ago
seems to be more than just messenger login; All of facebook is super flakey this morning.
JakeWesorick 8y ago
https://www.facebook.com/platform/api-status/ https://www.facebook.com/platform/api-status/ still returns "Facebook Platform is Healthy", but you can't even load https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/. Why have status pages if they are so susceptible to going down themselves?
- ceejayoz 8y ago
  I remember an S3 outage a number of years ago where AWS discovered that their status page was hosted on S3. Whoops! I believe this is why Github's status page is now on its own domain; so a github.com DNS outage won't take it down.
  - simlevesque 8y ago
    IIRC the AWS status page was fine but the indicator of the status of the different services was a red or green .PNG which were hosted on S3.
aviral 8y ago
https://downdetector.com/status/facebook https://downdetector.com/status/facebook
yeutterg 8y ago
Can confirm. Also, API integrations, such as Buffer, are not working.
JDiculous 8y ago
Is Facebook actually working for anyone?
jedberg 8y ago
So yesterday Google had a major (and out of character) outage, and today Facebook has a major (and also out of character) outage. I can't wait to see the RCA for both of these and if they're related.
jedberg 8y ago
So yesterday Google had a major (and out of character) outage across its apps, and today Facebook has a major (and also out of character) outage across its apps. I can't wait to see the RCA for both of these and if they're related.
- godzillabrennus 8y ago
  Private post Morten: The NSA middleware we are required to run (that took time to deploy to each of our social partners) is breaking something so let’s revert. Public post Mortem: Entirely believable technical cause.
  - samstave 8y ago
    Alternate Post Mortem: Cyber WWIII's first public skirmishes become visible... (Ignore Stuxnet, Ignore DUQU)
    - jmrobertson 8y ago
      yeah I wonder
    - wybiral 8y ago
      /me puts on tin foil hat It's the rice grain implants, man
  - popz41 8y ago
    I imagine the NSA uses an optical tap device. These devices create identical copies and require no power or management.
    - cronix 8y ago
      That's why it's called PRISM. It's exactly what you describe. Splitting an optical signal into 2 using, basically, a prism. One signal goes out to the net as normal, the other goes to their own datacenters, that they keep continually building and expanding. The newer ones are being build on military bases, for added security. Check em out. Look at the size and cost of them. Some are over a million sq. ft. That's a lot of data. They measure it in terms of zottabytes and zettabytes (in 2013, a lifetime ago in terms of storage space): https://modernsurvivalblog.com/government-gone-wild/nsa-locations-in-the-united-states/ https://modernsurvivalblog.com/government-gone-wild/nsa-loca... http://worldstopdatacenters.com/government-data-centers/ http://worldstopdatacenters.com/government-data-centers/
boshomi 8y ago
see also: Facebook, Instagram down: Social media sites not working for many, FB doing 'required maintenance' [1] https://www.abc15.com/news/national/facebook-down-social-media-site-not-working-for-many https://www.abc15.com/news/national/facebook-down-social-med...
boshomi 8y ago
the issue is also in EU: Is Facebook down? Messenger, site, app and Instagram hit by issues[1] [1]https://www.manchestereveningnews.co.uk/news/uk-news/facebook-down-app-website-crash-15969732 https://www.manchestereveningnews.co.uk/news/uk-news/faceboo...
g8oz 8y ago
...and nothing of value was lost.
arisAlexis 8y ago
yesterday Google and today Facebook. My conspirator says it's the Chinese government showcasing.
- mdkdog 8y ago
  Meanwhile in Russia they are talking about disconnecting their network from the rest of the world. Some test gone south?... Maybe... Someone has a traceroute handy?
BucketSort 8y ago
Google and FB having successive outages? Is this just a coincidence or is there some shared infrastructure that would explain this?
- js2 8y ago
  Verizon also had SMS issues yesterday: https://www.businessinsider.com/verizon-outage-on-east-coast-texts-arent-sending-2019-3 https://www.businessinsider.com/verizon-outage-on-east-coast...
- aviv 8y ago
  Likely a firmware issue with the NSA intercept devices.
  - Theodores 8y ago
    ...and their key infrastructure in Venezuela.
    - clydethefrog 8y ago
      Yes, I can't help to entertain the thought of another Stuxnet going wild.
  - 9999px 8y ago
    Are you basing that on anything or just BS'ing?
  - OscarTheGrinch 8y ago
    Hail Ghidra!
- frostyj 8y ago
  ^^^ I noticed this too. GCP is under all 50 shades of outages since past few days. Feel I might need to rush back to my house and start digging a bunker
- twothumbsup 8y ago
wybiral 8y ago
Can you imagine if Twitter and Google went down at the same time? People would be reactivating their Facebook accounts and having to sift through conspiracy theory posts about Hillary Clinton still just to figure out what was going on. Edit: The points on this post keep going up and down every time I check these comments. Yes, it was sarcasm, I was joking, but I was trying to point out that most people rely on a small set of services. "Cloud" has centralized things a lot.
- bargl 8y ago
  I think the only reason your comment gets down voted is because it's biased only to one side. I see about what Jon Oliver states as if I remember correct 1/2 as many conspiracy posts from about Trump (Edit: I meant to write conservatives write 2x as many conspiracy posts as democrats. This was not represented in my previous sentence.). It's annoying. I guess to accurately represent things on Facebook you'd have to say something like conspiracy posts about AOC, Bernie, and Dan Chrenshaw. 2x1 Ratio. You could have also said, "political conspiracy theories". I get annoyed by conspiracies on either side personally, but I can see how some people would see more of one than the other based on their friends. Edit: Re-read some of what I wrote and I want to clarify, this isn't me saying i down-voted you. Just an observation.
  - wybiral 8y ago
    It was a joke based on what people often say about FB. I don't use their platforms.
    - bargl 8y ago
      Jokes, typically hint at a truth or belief of the comedian. I'm not saying it's a bad joke, I actually liked it. I was merely making an observation which apparently a lot of people disliked, and it looks (based on the upvote/downvote feedback) I am probably wrong. It might just be because a lot of people around here read thishttp://www.paulgraham.com/hackernews.html http://www.paulgraham.com/hackernews.html and think that any joke means it's a bad comment. IDK. As I said I thought your comment was funny.
agosnell 8y ago
If you use their API and haven't seen it yet, their issue is listed here on their status page: https://developers.facebook.com/status/issues/559896447845433/ https://developers.facebook.com/status/issues/55989644784543...
- odomojuli 8y ago
  Looks like the status page is down for me as well.
  - yorwba 8y ago
    Increased Error Rates Created by Gary Fitzpatrick · · Facebook Team — Today at 10:32 AM Current State: Investigating Description: We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution. Start Time: 2 hours ago Last Update: about an hour ago Updates: There are currently no updates for this issue.
    - twic 8y ago
      > Increased Error Rates Created by Gary Fitzpatrick Well someone tell him to stop, for pete's sake!
      - samstave 8y ago
        Are you joshing me?
      - nogbit 8y ago
        I read it that way as well :)
      - jmts 8y ago
        Pete's in on it too!
nathan929 8y ago
...terrifying API developers everywhere...
njn 8y ago
Good! Keep them down.
jak92 8y ago
... and nothing of value was lost.
btown 8y ago
I've also seen issues uploading images to Whatsapp in the past half hour. I wonder if there's anything to do with the Google Cloud Storage outage that took down Gmail yesterday?
- twothumbsup 8y ago
  Facebook doesn't use GCS (at least they didn't in 2015), they have their own infrastructure/data centers.
earless1 8y ago
What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.
- phoe-krk 8y ago
  An expired certificate, for instance. https://www.thesslstore.com/blog/expired-certificate-ericsson-o2/ https://www.thesslstore.com/blog/expired-certificate-ericsso...
- rwultsch 8y ago
  Short duration: network, bad software deploy Long duration: db. If you break data, it takes a while to unbreak. Source: Me. My career has been spent managing db's for internet scale sites.
  - cheeze 8y ago
    Nothing worse than that sinking feeling of "oh fuck, we have to backfill a lot of data.
    - WrtCdEvrydy 8y ago
      Why did my username on this site just change to 'test123'... oh, where clauses.
      - jomkr 8y ago
        Nothing worse then the page on Friday night, oh there goes my weekend.
  - dsfyu404ed 8y ago
    I work for a smaller but comparably large platform. "If everything is down check the DB" is at the top of one of our internal monitoring websites in red. Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved but they have to spend a bunch of time synchronizing things and verifying the fix before they press the big red "go live" button.
tinyhouse 8y ago
From Google I got an error clicking on instagram and Quora links today.
- JohnJamesRambo 8y ago
  At least you are free of Quora now!
zomg 8y ago
All my funny cat videos and memes are loading fine. Did you try rebooting your router 62 times? All joking aside, is this news? :/
- blhack 8y ago
  One of the largest internet companies in the world having a massive, global outage? Yes, that seems like appropriate news for "hacker news", a website where people discuss technology news.
  - zomg 8y ago
    wow, tough crowd today. i didn't think people took facebook and instagram so seriously. i'll put the sarcasm machine back in the drawer.
    - icebraining 8y ago
      This is HN, we take everything seriously.
- pbhjpbhj 8y ago
  Do you want to explain why you believe it's not [HN worthy] news?
- deleted 8y ago
  [deleted]
rdtsc 8y ago
Google then Facebook and Instagram? My hunch is that it's the end of Q1 and people are trying to release code changes so they can pad their Q1 performance reviews "designed and delivered feature X on time in Q1".
- unfunco 8y ago
  "and brought down a service used by billions and losing potentially millions in revenue."
  - Qwertystop 8y ago
    Well, it's certainly impressive.
  - outside1234 8y ago
    Can't claim they didn't have impact then. :)
  - abtom 8y ago
    "Move fast break things"
- cheeze 8y ago
  Yeah I'm gonna go with about a 0% chance of this.
- toephu2 8y ago
  Year end performance reviews are the most important..those were already done at most companies.
  - rdtsc 8y ago
    I've seen all quarter ends being targets for releases. And things that have been delayed since start of the end of the year are usually pushed to the end of Q1.
nodesocket 8y ago
And yet Facebook's stock is still up on the day (+.74%)? You'd think being down for hours would be negative news and revenue impacting.
- derwiki 8y ago
  2 hours of downtime in a quarter is 0.09% of downtime -- probably very little effect on their monetization products.
  - farisjarrah 8y ago
    We're up to around 7 hours of partial outage now. I have spoken to FB employees in the past, every hour they are even partially down they are loosing millions and millions in revenue.
mizeandmen 8y ago
Bunch fb employees near pacific catch was talking about how fb was hacked
minimaxir 8y ago
Minor update: https://twitter.com/facebook/status/1105907126424109056?s=21 https://twitter.com/facebook/status/1105907126424109056?s=21 > We're focused on working to resolve the issue as soon as possible, but can confirm that the issue is not related to a DDoS attack.
- KennyFromIT 8y ago
  There's something ironic about a social media company being forced to rely on a competitor to facilitate communication between them and their users.
  - mic47 8y ago
    Why? When Facebook is down, you cannot use Facebook to communicate about the issue. That is also reason why FB still uses* irc instead of messenger for coordinating how to resolve such issues. * Or at least used 1 year ago, when I was working there.
    - nikanj 8y ago
      Hmm, I wonder if we could leverage that to make irc more popular again. "FB uses X" is often all the justification small startups need for picking a tech.
      - mic47 8y ago
        most startups can use their standard comm channels since most startups are not doing collaboration/communication tools.
linsomniac 8y ago
Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate UPS and generator. Last status update I had says that they are running off generators, but they've been shockingly tight-lipped about it. Don't get me wrong, it was hi-LAR-ious to call into their NOC and have them pretend that I was the only one having problems. "Can you tell me if there is a major data center outage going on?" "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." "... Why are you making a bunch of client calls if you aren't having an outage?"
- taurath 8y ago
  “Outage, what outage?” Is a sort of laughable response but all too common with tons of providers.
  - bradhe 8y ago
    I'm looking at you, AWS...
    - ceejayoz 8y ago
      A nuclear war could take out an AWS region and we'd get an "increased error rates" informational message.
      - bluetwo 8y ago
        Are you saying that a cold-war-era system like the internet/arpanet meant to survive a nuclear war might be vulnerable to an attack if we take all the code and data and store it in the same place? :-)
        8y ago
llamataboot 8y ago
Instagram seems to load the feed here fine (EU), but doesn't allow you to log in from any device or post anything new. FB is totally fine if you are logged in for reading, but also can't log in if logged out. VPN to US, insta can login, but still not post. Distributed services are weird man!
wybiral 8y ago
Look at how many service providers have increased incidents reported here: https://downdetector.com/ https://downdetector.com/ My bet is that people are having problems with FB/Insta and immediately assuming that the whole internet is messed up.
- deleted 8y ago
  [deleted]
- _bxg1 8y ago
  Especially if they can't sign in via OAuth. To an average user who signs into Spotify with their Facebook account, "I can't sign into Spotify" means Spotify is down, not Facebook.
- ashtonbaker 8y ago
  Looks like that site is having a problem logging reports! They all go to zero at around 17:00.
nabla9 8y ago
Serious question: Was any value lost? (this may appear sarcastic) Facebook obviously loses some ad revenue and Facebook customers may lose sales. But do Facebook/Instagram users suffer? But how does losing social media for several hours affect the quality of life of users?
- vokep 8y ago
  Potentially much value was gained (or stopped hemorrhaging)
- ZeroCool2u 8y ago
  Obviously this could be argued differently from a shareholder perspective, but I would say otherwise no. Interestingly, this might be one of the only times where a large outage could be claimed to be adding value. Again, not for FB, but for users, sure.
- nimir 8y ago
  I am not a big fan of social media too but you will be suprised ... For example here in Sudan (East Africa) the country has been under continuous protests for over 2 months now (53 dead, 4k+ detained, 500+ injured) with strong censorship from the regime & silents from the internatinal community. So facebook, whatsapp & twitter are the only media left for the people to fight for freedom —> every Thursday is the main protests in the week and this Wedensday night the outage might affect this as thousands around Sudan won't know about the meeting points of tomorrow!!! Actually the government did block all social media for over a month but that was fixable with vpn. (Follow hashtag #SudanUprising on twitter to learn more)
  - new_guy 8y ago
    No disrespect intended, but what's the rationale behind protesting just one day a week? It just seems completely unproductive, people can't affect any kind of change when they're scheduling it between watching the football and eating big macs.
    - 8y ago
red_admiral 8y ago
FB has been down for me for around an hour, but has just come back up again.
llamataboot 8y ago
Whatsapp now down across much of Europe it looks like. Cannot send/receive messages.
pragmaticalien8 8y ago
must be a super 0 day.
Implicated 8y ago
Facebook's own status dashboard (https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/) showed no issues or outtage just 30 min ago. I run a messenger bot platform - the webhooks stopped being delivered _hours_ ago... nothing on their status page until it had been down for hours. Their current issue... "We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution." What? lmao
- pbhjpbhj 8y ago
  I'm pretty sure businesses use status pages to divert attention from support resources, they never seem to give useful information about outages and half the time don't even mention the outage.
- yalok 8y ago
  that page is down now as well
- henrikschroder 8y ago
  I can't even reach that status page right now...
- buboard 8y ago
  you get what you pay for
  - knicholes 8y ago
    If you're inferring that Facebook is "free," I'd strongly disagree. Data is the currency of today, and they're swimming in it.
    - buboard 8y ago
      fb platform is free
      - 8y ago
kartan 8y ago
"This usually means we're making an improvement to the database your account is stored on. While this process won't affect your account, you temporarily won't be able to access the site." https://www.facebook.com/help/134401680031995 https://www.facebook.com/help/134401680031995 I guess that this is all that I will get. Facebook is never down, it is just making improvements (like restarting the services to make them work again).
- jfaat 8y ago
  We've always been at way with Eastasia
  - subcosmos 8y ago
    and autocorrect - it's doubleplus bad
    - T_D_K 8y ago
      *ungood
      - apolloWBrazil 8y ago
        They could be consolidating all of the DB infrastructure for their platforms. A zero down time dial-up would not be possible as they would need to nearly double their DB infrastructure. Short planned temporary outages of various features probably become long unplanned cross-platform outages. They probably decided to not rollback the migration after the first outage.
shrthnd 8y ago
I didn't notice.
aboutruby 8y ago
Hacker News might just have a "Major Internet Services" status board.
metahatem 8y ago
Elizabeth Warren, is that you?
- mxuribe 8y ago
  When EW speaks, the silos on the web tremble. (Ok, maybe not, but was fun to think of a bumper sticker-type phrase.)
cronix 8y ago
It looks like something much larger is going on. If you look at the front page of https://downdetector.com/ https://downdetector.com/ you'll see most major sites/backbones are having issues (Verizon/ATT/Sprint/CenturyLink/TMobile/Comcast/Level3/etc).
- blang 8y ago
  HN was down for a minute.
- ceejayoz 8y ago
  That site relies on user reports. I strongly suspect users are reporting "my Internet is having troubles" because their FB, Messenger, etc. isn't working right. For example, in the comments of the T-Mobile outage page, there's stuff like "Haven't been able to upload anything to social media all day" and "Cannot send pictures through whatsapp and fb messenger".
  - jnothing 8y ago
    Yeah in Turkey people thought government is slowing down/blocking the whole internet
    - ulucs 8y ago
      To be fair, the priors lean into that rather than Facebook and Whatsapp having troubles.
  - wybiral 8y ago
    I believe that too but some of the services seem unrelated (like Flickr and Capital One). Now, another interpretation is that the reports are simply false...
  - onychomys 8y ago
    That's a fair point, but those folks probably aren't reporting AWS problems, and it's showing a spike too.
casper345 8y ago
The real storm is realizing through Facebook OAuth you cannot access your affiliate accounts. Caution to move your accounts away from Facebook Edit: Or have other methods than just relying on Facebook authentication
- rmujica 8y ago
  realized that when trying to play Pokémon Go today, guess I'll break my weekly challenge...
johnchristopher 8y ago
> The team at Jefferies remains reasonably positive, and in the firm's top growth stock calls for the week we found four tech stocks that are offering more aggressive accounts good entry points. Carl Court / Getty Images What's that weird tagline about ?
Dangeranger 8y ago
Ironically as I've been checking this post HN has been experiencing errors loading new comments.
evesprini 8y ago
Argentina: Whatsapp works for text, but any type of media takes very long to send.
dstola 8y ago
The only things that I can think of that would cause this scale of being down is either a T1 center outage or (conspiracy hat on) a major hack and everyone is rush patching Would be interesting to read the post mortem if there is any regardless
- snazz 8y ago
  If rush patching were going on, we’d likely see some hints in commit messages of open source projects, like the Linux kernel commits that were tipoffs to Meltdown and Spectre. Edit: Has anyone seen anything of this sort in any of the projects they follow?
- ajflores1604 8y ago
  Did we ever get a post mortem for the global Youtube outage from late last year?
  - augbog 8y ago
    No but everyone is 99% sure it was related to killing Google+ which was announced not too long before and everyone who has used YouTube way back when knows they had to make a Google+ identity at one point to link em. HMMMMM....
benatkin 8y ago
I got my github two factor auth SMS two hours late. Fortunately it was just my old laptop. I wonder if it was related. Good reminder to set up an authenticator app on my new phone so I don’t have to rely on SMS!
- 0xffff2 8y ago
  Which you should do anyway because relying on SMS is only slightly more secure than not having 2FA enabled at all...
  - benatkin 8y ago
    SMS is still an option if you have a working 2FA Authenticator on GitHub. And even if I went through the trouble to disable it, I disagree. There are conceivable ways people could get to my email to initiate a password reset without getting to my phone, such as snatching my laptop from me while I'm working at a coffee shop.
subcosmos 8y ago
:/ Something fun happening in Germany? https://www.akamai.com/us/en/resources/visualizing-akamai/real-time-web-monitor.jsp https://www.akamai.com/us/en/resources/visualizing-akamai/re... And Level3 traffic going to Argentina? https://twitter.com/bgpstream/status/1105819050968580096 https://twitter.com/bgpstream/status/1105819050968580096 And GreatBritain going to cambodia? https://bgpstream.com/event/197968 https://bgpstream.com/event/197968
- deleted 8y ago
  [deleted]
- hokumguru 8y ago
  As a layperson, can you explain what exactly we're looking at here?
  - subcosmos 8y ago
    As a techperson .... barely - BGP is that complex I'm afraid, but its the wild wild west when it comes to potential nationstate attacks on internet traffic. I wonder if these patterns are typical or worrysome
- subcosmos 8y ago
  Hrmmm .... fun in Venezuela too ... https://twitter.com/bgpmon/status/1104919654441467904 https://twitter.com/bgpmon/status/1104919654441467904 Must be because of that Dam blowing up https://www.newsweek.com/sen-marco-rubio-blames-power-outage-german-dam-explosion-doesnt-realize-1357712 https://www.newsweek.com/sen-marco-rubio-blames-power-outage...
- ceejayoz 8y ago
  > Something fun happening in Germany? Major sporting events in the EU. BGP fuckups appear to happen regularly, based on the tweet history of that account.
akulbe 8y ago
In other news, productivity everywhere skyrocketed!
deca6cda37d0 8y ago
Unfortunately the AI software used to censor fake news from facebook has decided, again, to censor facebook :D
40acres 8y ago
I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.
- mtw 8y ago
  Good question is why oh why switch WhatsApp to Facebook tech when it was running perfectly ok on its own. Never crashed.
  - macintux 8y ago
    Speaking as an Erlang developer I approve of this question.
  - duado 8y ago
    So that engineers can be moved between product groups while carrying relevant knowledge and experience with them.
    - yellowapple 8y ago
      I'm pretty sure a bunch of Erlang programmers being shoehorned into a PHP codebase is the literal definition of Hell for all parties involved.
      - duado 8y ago
        Facebook is not a “PHP codebase.” I’d guess that fewer than 0.01% of CPU cycles at Facebook are used by PHP.
        nothal 8y ago
        Isn’t a significant amount of the codebase in Hack, a PHP derivative?
        Lt_Riza_Hawkeye 8y ago
snazz 8y ago
I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my workplace. This leaves me wondering what software all these places have in common. The application layers are all different, the databases are all different, the containerization and provisioning systems are different, but I imagine that all these systems rely on two things: the global Internet backbone, and maybe the Linux kernel. Have there been major security vulnerabilities patched lately in the Linux kernel that could have had unintended consequences?
- shafte 8y ago
  The best explanation is coincidence, I think. I have direct knowledge of two of the incidents in the past few weeks, and they have completely unrelated causes. Sometimes you just get unlucky!
  - snazz 8y ago
    That’s certainly possible. We’re probably still too early to tell, but the innate conspiracy theorist slash pattern-matching part of my brain wants to find a probable connection.
  - smacktoward 8y ago
    "Once is happenstance. Twice is coincidence. The third time it’s enemy action." -- Ian Fleming (in Goldfinger)
  - z3t4 8y ago
    If it can happen, it will happen.
- str33t_punk 8y ago
  Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques. It is one of the harder problems to solve and requires massive investment in tooling, refactoring, etc.
bluedino 8y ago
Instagram works for me on my iPhone, but the comments are all missing. I kind of like it that way.
- jjulius 8y ago
  You're already logged in so it's just going to show you old content. I'd be surprised if your able to post anything, or if any "like" you give during the outage is saved. Same thing is happening with both Instagram and FB apps on Android for me.
  - bluedino 8y ago
    I never do any of those things anyway
markstos 8y ago
I see they've made some progress putting Instagram on the same infrastructure as Facebook.
winrid 8y ago
Ironic: https://imgur.com/a/M2SzqIc https://imgur.com/a/M2SzqIc
Jansi 8y ago
Test Comment
hnruss 8y ago
Did they move too fast and break too many things?
armortech 8y ago
Whatever happens right now at Facebook is less important than the fact they will never say what affected them. Of course nobody would tell 'hey, outage right now due to 0day / mistake' but...
CodeWriter23 8y ago
This is bigger than Facebook. https://imgur.com/a/gePwi0i https://imgur.com/a/gePwi0i https://www.akamai.com/us/en/resources/visualizing-akamai/real-time-web-monitor.jsp https://www.akamai.com/us/en/resources/visualizing-akamai/re...
- ceejayoz 8y ago
  That chart doesn’t support your assertion. Akamai’s traffic and attack charts usually look like that, and the attack chart even says it’s currently low.
- ce4 8y ago
  The heavy traffic is due to sports events - champions league last-16 matchday live streaming: Bayern Munich vs. Liverpool, FC Barcelona vs Olympique Lyon. The heatmap matches the clubs' home countries UK, Germany, France & Spain quite well. Dont conflate that with fb/insta problems.
  - why-el 8y ago
    Indeed. The games are streamed like crazy! Lots of streams are in HD.
    - ce4 8y ago
      Yep, the actual bandwidth consumption is mindblowing :)
      - pferde 8y ago
        Multicast was designed exactly for this - same data streamed to many endpoints at the same time. Too bad it's not being more widely used, the bandwidth savings would likely be huge.
- dr1ggins 8y ago
  That traffic map seems to line up with time zones
js2 8y ago
I haven’t been able to post anything on Facebook, neither a new post to my wall nor add a comment on a friend’s post, since mid-morning US/Eastern and this is still the case. In addition I can’t login to the site - I am able to access the site only where I’m already logged-in.
FabHK 8y ago
Let's see whether we have a spike in the birth rate in 9 months. (Oh, turns out the Great Blackout Baby Boom was a myth: https://www.snopes.com/fact-check/from-here-to-maternity/ https://www.snopes.com/fact-check/from-here-to-maternity/ )
- Arbalest 8y ago
  What if there is a fall in birthrate because people can't organise risky hookups without their preferred communication platform?
  - anitil 8y ago
    Or a fall in STDs for the same reason.
davidsam 8y ago
Really?
- davidsam 8y ago
  FB
miloignis 8y ago
My fiance's uncle sent something today that because of a school shooting in Brazil, they were blocking all images and video shared to social networks like "WhatsApp, Instagram, Facebook and other social networks". I haven't been able to verify this myself or from any other sources, but I wonder if either people are misinterpreting the FB outage or if Brazil is blocking content it's having weird ripple effects.
nilskidoo 8y ago
Affecting more lives than any terrorist act.
maroonmed 8y ago
Facebook's Graph API and developer support pages seem to have completely gone to shit. My social media webapp's Facebook login flow is completely unusable ("getting a cannot link <app name> to Facebook" error), and I can't even report a bug or post a status update on my app's Facebook page without encountering an unknown error. It's crazy.
tinyhouse 8y ago
BTW, many apps are affected by this. I cannot log in to any app that uses Facebook for authentication.
- cheeze 8y ago
  This is one of the many many reasons I don't use fb for auth for any other websites.
entwife 8y ago
It's an experiment to see whether productivity improves when people aren't able to access FB and Instagram to slack off.
Endy 8y ago
Since it went down for PC and not mobile, I was concerned if it was just an idea of audience testing, in the process of moving to an app-only platform.
bradynapier 8y ago
Perhaps relevant that npm has been having issues although they only recently caught and fixed them. Scoped Private npm packages were getting cloudflare 503 errors
peepX 8y ago
I'm surprised there wasn't a national crisis alert sent out
evolvedcleaning 8y ago
Doesn’t their world class team make such a long outage to be quite unlikely? How hard would it be to devote ample resources to a cover story for the “incident report”? Is the timing relative to the plethora of indictments relevant at all? Reasonable that this may be related to shredding of data and/or code, or even a cooperation to turn over data to government in secret deal?
abbiya 8y ago
somebody please wipe off everything from all these big corps
goblin89 8y ago
Coincidentally, just watched The Social Network, the plot of which includes that quote by Mark: > Let me tell you difference between Facebook and everybody else. We don't crash ever! If the serves are down for even a day, our entire reputation is irreversibly destroyed. <…> > Even a few people leaving would reverberate through the entire use base. The users are interconnected. That is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go.
- ninth_ant 8y ago
  You’re quoting a movie, which played fast and loose with facts to tell a story (as movies do). In real life, Facebook had significant issues with uptime in the early years.
  - goblin89 8y ago
    Can’t argue with that, however the fact that Facebook used to go down before would not preclude that each time was (and maybe still is) seen as a major incident within the original corporate culture.
alien2003 8y ago
It's hard to believe that people simply can not live without fresh instagraphies
indigochill 8y ago
First thought when I heard the news was BGP hijacking (ignoring whether accidental or deliberate). Doesn't the symptom fit other known cases like the Telegram incident in Iran last year, just at a larger scale? Admittedly networking is not my strength, so perfectly happy for someone to shoot down this hypothesis.
revskill 8y ago
What's the cause of outage then ? Disk, memory, CPU, network bandwidth,... ?

610 points by ajiang 8y ago

This is the first time I experience this. Also note that current session on messenger.com still work, we can still send/receive message, but can't upload any image or send sticker. Looking forward for a post mortem analysis on this.

allannienhuis 8y ago

seems to be more than just messenger login; All of facebook is super flakey this morning.

JakeWesorick 8y ago

https://www.facebook.com/platform/api-status/ https://www.facebook.com/platform/api-status/ still returns "Facebook Platform is Healthy", but you can't even load https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/. Why have status pages if they are so susceptible to going down themselves?

ceejayoz 8y ago
I remember an S3 outage a number of years ago where AWS discovered that their status page was hosted on S3. Whoops! I believe this is why Github's status page is now on its own domain; so a github.com DNS outage won't take it down.
- simlevesque 8y ago
  IIRC the AWS status page was fine but the indicator of the status of the different services was a red or green .PNG which were hosted on S3.

aviral 8y ago

https://downdetector.com/status/facebook https://downdetector.com/status/facebook

yeutterg 8y ago

Can confirm. Also, API integrations, such as Buffer, are not working.

JDiculous 8y ago

Is Facebook actually working for anyone?

jedberg 8y ago

So yesterday Google had a major (and out of character) outage, and today Facebook has a major (and also out of character) outage. I can't wait to see the RCA for both of these and if they're related.

jedberg 8y ago

So yesterday Google had a major (and out of character) outage across its apps, and today Facebook has a major (and also out of character) outage across its apps. I can't wait to see the RCA for both of these and if they're related.

godzillabrennus 8y ago
Private post Morten: The NSA middleware we are required to run (that took time to deploy to each of our social partners) is breaking something so let’s revert. Public post Mortem: Entirely believable technical cause.
- samstave 8y ago
  Alternate Post Mortem: Cyber WWIII's first public skirmishes become visible... (Ignore Stuxnet, Ignore DUQU)
  - jmrobertson 8y ago
    yeah I wonder
  - wybiral 8y ago
    /me puts on tin foil hat It's the rice grain implants, man
- popz41 8y ago
  I imagine the NSA uses an optical tap device. These devices create identical copies and require no power or management.
  - cronix 8y ago
    That's why it's called PRISM. It's exactly what you describe. Splitting an optical signal into 2 using, basically, a prism. One signal goes out to the net as normal, the other goes to their own datacenters, that they keep continually building and expanding. The newer ones are being build on military bases, for added security. Check em out. Look at the size and cost of them. Some are over a million sq. ft. That's a lot of data. They measure it in terms of zottabytes and zettabytes (in 2013, a lifetime ago in terms of storage space): https://modernsurvivalblog.com/government-gone-wild/nsa-locations-in-the-united-states/ https://modernsurvivalblog.com/government-gone-wild/nsa-loca... http://worldstopdatacenters.com/government-data-centers/ http://worldstopdatacenters.com/government-data-centers/

boshomi 8y ago

see also: Facebook, Instagram down: Social media sites not working for many, FB doing 'required maintenance' [1] https://www.abc15.com/news/national/facebook-down-social-media-site-not-working-for-many https://www.abc15.com/news/national/facebook-down-social-med...

boshomi 8y ago

the issue is also in EU: Is Facebook down? Messenger, site, app and Instagram hit by issues[1] [1]https://www.manchestereveningnews.co.uk/news/uk-news/facebook-down-app-website-crash-15969732 https://www.manchestereveningnews.co.uk/news/uk-news/faceboo...

g8oz 8y ago

...and nothing of value was lost.

arisAlexis 8y ago

yesterday Google and today Facebook. My conspirator says it's the Chinese government showcasing.

mdkdog 8y ago
Meanwhile in Russia they are talking about disconnecting their network from the rest of the world. Some test gone south?... Maybe... Someone has a traceroute handy?

BucketSort 8y ago

Google and FB having successive outages? Is this just a coincidence or is there some shared infrastructure that would explain this?

js2 8y ago
Verizon also had SMS issues yesterday: https://www.businessinsider.com/verizon-outage-on-east-coast-texts-arent-sending-2019-3 https://www.businessinsider.com/verizon-outage-on-east-coast...
aviv 8y ago
Likely a firmware issue with the NSA intercept devices.
- Theodores 8y ago
  ...and their key infrastructure in Venezuela.
  - clydethefrog 8y ago
    Yes, I can't help to entertain the thought of another Stuxnet going wild.
- 9999px 8y ago
  Are you basing that on anything or just BS'ing?
- OscarTheGrinch 8y ago
  Hail Ghidra!
frostyj 8y ago
^^^ I noticed this too. GCP is under all 50 shades of outages since past few days. Feel I might need to rush back to my house and start digging a bunker
twothumbsup 8y ago

wybiral 8y ago

Can you imagine if Twitter and Google went down at the same time? People would be reactivating their Facebook accounts and having to sift through conspiracy theory posts about Hillary Clinton still just to figure out what was going on. Edit: The points on this post keep going up and down every time I check these comments. Yes, it was sarcasm, I was joking, but I was trying to point out that most people rely on a small set of services. "Cloud" has centralized things a lot.

bargl 8y ago
I think the only reason your comment gets down voted is because it's biased only to one side. I see about what Jon Oliver states as if I remember correct 1/2 as many conspiracy posts from about Trump (Edit: I meant to write conservatives write 2x as many conspiracy posts as democrats. This was not represented in my previous sentence.). It's annoying. I guess to accurately represent things on Facebook you'd have to say something like conspiracy posts about AOC, Bernie, and Dan Chrenshaw. 2x1 Ratio. You could have also said, "political conspiracy theories". I get annoyed by conspiracies on either side personally, but I can see how some people would see more of one than the other based on their friends. Edit: Re-read some of what I wrote and I want to clarify, this isn't me saying i down-voted you. Just an observation.
- wybiral 8y ago
  It was a joke based on what people often say about FB. I don't use their platforms.
  - bargl 8y ago
    Jokes, typically hint at a truth or belief of the comedian. I'm not saying it's a bad joke, I actually liked it. I was merely making an observation which apparently a lot of people disliked, and it looks (based on the upvote/downvote feedback) I am probably wrong. It might just be because a lot of people around here read thishttp://www.paulgraham.com/hackernews.html http://www.paulgraham.com/hackernews.html and think that any joke means it's a bad comment. IDK. As I said I thought your comment was funny.

agosnell 8y ago

If you use their API and haven't seen it yet, their issue is listed here on their status page: https://developers.facebook.com/status/issues/559896447845433/ https://developers.facebook.com/status/issues/55989644784543...

odomojuli 8y ago
Looks like the status page is down for me as well.
- yorwba 8y ago
  Increased Error Rates Created by Gary Fitzpatrick · · Facebook Team — Today at 10:32 AM Current State: Investigating Description: We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution. Start Time: 2 hours ago Last Update: about an hour ago Updates: There are currently no updates for this issue.
  - twic 8y ago
    > Increased Error Rates Created by Gary Fitzpatrick Well someone tell him to stop, for pete's sake!
    - samstave 8y ago
      Are you joshing me?
    - nogbit 8y ago
      I read it that way as well :)
    - jmts 8y ago
      Pete's in on it too!

nathan929 8y ago

...terrifying API developers everywhere...

njn 8y ago

Good! Keep them down.

jak92 8y ago

... and nothing of value was lost.

btown 8y ago

I've also seen issues uploading images to Whatsapp in the past half hour. I wonder if there's anything to do with the Google Cloud Storage outage that took down Gmail yesterday?

twothumbsup 8y ago
Facebook doesn't use GCS (at least they didn't in 2015), they have their own infrastructure/data centers.

earless1 8y ago

What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.

phoe-krk 8y ago
An expired certificate, for instance. https://www.thesslstore.com/blog/expired-certificate-ericsson-o2/ https://www.thesslstore.com/blog/expired-certificate-ericsso...
rwultsch 8y ago
Short duration: network, bad software deploy Long duration: db. If you break data, it takes a while to unbreak. Source: Me. My career has been spent managing db's for internet scale sites.
- cheeze 8y ago
  Nothing worse than that sinking feeling of "oh fuck, we have to backfill a lot of data.
  - WrtCdEvrydy 8y ago
    Why did my username on this site just change to 'test123'... oh, where clauses.
    - jomkr 8y ago
      Nothing worse then the page on Friday night, oh there goes my weekend.
- dsfyu404ed 8y ago
  I work for a smaller but comparably large platform. "If everything is down check the DB" is at the top of one of our internal monitoring websites in red. Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved but they have to spend a bunch of time synchronizing things and verifying the fix before they press the big red "go live" button.

tinyhouse 8y ago

From Google I got an error clicking on instagram and Quora links today.

JohnJamesRambo 8y ago
At least you are free of Quora now!

zomg 8y ago

All my funny cat videos and memes are loading fine. Did you try rebooting your router 62 times? All joking aside, is this news? :/

blhack 8y ago
One of the largest internet companies in the world having a massive, global outage? Yes, that seems like appropriate news for "hacker news", a website where people discuss technology news.
- zomg 8y ago
  wow, tough crowd today. i didn't think people took facebook and instagram so seriously. i'll put the sarcasm machine back in the drawer.
  - icebraining 8y ago
    This is HN, we take everything seriously.
pbhjpbhj 8y ago
Do you want to explain why you believe it's not [HN worthy] news?
deleted 8y ago
[deleted]

rdtsc 8y ago

Google then Facebook and Instagram? My hunch is that it's the end of Q1 and people are trying to release code changes so they can pad their Q1 performance reviews "designed and delivered feature X on time in Q1".

unfunco 8y ago
"and brought down a service used by billions and losing potentially millions in revenue."
- Qwertystop 8y ago
  Well, it's certainly impressive.
- outside1234 8y ago
  Can't claim they didn't have impact then. :)
- abtom 8y ago
  "Move fast break things"
cheeze 8y ago
Yeah I'm gonna go with about a 0% chance of this.
toephu2 8y ago
Year end performance reviews are the most important..those were already done at most companies.
- rdtsc 8y ago
  I've seen all quarter ends being targets for releases. And things that have been delayed since start of the end of the year are usually pushed to the end of Q1.

nodesocket 8y ago

And yet Facebook's stock is still up on the day (+.74%)? You'd think being down for hours would be negative news and revenue impacting.

derwiki 8y ago
2 hours of downtime in a quarter is 0.09% of downtime -- probably very little effect on their monetization products.
- farisjarrah 8y ago
  We're up to around 7 hours of partial outage now. I have spoken to FB employees in the past, every hour they are even partially down they are loosing millions and millions in revenue.

mizeandmen 8y ago

Bunch fb employees near pacific catch was talking about how fb was hacked

minimaxir 8y ago

Minor update: https://twitter.com/facebook/status/1105907126424109056?s=21 https://twitter.com/facebook/status/1105907126424109056?s=21 > We're focused on working to resolve the issue as soon as possible, but can confirm that the issue is not related to a DDoS attack.

KennyFromIT 8y ago
There's something ironic about a social media company being forced to rely on a competitor to facilitate communication between them and their users.
- mic47 8y ago
  Why? When Facebook is down, you cannot use Facebook to communicate about the issue. That is also reason why FB still uses* irc instead of messenger for coordinating how to resolve such issues. * Or at least used 1 year ago, when I was working there.
  - nikanj 8y ago
    Hmm, I wonder if we could leverage that to make irc more popular again. "FB uses X" is often all the justification small startups need for picking a tech.
    - mic47 8y ago
      most startups can use their standard comm channels since most startups are not doing collaboration/communication tools.

linsomniac 8y ago

Could this be related to the storm? I was out shoveling, and came back in to my phone blowing up. Our systems at IronMountain (formerly Fortrust) in Denver all rebooted at once. These are all on redundant power, each systems redundant power supplies connecting to different circuits entering the cabinet, and those two circuits fed from 3 PDUs (two separate, one share). Each of those is supposed to be fed by a separate UPS and generator. Last status update I had says that they are running off generators, but they've been shockingly tight-lipped about it. Don't get me wrong, it was hi-LAR-ious to call into their NOC and have them pretend that I was the only one having problems. "Can you tell me if there is a major data center outage going on?" "We are trying to gather information, we are making a bunch of client phone calls, we will know after we make those calls." "... Why are you making a bunch of client calls if you aren't having an outage?"

taurath 8y ago
“Outage, what outage?” Is a sort of laughable response but all too common with tons of providers.
- bradhe 8y ago
  I'm looking at you, AWS...
  - ceejayoz 8y ago
    A nuclear war could take out an AWS region and we'd get an "increased error rates" informational message.
    - bluetwo 8y ago
      Are you saying that a cold-war-era system like the internet/arpanet meant to survive a nuclear war might be vulnerable to an attack if we take all the code and data and store it in the same place? :-)
      - 8y ago

llamataboot 8y ago

Instagram seems to load the feed here fine (EU), but doesn't allow you to log in from any device or post anything new. FB is totally fine if you are logged in for reading, but also can't log in if logged out. VPN to US, insta can login, but still not post. Distributed services are weird man!

wybiral 8y ago

Look at how many service providers have increased incidents reported here: https://downdetector.com/ https://downdetector.com/ My bet is that people are having problems with FB/Insta and immediately assuming that the whole internet is messed up.

deleted 8y ago
[deleted]
_bxg1 8y ago
Especially if they can't sign in via OAuth. To an average user who signs into Spotify with their Facebook account, "I can't sign into Spotify" means Spotify is down, not Facebook.
ashtonbaker 8y ago
Looks like that site is having a problem logging reports! They all go to zero at around 17:00.

nabla9 8y ago

Serious question: Was any value lost? (this may appear sarcastic) Facebook obviously loses some ad revenue and Facebook customers may lose sales. But do Facebook/Instagram users suffer? But how does losing social media for several hours affect the quality of life of users?

vokep 8y ago
Potentially much value was gained (or stopped hemorrhaging)
ZeroCool2u 8y ago
Obviously this could be argued differently from a shareholder perspective, but I would say otherwise no. Interestingly, this might be one of the only times where a large outage could be claimed to be adding value. Again, not for FB, but for users, sure.
nimir 8y ago
I am not a big fan of social media too but you will be suprised ... For example here in Sudan (East Africa) the country has been under continuous protests for over 2 months now (53 dead, 4k+ detained, 500+ injured) with strong censorship from the regime & silents from the internatinal community. So facebook, whatsapp & twitter are the only media left for the people to fight for freedom —> every Thursday is the main protests in the week and this Wedensday night the outage might affect this as thousands around Sudan won't know about the meeting points of tomorrow!!! Actually the government did block all social media for over a month but that was fixable with vpn. (Follow hashtag #SudanUprising on twitter to learn more)
- new_guy 8y ago
  No disrespect intended, but what's the rationale behind protesting just one day a week? It just seems completely unproductive, people can't affect any kind of change when they're scheduling it between watching the football and eating big macs.
  - 8y ago

red_admiral 8y ago

FB has been down for me for around an hour, but has just come back up again.

llamataboot 8y ago

Whatsapp now down across much of Europe it looks like. Cannot send/receive messages.

pragmaticalien8 8y ago

must be a super 0 day.

Implicated 8y ago

Facebook's own status dashboard (https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/) showed no issues or outtage just 30 min ago. I run a messenger bot platform - the webhooks stopped being delivered _hours_ ago... nothing on their status page until it had been down for hours. Their current issue... "We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution." What? lmao

pbhjpbhj 8y ago
I'm pretty sure businesses use status pages to divert attention from support resources, they never seem to give useful information about outages and half the time don't even mention the outage.
yalok 8y ago
that page is down now as well
henrikschroder 8y ago
I can't even reach that status page right now...
buboard 8y ago
you get what you pay for
- knicholes 8y ago
  If you're inferring that Facebook is "free," I'd strongly disagree. Data is the currency of today, and they're swimming in it.
  - buboard 8y ago
    fb platform is free
    - 8y ago

kartan 8y ago

"This usually means we're making an improvement to the database your account is stored on. While this process won't affect your account, you temporarily won't be able to access the site." https://www.facebook.com/help/134401680031995 https://www.facebook.com/help/134401680031995 I guess that this is all that I will get. Facebook is never down, it is just making improvements (like restarting the services to make them work again).

jfaat 8y ago
We've always been at way with Eastasia
- subcosmos 8y ago
  and autocorrect - it's doubleplus bad
  - T_D_K 8y ago
    *ungood
    - apolloWBrazil 8y ago
      They could be consolidating all of the DB infrastructure for their platforms. A zero down time dial-up would not be possible as they would need to nearly double their DB infrastructure. Short planned temporary outages of various features probably become long unplanned cross-platform outages. They probably decided to not rollback the migration after the first outage.

shrthnd 8y ago

I didn't notice.

aboutruby 8y ago

Hacker News might just have a "Major Internet Services" status board.

metahatem 8y ago

Elizabeth Warren, is that you?

mxuribe 8y ago
When EW speaks, the silos on the web tremble. (Ok, maybe not, but was fun to think of a bumper sticker-type phrase.)

cronix 8y ago

It looks like something much larger is going on. If you look at the front page of https://downdetector.com/ https://downdetector.com/ you'll see most major sites/backbones are having issues (Verizon/ATT/Sprint/CenturyLink/TMobile/Comcast/Level3/etc).

blang 8y ago
HN was down for a minute.
ceejayoz 8y ago
That site relies on user reports. I strongly suspect users are reporting "my Internet is having troubles" because their FB, Messenger, etc. isn't working right. For example, in the comments of the T-Mobile outage page, there's stuff like "Haven't been able to upload anything to social media all day" and "Cannot send pictures through whatsapp and fb messenger".
- jnothing 8y ago
  Yeah in Turkey people thought government is slowing down/blocking the whole internet
  - ulucs 8y ago
    To be fair, the priors lean into that rather than Facebook and Whatsapp having troubles.
- wybiral 8y ago
  I believe that too but some of the services seem unrelated (like Flickr and Capital One). Now, another interpretation is that the reports are simply false...
- onychomys 8y ago
  That's a fair point, but those folks probably aren't reporting AWS problems, and it's showing a spike too.

casper345 8y ago

The real storm is realizing through Facebook OAuth you cannot access your affiliate accounts. Caution to move your accounts away from Facebook Edit: Or have other methods than just relying on Facebook authentication

rmujica 8y ago
realized that when trying to play Pokémon Go today, guess I'll break my weekly challenge...

johnchristopher 8y ago

> The team at Jefferies remains reasonably positive, and in the firm's top growth stock calls for the week we found four tech stocks that are offering more aggressive accounts good entry points. Carl Court / Getty Images What's that weird tagline about ?

Dangeranger 8y ago

Ironically as I've been checking this post HN has been experiencing errors loading new comments.

evesprini 8y ago

Argentina: Whatsapp works for text, but any type of media takes very long to send.

dstola 8y ago

The only things that I can think of that would cause this scale of being down is either a T1 center outage or (conspiracy hat on) a major hack and everyone is rush patching Would be interesting to read the post mortem if there is any regardless

snazz 8y ago
If rush patching were going on, we’d likely see some hints in commit messages of open source projects, like the Linux kernel commits that were tipoffs to Meltdown and Spectre. Edit: Has anyone seen anything of this sort in any of the projects they follow?
ajflores1604 8y ago
Did we ever get a post mortem for the global Youtube outage from late last year?
- augbog 8y ago
  No but everyone is 99% sure it was related to killing Google+ which was announced not too long before and everyone who has used YouTube way back when knows they had to make a Google+ identity at one point to link em. HMMMMM....

benatkin 8y ago

I got my github two factor auth SMS two hours late. Fortunately it was just my old laptop. I wonder if it was related. Good reminder to set up an authenticator app on my new phone so I don’t have to rely on SMS!

0xffff2 8y ago
Which you should do anyway because relying on SMS is only slightly more secure than not having 2FA enabled at all...
- benatkin 8y ago
  SMS is still an option if you have a working 2FA Authenticator on GitHub. And even if I went through the trouble to disable it, I disagree. There are conceivable ways people could get to my email to initiate a password reset without getting to my phone, such as snatching my laptop from me while I'm working at a coffee shop.

subcosmos 8y ago

:/ Something fun happening in Germany? https://www.akamai.com/us/en/resources/visualizing-akamai/real-time-web-monitor.jsp https://www.akamai.com/us/en/resources/visualizing-akamai/re... And Level3 traffic going to Argentina? https://twitter.com/bgpstream/status/1105819050968580096 https://twitter.com/bgpstream/status/1105819050968580096 And GreatBritain going to cambodia? https://bgpstream.com/event/197968 https://bgpstream.com/event/197968

deleted 8y ago
[deleted]
hokumguru 8y ago
As a layperson, can you explain what exactly we're looking at here?
- subcosmos 8y ago
  As a techperson .... barely - BGP is that complex I'm afraid, but its the wild wild west when it comes to potential nationstate attacks on internet traffic. I wonder if these patterns are typical or worrysome
subcosmos 8y ago
Hrmmm .... fun in Venezuela too ... https://twitter.com/bgpmon/status/1104919654441467904 https://twitter.com/bgpmon/status/1104919654441467904 Must be because of that Dam blowing up https://www.newsweek.com/sen-marco-rubio-blames-power-outage-german-dam-explosion-doesnt-realize-1357712 https://www.newsweek.com/sen-marco-rubio-blames-power-outage...
ceejayoz 8y ago
> Something fun happening in Germany? Major sporting events in the EU. BGP fuckups appear to happen regularly, based on the tweet history of that account.

akulbe 8y ago

In other news, productivity everywhere skyrocketed!

deca6cda37d0 8y ago

Unfortunately the AI software used to censor fake news from facebook has decided, again, to censor facebook :D

40acres 8y ago

I'm interviewing for a Production Engineer role at Facebook on Monday, thanks for providing relevant "do you have any questions for us" content.

mtw 8y ago
Good question is why oh why switch WhatsApp to Facebook tech when it was running perfectly ok on its own. Never crashed.
- macintux 8y ago
  Speaking as an Erlang developer I approve of this question.
- duado 8y ago
  So that engineers can be moved between product groups while carrying relevant knowledge and experience with them.
  - yellowapple 8y ago
    I'm pretty sure a bunch of Erlang programmers being shoehorned into a PHP codebase is the literal definition of Hell for all parties involved.
    - duado 8y ago
      Facebook is not a “PHP codebase.” I’d guess that fewer than 0.01% of CPU cycles at Facebook are used by PHP.
      - nothal 8y ago
        Isn’t a significant amount of the codebase in Hack, a PHP derivative?
        Lt_Riza_Hawkeye 8y ago

snazz 8y ago

I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my workplace. This leaves me wondering what software all these places have in common. The application layers are all different, the databases are all different, the containerization and provisioning systems are different, but I imagine that all these systems rely on two things: the global Internet backbone, and maybe the Linux kernel. Have there been major security vulnerabilities patched lately in the Linux kernel that could have had unintended consequences?

shafte 8y ago
The best explanation is coincidence, I think. I have direct knowledge of two of the incidents in the past few weeks, and they have completely unrelated causes. Sometimes you just get unlucky!
- snazz 8y ago
  That’s certainly possible. We’re probably still too early to tell, but the innate conspiracy theorist slash pattern-matching part of my brain wants to find a probable connection.
- smacktoward 8y ago
  "Once is happenstance. Twice is coincidence. The third time it’s enemy action." -- Ian Fleming (in Goldfinger)
- z3t4 8y ago
  If it can happen, it will happen.
str33t_punk 8y ago
Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques. It is one of the harder problems to solve and requires massive investment in tooling, refactoring, etc.

bluedino 8y ago

Instagram works for me on my iPhone, but the comments are all missing. I kind of like it that way.

jjulius 8y ago
You're already logged in so it's just going to show you old content. I'd be surprised if your able to post anything, or if any "like" you give during the outage is saved. Same thing is happening with both Instagram and FB apps on Android for me.
- bluedino 8y ago
  I never do any of those things anyway

markstos 8y ago

I see they've made some progress putting Instagram on the same infrastructure as Facebook.

winrid 8y ago

Ironic: https://imgur.com/a/M2SzqIc https://imgur.com/a/M2SzqIc

Jansi 8y ago

Test Comment

hnruss 8y ago

Did they move too fast and break too many things?

armortech 8y ago

Whatever happens right now at Facebook is less important than the fact they will never say what affected them. Of course nobody would tell 'hey, outage right now due to 0day / mistake' but...

CodeWriter23 8y ago

This is bigger than Facebook. https://imgur.com/a/gePwi0i https://imgur.com/a/gePwi0i https://www.akamai.com/us/en/resources/visualizing-akamai/real-time-web-monitor.jsp https://www.akamai.com/us/en/resources/visualizing-akamai/re...

ceejayoz 8y ago
That chart doesn’t support your assertion. Akamai’s traffic and attack charts usually look like that, and the attack chart even says it’s currently low.
ce4 8y ago
The heavy traffic is due to sports events - champions league last-16 matchday live streaming: Bayern Munich vs. Liverpool, FC Barcelona vs Olympique Lyon. The heatmap matches the clubs' home countries UK, Germany, France & Spain quite well. Dont conflate that with fb/insta problems.
- why-el 8y ago
  Indeed. The games are streamed like crazy! Lots of streams are in HD.
  - ce4 8y ago
    Yep, the actual bandwidth consumption is mindblowing :)
    - pferde 8y ago
      Multicast was designed exactly for this - same data streamed to many endpoints at the same time. Too bad it's not being more widely used, the bandwidth savings would likely be huge.
dr1ggins 8y ago
That traffic map seems to line up with time zones

js2 8y ago

I haven’t been able to post anything on Facebook, neither a new post to my wall nor add a comment on a friend’s post, since mid-morning US/Eastern and this is still the case. In addition I can’t login to the site - I am able to access the site only where I’m already logged-in.

FabHK 8y ago

Let's see whether we have a spike in the birth rate in 9 months. (Oh, turns out the Great Blackout Baby Boom was a myth: https://www.snopes.com/fact-check/from-here-to-maternity/ https://www.snopes.com/fact-check/from-here-to-maternity/ )

Arbalest 8y ago
What if there is a fall in birthrate because people can't organise risky hookups without their preferred communication platform?
- anitil 8y ago
  Or a fall in STDs for the same reason.

davidsam 8y ago

Really?

davidsam 8y ago
FB

miloignis 8y ago

My fiance's uncle sent something today that because of a school shooting in Brazil, they were blocking all images and video shared to social networks like "WhatsApp, Instagram, Facebook and other social networks". I haven't been able to verify this myself or from any other sources, but I wonder if either people are misinterpreting the FB outage or if Brazil is blocking content it's having weird ripple effects.

nilskidoo 8y ago

Affecting more lives than any terrorist act.

maroonmed 8y ago

Facebook's Graph API and developer support pages seem to have completely gone to shit. My social media webapp's Facebook login flow is completely unusable ("getting a cannot link <app name> to Facebook" error), and I can't even report a bug or post a status update on my app's Facebook page without encountering an unknown error. It's crazy.

tinyhouse 8y ago

BTW, many apps are affected by this. I cannot log in to any app that uses Facebook for authentication.

cheeze 8y ago
This is one of the many many reasons I don't use fb for auth for any other websites.

entwife 8y ago

It's an experiment to see whether productivity improves when people aren't able to access FB and Instagram to slack off.

Endy 8y ago

Since it went down for PC and not mobile, I was concerned if it was just an idea of audience testing, in the process of moving to an app-only platform.

bradynapier 8y ago

Perhaps relevant that npm has been having issues although they only recently caught and fixed them. Scoped Private npm packages were getting cloudflare 503 errors

peepX 8y ago

I'm surprised there wasn't a national crisis alert sent out

evolvedcleaning 8y ago

Doesn’t their world class team make such a long outage to be quite unlikely? How hard would it be to devote ample resources to a cover story for the “incident report”? Is the timing relative to the plethora of indictments relevant at all? Reasonable that this may be related to shredding of data and/or code, or even a cooperation to turn over data to government in secret deal?

abbiya 8y ago

somebody please wipe off everything from all these big corps

goblin89 8y ago

Coincidentally, just watched The Social Network, the plot of which includes that quote by Mark: > Let me tell you difference between Facebook and everybody else. We don't crash ever! If the serves are down for even a day, our entire reputation is irreversibly destroyed. <…> > Even a few people leaving would reverberate through the entire use base. The users are interconnected. That is the whole point. College kids are online because their friends are online, and if one domino goes, the other dominos go.

ninth_ant 8y ago
You’re quoting a movie, which played fast and loose with facts to tell a story (as movies do). In real life, Facebook had significant issues with uptime in the early years.
- goblin89 8y ago
  Can’t argue with that, however the fact that Facebook used to go down before would not preclude that each time was (and maybe still is) seen as a major incident within the original corporate culture.

alien2003 8y ago

It's hard to believe that people simply can not live without fresh instagraphies

indigochill 8y ago

First thought when I heard the news was BGP hijacking (ignoring whether accidental or deliberate). Doesn't the symptom fit other known cases like the Telegram incident in Iran last year, just at a larger scale? Admittedly networking is not my strength, so perfectly happy for someone to shoot down this hypothesis.

revskill 8y ago

What's the cause of outage then ? Disk, memory, CPU, network bandwidth,... ?