23 ms·

Facebook, Instagram, and WhatsApp outages

345 points by skilled 7y ago

nilayj 7y ago
Looks like Facebook image store is having issues. Facebook, Instagram and WhatsApp use the same image store, so all the services are affected.
- mtgx 7y ago
  Probably a good opportunity for more people to realize that all three services belong to the same company.
  - hbosch 7y ago
    It’s common knowledge.
    - 3stripe 7y ago
      For HN readers sure
- ceocoder 7y ago
  However image metadata service seems to be up, this is what I see for most images on FB now. https://imgur.com/a/uC2L354 https://imgur.com/a/uC2L354
  - thrusong 7y ago
    I'm pretty sure this data is kept in MySQL and memcached.
- edwintorok 7y ago
  https://developers.facebook.com/status/issues/343373056594853/ https://developers.facebook.com/status/issues/34337305659485...
- londons_explore 7y ago
  Could also be a networking issue. If network capacity is severely reduced, the bulk static content (images/media) is what you'd switch off first. The fact it's global makes that unlikely tho.
- jug 7y ago
  That’s also what it looks like to me. The services themselves work and are responsive here. But oh boy, images have been down for the longest time now for being top websites.
- giancarlostoro 7y ago
  My concern there is my understanding has always been that WhatsApp implements end to end encryption through the Signal protocol, how can images be stored and be considered end-to-end encrypted? So are only text messages end-to-end? Ignoring logs being uploaded to the cloud. I wouldn't send any sensitive pictures over WhatsApp then.
  - kabwj 7y ago
    Probably blob store, not just image store.
  - justapassenger 7y ago
    E2E encryption doesn't mean P2P networking. Data still flows through servers.
  - octorian 7y ago
    Because the actual data for those images, that is stored on Facebook's media servers, is just a blob of ciphertext. Only the sending and receiving clients actually have the keys, not the servers.
- perrwa 7y ago
  On Instagram, ad posts still fully display. Would that be a different image store or delivery?
functionCall 7y ago
I am see WhatsApp issues as well. Can not download media.
- robjan 7y ago
  Yep, their CDN is down. All Facebook properties are affected.
sonofgod 7y ago
Thanks, was wondering what was up.
manjana 7y ago
I'm having issues with streaming a series from HBO Nordic as well.
the-dude 7y ago
I find the outages of the last few days a curious coincidence : Slack, Google, Cloudflare, Azure, now FB. edit : added Slack after @kache_ mentioned it, added Azure after stevehubertron mentioned it.
- samstave 7y ago
  So the options are: 1) a secret cyber-war is going on... 2) Some entity is installing new spying slurps 3) An (critical) IX that we werent aware of is having major issues... and we dont know who to blame on that one...
  - lovecg 7y ago
    4) People love to find patterns where none exist
    - anon_z88 7y ago
      This is a fallacy all humans experience, however, what is the probability specifically that all these problems are unrelated? At what probability is it mathematically impossible that these problems are unrelated? AFAF
      - NateEag 7y ago
        The answer to your last question is 0. "Impossible" means "impossible", not "unbelievably improbable". That's even more true when you ask about "mathematically impossible", as it reinforces the idea of formal logic being the relevant domain, where precise meaning of words is fundamental. If you adjust your question to be "at what probability is it unreasonable to claim these problems are unrelated?", then the answer is subjective - different people have different standards for reasonability. I think we'd need mountains more data than we have about the incidents to compute a meaningful probability, anyway.
    - lucasverra 7y ago
      is that a patern ?
anaidjm1 7y ago
It's being atacked https://subefotos.com/ver/?263b876bdf09824089e4b118bb4da51bo.png https://subefotos.com/ver/?263b876bdf09824089e4b118bb4da51bo...
- vernie 7y ago
  What does this show?
  - the-dude 7y ago
    A worldmap with some shiny dots.
    - trentlott 7y ago
      If you're lucky! I only see the dots
- ceejayoz 7y ago
  These "attack maps" tend to just be a map of population centers cross-referenced against where it's daytime. They're worthless for actually assessing anything about potential threats.
_nickwhite 7y ago
Not only Facebook, but Instagram (which shares common infrastructure) has also been mostly down (read-only mode it seems). The most popular Twitter hashtag right now is #instagramdown
- parthdesai 7y ago
  WhatsApp as well.
kache_ 7y ago
I wonder what kind of economic damage all these recent outages add up to. How much harm are they causing, in terms of productivity and value. Facebook's social networks I'd imagine are less impactful than the slack outages, which could completely cripple a company if they were primarily remote.
- jressey 7y ago
  I think we should think about it more as the economic gain being experienced when people don't have these things to distract them.
  - aphextim 7y ago
    There was an old joke from I believe Jerry Seinfeld which came out around the time the Y2K scare was in full force. I am probably going to butcher it as I am no comedian but it went something like: >All these people are freaking out about this whole Y2K thing. What are we going to do when the internet and computers all stop working?! I always respond with, "I know it will be horrible, it will be like living in the 80s again" Now clearly I realize if all computers went down there would be real world consequences and issues, however I still like the joke.
    - navaati 7y ago
      Damn, of course ! It's a Stranger Things marketing ploy !
      - misterprime 7y ago
        In The Morning, Dude Named Ben.
- the-dude 7y ago
  You ain't seen nothing yet : https://en.wikipedia.org/wiki/SQL_Slammer https://en.wikipedia.org/wiki/SQL_Slammer
- JMTQp8lwXL 7y ago
cronix 7y ago
One interesting thing I noticed is the ALT text that is displayed in place of the missing images. It shows how their image recognition is classifying everything. I guess a lot of my friends drink as a lot of the alt text was like "Image may contain: drink"
- the-dude 7y ago
  I noticed those too. Except it contained something else.
- amputect 7y ago
  I noticed that too (mtg meme group just showed up as a wall of "image may contain:text" placeholders), but I didn't realize it was an outage and I thought that I'd accidentally turned on some kind of really annoying safe-mode.
- oceliker 7y ago
  That's an accessibility feature and has been around for at least a couple of years, I believe.
  - chirau 7y ago
    I think OP is referring to the fact that the alt-text is now all ML generated. But yes, you are right, it's been around for a while
smaili 7y ago
One interesting thing I've noticed from this (not sure if this is just my experience or if others noticed as well) but none of the ads seem to be impacted by this. Certainly makes me wonder if those are on a more prioritized and entirely separate infra and SLA that is designed to be more resilient and highly available.
- Mountain_Skies 7y ago
  Our local transit system installed video boards showing arrival times for the next trains. It was paid for and operated by an advertising company who in return were able to run ads on their video boards. The functionality for arrival times rarely worked but the displaying of ads never failed. It was obvious where the company's priorities were at.
  - adjkant 7y ago
    Intersection right?
- jedberg 7y ago
  It's a lot easier to serve ads than your custom photos. The ads are more generic, easier to cache globally, and show to a lot of people. They can also fall back to super generic ads if the database is unavailable to customize them. But your photos don't have a fallback.
  - londons_explore 7y ago
    Fallback is the main reason. Some webservices even fallback to non-moneymaking ads for charities in the case of a technical fault, because shareholders react badly to an ads outage, but they don't notice an hour or two of charity ads...
- klodolph 7y ago
  > Certainly makes me wonder if those are on a more prioritized and entirely separate infra and SLA that is designed to be more resilient and highly available. There's absolutely no doubt that the ads are served differently. Think about it this way: - Ads: Small number, each served to many people, funded with real cash, but you can't get the cash if you don't serve the ads. - Photos: Large number, each served to few people, funded by money left over from ads, and if you don't serve them your (non-paying) customers get frustrated. My guess is that ads is funded well enough that they can run at lower resource utilization, and much more effort is made to run photos at high resource utilization. That's how I'd run it.
legohead 7y ago
Noticed it this morning (~2 hours ago) on WhatsApp, voice messages weren't working, and still aren't. So, code deploy or server configuration change?
- sna1l 7y ago
  Seems like if it is affecting their services so broadly it is a server configuration change
  - pratap103 7y ago
    Yup, seems like it. This is second time in a couple of months that they've been down.
- scohesc 7y ago
  They're implementing their personal data harvesting tools into WhatsApp and the sheer amount of content being pulled from their servers was too much to handle and it crashed /s but probably true eventually
heavymark 7y ago
I first noticed Instagram then realized it was Facebook and their other properties such as What's App as well. Is very odd all of these major outages this week. By Cloudflare at least said it was an employee error, so assuming it's not subtle attacks by other countries unless companies like Cloudflare are required to legally provide an alternate story for national security.
- ksec 7y ago
  According to Digital Attack [1] it seems the world is constantly on DDoS. [1] https://www.digitalattackmap.com https://www.digitalattackmap.com
  - Radle 7y ago
    "Notable Recent Attacks" from 2016, the site is dead and shows fancy graphic.
  - deleted 7y ago
    [deleted]
- dboreham 7y ago
  People went on vacation and left interns running the show?
tcarn 7y ago
Surprised this impacts Whatsapp as well which seemed to store everything encrypted on your device...
- robjan 7y ago
  It needs to download it from the CDN first. The actual implementation is your media are stored encrypted on a CDN for a certain amount of time then you receive the keys as an encrypted message.
- oasisbob 7y ago
  Likewise. If it's a well-thought out encrypted blob store, fine... But given that Facebook, Instagram, and WhatsApp would seem to have vastly different caching and security requirements - its hard to believe. Is any of this architecture publicly documented?
  - thrusong 7y ago
    I think Zuck himself said recently they're moving to encrypt everything and merge infrastructures for single login across properties (which a lot of us think is them sneakily trying to make their web properties a lot harder to split up in an anti-trust case).
supergauntlet 7y ago
Feels like every day there's another one of these outages. Shame that everything is so centralized that there's a single point of failure that affects 3 of the biggest communication platforms around.
kevlawrence 7y ago
This gives me a lot of confidence in their crypto currency.
- smt88 7y ago
  1) Cryptocurrency ops are so vastly different from running a social website that I can't even think of any overlap. 2) I hate Facebook as a company, but as a builder/scaler of web apps for many years, I'm continually blown away by the speed and reliability of their website. Their operations are mind-blowing. The only comparable apps (in terms of scale) are Gmail and YouTube, and Gmail is simpler in certain key areas (e.g. mail delivery isn't millisecond-sensitive for a user).
  - reaperducer 7y ago
    Cryptocurrency ops are so vastly different from running a social website that I can't even think of any overlap I know nothing about how cryptocurrency works, but wouldn't social media outage sources like multiple server failures, hurricane, tornado, sliced fiber line, etc... affect the kind of cryptocurrency that Facebook is embarking on? Or is there something in the "distributed" nature of cryptocurrency that makes it more resilient? Is Facebook using that model, too?
    - dboreham 7y ago
      No. Yes.
  - dullgiulio 7y ago
    And Google Search. That's actually treated even more specially (compared to GMail and YT) inside Google and it has amazing reliability. It would be interesting to know more details. I bet results are sometimes incomplete, but somehow Google manages to keep the system correct enough nobody notices a sudden quality drop. I also understood Search has some special QoS at all levels: from the network to the scheduling of jobs...
  - basch 7y ago
    How would bgp routing failures and cut fiber NOT also impact digital currency networks?
kofisarfo 7y ago
Imagine if this meant not being able to send or receive currency.
- ceejayoz 7y ago
  Not hard. My bank's had the occasional (twice in four years, IIRC) debit/credit swipe outage of a couple of hours. It happens.
- anbop 7y ago
  And? That’s why cash exists.
  - the-dude 7y ago
    In NL cash transactions of 10k+ already needed to be reported to the authorities. But a new law will forbid any cash transactions higher than 3k.
    - dlphn___xyz 7y ago
      whats the reasoning behind the new law?
      - Spearchucker 7y ago
        The revenue service is unaware of cash payments/purchases made by private parties and so cannot tax such transactions. Unless those private parties volunteer the information.
      - i_cant_speel 7y ago
        It looks like an attempt to prevent money laundering https://nltimes.nl/2019/07/01/dutch-govt-ban-cash-payments-eu3000-fight-money-laundering https://nltimes.nl/2019/07/01/dutch-govt-ban-cash-payments-e...
    - urtrs 7y ago
      In Greece it is forbidden by law any cash transaction higher than 500 euros.
samstave 7y ago
I am skeptical about all these outages occurring over the last week... Many outages. I think a silent cyber-war is occurring between the US and China.
jbverschoor 7y ago
Duplicate of https://news.ycombinator.com/item?id=20344749#20344824 https://news.ycombinator.com/item?id=20344749#20344824
picacho 7y ago
does leetcode help solving the outages?
slimshady93 7y ago
It seems intern season has begun.
buboard 7y ago
The fact that these are called outages is kind of alarming. Do people rely on any of these as their exclusive communication when in danger?
- wybiral 7y ago
  I don't think the service needs to be critical for it to be an "outage". If the service is unavailable to people who use it that's an "outage". You could have a soft-serve ice cream machine outage.
- maw 7y ago
  I hope not. But in some countries many small to medium businesses use WhatsApp extensively for communication with their customers. For many people it's as bad, arguably worse, than their main ISP going down.
- renaudg 7y ago
  Of course they do. In many developing countries, Facebook IS the Internet. https://qz.com/333313/milliions-of-facebook-users-have-no-idea-theyre-using-the-internet/ https://qz.com/333313/milliions-of-facebook-users-have-no-id...
y04nn 7y ago
All those outages are strange, I wasn't able to access my personal server for few hours today, I was not even capable to ping ip). When I was able to reconnect, everything seem to be ok. Maybe there is some routing/BGP issues/attack (again). Sadly, I didn't try to traceroute the serveur IP.
- cfitz 7y ago
  Second this, with my DigitalOcean VPS's. DigitalOcean posted an incident report yesterday. They say "global networking issues" were caused by a "major provider", but unfortunately there isn't much more detail than that [1]. [1] https://status.digitalocean.com/incidents/qvdcj7yx4030 https://status.digitalocean.com/incidents/qvdcj7yx4030
  - devin 7y ago
    That was due to google cloud networking issues.
    - jammygit 7y ago
      GCP provides infrastructure for digital ocean?
      - captn3m0 7y ago
        No, but their Data Centres may be sharing upstream fibre provider? Google mentioned a Fiber cut upstream.
awirth 7y ago
Some popular images seem to load for me, but longer tail stuff doesn't. Maybe they accidentally flushed their edge cache and then their origin couldn't handle the load.
- londons_explore 7y ago
  Could they be in your own browser's cache? Or maybe in your ISP's cache? Considering the type of content Facebook, WhatsApp, and Instagram host, I'd imagine the cache hit rates to be pretty low for media. They'd be effective for JavaScript/CSS though, which might also be served from the same system
  - awirth 7y ago
    No, I checked if it was my own cache, and it's all HTTPS so it couldn't have been my ISP's cache either. I compared a few celebrity's instagrams with accounts of friends with many fewer followers. Celeb's images loaded but less and less as you scroll back through time, friends images all 503'd. Still seems to be the case.
    - londons_explore 7y ago
      By ISP cache, I meant fbcdn running off a dell server in a rack at your ISP...
      - awirth 7y ago
        Ah, sure. Down detector's map suggests it's global: https://downdetector.com/status/facebook/map/ https://downdetector.com/status/facebook/map/ Instagram also doesn't seem to be serving hot and cold content from different hostnames/IPs for me, so it couldn't be that either. Also check out the official status update at https://developers.facebook.com/status/issues/343373056594853/ https://developers.facebook.com/status/issues/34337305659485... and https://developers.facebook.com/support/bugs/610822019411772/ https://developers.facebook.com/support/bugs/610822019411772...
cyanbane 7y ago
Browsing it right now is a good exercise for devs who need to make sure they think about low vision/blind users who browse via audio, etc. ie "Image may contain: a dog, sunglasses"
- rzzzt 7y ago
  Automatic alt text is a nifty feature: https://www.facebook.com/help/216219865403298?helpref=faq_content https://www.facebook.com/help/216219865403298?helpref=faq_co...
- deleted 7y ago
  [deleted]
edwintorok 7y ago
Do we need a 'Show Outage:' tag? HN seems to become the defacto outage reporting place...
- eneveu 7y ago
  Because it's interesting to discuss the impact of those outages and the reasons for them, to learn from those experiences.
o_p 7y ago
A perfect moment to evangelize my contacts about Telegram...
- edwintorok 7y ago
  I've heard good things about Signal and Wire too.
bitL 7y ago
Anonymous working on post-FAANG world?
- gsich 7y ago
  Gmafia is a better term.
twsted 7y ago
Do you remember when we first heard about "the cloud"? I remind about a video when Larry Ellison mentioned the cloud and I thought: "What is he speaking about? It's the internet". How I was wrong... Now we are in a situation where we have too much concentration and we are experiencing that. It's a problem.
- supergauntlet 7y ago
  I wonder if there's a large enough market for a "decentralized cloud" along the lines of IPFS but for compute. And, yknow, actually working decently. Something where you could pay compute/storage operators for resources and also be a server host and share your excess compute power.
  - nexuist 7y ago
    This is effectively the idea behind the Ethereum blockchain model. You can make "smart contracts" (think APIs with persistent state) using languages like Solidity[1] and deploy them onto the blockchain. After that, you can invoke individual functions by paying "gas" (small amounts of ETH) that goes to the node operator's account. Smart contracts also get their own blockchain addresses, so they're capable of sending and receiving transactions. Meaning you can build financial applications with little to no barriers to entry! (whether this is a good thing or not, I will leave up to the reader). The really cool thing about it is that nobody owns a smart contract once it's deployed. You can't edit the code or even delete the contract itself. It's a truly autonomous entity that will continue to operate the same way forever (unless there's a 51% attack or something of that nature). The obvious benefit for this is that you can mathematically ensure trust. For example, if you hosted a lottery app on a LAMP stack, you could steal the money, hackers could get into your server, your database could get corrupted, etc. On the blockchain, nobody can access your lottery funds or business logic, not even yourself, meaning that as long as you developed the application correctly (to be fair, that is a big assumption), it is truly fair. [1] https://solidity.readthedocs.io/en/v0.5.10/ https://solidity.readthedocs.io/en/v0.5.10/
    - giancarlostoro 7y ago
hn23 7y ago
So what is next, productivity peak at maximum?
thrusong 7y ago
Seems to be completely back online from my end in central Canada.
agoodthrowaway 7y ago
This outage coincides with FBs PSC (performance summary cycle) time. I wonder if this is folks trying to push features so they get “impact” for PSC.
- lopespm 7y ago
  Very good point. I wonder if the recent outages on other well known services could be heavily influenced by a similar phenomenon. If this holds water, it would be interesting to have an article or study around this issue. I certainly would be interested in reading it.

345 points by skilled 7y ago

Looks like Facebook image store is having issues. Facebook, Instagram and WhatsApp use the same image store, so all the services are affected.

mtgx 7y ago
Probably a good opportunity for more people to realize that all three services belong to the same company.
- hbosch 7y ago
  It’s common knowledge.
  - 3stripe 7y ago
    For HN readers sure
ceocoder 7y ago
However image metadata service seems to be up, this is what I see for most images on FB now. https://imgur.com/a/uC2L354 https://imgur.com/a/uC2L354
- thrusong 7y ago
  I'm pretty sure this data is kept in MySQL and memcached.
edwintorok 7y ago
https://developers.facebook.com/status/issues/343373056594853/ https://developers.facebook.com/status/issues/34337305659485...
londons_explore 7y ago
Could also be a networking issue. If network capacity is severely reduced, the bulk static content (images/media) is what you'd switch off first. The fact it's global makes that unlikely tho.
jug 7y ago
That’s also what it looks like to me. The services themselves work and are responsive here. But oh boy, images have been down for the longest time now for being top websites.
giancarlostoro 7y ago
My concern there is my understanding has always been that WhatsApp implements end to end encryption through the Signal protocol, how can images be stored and be considered end-to-end encrypted? So are only text messages end-to-end? Ignoring logs being uploaded to the cloud. I wouldn't send any sensitive pictures over WhatsApp then.
- kabwj 7y ago
  Probably blob store, not just image store.
- justapassenger 7y ago
  E2E encryption doesn't mean P2P networking. Data still flows through servers.
- octorian 7y ago
  Because the actual data for those images, that is stored on Facebook's media servers, is just a blob of ciphertext. Only the sending and receiving clients actually have the keys, not the servers.
perrwa 7y ago
On Instagram, ad posts still fully display. Would that be a different image store or delivery?

functionCall 7y ago

I am see WhatsApp issues as well. Can not download media.

robjan 7y ago
Yep, their CDN is down. All Facebook properties are affected.

sonofgod 7y ago

Thanks, was wondering what was up.

manjana 7y ago

I'm having issues with streaming a series from HBO Nordic as well.

the-dude 7y ago

I find the outages of the last few days a curious coincidence : Slack, Google, Cloudflare, Azure, now FB. edit : added Slack after @kache_ mentioned it, added Azure after stevehubertron mentioned it.

samstave 7y ago
So the options are: 1) a secret cyber-war is going on... 2) Some entity is installing new spying slurps 3) An (critical) IX that we werent aware of is having major issues... and we dont know who to blame on that one...
- lovecg 7y ago
  4) People love to find patterns where none exist
  - anon_z88 7y ago
    This is a fallacy all humans experience, however, what is the probability specifically that all these problems are unrelated? At what probability is it mathematically impossible that these problems are unrelated? AFAF
    - NateEag 7y ago
      The answer to your last question is 0. "Impossible" means "impossible", not "unbelievably improbable". That's even more true when you ask about "mathematically impossible", as it reinforces the idea of formal logic being the relevant domain, where precise meaning of words is fundamental. If you adjust your question to be "at what probability is it unreasonable to claim these problems are unrelated?", then the answer is subjective - different people have different standards for reasonability. I think we'd need mountains more data than we have about the incidents to compute a meaningful probability, anyway.
  - lucasverra 7y ago
    is that a patern ?

anaidjm1 7y ago

It's being atacked https://subefotos.com/ver/?263b876bdf09824089e4b118bb4da51bo.png https://subefotos.com/ver/?263b876bdf09824089e4b118bb4da51bo...

vernie 7y ago
What does this show?
- the-dude 7y ago
  A worldmap with some shiny dots.
  - trentlott 7y ago
    If you're lucky! I only see the dots
ceejayoz 7y ago
These "attack maps" tend to just be a map of population centers cross-referenced against where it's daytime. They're worthless for actually assessing anything about potential threats.

_nickwhite 7y ago

Not only Facebook, but Instagram (which shares common infrastructure) has also been mostly down (read-only mode it seems). The most popular Twitter hashtag right now is #instagramdown

parthdesai 7y ago
WhatsApp as well.

kache_ 7y ago

I wonder what kind of economic damage all these recent outages add up to. How much harm are they causing, in terms of productivity and value. Facebook's social networks I'd imagine are less impactful than the slack outages, which could completely cripple a company if they were primarily remote.

jressey 7y ago
I think we should think about it more as the economic gain being experienced when people don't have these things to distract them.
- aphextim 7y ago
  There was an old joke from I believe Jerry Seinfeld which came out around the time the Y2K scare was in full force. I am probably going to butcher it as I am no comedian but it went something like: >All these people are freaking out about this whole Y2K thing. What are we going to do when the internet and computers all stop working?! I always respond with, "I know it will be horrible, it will be like living in the 80s again" Now clearly I realize if all computers went down there would be real world consequences and issues, however I still like the joke.
  - navaati 7y ago
    Damn, of course ! It's a Stranger Things marketing ploy !
    - misterprime 7y ago
      In The Morning, Dude Named Ben.
the-dude 7y ago
You ain't seen nothing yet : https://en.wikipedia.org/wiki/SQL_Slammer https://en.wikipedia.org/wiki/SQL_Slammer
JMTQp8lwXL 7y ago

cronix 7y ago

One interesting thing I noticed is the ALT text that is displayed in place of the missing images. It shows how their image recognition is classifying everything. I guess a lot of my friends drink as a lot of the alt text was like "Image may contain: drink"

the-dude 7y ago
I noticed those too. Except it contained something else.
amputect 7y ago
I noticed that too (mtg meme group just showed up as a wall of "image may contain:text" placeholders), but I didn't realize it was an outage and I thought that I'd accidentally turned on some kind of really annoying safe-mode.
oceliker 7y ago
That's an accessibility feature and has been around for at least a couple of years, I believe.
- chirau 7y ago
  I think OP is referring to the fact that the alt-text is now all ML generated. But yes, you are right, it's been around for a while

smaili 7y ago

One interesting thing I've noticed from this (not sure if this is just my experience or if others noticed as well) but none of the ads seem to be impacted by this. Certainly makes me wonder if those are on a more prioritized and entirely separate infra and SLA that is designed to be more resilient and highly available.

Mountain_Skies 7y ago
Our local transit system installed video boards showing arrival times for the next trains. It was paid for and operated by an advertising company who in return were able to run ads on their video boards. The functionality for arrival times rarely worked but the displaying of ads never failed. It was obvious where the company's priorities were at.
- adjkant 7y ago
  Intersection right?
jedberg 7y ago
It's a lot easier to serve ads than your custom photos. The ads are more generic, easier to cache globally, and show to a lot of people. They can also fall back to super generic ads if the database is unavailable to customize them. But your photos don't have a fallback.
- londons_explore 7y ago
  Fallback is the main reason. Some webservices even fallback to non-moneymaking ads for charities in the case of a technical fault, because shareholders react badly to an ads outage, but they don't notice an hour or two of charity ads...
klodolph 7y ago
> Certainly makes me wonder if those are on a more prioritized and entirely separate infra and SLA that is designed to be more resilient and highly available. There's absolutely no doubt that the ads are served differently. Think about it this way: - Ads: Small number, each served to many people, funded with real cash, but you can't get the cash if you don't serve the ads. - Photos: Large number, each served to few people, funded by money left over from ads, and if you don't serve them your (non-paying) customers get frustrated. My guess is that ads is funded well enough that they can run at lower resource utilization, and much more effort is made to run photos at high resource utilization. That's how I'd run it.

legohead 7y ago

Noticed it this morning (~2 hours ago) on WhatsApp, voice messages weren't working, and still aren't. So, code deploy or server configuration change?

sna1l 7y ago
Seems like if it is affecting their services so broadly it is a server configuration change
- pratap103 7y ago
  Yup, seems like it. This is second time in a couple of months that they've been down.
scohesc 7y ago
They're implementing their personal data harvesting tools into WhatsApp and the sheer amount of content being pulled from their servers was too much to handle and it crashed /s but probably true eventually

heavymark 7y ago

I first noticed Instagram then realized it was Facebook and their other properties such as What's App as well. Is very odd all of these major outages this week. By Cloudflare at least said it was an employee error, so assuming it's not subtle attacks by other countries unless companies like Cloudflare are required to legally provide an alternate story for national security.

ksec 7y ago
According to Digital Attack [1] it seems the world is constantly on DDoS. [1] https://www.digitalattackmap.com https://www.digitalattackmap.com
- Radle 7y ago
  "Notable Recent Attacks" from 2016, the site is dead and shows fancy graphic.
- deleted 7y ago
  [deleted]
dboreham 7y ago
People went on vacation and left interns running the show?

tcarn 7y ago

Surprised this impacts Whatsapp as well which seemed to store everything encrypted on your device...

robjan 7y ago
It needs to download it from the CDN first. The actual implementation is your media are stored encrypted on a CDN for a certain amount of time then you receive the keys as an encrypted message.
oasisbob 7y ago
Likewise. If it's a well-thought out encrypted blob store, fine... But given that Facebook, Instagram, and WhatsApp would seem to have vastly different caching and security requirements - its hard to believe. Is any of this architecture publicly documented?
- thrusong 7y ago
  I think Zuck himself said recently they're moving to encrypt everything and merge infrastructures for single login across properties (which a lot of us think is them sneakily trying to make their web properties a lot harder to split up in an anti-trust case).

supergauntlet 7y ago

Feels like every day there's another one of these outages. Shame that everything is so centralized that there's a single point of failure that affects 3 of the biggest communication platforms around.

kevlawrence 7y ago

This gives me a lot of confidence in their crypto currency.

smt88 7y ago
1) Cryptocurrency ops are so vastly different from running a social website that I can't even think of any overlap. 2) I hate Facebook as a company, but as a builder/scaler of web apps for many years, I'm continually blown away by the speed and reliability of their website. Their operations are mind-blowing. The only comparable apps (in terms of scale) are Gmail and YouTube, and Gmail is simpler in certain key areas (e.g. mail delivery isn't millisecond-sensitive for a user).
- reaperducer 7y ago
  Cryptocurrency ops are so vastly different from running a social website that I can't even think of any overlap I know nothing about how cryptocurrency works, but wouldn't social media outage sources like multiple server failures, hurricane, tornado, sliced fiber line, etc... affect the kind of cryptocurrency that Facebook is embarking on? Or is there something in the "distributed" nature of cryptocurrency that makes it more resilient? Is Facebook using that model, too?
  - dboreham 7y ago
    No. Yes.
- dullgiulio 7y ago
  And Google Search. That's actually treated even more specially (compared to GMail and YT) inside Google and it has amazing reliability. It would be interesting to know more details. I bet results are sometimes incomplete, but somehow Google manages to keep the system correct enough nobody notices a sudden quality drop. I also understood Search has some special QoS at all levels: from the network to the scheduling of jobs...
- basch 7y ago
  How would bgp routing failures and cut fiber NOT also impact digital currency networks?

kofisarfo 7y ago

Imagine if this meant not being able to send or receive currency.

ceejayoz 7y ago
Not hard. My bank's had the occasional (twice in four years, IIRC) debit/credit swipe outage of a couple of hours. It happens.
anbop 7y ago
And? That’s why cash exists.
- the-dude 7y ago
  In NL cash transactions of 10k+ already needed to be reported to the authorities. But a new law will forbid any cash transactions higher than 3k.
  - dlphn___xyz 7y ago
    whats the reasoning behind the new law?
    - Spearchucker 7y ago
      The revenue service is unaware of cash payments/purchases made by private parties and so cannot tax such transactions. Unless those private parties volunteer the information.
    - i_cant_speel 7y ago
      It looks like an attempt to prevent money laundering https://nltimes.nl/2019/07/01/dutch-govt-ban-cash-payments-eu3000-fight-money-laundering https://nltimes.nl/2019/07/01/dutch-govt-ban-cash-payments-e...
  - urtrs 7y ago
    In Greece it is forbidden by law any cash transaction higher than 500 euros.

samstave 7y ago

I am skeptical about all these outages occurring over the last week... Many outages. I think a silent cyber-war is occurring between the US and China.

jbverschoor 7y ago

Duplicate of https://news.ycombinator.com/item?id=20344749#20344824 https://news.ycombinator.com/item?id=20344749#20344824

picacho 7y ago

does leetcode help solving the outages?

slimshady93 7y ago

It seems intern season has begun.

buboard 7y ago

The fact that these are called outages is kind of alarming. Do people rely on any of these as their exclusive communication when in danger?

wybiral 7y ago
I don't think the service needs to be critical for it to be an "outage". If the service is unavailable to people who use it that's an "outage". You could have a soft-serve ice cream machine outage.
maw 7y ago
I hope not. But in some countries many small to medium businesses use WhatsApp extensively for communication with their customers. For many people it's as bad, arguably worse, than their main ISP going down.
renaudg 7y ago
Of course they do. In many developing countries, Facebook IS the Internet. https://qz.com/333313/milliions-of-facebook-users-have-no-idea-theyre-using-the-internet/ https://qz.com/333313/milliions-of-facebook-users-have-no-id...

y04nn 7y ago

All those outages are strange, I wasn't able to access my personal server for few hours today, I was not even capable to ping ip). When I was able to reconnect, everything seem to be ok. Maybe there is some routing/BGP issues/attack (again). Sadly, I didn't try to traceroute the serveur IP.

cfitz 7y ago
Second this, with my DigitalOcean VPS's. DigitalOcean posted an incident report yesterday. They say "global networking issues" were caused by a "major provider", but unfortunately there isn't much more detail than that [1]. [1] https://status.digitalocean.com/incidents/qvdcj7yx4030 https://status.digitalocean.com/incidents/qvdcj7yx4030
- devin 7y ago
  That was due to google cloud networking issues.
  - jammygit 7y ago
    GCP provides infrastructure for digital ocean?
    - captn3m0 7y ago
      No, but their Data Centres may be sharing upstream fibre provider? Google mentioned a Fiber cut upstream.

awirth 7y ago

Some popular images seem to load for me, but longer tail stuff doesn't. Maybe they accidentally flushed their edge cache and then their origin couldn't handle the load.

londons_explore 7y ago
Could they be in your own browser's cache? Or maybe in your ISP's cache? Considering the type of content Facebook, WhatsApp, and Instagram host, I'd imagine the cache hit rates to be pretty low for media. They'd be effective for JavaScript/CSS though, which might also be served from the same system
- awirth 7y ago
  No, I checked if it was my own cache, and it's all HTTPS so it couldn't have been my ISP's cache either. I compared a few celebrity's instagrams with accounts of friends with many fewer followers. Celeb's images loaded but less and less as you scroll back through time, friends images all 503'd. Still seems to be the case.
  - londons_explore 7y ago
    By ISP cache, I meant fbcdn running off a dell server in a rack at your ISP...
    - awirth 7y ago
      Ah, sure. Down detector's map suggests it's global: https://downdetector.com/status/facebook/map/ https://downdetector.com/status/facebook/map/ Instagram also doesn't seem to be serving hot and cold content from different hostnames/IPs for me, so it couldn't be that either. Also check out the official status update at https://developers.facebook.com/status/issues/343373056594853/ https://developers.facebook.com/status/issues/34337305659485... and https://developers.facebook.com/support/bugs/610822019411772/ https://developers.facebook.com/support/bugs/610822019411772...

cyanbane 7y ago

Browsing it right now is a good exercise for devs who need to make sure they think about low vision/blind users who browse via audio, etc. ie "Image may contain: a dog, sunglasses"

rzzzt 7y ago
Automatic alt text is a nifty feature: https://www.facebook.com/help/216219865403298?helpref=faq_content https://www.facebook.com/help/216219865403298?helpref=faq_co...
deleted 7y ago
[deleted]

edwintorok 7y ago

Do we need a 'Show Outage:' tag? HN seems to become the defacto outage reporting place...

eneveu 7y ago
Because it's interesting to discuss the impact of those outages and the reasons for them, to learn from those experiences.

o_p 7y ago

A perfect moment to evangelize my contacts about Telegram...

edwintorok 7y ago
I've heard good things about Signal and Wire too.

bitL 7y ago

Anonymous working on post-FAANG world?

gsich 7y ago
Gmafia is a better term.

twsted 7y ago

Do you remember when we first heard about "the cloud"? I remind about a video when Larry Ellison mentioned the cloud and I thought: "What is he speaking about? It's the internet". How I was wrong... Now we are in a situation where we have too much concentration and we are experiencing that. It's a problem.

supergauntlet 7y ago
I wonder if there's a large enough market for a "decentralized cloud" along the lines of IPFS but for compute. And, yknow, actually working decently. Something where you could pay compute/storage operators for resources and also be a server host and share your excess compute power.
- nexuist 7y ago
  This is effectively the idea behind the Ethereum blockchain model. You can make "smart contracts" (think APIs with persistent state) using languages like Solidity[1] and deploy them onto the blockchain. After that, you can invoke individual functions by paying "gas" (small amounts of ETH) that goes to the node operator's account. Smart contracts also get their own blockchain addresses, so they're capable of sending and receiving transactions. Meaning you can build financial applications with little to no barriers to entry! (whether this is a good thing or not, I will leave up to the reader). The really cool thing about it is that nobody owns a smart contract once it's deployed. You can't edit the code or even delete the contract itself. It's a truly autonomous entity that will continue to operate the same way forever (unless there's a 51% attack or something of that nature). The obvious benefit for this is that you can mathematically ensure trust. For example, if you hosted a lottery app on a LAMP stack, you could steal the money, hackers could get into your server, your database could get corrupted, etc. On the blockchain, nobody can access your lottery funds or business logic, not even yourself, meaning that as long as you developed the application correctly (to be fair, that is a big assumption), it is truly fair. [1] https://solidity.readthedocs.io/en/v0.5.10/ https://solidity.readthedocs.io/en/v0.5.10/
  - giancarlostoro 7y ago

hn23 7y ago

So what is next, productivity peak at maximum?

thrusong 7y ago

Seems to be completely back online from my end in central Canada.

agoodthrowaway 7y ago

This outage coincides with FBs PSC (performance summary cycle) time. I wonder if this is folks trying to push features so they get “impact” for PSC.

lopespm 7y ago
Very good point. I wonder if the recent outages on other well known services could be heavily influenced by a similar phenomenon. If this holds water, it would be interesting to have an article or study around this issue. I certainly would be interested in reading it.