8 ms·

Can't wait to find out what happened here. This seems to be a massive outage. Interesting how fragile things become when so much technology is concentrated into

Can't wait to find out what happened here. This seems to be a massive outage. Interesting how fragile things become when so much technology is concentrated into just a few companies.

theshrike79 2y ago
It's always DNS.
- jeffrallen 2y ago
  Unless it's Nagle. (Sorry animats.)
- onion2k 2y ago
  Or BGP
us0r 2y ago
My money is on expired domain somewhere or security certificate.
- jraph 2y ago
  A Windows update restarted a critical server automatically. A core service is blocked from starting by a Candy Crush ad installed by the update. The Crandy Crush ad is somehow expecting a Copilot key to be pressed on the keyboard to let the system keep going. MS engineers are waiting for an online purchase of a new $400 keyboard with the Copilot key to complete and planning to run to the data center to plug the keyboard. However, the Bing outage is preventing the purchase to go through, because the payment somehow relies on Bing suggestions to load for obscure reasons.
  - Kerbonut 2y ago
    No way Microsoft could be this inept… could they?
    - deleted 2y ago
      [deleted]
  - exikyut 2y ago
    Very very technically, if RDP is enabled and working, this could be fixed by rdesktop-ing to the machine from a Linux box and using xdotool to experiment with typing raw keyboard scancodes through the RDP session in the hope you figure out the encoding of the Copilot key.
    - r2_pilot 2y ago
      The Copilot key is actually Left-Shift + Windows key + F23
      - adamomada 2y ago
        And you can remap it but can’t use it as a modifier key https://www.tomshardware.com/software/windows/windows-copilot-key-is-secretly-from-the-ibm-era-but-you-can-remap-it-with-the-right-tools https://www.tomshardware.com/software/windows/windows-copilo...
    - jraph 2y ago
      Neat answer :-) I also appreciate that you prevented me from countering using a guard like "if RDP is enabled and working", and that a follow up answer actually provides the missing piece xD.
  - rchaud 2y ago
    Update: the payment gateway is Stripe, which is not processing any transactions associated with the MS account. A developer has posted the issue to HN in the hope that a Stripe employee will see it and escalate the issue. /s
    - manuelmoreale 2y ago
      I love that you had to include the /s because someone might actually believe that is indeed the case. What a bizarre world we live in.
      - adamomada 2y ago
        Poe’s Law is pretty old by now
        manuelmoreale 2y ago
        I get that but still. I found it amusing.
  - ornornor 2y ago
    MS knows better than to run windows on their servers. They’ve famously been running Linux on their public web facing stuff for years, including when they were publicly discrediting Linux (because IIS and MS server were so good they couldn’t run their web services reliably and securely).
radiorental 2y ago
Genuine question, are distributed systems naturally more resilient? I can see arguments for both sides. Your point and then the hidden failure modes without central observability and ownership. Nothing exists in isolation.
- Dalewyn 2y ago
  >are distributed systems naturally more resilient? All else being equal: Yes. It's like asking if a RAID1 is more resilient than a single drive.
  - CWuestefeld 2y ago
    To the GP's point - if you lose the RAID controller, then you've lost a whole lot more than just a single drive failure.
    - Dalewyn 2y ago
      Yes, RAID isn't a backup, but it is resilient. You will have a better chance at uptime with a RAID than a single drive so you hopefully don't have to climb up ventilation ducts, walk across broken glass, and kill anyone sent to stop you on your quest to reconnect those cables that were cut.
    - Gormo 2y ago
      The controller isn't stateful; it's just an interface to the disks. If the controller fails, but the disks haven't, then all you've lost is the time it takes to plug the disks into a new controller. With RAID1, there's also nothing specific to the RAID configuration inherent in the way the data is encoded on the disk. You might have to carefully replicate your configuration to access the filesystem from a failed RAID0 array, but you can just pull and individual disk out of a RAID1 array and use it normally as a standalone disk.
  - steve1977 2y ago
    RAID1 is mirrored. That is not what I would call a typical distributed system. It is a very redundant system. Like a cluster. A distributed system without redundancy would rather be something like data stripped across disks without parity. And that actually makes it less resilient, because failure of one component can bring down the whole system and the likelihood of failure is statistically higher because of the higher number of components.
re-thc 2y ago
Used a Co-pilot enabled PC.
the_biot 2y ago
Not necessarily massive. Given that Bing works again now, this seems more like an API frontend failure, or an internal routing failure at some level. Note they seem to have managed to fix the Bing frontend hours ago, but DDG is still dead in the water. Priorities... :-)
- deleted 2y ago
  [deleted]
- pythux 2y ago
  Bing does not work me at the moment (maybe they can service partial traffic? Unclear if it’s better/worse than a few hours ago when the outage started)
simias 2y ago
My subjective impression as a web user since the late 90s is that now things break relatively rarely (I think it's the first time I have any such issue with DDG for instance) but when they do a huge chunk of the web becomes unreachable. Back when things were more decentralized individual websites and services would have issues much more regularly because the individual software and hardware stacks weren't as robust and fault-tolerant, but then usually the problem would always be limited to a single website/service.
- AbstractH24 2y ago
  And America only uses the term “Too Big Toto Fail” when it comes to banks.
  - lelanthran 2y ago
    > And America only uses the term “Too Big Toto Fail” when it comes to banks. And only when they're not in Kansas, anymore.

Can't wait to find out what happened here. This seems to be a massive outage. Interesting how fragile things become when so much technology is concentrated into just a few companies.

theshrike79 2y ago

It's always DNS.

jeffrallen 2y ago
Unless it's Nagle. (Sorry animats.)
onion2k 2y ago
Or BGP

us0r 2y ago

My money is on expired domain somewhere or security certificate.

jraph 2y ago
A Windows update restarted a critical server automatically. A core service is blocked from starting by a Candy Crush ad installed by the update. The Crandy Crush ad is somehow expecting a Copilot key to be pressed on the keyboard to let the system keep going. MS engineers are waiting for an online purchase of a new $400 keyboard with the Copilot key to complete and planning to run to the data center to plug the keyboard. However, the Bing outage is preventing the purchase to go through, because the payment somehow relies on Bing suggestions to load for obscure reasons.
- Kerbonut 2y ago
  No way Microsoft could be this inept… could they?
  - deleted 2y ago
    [deleted]
- exikyut 2y ago
  Very very technically, if RDP is enabled and working, this could be fixed by rdesktop-ing to the machine from a Linux box and using xdotool to experiment with typing raw keyboard scancodes through the RDP session in the hope you figure out the encoding of the Copilot key.
  - r2_pilot 2y ago
    The Copilot key is actually Left-Shift + Windows key + F23
    - adamomada 2y ago
      And you can remap it but can’t use it as a modifier key https://www.tomshardware.com/software/windows/windows-copilot-key-is-secretly-from-the-ibm-era-but-you-can-remap-it-with-the-right-tools https://www.tomshardware.com/software/windows/windows-copilo...
  - jraph 2y ago
    Neat answer :-) I also appreciate that you prevented me from countering using a guard like "if RDP is enabled and working", and that a follow up answer actually provides the missing piece xD.
- rchaud 2y ago
  Update: the payment gateway is Stripe, which is not processing any transactions associated with the MS account. A developer has posted the issue to HN in the hope that a Stripe employee will see it and escalate the issue. /s
  - manuelmoreale 2y ago
    I love that you had to include the /s because someone might actually believe that is indeed the case. What a bizarre world we live in.
    - adamomada 2y ago
      Poe’s Law is pretty old by now
      - manuelmoreale 2y ago
        I get that but still. I found it amusing.
- ornornor 2y ago
  MS knows better than to run windows on their servers. They’ve famously been running Linux on their public web facing stuff for years, including when they were publicly discrediting Linux (because IIS and MS server were so good they couldn’t run their web services reliably and securely).

radiorental 2y ago

Genuine question, are distributed systems naturally more resilient? I can see arguments for both sides. Your point and then the hidden failure modes without central observability and ownership. Nothing exists in isolation.

Dalewyn 2y ago
>are distributed systems naturally more resilient? All else being equal: Yes. It's like asking if a RAID1 is more resilient than a single drive.
- CWuestefeld 2y ago
  To the GP's point - if you lose the RAID controller, then you've lost a whole lot more than just a single drive failure.
  - Dalewyn 2y ago
    Yes, RAID isn't a backup, but it is resilient. You will have a better chance at uptime with a RAID than a single drive so you hopefully don't have to climb up ventilation ducts, walk across broken glass, and kill anyone sent to stop you on your quest to reconnect those cables that were cut.
  - Gormo 2y ago
    The controller isn't stateful; it's just an interface to the disks. If the controller fails, but the disks haven't, then all you've lost is the time it takes to plug the disks into a new controller. With RAID1, there's also nothing specific to the RAID configuration inherent in the way the data is encoded on the disk. You might have to carefully replicate your configuration to access the filesystem from a failed RAID0 array, but you can just pull and individual disk out of a RAID1 array and use it normally as a standalone disk.
- steve1977 2y ago
  RAID1 is mirrored. That is not what I would call a typical distributed system. It is a very redundant system. Like a cluster. A distributed system without redundancy would rather be something like data stripped across disks without parity. And that actually makes it less resilient, because failure of one component can bring down the whole system and the likelihood of failure is statistically higher because of the higher number of components.

re-thc 2y ago

Used a Co-pilot enabled PC.

the_biot 2y ago

Not necessarily massive. Given that Bing works again now, this seems more like an API frontend failure, or an internal routing failure at some level. Note they seem to have managed to fix the Bing frontend hours ago, but DDG is still dead in the water. Priorities... :-)

deleted 2y ago
[deleted]
pythux 2y ago
Bing does not work me at the moment (maybe they can service partial traffic? Unclear if it’s better/worse than a few hours ago when the outage started)

simias 2y ago

My subjective impression as a web user since the late 90s is that now things break relatively rarely (I think it's the first time I have any such issue with DDG for instance) but when they do a huge chunk of the web becomes unreachable. Back when things were more decentralized individual websites and services would have issues much more regularly because the individual software and hardware stacks weren't as robust and fault-tolerant, but then usually the problem would always be limited to a single website/service.

AbstractH24 2y ago
And America only uses the term “Too Big Toto Fail” when it comes to banks.
- lelanthran 2y ago
  > And America only uses the term “Too Big Toto Fail” when it comes to banks. And only when they're not in Kansas, anymore.