4 ms·

I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organiza

by snazz 8y ago

I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my workplace. This leaves me wondering what software all these places have in common. The application layers are all different, the databases are all different, the containerization and provisioning systems are different, but I imagine that all these systems rely on two things: the global Internet backbone, and maybe the Linux kernel. Have there been major security vulnerabilities patched lately in the Linux kernel that could have had unintended consequences?

shafte 8y ago
The best explanation is coincidence, I think. I have direct knowledge of two of the incidents in the past few weeks, and they have completely unrelated causes. Sometimes you just get unlucky!
- snazz 8y ago
  That’s certainly possible. We’re probably still too early to tell, but the innate conspiracy theorist slash pattern-matching part of my brain wants to find a probable connection.
- smacktoward 8y ago
  "Once is happenstance. Twice is coincidence. The third time it’s enemy action." -- Ian Fleming (in Goldfinger)
- z3t4 8y ago
  If it can happen, it will happen.
str33t_punk 8y ago
Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques. It is one of the harder problems to solve and requires massive investment in tooling, refactoring, etc.
- snazz 8y ago
  You’re undeniably right about not looking at Facebook or Google as one whole system, but there have also been what seems like an unprecedented number of strange little outages (see the ones mentioned by https://news.ycombinator.com/item?id=19382418 https://news.ycombinator.com/item?id=19382418) that aren’t huge companies. My workplace had some of their own today that I haven’t heard an incident report about (it’s a pretty large company and I’m not in IT).
- hideo 8y ago
  >>Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques Curious what makes you think this. Are there specific job postings in either company that are focused on this?
  - bpye 8y ago
    I work for Microsoft, I know of at least CrystalNet [1]. [1] - https://www.microsoft.com/en-us/research/blog/eliminating-network-downtime/ https://www.microsoft.com/en-us/research/blog/eliminating-ne...
madrox 8y ago
> This leaves me wondering what software all these places have in common. Cisco or Arista
- happythought 8y ago
  FANGs all use white box hardware with “merchant silicon” meaning they buy the chipsets directly from Broadcom, Mellanox, etc. and build their own devices. However, they do all have Broadcom and Mellanox in common and Cisco, juniper, and arista do too.
  - nwlieb 8y ago
    Some FANGs definitely use Arista/Cisco. As far as I know white box hardware is mostly for top of the rack switches (as opposed to backbone infra).
    - packetslave 8y ago
      Depends on which backbone infra :-) Take a look at the B4 and Jupiter papers from Google, for example.
felix246 8y ago
It could also just be coincidence.
taneq 8y ago
Maybe we're in the first week of a rogue AI's hard takeoff. ;)
nubslayer2 8y ago
>all of these organizations have fairly diverse tech stacks to my knowledge >This leaves me wondering what software all these places have in common. dunno what systems you're talking about, but seems likely they are mostly x86 systems and maybe even mostly using Intel hardware and microcode those systems can-be/are rooted and more, to my knowledge

by snazz 8y ago

shafte 8y ago

The best explanation is coincidence, I think. I have direct knowledge of two of the incidents in the past few weeks, and they have completely unrelated causes. Sometimes you just get unlucky!

snazz 8y ago
That’s certainly possible. We’re probably still too early to tell, but the innate conspiracy theorist slash pattern-matching part of my brain wants to find a probable connection.
smacktoward 8y ago
"Once is happenstance. Twice is coincidence. The third time it’s enemy action." -- Ian Fleming (in Goldfinger)
z3t4 8y ago
If it can happen, it will happen.

str33t_punk 8y ago

Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques. It is one of the harder problems to solve and requires massive investment in tooling, refactoring, etc.

snazz 8y ago
You’re undeniably right about not looking at Facebook or Google as one whole system, but there have also been what seems like an unprecedented number of strange little outages (see the ones mentioned by https://news.ycombinator.com/item?id=19382418 https://news.ycombinator.com/item?id=19382418) that aren’t huge companies. My workplace had some of their own today that I haven’t heard an incident report about (it’s a pretty large company and I’m not in IT).
hideo 8y ago
>>Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques Curious what makes you think this. Are there specific job postings in either company that are focused on this?
- bpye 8y ago
  I work for Microsoft, I know of at least CrystalNet [1]. [1] - https://www.microsoft.com/en-us/research/blog/eliminating-network-downtime/ https://www.microsoft.com/en-us/research/blog/eliminating-ne...

madrox 8y ago

> This leaves me wondering what software all these places have in common. Cisco or Arista

happythought 8y ago
FANGs all use white box hardware with “merchant silicon” meaning they buy the chipsets directly from Broadcom, Mellanox, etc. and build their own devices. However, they do all have Broadcom and Mellanox in common and Cisco, juniper, and arista do too.
- nwlieb 8y ago
  Some FANGs definitely use Arista/Cisco. As far as I know white box hardware is mostly for top of the rack switches (as opposed to backbone infra).
  - packetslave 8y ago
    Depends on which backbone infra :-) Take a look at the B4 and Jupiter papers from Google, for example.

felix246 8y ago

It could also just be coincidence.

taneq 8y ago

Maybe we're in the first week of a rogue AI's hard takeoff. ;)

nubslayer2 8y ago

>all of these organizations have fairly diverse tech stacks to my knowledge >This leaves me wondering what software all these places have in common. dunno what systems you're talking about, but seems likely they are mostly x86 systems and maybe even mostly using Intel hardware and microcode those systems can-be/are rooted and more, to my knowledge