4 ms·

> The robots went silent. What about the traffic coming to our CDN sites from Facebook User-Agents? The gap is indisputable. We see about 30% of a typical reque

> The robots went silent. What about the traffic coming to our CDN sites from Facebook User-Agents? The gap is indisputable. We see about 30% of a typical request rate hitting us. But it's not zero; why is that? We'll let you know a little secret. Never trust User-Agent information; it's broken. User-Agent spoofing is everywhere. That is an interesting way of finding out how many spoof connections exist for Facebook and who the spoofers are. Just take down Facebook and all the traffic that says to be Facebook is spoofing. Simple as that.

jgrahamc 5y ago
Yes. And we are using that information to better target those bots. But guess what? iMessage adds Facebot and Twitterbot to its User—Agent when getting a preview when you send someone a link. So, as ever, shit’s complicated in the real world.
- detaro 5y ago
  Interesting, but makes sense. want to do previews, look like a preview bot - presumably there's also some sites that serve preview metadata only to such user agents.
addingnumbers 5y ago
You don't need anything to go down, just get a list of the subnets within their BGP ASN and filter those out, the rest are these spoofs. You can do this trivially at any time.
jefftk 5y ago
It's not necessarily spoofing. During the outage you could still open the FB app, at least on Android, and it would show you cached content. By default it opens links in the FB Browser, which was probably still working during the outage. So it shouldn't go to zero. Example in-app browser UA, pulled from my server logs just now: Mozilla/5.0 (Linux; Android 11; Pixel 3 XL Build/RQ3A.210905.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/93.0.4577.82 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/337.0.0.32.118;]
- cldellow 5y ago
  I don't think the user agents under discussion are the UAs sent by the Facebook browser, but rather the ones sent by Facebook's crawlers. The Facebook browser is distributed across all of humanity, so it'd be expected to have legitimate activity during the outage. The Facebook crawlers, however, are presumably all behind the blackholed Facebook network, so any request purporting to be from a Facebook crawler is likely up to no good.
- deleted 5y ago
  [deleted]

jgrahamc 5y ago

Yes. And we are using that information to better target those bots. But guess what? iMessage adds Facebot and Twitterbot to its User—Agent when getting a preview when you send someone a link. So, as ever, shit’s complicated in the real world.

detaro 5y ago
Interesting, but makes sense. want to do previews, look like a preview bot - presumably there's also some sites that serve preview metadata only to such user agents.

addingnumbers 5y ago

You don't need anything to go down, just get a list of the subnets within their BGP ASN and filter those out, the rest are these spoofs. You can do this trivially at any time.

jefftk 5y ago

It's not necessarily spoofing. During the outage you could still open the FB app, at least on Android, and it would show you cached content. By default it opens links in the FB Browser, which was probably still working during the outage. So it shouldn't go to zero. Example in-app browser UA, pulled from my server logs just now: Mozilla/5.0 (Linux; Android 11; Pixel 3 XL Build/RQ3A.210905.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/93.0.4577.82 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/337.0.0.32.118;]

cldellow 5y ago
I don't think the user agents under discussion are the UAs sent by the Facebook browser, but rather the ones sent by Facebook's crawlers. The Facebook browser is distributed across all of humanity, so it'd be expected to have legitimate activity during the outage. The Facebook crawlers, however, are presumably all behind the blackholed Facebook network, so any request purporting to be from a Facebook crawler is likely up to no good.
deleted 5y ago
[deleted]