23 ms·
We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (u
by lun4r 6y ago
We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks.
I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem.
I found out that there is an option in the FB Catalog manager that lets FB auto-remove items from the catalog when their destination page is gone. Disabling this option solved the issue.
- napolux 6y agoThanks man! I'll have a look.
- cdown 6y agoHey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
- deleted 6y ago[deleted]
- traverseda 6y agoI'm not sure I'd publicly post my email like that, if I worked at FB. But congratulations on your promotion to "official technical contact for all facebook issues forever".
- cdown 6y agoMy e-mail address is already public from my kernel commits and upstream work. :-)
- doommius 6y agoDon't think I used my email for anything important doing my time at FB. If it gets out of hand he could just make a request to have a new primary email made and use the above one for "spam"
- thirtythree 6y agoCuriousity question: does FB use Gmail/Google suite?
- callalex 6y agoMy impression is that they pretty much roll their own communication suite.
- rock_hard 6y agoThat’s somewhat correct But at least for email/calendar backend its exchange The internal replacement clients for calendar and other things are killer...have yet to find replacements For the most part though they use Facebook internally for messaging and regular communication (technically now Worplace but before it was just Facebook) Email is really just for external folks
- dan15 6y agoFB uses Office365 for email. It was on-premise Exchange many many years ago, but moved "to the cloud" a while back.
- hirako2000 6y agoFeels odd to read Facebook uses office365/exchange for emails. they haven't built their fsuite yet, I thought they would simply promote Facebook messenger internally. I'm only half joking.
- dan15 6y agoMost communication is via Workplace (group posts and chat). Emails aren't very common any more - mainly for communication with external people and alerting.
- napolux 6y agoI don't want to share my website for personal reasons, but here is some data from cloudflare dashboard (a request made on 11 Jun, 2020 21:30:55 from Ireland, I have 3 requests in the same second from 2 different IPs) user-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php http://www.facebook.com/externalhit_uatext.php) ip 1: 2a03:2880:22ff:3::face:b00c (1 request) ip 2: 2a03:2880:22ff:b::face:b00c (2 requests) ASN: AS32934 FACEBOOK
- napolux 6y agoYes requests are still coming. Thanks CloudFlare for saving my ass.
- fsociety 6y agoThanks for looking at this!
- qroshan 6y agoyeah, for all you know, it could just a naive/rogue employee building a crawler for his personal/20% project
- purple_ferret 6y agoPlease ask Zuck to resign while you're at it. Thanks.
- Ueland 6y agoHey, I can by the way confirm this issue. I work in a large newspaper in Norway and around a year ago we saw the same issue. Thousands of requests per second until we blocked it . And after we blocked it, traffic to our Facebook page also plummeted. I assume Facebook considered our website down and thus wouldn't give users content from our Facebook page either as that would serve them content that would give a bad user experience. The Facebook traffic did not normalize before the attack stopped AND after we told Facebook to reindex all our content. I'd you want more info, send me a email and il dig out some logs etc. thu at db.no
- pera 6y agoI just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address. By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.
- gerdesj 6y agoNice idea, I like the thinking. I'll tuck that away for use later. PowerDNS has LUA built in amongst a few other things. My stack of projects to do is growing at a hell of a rate and I'm not popping them off the stack fast enough.
- verdverm 6y agoI know the feeling, it's one of the reasons I'm working on https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof Check out the code generation parts and modules, they are the most mature. We have HRDs (like CRDs in k8s for anything) and a scripting language between bash and Python coming out soon too.
- gerdesj 6y agoI've tried to work out what your project does but I'm none the wiser. GEB is prominent on my bookshelf. I'm a syadmin and I got as far as "hollow wold" in Go or was it "Hail Marrow"? Can't remember. I've checked out your repo for a look over tomorrow when I'm cough sober!
- verdverm 6y agoStop by gitter and I'd be happy to explain more
- foobiekr 6y agoif only there was a way to accidentally amplify that.
- 6y ago
- smallgovt 6y ago>> The bots didn't identify as FB (used "spoofed" UAs) That's surprising. What were the spoofed user agents that they used? We've run into this issue also, but all Facebook bot activity had user agents that contained the string "facebookexternalhit".
- rob-olmos 6y agoI've seen these these two user-agents from FB IPs, maybe others: Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53 Mozilla/5.0 (iPhone; CPU iPhone OS 13_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1 Which also execute javascript in some modified sandbox or something, causing errors, and executing error handlers. Interesting attempt to analyze the crawler here: https://github.com/aFarkas/lazysizes/issues/520#issuecomment-484527859 https://github.com/aFarkas/lazysizes/issues/520#issuecomment...
- lun4r 6y agoYep, these were the UAs we also saw (amongst others). And also in our case those bots were executing the JS, even hitting our Google Analytics. For some reason GA reported this traffic to come from Peru and Philippines, while an IP lookup showed it belonged to FB registered in the US or Ireland
- paulpauper 6y agotwitter does the same thing. it sends a bunch of spoofed US visitors from Korea and Germany and US. The bots are spoofed to make it harder to filter them.
- rapind 6y agoI wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?
- detaro 6y agoPossibly trying to avoid people sending them a different version of the page than users would see (of course they could change the page after the initial caching of a preview, but Twitter might refresh/check them later). Also, you often need an impressive amount of the stuff thats in a normal UA string for random sites to not break/send you the "unsupported browser, please use Netscape 4 or newer!!!" page/..., although you normally can fit an identifier of what you really are at the end. (As an example, here's an iOS Safari user agent: "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1" - it's not Mozilla, it's not Gecko, but it has those keywords and patterns because sites expect to see that)
- rapind 6y agoThat makes sense. I couldn't come up with a shady reason why they would do it to be honest, but I was curious.
- yjftsjthsd-h 6y agoYeah, I once tried to tell my browser to send... I forget; either no UA, or a blank UA string, or nonsense or just "Firefox" or something. I figured, "hey, some sites might break, but it can't be that important!" It broke everything. IIRC, the breaking point was that my own server refused to talk to me. Now, I still think this is insane, but apparently this really is how it is right now.
- stiray 6y agoI dont really understand what is the issue. On my welcome page (while all other urls are impossible to guess) i give browser something that requires a few seconds of cpu at 100% to crunch. And tracking some user action in between, visting tarpitted urls etc. In last few years no bot came through. Why bother with robots.txt, just give them something to break their teeths... (I would give you the url, but I just dont want ti be visited)
- cblades 6y agoI don't think most users would appreciate having a site spike their CPU for a few seconds when they visit...at least I wouldn't.
- aetherspawn 6y agoI don’t think the solution is flatlining everyone’s CPUs.
- trashburger 6y agoSounds like terrible UX! Won't browsers give you a "Stop script" prompt if you do that?
- xg15 6y ago> but I just dont want ti be visited Maybe you should just take your website offline?
- quickthrower2 6y agoHow much have you mined?
- hombre_fatal 6y agoThe far-fetched "solution" along with the "I don't understand everyone doesn't do <thing that is easy to understand why everyone wouldn't do it>" make it hard for me to believe you're actually doing this. Sounds more like a shower thought, maybe fun weekend project, than something in production.
- wheresmycraisin 6y agoSide question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.
- missedthecue 6y agoReally? I've never had problems contacting them by email. They're one of the easiest tech companies to talk to.
- redeux 6y agoIn my experience they're one of the most useless and difficult companies I've ever tried to interact with.
- corobo 6y agoThe entirety of the universe is contained in the preceding two comments.
- peanutz454 6y agoAah, finally it all makes sense!
- lvs 6y agohttps://twitter.com/james_kpatrick/status/320150923336892416 https://twitter.com/james_kpatrick/status/320150923336892416
- deadwing0 6y agoWhat did you contact them for? Just curious as I've almost always heard they're like Google and impossible to get a human response from.
- wheresmycraisin 6y agoWhat is their email? I've never found one that they'd respond to.
- adrianN 6y agoSet up a robots.txt that disallows Facebook crawlers, sue Facebook if the crawling continues for unauthorized access to computer systems, profit.
- albatross 6y agoI believe this only works if it is coming from a corporation and targeted at an individual -_-
- justinph 6y agorobots.txt is not a legal document. It is asking nicely, and plenty of crawlers purposefully ignore it.
- adrianN 6y agoI'm not a lawyer, but I believe I remember people being sued for essentially making a GET request to a URL they weren't supposed to GET.
- greggyb 6y agoLegal document is a tricky phrase to use. "No trespassing" signs are usually considered sufficient to justify prosecution for trespassing. If the sign is conspicuously placed, it does not usually matter if you actually see the sign or not. I am not as familiar with law around accessing computer systems, but I imagine that given some of the draconian enforcement we've seen in the past that a robots.txt should be sufficient to support some legal action against someone who disregards it.
- dungdang 6y agothe no tresspassing sign does not mean i can't yell at you from the street 'tell me your life story.' which you are free to ignore.
- greggyb 6y agoSure, but federal law does prevent you from repeatedly placing that request into my mailbox. I don't even need a sign for that. Making an http request does not fit cleanly as an analogue to yelling from the street, nor does it fit as an analogue to throwing a written request on a brick through a window. It is something different that must be understood on its own terms.
- 6c696e7578 6y agoThank goodness for mod_rewrite, which makes blocking/redirecting traffic on basic things like headers pretty easy. https://www.usenix.org.uk/content/rewritemap.html https://www.usenix.org.uk/content/rewritemap.html You could of course block upstream by IP, but if you want to send the traffic away from a CPU heavy dynamic page to something static that 2xx's or 301's to https://developers.facebook.com/docs/sharing/webmasters/crawler/ https://developers.facebook.com/docs/sharing/webmasters/craw... then this could be the answer.
- lun4r 6y agoHere are some more details from my report to FB: "My webserver is getting hit with bursts of hundreds of requests from Facebook's IP ranges. Google Analytics also reports these hits and shows them as coming from (mostly) Philippines and Peru, however, IP lookup shows that these IPs belong to Facebook (TFBNET3). The number of these hits during a burst typically exceeds my normal traffic by 200%, putting a lot of stress at our infrastructure, putting our business at risk. This started happening after the Facebook Support team resolved a problem I reported earlier regarding connecting my Facebook Pixel as a data source to my Catalog. It seems Facebook is sending a bot to fetch information from the page, but does so very aggressively and apparently call other trackers on the page (such as Google Analytics)" 69.171.240.19 - - [13/Aug/2018:11:09:52 +0200] "GET /items/ley3xk/ford-dohc-20-sierra-mondeo-scorpio-luk-set.html HTTP/1.1" 200 15181 "https://www.facebook.com/" https://www.facebook.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134" "e.g. IP addresses 173.252.87.* performed 15,211 hits between Aug 14 12:00 and 12:59, followed by 13,946 hits from 31.13.115.*" "What is also interesting is that the user agents are very diverse. I would expect a Facebook crawler to identify itself with a unique User-Agent header (as suggested by the documentation page mentioned earlier), but instead I see User-Agent strings that belong to many different browsers. E.g. this file contains 53,240 hits from Facebook's IP addresses with User-Agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134" There are a few Facebook useragents in there, but far less than browser useragents: 7,310 hits: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php http://www.facebook.com/externalhit_uatext.php) 2,869 hits: facebookexternalhit/1.1 1,439 hits: facebookcatalog/1.0 120 hits: facebookexternalua Surprisingly, there is even a useragent string that mentions Bing: 6,280 hits: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b These IPs don't only fetch the HTML page, but load all the page's resources (images, css, ..) including all third-party trackers (such as Google Analytics). Not only does this put unnecessary stress at our infrastructure, it drives up the usage costs of 3rd party tracking services and renders some of our reports unreliable." final response from FB: "Thanks for your patience while our team looked into this. They've added measures to reduce the amount of crawler calls made. Further optimizations are being worked on as well, but for now, this issue should be resolved." <- NOT.