23 ms·
A Facebook crawler was making 7M requests per day to my stupid website
- sradman 6y agoInteresting. According to the Facebook Crawler link provided by the OP [1], it makes a range request of compressed resources. I wonder if the response from the OP’s PHP/SQLite app isn’t triggering a crawler bug. Maybe Cloudflare can step up and act as an intermediary to isolate the problem. Isolating DoS attacks is one of their comparative advantages. [1] https://developers.facebook.com/docs/sharing/webmasters/crawler/ https://developers.facebook.com/docs/sharing/webmasters/craw...
- spiderfarmer 6y agoWas it a real FB crawler or a random buggy homemade bot masquerading as one? Check here: https://developers.facebook.com/docs/sharing/webmasters/crawler/ https://developers.facebook.com/docs/sharing/webmasters/craw...
- hinkley 6y agoSomeone up to no good using a false flag would not surprise me in the slightest. "who do people already like to hate? I'll pretend to be them."
- napolux 6y agoDoesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:2880:20ff:d::face:b00c https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...
- Wingy 6y agoIP packets have a source field, which can be fake and not their actual IP. That's a Facebook IP, but the packet might not have actually come from it.
- detaro 6y agoIf they made actual requests that went through, a connection got established. That won't happen with a faked source.
- thephyber 6y agoUDP packets can fake their source IP. TCP packets realistically can't.
- hinkley 6y agoMore precisely, spoofing a TCP handshake is a problem that we know how to prevent. Odds are pretty good that your kernel has these protections enabled by default, but it's not guaranteed and you should check as a matter of due diligence.
- draw_down 6y agoWhy is it so hard for people here to believe that Facebook could have a crawler misbehaving? It is all over this thread, even though multiple people are saying they saw the same thing and confirmed the IP belonged to FB. What gives?
- napolux 6y agoOne IP (ipv6) I just got from CloudFlare. https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a03%3A2880%3A20ff%3A1b%3A%3Aface%3Ab00c&as_sfid=AAAAAAVa_685ehdliHL-fKlh3v8lQwcfqf1MvjqW2qgkQyyv8g2-z-m-XBtLtF3u6thsALQbH7xaIC7fHiD2oxyY4P-1HH-WSJe-3D4Cxyl64YVHDn9XR25Hm8GScudbWCNEWUw%3D&as_fid=4cf20d4e5d982453717d1efee86a730cba0705ee https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...
- namanaggarwal 6y agoDid you check the ip and not just the UA? Does it matches with the one from fb?
- napolux 6y agoYes, one random ip from cloudflare: https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a03%3A2880%3A20ff%3A1b%3A%3Aface%3Ab00c&as_sfid=AAAAAAVa_685ehdliHL-fKlh3v8lQwcfqf1MvjqW2qgkQyyv8g2-z-m-XBtLtF3u6thsALQbH7xaIC7fHiD2oxyY4P-1HH-WSJe-3D4Cxyl64YVHDn9XR25Hm8GScudbWCNEWUw%3D&as_fid=4cf20d4e5d982453717d1efee86a730cba0705ee https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...
- ablanco 6y agoIt would be nice to understand how you came to the conclusion that this was a Facebook bot.
- akerro 6y agoYou can check list of IP addresses?
- huac 6y agothe OP did not (explicitly) check, but you can check if the IP falls into a range allocated to Facebook e.g. https://ipinfo.io/AS32934 https://ipinfo.io/AS32934
- napolux 6y agoOf course I've checked. Here is a recent example. https://news.ycombinator.com/item?id=23491455 https://news.ycombinator.com/item?id=23491455
- napolux 6y agoIp addresses. for example https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a03%3A2880%3A20ff%3A1b%3A%3Aface%3Ab00c&as_sfid=AAAAAAVa_685ehdliHL-fKlh3v8lQwcfqf1MvjqW2qgkQyyv8g2-z-m-XBtLtF3u6thsALQbH7xaIC7fHiD2oxyY4P-1HH-WSJe-3D4Cxyl64YVHDn9XR25Hm8GScudbWCNEWUw%3D&as_fid=4cf20d4e5d982453717d1efee86a730cba0705ee https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...
- k2xl 6y agoThis is actually a common tactic from malicious actors - pretend to be other bots to get through to websites. If you reverse lookup the IP address you can see if it part of the facebook network. UA is very untrustworthy
- napolux 6y agoHere is another one https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:2880:20ff:d::face:b00c https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...
- napolux 6y agoThis is one of the IPs https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a03%3A2880%3A20ff%3A1b%3A%3Aface%3Ab00c&as_sfid=AAAAAAVa_685ehdliHL-fKlh3v8lQwcfqf1MvjqW2qgkQyyv8g2-z-m-XBtLtF3u6thsALQbH7xaIC7fHiD2oxyY4P-1HH-WSJe-3D4Cxyl64YVHDn9XR25Hm8GScudbWCNEWUw%3D&as_fid=4cf20d4e5d982453717d1efee86a730cba0705ee https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...
- dylan604 6y agoWhy does FB need a crawler? Their users provide the content for their site. Is there an FB web search engine?
- youeseh 6y agoWhen you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.
- toomuchtodo 6y agoWhat happens if that link performs an action upon a GET request? Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.
- vasco 6y agoThat "link" doesn't respect the GET semantics then.
- thinkindie 6y agothat's your own problem because you are going against the HTTP protocol standard. GET should be idempotent.
- thaumasiotes 6y agoGoing against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately. The problem with unsafe GET is that it conflicts with reality, not that it conflicts with the standard.
- dragonwriter 6y ago> Going against the HTTP standard isn't a problem. For example, it's a good practice to ignore HEAD requests as opposed to responding appropriately. That's only against the standard of you advertise HEAD as a supported method on the resource, which converts it from a good idea in some circumstances to a bad one, so if there is a good example to support your claim, that isn't it.
- laurentdc 6y agoUnrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent? I wonder if you can bypass paywalls or see things that aren't generally presented to regular users
- spiderfarmer 6y agoSafari has this built in: https://osxdaily.com/2013/01/16/change-user-agent-chrome-safari-firefox/ https://osxdaily.com/2013/01/16/change-user-agent-chrome-saf...
- monkpit 6y agoYou chose to mention Safari, but link to an article about Chrome, Firefox, and Safari? Looks like all major browsers (probably Edge too) have this built in.
- danemacaulay 6y agohttps://github.com/iamadamdev/bypass-paywalls-chrome https://github.com/iamadamdev/bypass-paywalls-chrome
- Sohcahtoa82 6y agoYou wouldn't even need an extension. Both browsers allow you to change your User-agent. An extension just makes it a little easier. I have a UA switcher on Firefox so I can change my UA in two clicks. One of the built-in UA options is indeed the Googlebot.
- Ambroos 6y agoI surfed through a Google network for a bit a few years ago, and it does reveal things. The most common was Drupal and WordPress sites with some malicious code running that would link to online pharmacies on every page, but only to Google user agents / networks, I assume to boost their ranking. I let a bunch of webmasters know, most of them didn't believe me because they didn't see it themselves.
- leafo 6y agoSame thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834 https://twitter.com/moonscript/status/1124888489298808834 The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.
- _jal 6y agoThey've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off. Of course, sample size of 1, etc. It could have been coincidental.
- hinkley 6y agoTarpits are an underappreciated solution to a pool of bad actors. You can add artificial wait times to responses, or you can just route all of the 'bad' traffic to one machine, which becomes oversubscribed (be sure to segregate your stats!). All bad actors fighting over the same scraps creates proportional backpressure. Just adding 2 second delays to each request won't necessarily achieve that if multiple user agents are hitting you at once.
- apocalyptic0n3 6y agoI never looked into the TARPIT option in iptables before reading your comment. That seems really useful. I've been dealing with on and off bursts of traffic from a single AWS region for the last month. They usually keep going for about 90 minutes every day, regardless of how many IPs I block, and consume every available resource with about 250 requests per second (not a big server and I'm still waiting for approval to just outright block the AWS region). I'm going to try a tarpit next time rather than a DROP and see if it makes a difference.
- rocho 6y ago
- avian 6y agoThis reminds me of a case years ago when I had a small-ish video file on my server that would cause Google Bot to just keep fetching it in a loop. It wasted a huge amount of bandwidth for me since I only noticed it in a report at the end of the month. I had no way of figuring out what was wrong with it, so I just deleted the file and the bot went away.
- crazygringo 6y agoIf you can verify it's actually coming from Facebook's IP address (as many other comments here suggest), you should absolutely get in touch with them, though I'm not sure how. Perhaps their security team. That's a very serious bug that they'd be glad to catch -- if it's affecting you it's surely affecting others. Otherwise it's a bot/malware/etc. spoofing Facebook and gone wrong, which sucks. And yeah just block it by UA, and hopefully eventually it goes away.
- lun4r 6y agoWe've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove items from the catalog when their destination page is gone. Disabling this option solved the issue.
- napolux 6y agoThanks man! I'll have a look.
- cdown 6y agoHey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too. I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)
- deleted 6y ago[deleted]
- traverseda 6y agoI'm not sure I'd publicly post my email like that, if I worked at FB. But congratulations on your promotion to "official technical contact for all facebook issues forever".
- liveoneggs 6y agocheck these hits for an X-Purpose header or similar
- forgotmypw17 6y agoMaybe someone at Facebook is testing a project that didn't work as expected?
- hinkley 6y agoI wish we had better tools for things like this. What I've heard happen is someone puts a governor on the system, and much later your new big customer starts having errors because your aggregate traffic exceeds capacity planning you did 2 years ago and forgot about.
- president 6y agoI just thought of a malicious idea. If I were Amazon or some other cloud provider, I could hammer my customers' web sites with tons of requests to make them pay more for resource usage (network bandwidth, s3 calls, misc per/unit usage etc). It would be hard to trace as well. Wonder if people are already doing that today.
- eloff 6y agoThe amount of bad will that would generate when, not if, it came to light would dwarf the possible gains. So ethical concerns aside, it would be idiotic. Sometimes that doesn't stop the pointy haired bosses, because sometimes they are the perfect storm of unethical and moronic.
- lun4r 6y agoRelated thread at SO: https://stackoverflow.com/questions/49577546/facebook-crawler-is-hitting-my-server-hard-and-ignoring-directives-accessing-sa https://stackoverflow.com/questions/49577546/facebook-crawle...
- lun4r 6y agoAnd FB bug report: https://developers.facebook.com/support/bugs/1894024420610804/ https://developers.facebook.com/support/bugs/189402442061080...
- nickysielicki 6y agoIt bugs me that they require a login to see a bug report, especially in the case of this bug where those affected aren't necessarily facebook users.
- napolux 6y agoI've stumbled in that too.
- JoshMcguigan 6y agoThanks for sharing. I’d like to hear more about the website described here, it sounds very interesting.
- bberenberg 6y agoDoesn't ignoring headers and taking someones site down fall afoul of the CFAA? Especially given how comments here are showing that this is a recurring issue? Depending on which side of the fence you're on, there could be standing to go after FB for this either for money, or for their lawyers to help set a precedent to limit CFAA further.
- meritt 6y agoNo. The owner needs send Facebook a cease and desist and block their IPs in order to revoke authorization under the CFAA, and then Facebook would need to continue accessing his website despite explicitly being told not to. That's the precedent set by Craigslist v. 3Taps. Whether or not the above even holds today (hiQ Labs v. LinkedIn argues it does not) remains to be decided in a current appeal to the Supreme Court.
- yupyup54133 6y agoWhen someone does this to Facebook its malicious and they go to jail. WHen Facebook does it to someone else... "oops".
- yupyup54133 6y agoNever forget Aaron Swartz. All he did was send download requests to JSTOR. https://www.youtube.com/watch?v=9vz06QO3UkQ https://www.youtube.com/watch?v=9vz06QO3UkQ
- gregf 6y agoI'm going to get a ton of hate for this, but he was doing it with the intent to redistribute the content for free. He didn't own the content. There's a big difference. That being said it's very sad what came about of that. I really don't think the FBI needed to be involved.
- yupyup54133 6y ago> he was doing it with the intent to redistribute the content for free There is actually no proof of this.
- alpb 6y agoHate to say but this is probably a FB engineer running a test/experiment and not a production crawler taking robots.txt etc into account.
- 6510 6y agoHow do you monetize a robot?
- napolux 6y agoThat's a good question. Even better if Facebook can pay me some money :P
- machbio 6y agodont know whether the facebook bots execute javascript - https://www.rdegges.com/2017/how-to-monetize-your-website-with-cryptocurrency-mining/ https://www.rdegges.com/2017/how-to-monetize-your-website-wi...
- thephyber 6y agoEstablish contractual damages in a ToS for the site. Prove violation and offender. Take to court and collect damages. Converting the effort into cash is tough, but the strategy exists. Project HoneyPot is an API which allows any website to do this for honeypot email addresses which are injected the website, along with a ToS which says: > By continuing to access the Website, You acknowledge and agree that each email address the Website contains has a value not less than US $50 derived from their relative secrecy.[1] [1] https://www.projecthoneypot.org/terms_of_use.php https://www.projecthoneypot.org/terms_of_use.php
- the_jeremy 6y agoHas that ever worked? I can't find any record of judgments one way or the other, on their website or elsewhere.
- thephyber 6y agoI know ProjectHoneyPot was pushing a $x billion litigation against a spammer. I don't remember if/how that was resolved. This guy[1] apparently spent years just suing email spammers and occasionally winning. [1] https://www.danhatesspam.com/index.html https://www.danhatesspam.com/index.html
- amelius 6y agoWhat would happen if you created a "rabbithole" for webcrawlers? Would they fall for it?
- sgift 6y agoModern web crawlers usually have mechanisms against rabbit holes, e.g. for Apache Nutch you can configure it should only follow to a certain depth of links from a root, it should only follow a redirect a few times and so on. It's always a trade-off though. If you cut off too aggressive you don't get everything, if you cut off too late you waste resources.
- danschumann 6y agoIt's an interesting thing to think about, that someone can make a request to your website and you pay for it. This is not how the mail works, where someone needs to buy a stamp to spam you.
- thephyber 6y agoThis is a common pattern in (blocking i/o versus non-blocking i/o) and (sync task versus async task). The typical analogy I hear for DDoS attacks is if you own a business and a giant group of protesters drives traffic to your store, but no one is interested in buying. They create a backlog to enter the store, they crowd the isles, they pick up your inventory, and the lines and crowds scare off your legitimate visitors/customers.
- ortusdux 6y agoMy office still gets spam faxes to this day. The paper and toner only add up to a few cents a month, so it's not worth doing anything about. I knew a realtor that had a sheet of black paper with a few choice expletives written on it that they would send back to spammers. There was an art to taping it into a loop so it would continuously feed. This was a few decades ago when a the spam faxes could cost more than a stamp.
- mulmen 6y agoMy desk phone at an old job used to get dialed by a fax machine. Not fun picking that up. I redirected it to a virtual fax line and it turns out it was a local clinic faxing medical records. I faxed them back with some message about you have the wrong number but they never stopped.
- fennecfoxen 6y agoIf you really wanted to make them stop, you might find a contact for their lawyer and make HIPAA noises at them ;)
- 6y ago
- lclarkmichalek 6y agoMake sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.
- napolux 6y agoWhere? Here? https://developers.facebook.com/support/bugs/ https://developers.facebook.com/support/bugs/
- falcolas 6y agoWhy is this the owner's problem? Someone at Facebook should be filing the bug, or better yet instrumenting their systems so that incidents like this issue a wake-up-an-engineer alert. Fuck this culture of "it's up to the victim of our fuckup to file a bug report with us".
- nhooyr 6y ago100%
- mulmen 6y agoNot just “file a bug report” but also compile a time series graph (lol) and then pray that Facebook triages it correctly, which they have no incentive to do.
- rurp 6y agoAnd to make things even more ridiculous, you need to sign in with a facebook account to even file a bug there.
- shultays 6y agoisn't this what robots.txt for? can't you block facebook?
- jokoon 6y agoThat's 81 request per second on average. Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages? That sounds like a lot of bandwidth (and server stress).
- napolux 6y agoI saw also of 300reqs per second.
- Traster 6y agoYes, this was my first thought too - Facebook is essentially denial of servicing people's sites and it sounds like they're aware of it. Move fast and break your things. If this were your average Joe I'm fairly certain something like the DMCA could be used.
- sprayk 6y agoWhat? How does the DCMA apply here at all?
- sirclueless 6y agoThe DMCA has a couple major parts. The part most people are familiar with deals with safe harbor provisions and "DMCA takedowns". The relevant part for this discussion makes it illegal to circumvent digital access-control technology. So you could argue that if you have reasonable measures in place to stop automated access of your content such as robots.txt then Facebook continuing to send automated requests would be a circumvention of your access controls.
- jrockway 6y agoShouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it? (I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated requests, for things like "link unfurling" in things like Slack. I am guessing the crawler ignores robots.txt because it is doing the request on behalf of a human user, not to create some sort of index. Google is attempting to standardize this widely-understood convention: https://developers.google.com/search/reference/robots_txt https://developers.google.com/search/reference/robots_txt)
- rcfox 6y agoDid anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c "face:b00c"
- sprayk 6y agoif only they could have convinced IANA to give them a vanity prefix.
- napolux 6y agoThe mighty power of IPV6 :P
- tombrossman 6y agoMy "404-pests" fail2ban-client status, which drops everything making *.php requests (This machine has never had PHP installed...): 2a03:2880:10ff:14::face:b00c 2a03:2880:10ff:21::face:b00c 2a03:2880:11ff:1a::face:b00c 2a03:2880:11ff:1f::face:b00c 2a03:2880:11ff:2::face:b00c 2a03:2880:12ff:10::face:b00c 2a03:2880:12ff:1::face:b00c 2a03:2880:12ff:9::face:b00c 2a03:2880:12ff:d::face:b00c 2a03:2880:13ff:3::face:b00c 2a03:2880:13ff:4::face:b00c 2a03:2880:20ff:12::face:b00c 2a03:2880:20ff:1e::face:b00c 2a03:2880:20ff:4::face:b00c 2a03:2880:20ff:5::face:b00c 2a03:2880:20ff:75::face:b00c 2a03:2880:20ff:77::face:b00c 2a03:2880:20ff:e::face:b00c 2a03:2880:21ff:30::face:b00c 2a03:2880:22ff:11::face:b00c 2a03:2880:22ff:12::face:b00c 2a03:2880:22ff:14::face:b00c 2a03:2880:23ff:5::face:b00c 2a03:2880:23ff:b::face:b00c 2a03:2880:23ff:c::face:b00c 2a03:2880:30ff:10::face:b00c 2a03:2880:30ff:11::face:b00c 2a03:2880:30ff:17::face:b00c 2a03:2880:30ff:1::face:b00c 2a03:2880:30ff:71::face:b00c 2a03:2880:30ff:a::face:b00c 2a03:2880:30ff:b::face:b00c 2a03:2880:30ff:c::face:b00c 2a03:2880:30ff:d::face:b00c 2a03:2880:30ff:f::face:b00c 2a03:2880:31ff:10::face:b00c 2a03:2880:31ff:11::face:b00c 2a03:2880:31ff:12::face:b00c 2a03:2880:31ff:13::face:b00c 2a03:2880:31ff:17::face:b00c 2a03:2880:31ff:1::face:b00c 2a03:2880:31ff:2::face:b00c 2a03:2880:31ff:3::face:b00c 2a03:2880:31ff:4::face:b00c 2a03:2880:31ff:5::face:b00c 2a03:2880:31ff:6::face:b00c 2a03:2880:31ff:71::face:b00c 2a03:2880:31ff:7::face:b00c 2a03:2880:31ff:8::face:b00c 2a03:2880:31ff:c::face:b00c 2a03:2880:31ff:d::face:b00c 2a03:2880:31ff:e::face:b00c 2a03:2880:31ff:f::face:b00c 2a03:2880:32ff:4::face:b00c 2a03:2880:32ff:5::face:b00c 2a03:2880:32ff:70::face:b00c 2a03:2880:32ff:d::face:b00c 2a03:2880:ff:16::face:b00c 2a03:2880:ff:17::face:b00c 2a03:2880:ff:1a::face:b00c 2a03:2880:ff:1c::face:b00c 2a03:2880:ff:1d::face:b00c 2a03:2880:ff:25::face:b00c 2a03:2880:ff::face:b00c 2a03:2880:ff:b::face:b00c 2a03:2880:ff:c::face:b00c 2a03:2880:ff:d::face:b00c
- chirau 6y agoI think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.
- Accacin 6y agoSomeone purporting to be working in SEO and getting hit by 7M requests, but offers not real proof and cannot link the webpage either? Am I just getting too suspicious? Edit: Was trying to make a (bad) joke.
- napolux 6y agoAsk me anything. That specific website is a personal project of mine which I'd like not to disclose. As you can see from my blog I'm not selling anything, I don't even have a banner on my blog, so where your suspect is coming from?
- bkanber 6y ago> Am I just getting too suspicious? Yes, I think so. What's the conspiracy supposed to be here?
- tlrobinson 6y ago> And then they’ll probably ask you to pay the bill Can anyone comment on whether there would be any legal basis for this?
- ruuda 6y agoHi Napolux, It looks like your site is using a theme based on my website (https://ruudvanasseldonk.com/ https://ruudvanasseldonk.com/, source at https://github.com/ruuda/blog https://github.com/ruuda/blog). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?
- RhodesianHunter 6y agoDespite how it sounds, I ask this with zero judgment and pure curiosity. Why do you care?
- kungato 6y agoWell I respect the question of GP if nothing then for principle. People should get their shit together and start respecting the license. The state today is horrendous and the culture is evolving into less and less respect of the license. If people don't follow them personally they wont care to follow them professionally either
- timy2shoes 6y agoPlease read https://sfconservancy.org/copyleft-compliance/principles.html https://sfconservancy.org/copyleft-compliance/principles.htm.... My understanding is that if you do not enforce your copyright (or copyleft in this case) you can lose the copyright.
- 6y ago
- annoyingnoob 6y agofacebook - scourge of the internet
- staycoolboy 6y agoI'm new to webadmining in the cloud and my website is getting hammered by baidu, google, fb, et others causing traffic I/O costs to increase. What's an AWS LoadBalancer way of blocking this traffic? Again, noob here. THanks.
- namibj 6y agoDon't host this stuff on AWS if you care about cost.
- staycoolboy 6y agoSo you don't know how to do this in AWS is what I'm hearing?
- namibj 6y agoMy point is, that you don't want to do this inside of a load balancer there. I don't recall any traffic filtering abilities that would suffice, but I'm not fully up-to-date with the configurability either. If the load balancer supports it, a short search in the net or docs should surface an easily-applicable guide, and if not, I'd probably put that blocking closer to my app server. And the reason against AWS for this would be both the general cost (AWS is not cost-efficient in many cases, unless you have complicated infrastructure that takes a lot of management, where the Infrastructure-as-code approach can give you a sizable benefit), the bandwidth cost in particular, and the lack of configurability of their services to e.g. apply suitable tar-pitting against such crawlers.
- pmlnr 6y agoAre you sure it's a crawler and not a proxy of some sort? Eg. one of your links is on something high traffic in facebook, and all requests are human, running through fb machines.
- napolux 6y agoNah, it's a specific user-agent for their crawler
- pier25 6y agoShouldn't have Cloudflare considered this a DDOS attack?
- TheChaplain 6y agoReading through the article and comments here I find it odd that a company like FB can't get a crawler to respect robots.txt or 429 status codes. Even I would stop and think "maybe I should put in some guards against large amounts of traffic" when writing a crawler, and I'm certainly not one of those brilliant minds who manage to pass their interview process.
- callalex 6y agoIf the dev respected others’ boundaries and norms of expected behavior, they would probably work somewhere else.
- tejas5 6y agoDid you reach out to the fb contact?
- tejas5 6y agoDo you have some sort of fb plugin in your SEO site?
- snorrah 6y agoSurprise, Facebook not caring about playing nice with others! Anyway, what reason would they have for so many requests? Why would they need to do 300 per second?
- nraynaud 6y agoWe had that with google in the past. We put a kml database on a web server and made the google maps API read them. Google went to get them without any throttling (my guess is that they primed their edge servers straight from us), the server went down so fast we sought there had been a hardware failure. We ended up putting a cdn just for google servers.
- _drimzy 6y agoWhy would a crawler want to crawl the same page every second (let alone 71 times a second). Even if I were Google, wouldn't crawling any page every few mins (or say every minute) be way more than enough for freshness?
- cascader 6y agoIs it worth duplicating the fb specific parts of your website at another web address as a test site, so that If it was similarly affected, you could provide the address info to help troubleshoot?
- itsjloh 6y agoWhen I enabled IPv6 traffic at home and logged default denies I constantly see Facebook owned IPv6 blocks trying to reach addresses browsing Facebook. I have no idea what its trying to do but its legitimately the only IPv6 inbound traffic that isn't related to normal browsing.
- hank_z 6y agoI’m a bit confused. Facebook is social media company, why do they send out crawlers?
- thenickdude 6y agoFacebook shows previews of pretty much everything you share a link to, so at a minimum they have to fetch the page/image to build those previews.
- hank_z 6y agoI see, thanks for the explanation.
- shadykiller 6y agoThen your website isn't stupid but rather the FB crawler
- keevitaja 6y agowe had a similar experience with yandex. had to block it. crawler started to crawl on a larger e-shop by trying out every single possible filter combination. in other words we got ddos attack from yandex.
- chrismarlow9 6y agobit rot? 1 off anywhere in a DNS request could slam someone
- dan15 6y agoFacebook caches URLs pretty aggressively (often you need to explicitly purge the cache to pick up updates) so I don't quite understand exactly what happened here. The article is very light on details. Do you have 7 million unique URLs that have been shared on Facebook? Or is it the same URL being scraped over and over? Are you correctly sending caching headers?
- drewmol 6y agoTitle change? A stupid Facebook crawler was making 7M requests per day to my website
- napolux 6y agoNice one. ;)
- accurrent 6y agoWhy does facebook have crawlers in the first place?
- omershapira 6y agoI once made a PHP page which fetches a random wikipedia article, but changes the title to "What the fuck is [TOPIC]?". It was incredibly funny, until I started getting angry emails from people, mostly threatening a form of libel lawsuit. Turns out, since it was a front for all of wikipedia[1], google was agressively indexing it, but the results rarely made it to the first search page. And since this isn't exactly an important site, old results would stick around. Hence a pattern: 1. Some Rando creates a page about themselves 2. Wikipedia editors, being holy and good, extinguish that nonsense 1.5. GOOGLE INDEXES IT 3. Rando, by nature of being a rando, googles themselves, doesn't find their wikipedia page anymore (that's gone), but does find a link to my site in the first page of google results with their name. 4. Lawyers get involved, somehow Details: http://omershapira.com/blog/2013/03/randomfax-net/ http://omershapira.com/blog/2013/03/randomfax-net/ [1] Hebrew wikipedia. I can't imagine what would've happened on the English version.
- smileybarry 6y agoI understand why people got angry, though. The title was rougly "What is this '<article name>' shit?"[1], with "shit" in this Hebrew context also able to be interpreted as calling the subject "shit". Saying "מה זה לעזאזל X?"[2] is closer to "what the fuck is X?" (except it's more "what the hell" than "what the fuck"). [1] I wasn't sure how to portray this to non-Hebrew speakers but, surprisingly, Google Translate actually nailed it: https://translate.google.com/#view=home&op=translate&sl=auto&tl=en&text=%D7%9E%D7%94%20%D7%96%D7%94%20%D7%94%D7%97%D7%A8%D7%90%20%D7%94%D7%96%D7%94%20%22%D7%90%D7%A4%D7%9C%22%3F https://translate.google.com/#view=home&op=translate&sl=auto... [2] Google Translate got this example right, too: https://translate.google.com/#view=home&op=translate&sl=auto&tl=en&text=%D7%9E%D7%94%20%D7%96%D7%94%20%D7%9C%D7%A2%D7%96%D7%90%D7%96%D7%9C%20%22%D7%90%D7%A4%D7%9C%22%3F https://translate.google.com/#view=home&op=translate&sl=auto...
- Too 6y ago> I own a little website I use for some SEO experiments [...] can generate thousands of different pages. Do you have more details on this? One could read that as if the page is designed to attract crawlers.
- kuu 6y agoI wonder what the crawler does to it requires so many requests per day to the same site...
- magwa101 6y agoTheir 2FA page was broken yesterday...it goes on
- historyremade 6y agoSomeone could use this for DDOS or Coinhive
- zippy786 6y agoThere was a very old post: https://news.ycombinator.com/item?id=7649025 https://news.ycombinator.com/item?id=7649025, lots of ways to abuse Facebook network.
- hodgesrm 6y agoThis is one of the better HN threads in a while. Starting from an easily described problem, it goes into a number of interesting directions including how FB seems to DDOS certain sites for reasons unknown, defenses against such attacks and DDOS in general, GPL licensing, how to collapse HN conversations, etc. Not a profound comment. But it was really fun to follow the path down the various rabbit holes. This is why I like HN.
- mesozoic 6y agoDetect the crawler and delay the returns as long as you can. Have it use as many resources as you can that should be interesting.