5 ms·
Good points. According to OP, the goal would be to persuade FB to back off crawling at 300 qps, not to bring down the FB crawl. Having each request expand to 1G
by golem14 6y ago
Good points. According to OP, the goal would be to persuade FB to back off crawling at 300 qps, not to bring down the FB crawl. Having each request expand to 1GB is probably already enough to do that, since a crawler implementing the mitigations you mentioned will likely reduce the crawl rate. Should be easily testable.