3 ms·
Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider. https://www.nerdcrawler
by jsnell 2y ago
Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider.
https://www.nerdcrawler.com/robots.txt https://www.nerdcrawler.com/robots.txt
The domain serving the images is allowing everything:
https://res.cloudinary.com/robots.txt https://res.cloudinary.com/robots.txt
- catoc 2y agoI just assumed that would have been the first thing they tried…
- chptung 2y agoSo I decided to google around and I think TikTok does ignore robots.txt. A few posts from around: - https://www.reddit.com/r/flask/comments/161fqml/what_to_do_about_bytespider/ https://www.reddit.com/r/flask/comments/161fqml/what_to_do_a... - https://www.reddit.com/r/sysadmin/comments/1ahhzdg/has_anyone_noticed_odd_bytedance_web_crawler/ https://www.reddit.com/r/sysadmin/comments/1ahhzdg/has_anyon...