4 ms·

Can large companies not be faulted for ignoring robots.txt? Seems like something GDPR could enforce for personal(ly owned) sites?

• deleted 2y ago
  [deleted]
• jsnell 2y ago
  Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider. https://www.nerdcrawler.com/robots.txt https://www.nerdcrawler.com/robots.txt The domain serving the images is allowing everything: https://res.cloudinary.com/robots.txt https://res.cloudinary.com/robots.txt
  • catoc 2y ago
    I just assumed that would have been the first thing they tried…
  • chptung 2y ago
    So I decided to google around and I think TikTok does ignore robots.txt. A few posts from around: - https://www.reddit.com/r/flask/comments/161fqml/what_to_do_about_bytespider/ https://www.reddit.com/r/flask/comments/161fqml/what_to_do_a... - https://www.reddit.com/r/sysadmin/comments/1ahhzdg/has_anyone_noticed_odd_bytedance_web_crawler/ https://www.reddit.com/r/sysadmin/comments/1ahhzdg/has_anyon...