3 ms·

Wait, so a simple web scraper script has to comply with robots.txt. But if I want to completely ignore the robots.txt, I only have to make my script more compli

• seanthemon 3y ago
  And make it complicated enough (a human) and no .txt can stop them!
• kelseyfrog 3y ago
  Yeah, it's like one weird trick for web scraping.
• pixl97 3y ago
  I'd like to consider this a difference between script action and user action. For example if you make a web page a user pulls up that calls another webpage, is that a user action, a script action, a mix of both? I personally would consider it a user action.
• JohnFen 3y ago
  Nobody actually has to comply with robots.txt.