2 ms·

Regardless. You should still respect robots.txt..

• andrew_zhong 6mo ago
  We do respect robots.txt production - also scraping browser providers like BrightData enforces that. I will add a PR to enforce robots.txt before the actual scraping.
  • plastic041 6mo ago
    How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?
    • andrew_zhong 6mo ago
      I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.