4 ms·
there's no way you can convince me that if you're building a bot to scrape websites, you've never heard of robots.txt.
by dylan604 3y ago
there's no way you can convince me that if you're building a bot to scrape websites, you've never heard of robots.txt.