5 ms·
Your browser is also automating things, you're not resolving DNS and writing all the http requests by hand.
by rolisz 3y ago
Your browser is also automating things, you're not resolving DNS and writing all the http requests by hand.
- worthless-trash 3y agoI have found the next topic for my blog post. Challenge accepted!
- jtbayly 3y agoSo because something is automated somewhere, nobody has to follow robots.TXT anymore?
- Spivak 3y agoYou have it backwards, when a human initiates the request, like clicking on a link, your browser (rightfully) will ignore robots.txt. Unattended requests like scrapers do respect robots.txt. This is a case where ChatGPT is acting more like a browser with a funny way of displaying the final result. Each request is initiated by a human so it would likely be reasonable to ignore it in this case.