3 ms·
It's not super-difficult to add HTTPS and HTTP basic auth (or similar) to websites these days. That's enough to keep automated scrapers from getting into the p
by NateEag 6y ago
It's not super-difficult to add HTTPS and HTTP basic auth (or similar) to websites these days.
That's enough to keep automated scrapers from getting into the private parts of your website.
- 29athrowaway 6y agoHTTPS is not a deterrent for bot. Authentication is. Unless the bot identifies itself as such by setting a user-agent header honestly you cannot distinguish a browser from a bot.
- NateEag 6y agoRight. That's why I said HTTPS and auth. Auth to keep the bots out, and HTTPS because it's free and easy to add, and keeps network admins from sniffing your username and password.