3 ms·
Being indexed in search engines has zero benefit? Also robots.txt is a suggestion but hashcash is enforced server side. I agree it's a tragedy people have star
by msgodel 1y ago
Being indexed in search engines has zero benefit?
Also robots.txt is a suggestion but hashcash is enforced server side. I agree it's a tragedy people have started to completely ignore it but you can't ignore server side behavior.
- MyPasswordSucks 1y agoHow do you propose the server distinguish between a bot and a human visitor?
- theamk 1y agoBeing indexed in search engines is very useful, but does not need any filtering - Google and all other major search engines respect robots.txt, use well-known UA and even publish IP address. AI bots are not search engines, and they have no benefits for the website owners. This can be very clearly seen because they ignore robots.txt, pretend to be regular browsers, and use multiple proxies to avoid IP bans.