4 ms·
why not use robots.txt
by brainzap 3y ago
why not use robots.txt
- tjpnz 3y agoThat assumes they'll honour it, either now or in the future.
- lapser 3y agoIf they'd be that malicious, they could also change their bot's UA, so would it really matter?
- tenebrisalietum 3y agoThere are a lot of dumb bots.
- IceWreck 3y agoAnd you think Meta or Twitter's bots would be dumb ? If they wanted to scrape your site, nobody can prevent them.
- tenebrisalietum 3y agoKinda. Meta and Twitter want you to join their platforms, they aren't general purpose search engines scraping the entire Internet - they're scraping people that join them. Requests from Meta/Twitter are probably from a link someone put in a post. ChatGPT can't be an impolite Internet citizen (spoofing UA's) and claim to be using AI for the good for humanity, so they're not going to be dishonest with their user-agent.
- eindiran 3y ago> ChatGPT can't be an impolite Internet citizen (spoofing UA's) and claim to be using AI for the good for humanity, so they're not going to be dishonest with their user-agent. That reads an awful-lot like "Google can't be evil and claim that their motto is 'Don't be evil', so they're not going to be evil" but here we are. The profit motive eventually undoes any principled claim by a company.
- tenebrisalietum 3y agoAbsolutely, but until then, adding bot UA's to a blocklist is somewhat useful. Like anything else in IT security it's never "set and forget" permanently; the effectiveness of things like that decay over time and must be periodically re-evaluated. But if something can be used to your advantage now, even if for a while, then why not use it.
- xgbi 3y agowhy not both? and check your logs to see who is not complying
- mr90210 3y agoThat’s a good one. Can non-complaints be subject of a lawsuit?
- tourmalinetaco 3y agoWhy would they be? No contract was signed, no DRM was being broken, they’re simply accessing the website you’re publicly hosting.
- sMarsIntruder 3y agorobots.txt isn’t honoured only by bad bots and scrapers, so I agree with you that this nginx configuration is pretty useless and doesn’t even solve the problem on the other side!