4 ms·
> robots.txt violations being considered hacking Hm, I disagree. Either information is public, no matter for who. Or the information is private, and you should
by diggan 9y ago
> robots.txt violations being considered hacking
Hm, I disagree. Either information is public, no matter for who. Or the information is private, and you should have ACL for accessing the information. I don't think it's fair to say that information is public if you're a human but private if you're a machine, or vice versa.
It's not about if it's difficult to build but rather the principle behind if you can just allow humans to read something.
- ikeboy 9y agoWhy is discriminating against robots unfair? There are valid reasons (for instance, robots take a lot of resources to serve and don't lead to revenue).
- diggan 9y agoJust because it's a robot doesn't mean that it takes more resources to load a page. A robot that loads 1000x more pages than a normal user, sure. But then rate-limit everyone rather than blocking specifically bots. And that bots don't lead to revenue depends on why the bot is navigating on your page no? If it's some indexer that links back to your website and it's a popular index, then you'll maybe end up with more revenue thanks to that bot than a normal user.
- ikeboy 9y agoAccepting robots + humans takes more resources than only accepting humans. Your arguments about revenue are website-dependant and it's the website owner who is in the best position to decide whether robots are good for them or not (and plenty of sites don't ban bots in their robots.txt). In this case, the company that ran the bots is directly competing with Linkedin's products that sell aggregated data to employers and such, and linkedin clearly decided it's not going to lead to more revenue for them.
- ouid 9y agomy browser is a robot that renders your page.
- marcosdumay 9y agoWhat is exactly the difference between a robot and a person using a browser? Does an ad-blocking browser counts as a bot or as a human? And what is something that concatenates all of your infinite scrolling to represent a paginated view? What is something that changes the structure of your page? What is something that concatenates different pages before displaying?
- rtpg 9y agoThe real life equivalent of this is "if I leave my door unlocked, should someone be allowed to walk in anyways?" I would definitely want some intent provisions in, but saying something is accessible therefore free game seems too wide.
- MereInterest 9y ago> The real life equivalent of this is "if I leave my door unlocked, should someone be allowed to walk in anyways?" The problem with analogies is that many equally valid analogies that can be made, but with many different points. I would argue that the real life equivalent is "Have this free book, but you may not read Chapter 4."
- rtpg 9y agoWell I suppose both are possible, and it's on a case by case basis. If I put on my website terms of service "please don't try to go everywhere" , and then you do... seems like you did _something_. I don't really get what sort of stuff is enforceable, though.
- scooble 9y agoOr putting up a poster that only some people are allowed to look at (or that the google maps car isn't allowed to photograph).
- chii 9y ago> ACL for accessing the information the ACL is the robots.txt. A door with or without a lock doesn't determine whether the place is public or not.
- joelanford 9y agorobots.txt is more like a sign that asks certain people not to look at a bunch of other publicly visible signs. One can't post a sign in public that tells people not to look at other publicly visible signs and expect the government to arrest or fine them for ignoring it.
- lloeki 9y agorobot != UA What if I user curl to pipe web content to my mail so that I can read it in a quirky way? What if I write a Chrome extension to crawl a site? Where does w3m stands? This is not a question of the tool (UA) but of the intent (mass crawling, indexing, mass-replicating stuff). robots.txt is made as hints for crawlers and the like, not optimistically ACL whether something is public or not.
- sgtmas2006 9y agoif my bot is actually my cat actuating a switch for it to load a page, does it have to follow robots.txt?
- Dylan16807 9y agorobots.txt cannot change whether something is public, because it doesn't apply to humans.