7 ms·

>good bots You mean, bots that obey robots.txt? https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt very specifically prohibits scraping

by ikeboy 9y ago

>good bots You mean, bots that obey robots.txt? https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt very specifically prohibits scraping by any bot besides a small whitelist. robots.txt compliance is not difficult to build. I'm fine with robots.txt violations being considered hacking.

diggan 9y ago
> robots.txt violations being considered hacking Hm, I disagree. Either information is public, no matter for who. Or the information is private, and you should have ACL for accessing the information. I don't think it's fair to say that information is public if you're a human but private if you're a machine, or vice versa. It's not about if it's difficult to build but rather the principle behind if you can just allow humans to read something.
- ikeboy 9y ago
  Why is discriminating against robots unfair? There are valid reasons (for instance, robots take a lot of resources to serve and don't lead to revenue).
  - diggan 9y ago
    Just because it's a robot doesn't mean that it takes more resources to load a page. A robot that loads 1000x more pages than a normal user, sure. But then rate-limit everyone rather than blocking specifically bots. And that bots don't lead to revenue depends on why the bot is navigating on your page no? If it's some indexer that links back to your website and it's a popular index, then you'll maybe end up with more revenue thanks to that bot than a normal user.
    - ikeboy 9y ago
      Accepting robots + humans takes more resources than only accepting humans. Your arguments about revenue are website-dependant and it's the website owner who is in the best position to decide whether robots are good for them or not (and plenty of sites don't ban bots in their robots.txt). In this case, the company that ran the bots is directly competing with Linkedin's products that sell aggregated data to employers and such, and linkedin clearly decided it's not going to lead to more revenue for them.
  - ouid 9y ago
    my browser is a robot that renders your page.
  - marcosdumay 9y ago
    What is exactly the difference between a robot and a person using a browser? Does an ad-blocking browser counts as a bot or as a human? And what is something that concatenates all of your infinite scrolling to represent a paginated view? What is something that changes the structure of your page? What is something that concatenates different pages before displaying?
- rtpg 9y ago
  The real life equivalent of this is "if I leave my door unlocked, should someone be allowed to walk in anyways?" I would definitely want some intent provisions in, but saying something is accessible therefore free game seems too wide.
  - MereInterest 9y ago
    > The real life equivalent of this is "if I leave my door unlocked, should someone be allowed to walk in anyways?" The problem with analogies is that many equally valid analogies that can be made, but with many different points. I would argue that the real life equivalent is "Have this free book, but you may not read Chapter 4."
    - rtpg 9y ago
      Well I suppose both are possible, and it's on a case by case basis. If I put on my website terms of service "please don't try to go everywhere" , and then you do... seems like you did _something_. I don't really get what sort of stuff is enforceable, though.
    - scooble 9y ago
      Or putting up a poster that only some people are allowed to look at (or that the google maps car isn't allowed to photograph).
- chii 9y ago
  > ACL for accessing the information the ACL is the robots.txt. A door with or without a lock doesn't determine whether the place is public or not.
  - joelanford 9y ago
    robots.txt is more like a sign that asks certain people not to look at a bunch of other publicly visible signs. One can't post a sign in public that tells people not to look at other publicly visible signs and expect the government to arrest or fine them for ignoring it.
    - lloeki 9y ago
      robot != UA What if I user curl to pipe web content to my mail so that I can read it in a quirky way? What if I write a Chrome extension to crawl a site? Where does w3m stands? This is not a question of the tool (UA) but of the intent (mass crawling, indexing, mass-replicating stuff). robots.txt is made as hints for crawlers and the like, not optimistically ACL whether something is public or not.
  - sgtmas2006 9y ago
    if my bot is actually my cat actuating a switch for it to load a page, does it have to follow robots.txt?
  - Dylan16807 9y ago
    robots.txt cannot change whether something is public, because it doesn't apply to humans.
anilgulecha 9y ago
robots.txt have no legal validity.
- ikeboy 9y ago
  I mean, it seems to have been cited in the lawsuit. See e.g. https://static1.squarespace.com/static/5803b57737c581885cbd0667/t/597779d26a49634b3d745b33/1501002195360/062917+TRO+Hearing+hiQ+Labs+v+LinkedIn+Corp_.pdf https://static1.squarespace.com/static/5803b57737c581885cbd0... and search for it.
  - dognotdog 9y ago
    I doubt a bot could legally agree to a license put into robots.txt, even if it were able to make sense of it, and a human is never expected to read it. The purpose of robots.txt is to guide bots away from circular links and such that would result in bogging down the site and causing undue amounts of nonsense traffic. The purpose of robots.txt not access control. EDIT: typo fix
    - ikeboy 9y ago
      The human is expected to use it to not scrape sites that prohibit it.
  - scoot 9y ago
    Although it appears the court found for HiQ (against LinkedIn): https://regmedia.co.uk/2017/08/14/hiqlinkedintro.pdf https://regmedia.co.uk/2017/08/14/hiqlinkedintro.pdf
    - ikeboy 9y ago
      Temporary injunction, not final decision.
rhema 9y ago
I don't know. The pathological case could include a rapidly changing robots.txt. Think about archive.org's policy. If they suddenly find new restrictions on a domain, they hide it in their waybackmachine. Sometimes an old site will go down and be replaced by totally new owners. This breaks some domains of the waybackmachine retroactively.
- ikeboy 9y ago
  I think judges are able to ask some questions and tell the difference between an honest mistake and a flagrant disregard for robots.txt, if that were to be the legal standard.
martin-adams 9y ago
For the most part I agree, but I feel there are grey areas. Things like web browsers (which are not robots) can access the content as though they are from a human. But what about extensions or apps that do things in the background, such as caching the contents of several pages for offline viewing. Is that now considered a bot. The robotstxt.org site states that a robot "should" obey the rules. "should" is not a legal term that implies compliance. "must" would have been more appropriate to indicate enforcement.
mcguire 9y ago
OTOH, if HiQ employed a team of people to surf to Linkedin and physically type the information into their databases, that would be ok?
bambax 9y ago
> I'm fine with robots.txt violations being considered hacking Really?? That would mean private corporations, or private citizens, can write laws.
- ikeboy 9y ago
  You can put up a "no trespassing sign" on your property (although there's some debate as to how much that actually counts for - a quick search pulls up https://www.washingtonpost.com/news/volokh-conspiracy/wp/2017/04/12/the-fourth-amendment-and-no-trespassing-signs/?utm_term=.4e4998979dce https://www.washingtonpost.com/news/volokh-conspiracy/wp/201...)
  - jstarfish 9y ago
    Robots.txt is not a 'no-trespassing' sign. Robots.txt is a 'whites-only' sign. The information is available to the public, just not for certain classes. This is and should be legally unenforceable. If something is truly meant to be private it should not be referenced from a public-facing page or it should have access control enabled.
    - ikeboy 9y ago
      Except race /skin color is a legally protected class, and robots aren't (and why should they be? They can't enter into contracts, conduct business, etc. So it's perfectly legitimate to exclude them from a site where they cannot use it in the intended manner). "If you truly didn't want trespassers you should've put up a gate."
      - jstarfish 9y ago
        Bots being legally protected as a class or not, using robots.txt as the ultimate test of what distinguishes normal traffic from CFAA violations is a very flawed mechanism. It turns your website into a minefield. As a property owner, a no-trespassing sign won't protect you from the lawsuits that result when a toddler drowns in your pool. You're expected to do more (like putting up that gate). Equifax's systems are peppered with "no-trespassing" motds at login. They also have a robots.txt file. We expected them to do more. Same for leaving keys in your ignition, guns unlocked on your nightstand, etc. "Don't touch" signs won't absolve you of responsibility when either gets stolen and used in a spree killing. So yes, as the owner of any sort of asset, in most contexts it is your responsibility to implement access controls to keep unauthorized traffic out.
FRex 9y ago
That file includes at least two non-standard syntax extensions[0]. Robots is just a de facto standard and respect of some directives varies[1]. So much for it being 'not difficult' while the task is not even clear because there isn't even a clear standard. Archive.org also dislikes how robots.txt is being used mainly for search engines and goes against their mission in particular[2]. Are they now hackers for not throwing away information just because someone was overzealous with robots.txt or retired a certain website and uses robots.txt as SEO to let another one take its place in Google search results? If some big corp wants to cry and bring legal matters into software they should first be accountable themselves for not securing themselves and the data of their clients (see the LinkedIn hack people mentioned elsewhere here and in general the high profile hacks like Equifax, Sony, etc.). Or should software shape up to be like many other areas today are - multi-million corporations are free to play fast and loose and endanger people while small guys get fried over meaningless bullshit and vaguely defined "crimes". [0] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nons... [1] - https://intoli.com/blog/analyzing-one-million-robots-txt-files/ https://intoli.com/blog/analyzing-one-million-robots-txt-fil... [2] - https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- ikeboy 9y ago
  It contains User-agent: * Disallow: / I am pretty sure none of the standard libraries/ tools that respect robots.txt would continue after being fed that file. >throwing away information This is entirely irrelevant. If they receive data from someone they have no obligation to discard it because of the current status of robots.txt. The question would be if they should continue to actively scrape that website. It seems like they've done that for gov sites, but nobody particularly cares about enforcing gov robots.txt. It would've been interesting if the government sued them, although if they cared they probably would've just told them to stop.
  - FRex 9y ago
    So we have an unclear "standard" that is only a de facto standard (and still varies in more advances directives between few big bots) that you're "pretty sure" about but that's seemingly not written down in its entirety anywhere and it'd also be enforced selectively depending on whether or not "someone particularly cares". Truly perfect and foolproof law that would be. And all this to protect some corp's business model of not letting others collect automatically the public information they provide, while they are free to use outdated or buggy software, store passwords in plaintext, etc. and get away with leaking data of millions of customers that should never be public. And it'd fail to stop anyone except benign, private and low fund actors because instantly Indian (or other low wage country) services for "scraping by human thus not a bot ignoring robots.txt" would pop up, just like there are captcha solving services that employ humans already, and malicious bots wouldn't care anyway just like they make 0 effort to respect it now and run from servers in some country that isn't friendly towards USA so there is 0 potential for catching the perpetrators.
cabaalis 9y ago
I was about to agree that robots.txt prohibitions should be considered a form of authorization. But I think what is being argued is that "if it's publicly available on a URL, it's available for any client to download and use." I think the latter argument holds more water, as since they are making it publicly available it is implicit authorization.
cmiles74 9y ago
Honoring the robots.txt file is voluntary and ignoring it should in no way be considered hacking. I would go so far as to say that any activity that someone could engage in, simply by loading a URL, should in no way be considered hacking. Not only does it make it way too easy to prosecute software developers, it really devalues the term "hacking".
- dec0dedab0de 9y ago
  Sometimes you can do SQL injections just by loading a URL
  - cmiles74 9y ago
    Perhaps that shouldn't be construed as hacking either. If I send a link to someone via email, they shouldn't need to worry about breaking the law if they click it. I do think that a company who has been victim of a SQL injection attack will have a better chance in court then, say, LinkedIn in this specific case. At least this theoretical company has made some small effort to protect their data, however inept.
ThrustVectoring 9y ago
>robots.txt compliance is not difficult to build. I'm fine with robots.txt violations being considered hacking. I'm not. You can set up a server to serve different versions of robots.txt to different folks. A malicious actor could deliberately feed inputs to a specific crawler that convince it to violate the terms of the robots.txt it serves to everyone else, and then press for criminal charges against the operator of the scraper. In a sufficiently adversarial relationship, this lets website owners turn any well-behaved site scraper into criminal activity. That's not a power we want to grant.
alasdair_ 9y ago
>I'm fine with robots.txt violations being considered hacking. Okay. Start with something simple then - how would you define a "bot" and thus subject to your robots.txt rule? Is my web-browser a bot? What about a proxy? What about a deaf persons screen reader? If my web-browser pre-fetches links near my mouse pointer, is that a bot? What if it downloads the whole of an article split over, say, ten pages? I think of robots.txt similar to posting a "No Trespassing" sign. For a private residence, it's almost not even required, yet for something like a shopping mall during opening hours, the default assumption is that anyone is allowed to be there without a specific invitation, until they are expressly asked to leave and not come back. Trying to nail down the exact line is a tough issue.
hardtke 9y ago
If robots.txt allows Google and Bing but nobody else, it should be ignored. If it blocks everyone, then I agree. We need to make sure that the next Google has a chance to succeed.
decasteve 9y ago
Do Not Track compliance is even easier to build. Does the same logic apply?
- TheCoelacanth 9y ago
  Yes, this is an extremely good point. If failing to follow robots.txt is a criminal violation of CFAA, then using any of my computers resources (cookies, javascript, etc) to track me while I am sending a DNT header is also a criminal violation of CFAA. I would almost be willing to concede making not following robots.txt a violation of CFAA if the trade-off was Mark Zuckerberg being brought up on several billion felony charges every year.
grkvlt 9y ago
Interesting. They say that crawling is prohibited there, actually, and have a blanked 'Disallow' at the end. # Notice: The use of robots or other automated means to access LinkedIn without # the express permission of LinkedIn is strictly prohibited. ... User-agent: * Disallow: / All the listed bots are only able to access a small subset of pages, the same for each bot apart from one. The 'deepcrawl' bot is privileged, and gets to see the '/profinder' pages, for some reason? # Profinder only for deepcrawl Allow: /profinder* Anyone know who operates this bot?

by ikeboy 9y ago

diggan 9y ago

> robots.txt violations being considered hacking Hm, I disagree. Either information is public, no matter for who. Or the information is private, and you should have ACL for accessing the information. I don't think it's fair to say that information is public if you're a human but private if you're a machine, or vice versa. It's not about if it's difficult to build but rather the principle behind if you can just allow humans to read something.

ikeboy 9y ago
Why is discriminating against robots unfair? There are valid reasons (for instance, robots take a lot of resources to serve and don't lead to revenue).
- diggan 9y ago
  Just because it's a robot doesn't mean that it takes more resources to load a page. A robot that loads 1000x more pages than a normal user, sure. But then rate-limit everyone rather than blocking specifically bots. And that bots don't lead to revenue depends on why the bot is navigating on your page no? If it's some indexer that links back to your website and it's a popular index, then you'll maybe end up with more revenue thanks to that bot than a normal user.
  - ikeboy 9y ago
    Accepting robots + humans takes more resources than only accepting humans. Your arguments about revenue are website-dependant and it's the website owner who is in the best position to decide whether robots are good for them or not (and plenty of sites don't ban bots in their robots.txt). In this case, the company that ran the bots is directly competing with Linkedin's products that sell aggregated data to employers and such, and linkedin clearly decided it's not going to lead to more revenue for them.
- ouid 9y ago
  my browser is a robot that renders your page.
- marcosdumay 9y ago
  What is exactly the difference between a robot and a person using a browser? Does an ad-blocking browser counts as a bot or as a human? And what is something that concatenates all of your infinite scrolling to represent a paginated view? What is something that changes the structure of your page? What is something that concatenates different pages before displaying?
rtpg 9y ago
The real life equivalent of this is "if I leave my door unlocked, should someone be allowed to walk in anyways?" I would definitely want some intent provisions in, but saying something is accessible therefore free game seems too wide.
- MereInterest 9y ago
  > The real life equivalent of this is "if I leave my door unlocked, should someone be allowed to walk in anyways?" The problem with analogies is that many equally valid analogies that can be made, but with many different points. I would argue that the real life equivalent is "Have this free book, but you may not read Chapter 4."
  - rtpg 9y ago
    Well I suppose both are possible, and it's on a case by case basis. If I put on my website terms of service "please don't try to go everywhere" , and then you do... seems like you did _something_. I don't really get what sort of stuff is enforceable, though.
  - scooble 9y ago
    Or putting up a poster that only some people are allowed to look at (or that the google maps car isn't allowed to photograph).
chii 9y ago
> ACL for accessing the information the ACL is the robots.txt. A door with or without a lock doesn't determine whether the place is public or not.
- joelanford 9y ago
  robots.txt is more like a sign that asks certain people not to look at a bunch of other publicly visible signs. One can't post a sign in public that tells people not to look at other publicly visible signs and expect the government to arrest or fine them for ignoring it.
  - lloeki 9y ago
    robot != UA What if I user curl to pipe web content to my mail so that I can read it in a quirky way? What if I write a Chrome extension to crawl a site? Where does w3m stands? This is not a question of the tool (UA) but of the intent (mass crawling, indexing, mass-replicating stuff). robots.txt is made as hints for crawlers and the like, not optimistically ACL whether something is public or not.
- sgtmas2006 9y ago
  if my bot is actually my cat actuating a switch for it to load a page, does it have to follow robots.txt?
- Dylan16807 9y ago
  robots.txt cannot change whether something is public, because it doesn't apply to humans.

anilgulecha 9y ago

robots.txt have no legal validity.

ikeboy 9y ago
I mean, it seems to have been cited in the lawsuit. See e.g. https://static1.squarespace.com/static/5803b57737c581885cbd0667/t/597779d26a49634b3d745b33/1501002195360/062917+TRO+Hearing+hiQ+Labs+v+LinkedIn+Corp_.pdf https://static1.squarespace.com/static/5803b57737c581885cbd0... and search for it.
- dognotdog 9y ago
  I doubt a bot could legally agree to a license put into robots.txt, even if it were able to make sense of it, and a human is never expected to read it. The purpose of robots.txt is to guide bots away from circular links and such that would result in bogging down the site and causing undue amounts of nonsense traffic. The purpose of robots.txt not access control. EDIT: typo fix
  - ikeboy 9y ago
    The human is expected to use it to not scrape sites that prohibit it.
- scoot 9y ago
  Although it appears the court found for HiQ (against LinkedIn): https://regmedia.co.uk/2017/08/14/hiqlinkedintro.pdf https://regmedia.co.uk/2017/08/14/hiqlinkedintro.pdf
  - ikeboy 9y ago
    Temporary injunction, not final decision.

rhema 9y ago

I don't know. The pathological case could include a rapidly changing robots.txt. Think about archive.org's policy. If they suddenly find new restrictions on a domain, they hide it in their waybackmachine. Sometimes an old site will go down and be replaced by totally new owners. This breaks some domains of the waybackmachine retroactively.

ikeboy 9y ago
I think judges are able to ask some questions and tell the difference between an honest mistake and a flagrant disregard for robots.txt, if that were to be the legal standard.

martin-adams 9y ago

For the most part I agree, but I feel there are grey areas. Things like web browsers (which are not robots) can access the content as though they are from a human. But what about extensions or apps that do things in the background, such as caching the contents of several pages for offline viewing. Is that now considered a bot. The robotstxt.org site states that a robot "should" obey the rules. "should" is not a legal term that implies compliance. "must" would have been more appropriate to indicate enforcement.

mcguire 9y ago

OTOH, if HiQ employed a team of people to surf to Linkedin and physically type the information into their databases, that would be ok?

bambax 9y ago

> I'm fine with robots.txt violations being considered hacking Really?? That would mean private corporations, or private citizens, can write laws.

ikeboy 9y ago
You can put up a "no trespassing sign" on your property (although there's some debate as to how much that actually counts for - a quick search pulls up https://www.washingtonpost.com/news/volokh-conspiracy/wp/2017/04/12/the-fourth-amendment-and-no-trespassing-signs/?utm_term=.4e4998979dce https://www.washingtonpost.com/news/volokh-conspiracy/wp/201...)
- jstarfish 9y ago
  Robots.txt is not a 'no-trespassing' sign. Robots.txt is a 'whites-only' sign. The information is available to the public, just not for certain classes. This is and should be legally unenforceable. If something is truly meant to be private it should not be referenced from a public-facing page or it should have access control enabled.
  - ikeboy 9y ago
    Except race /skin color is a legally protected class, and robots aren't (and why should they be? They can't enter into contracts, conduct business, etc. So it's perfectly legitimate to exclude them from a site where they cannot use it in the intended manner). "If you truly didn't want trespassers you should've put up a gate."
    - jstarfish 9y ago
      Bots being legally protected as a class or not, using robots.txt as the ultimate test of what distinguishes normal traffic from CFAA violations is a very flawed mechanism. It turns your website into a minefield. As a property owner, a no-trespassing sign won't protect you from the lawsuits that result when a toddler drowns in your pool. You're expected to do more (like putting up that gate). Equifax's systems are peppered with "no-trespassing" motds at login. They also have a robots.txt file. We expected them to do more. Same for leaving keys in your ignition, guns unlocked on your nightstand, etc. "Don't touch" signs won't absolve you of responsibility when either gets stolen and used in a spree killing. So yes, as the owner of any sort of asset, in most contexts it is your responsibility to implement access controls to keep unauthorized traffic out.

FRex 9y ago

That file includes at least two non-standard syntax extensions[0]. Robots is just a de facto standard and respect of some directives varies[1]. So much for it being 'not difficult' while the task is not even clear because there isn't even a clear standard. Archive.org also dislikes how robots.txt is being used mainly for search engines and goes against their mission in particular[2]. Are they now hackers for not throwing away information just because someone was overzealous with robots.txt or retired a certain website and uses robots.txt as SEO to let another one take its place in Google search results? If some big corp wants to cry and bring legal matters into software they should first be accountable themselves for not securing themselves and the data of their clients (see the LinkedIn hack people mentioned elsewhere here and in general the high profile hacks like Equifax, Sony, etc.). Or should software shape up to be like many other areas today are - multi-million corporations are free to play fast and loose and endanger people while small guys get fried over meaningless bullshit and vaguely defined "crimes". [0] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nons... [1] - https://intoli.com/blog/analyzing-one-million-robots-txt-files/ https://intoli.com/blog/analyzing-one-million-robots-txt-fil... [2] - https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

ikeboy 9y ago
It contains User-agent: * Disallow: / I am pretty sure none of the standard libraries/ tools that respect robots.txt would continue after being fed that file. >throwing away information This is entirely irrelevant. If they receive data from someone they have no obligation to discard it because of the current status of robots.txt. The question would be if they should continue to actively scrape that website. It seems like they've done that for gov sites, but nobody particularly cares about enforcing gov robots.txt. It would've been interesting if the government sued them, although if they cared they probably would've just told them to stop.
- FRex 9y ago
  So we have an unclear "standard" that is only a de facto standard (and still varies in more advances directives between few big bots) that you're "pretty sure" about but that's seemingly not written down in its entirety anywhere and it'd also be enforced selectively depending on whether or not "someone particularly cares". Truly perfect and foolproof law that would be. And all this to protect some corp's business model of not letting others collect automatically the public information they provide, while they are free to use outdated or buggy software, store passwords in plaintext, etc. and get away with leaking data of millions of customers that should never be public. And it'd fail to stop anyone except benign, private and low fund actors because instantly Indian (or other low wage country) services for "scraping by human thus not a bot ignoring robots.txt" would pop up, just like there are captcha solving services that employ humans already, and malicious bots wouldn't care anyway just like they make 0 effort to respect it now and run from servers in some country that isn't friendly towards USA so there is 0 potential for catching the perpetrators.

cabaalis 9y ago

I was about to agree that robots.txt prohibitions should be considered a form of authorization. But I think what is being argued is that "if it's publicly available on a URL, it's available for any client to download and use." I think the latter argument holds more water, as since they are making it publicly available it is implicit authorization.

cmiles74 9y ago

Honoring the robots.txt file is voluntary and ignoring it should in no way be considered hacking. I would go so far as to say that any activity that someone could engage in, simply by loading a URL, should in no way be considered hacking. Not only does it make it way too easy to prosecute software developers, it really devalues the term "hacking".

dec0dedab0de 9y ago
Sometimes you can do SQL injections just by loading a URL
- cmiles74 9y ago
  Perhaps that shouldn't be construed as hacking either. If I send a link to someone via email, they shouldn't need to worry about breaking the law if they click it. I do think that a company who has been victim of a SQL injection attack will have a better chance in court then, say, LinkedIn in this specific case. At least this theoretical company has made some small effort to protect their data, however inept.

ThrustVectoring 9y ago

>robots.txt compliance is not difficult to build. I'm fine with robots.txt violations being considered hacking. I'm not. You can set up a server to serve different versions of robots.txt to different folks. A malicious actor could deliberately feed inputs to a specific crawler that convince it to violate the terms of the robots.txt it serves to everyone else, and then press for criminal charges against the operator of the scraper. In a sufficiently adversarial relationship, this lets website owners turn any well-behaved site scraper into criminal activity. That's not a power we want to grant.

alasdair_ 9y ago

>I'm fine with robots.txt violations being considered hacking. Okay. Start with something simple then - how would you define a "bot" and thus subject to your robots.txt rule? Is my web-browser a bot? What about a proxy? What about a deaf persons screen reader? If my web-browser pre-fetches links near my mouse pointer, is that a bot? What if it downloads the whole of an article split over, say, ten pages? I think of robots.txt similar to posting a "No Trespassing" sign. For a private residence, it's almost not even required, yet for something like a shopping mall during opening hours, the default assumption is that anyone is allowed to be there without a specific invitation, until they are expressly asked to leave and not come back. Trying to nail down the exact line is a tough issue.

hardtke 9y ago

If robots.txt allows Google and Bing but nobody else, it should be ignored. If it blocks everyone, then I agree. We need to make sure that the next Google has a chance to succeed.

decasteve 9y ago

Do Not Track compliance is even easier to build. Does the same logic apply?

TheCoelacanth 9y ago
Yes, this is an extremely good point. If failing to follow robots.txt is a criminal violation of CFAA, then using any of my computers resources (cookies, javascript, etc) to track me while I am sending a DNT header is also a criminal violation of CFAA. I would almost be willing to concede making not following robots.txt a violation of CFAA if the trade-off was Mark Zuckerberg being brought up on several billion felony charges every year.

grkvlt 9y ago

Interesting. They say that crawling is prohibited there, actually, and have a blanked 'Disallow' at the end. # Notice: The use of robots or other automated means to access LinkedIn without # the express permission of LinkedIn is strictly prohibited. ... User-agent: * Disallow: / All the listed bots are only able to access a small subset of pages, the same for each bot apart from one. The 'deepcrawl' bot is privileged, and gets to see the '/profinder' pages, for some reason? # Profinder only for deepcrawl Allow: /profinder* Anyone know who operates this bot?