3 ms·
Does their robots.txt say don't crawl this part of the site? If it does, this ruling is catastrophic. If it doesn't then there is hope.
by bryanrasmussen 7y ago
Does their robots.txt say don't crawl this part of the site? If it does, this ruling is catastrophic. If it doesn't then there is hope.
- johnny99 7y agoCompliance with robots.txt is and always has been voluntary, and many crawlers have long ignored it, including Archive.org.
- shkkmo 7y ago> If it does, this ruling is catastrophic I know it is a generally considered bad form to ask, but did you read much of the ruling? I feel like a lot of people on this thread are just going off of Animats' comment and haven't spent much time looking at the opinion. I didn't read the whole thing, but skimmed through it and read what seemed to be the relevant parts of the argument. (Including the bit that talks about LinkedIn's robots.txt) The ruling doesn't really support your claim of catastrophe and doesn't claim to pass any sort of final judgement. The judge makes a specific point about not reading too much into him upholding the injunction saying: >> These appeals generally provide “little guidance” because “of the limited scope of our review of the law” and “because the fully developed factual record may be materially different from that initially before the district court.”
- paulie_a 7y agoDo any scrapers actually pay attention to robots.txt?