6 ms·

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

bakql 1y ago
Stop your http server if you do not wish to receive http requests.
- vkou 1y ago
  Turn off your phone if you don't want to receive robo-dialed calls and unsolicited texts 300 times a day. Fence off your yard if you don't want people coming by and dumping a mountain of garbage on it every day. You can certainly choose to live in a society that thinks these are acceptable solutions. I think it's bullshit, and we'd all be better off if anyone doing these things would be breaking rocks with their teeth in a re-education camp, until they learn how to be a decent human being.
- bigbuppo 1y ago
  Ah yes, and unplug the mail server to stop all spam. Great idea!
Calavar 1y ago
If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape
- yuliyp 1y ago
  Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.
  - halJordan 1y ago
    No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.
    - oytis 1y ago
      > door to door salesman regularly walk right past them. Oh, now I understand why Americans can't see a problem here.
    - ahtihn 1y ago
      If a company was sending hundreds of salesmen to knock at a door one after the other, I'm pretty sure they could successfully get sued for harassment.
      - hsbauauvhabzb 1y ago
        Can’t Americans literally shoot each other for trespassing?
        dragonwriter 1y ago
        Generally, legally, no, not just for ignoring a “no soliciting” sign.
        hsbauauvhabzb 1y ago
        But they’re presumably trespassing.
- munk-a 1y ago
  I disagree strongly here - though not from a technical perspective. There's absolutely a legal concept of making your work available for viewing without making it available for copying and AI scraping (while we can technically phrase it as just viewing a bunch of times) is effectively copying. Lets say a large art hosting site realizes how damaging AI training on their data can be - should they respond by adding a paywall before any of their data is visible? If that paywall is added (let's just say $5/mo) can most of the artists currently on their site afford to stay there? Can they afford it if their potential future patrons are limited to just those folks who can pay $5/mo? Would the scraper be able to afford a one time cost of $5 to scrape all of that data? I think, as much they are a deeply flawed concept, this is a case where EULAs or an assumption of no-access for training unless explicitly granted that's actually enforced through the legal system is required. There are a lot of small businesses and side projects that are dying because of these models and I think that creative outlet has societal value we would benefit from preserving.
  - jMyles 1y ago
    > There's absolutely a legal concept of making your work available for viewing without making it available for copying This "legal concept" is enforceable through legacy systems of police and violence. The internet does not recognize it. How much more obvious can this get? If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just come to completely dominate the entire economy of the internet? If I can spin up a server in copyleftistan with a complete copy of every album and film ever made, available for free download, why would users in copyrightistan use the locked down services of their domestic economy?
    - kelnos 1y ago
      > legacy systems of police and violence You use "legacy" as if these systems are obsolete and on their way out. They're not. They're here to stay, and will remain dominant, for better or worse. Calling them "legacy" feels a bit childish, as if you're trying to ignore reality and base arguments on your preferred vision of how things should be. > The internet does not recognize it. Sure it does. Not universally, but there are a lot of things governments and law enforcement can do to control what people see and do on the internet. > If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just come to completely dominate the entire economy of the internet? No, of course not, that's silly. That only really works on the margins. Any other country would immediately slap economic sanctions on that free-for-all jurisdiction and cripple them. If that fails, there's always a military response they can resort to. > If I can spin up a server in copyleftistan with a complete copy of every album and film ever made, available for free download, why would users in copyrightistan use the locked down services of their domestic economy? Because the governments of all the copyrightistans will block all traffic going in and out of copyleftistan. While this may not stop determined, technically-adept people, it will work for the most part. As I said, this sort of thing only really works on the margins.
- andoando 1y ago
  Well yes this is exactly what's happening as of now. But there SHOULD be a way to upload content without giving it access to scrapers.
- kelnos 1y ago
  > If you are serving web pages, you are soliciting GET requests So what's the solution? How do I host a website that welcomes human visitors, but rejects all scrapers? There is no mechanism! The best I can do is a cat-and-mouse arms race where I try to detect the traffic I don't want, and block it, while the people generating the traffic keep getting more sophisticated about hiding from my detection. No, putting up a paywall is not a reasonable response to this. > The question is are you expressing a preference on etiquette versus a hard rule that must be followed. Well, there really aren't any hard rules that must be followed, because there are no enforcement mechanisms outside of going nuclear (requiring login). Everything is etiquette. And I agree that robots.txt is also etiquette, and it is super messed up that we tolerate "AI" companies stomping all over that etiquette. Do we maybe want laws that say everyone must respect robots.txt? Maybe? But then people will just move their scrapers to a jurisdiction without those laws. And I'm sure someone could make the argument that robots.txt doesn't apply to them because they spoofed a browser user-agent (or another user-agent that a site explicitly allows). So perhaps we have a new mechanism, or new laws, or new... something. But this all just highlights the point I'm making here: there is no reasonable mechanism (no, login pages and http auth don't count) for site owners to restrict access to their site based on these sorts of criteria. And that's a problem.
- davesque 1y ago
  If I order a package from a company selling a good, am I inviting all that company's competitors to show up at my doorstep to try and outbid the delivery person from the original company when they arrive, and maybe they all show up at the same time and cause my porch to collapse? No, because my front porch is a limited resource for which I paid for an intended purpose. Is it illegal for those other people to show up? Maybe not by the letter of the law.
- pluto_modadic 1y ago
  ignoring a rate limit gets you blocked.
  - hsbauauvhabzb 1y ago
    Scrapers actively bypass this by rotating IP addresses.
davsti4 1y ago
Its simple, and I'll quote myself - "robots.txt isn't the law".
- ColinWright 1y ago
  Quoting Cervisia : > robots.txt. This is not the law In Germany, it is the law. § 44b UrhG says (translated): (1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations. (2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for text and data mining. (3) Uses pursuant to paragraph 2, sentence 1, are only permitted if the rights holder has not reserved these rights. A reservation of rights for works accessible online is only effective if it is in machine-readable form. -- https://news.ycombinator.com/item?id=45776825 https://news.ycombinator.com/item?id=45776825
  - davsti4 1y ago
    Its the law pertaining to copyright. https://www.gesetze-im-internet.de/englisch_urhg/englisch_urhg.html https://www.gesetze-im-internet.de/englisch_urhg/englisch_ur... ... and that's only in Germany. If what you're protecting with robots.txt isn't copyright-able, then you'll need to find another legal means.
- bigbuppo 1y ago
  Violating norms makes you an abusive jerk at best.
  - davsti4 1y ago
    That's a slippery slope opinion. One person's norm can oppose another's norm. Who defines them? If my "norm" is to stop at a stop sign, and yours is to roll through it, are you the abusive jerk?
    - bigbuppo 1y ago
nkrisc 1y ago
Put your content behind authentication if you don’t want it to be requested by just anyone.
- kelnos 1y ago
  But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!
  - 1gn15 1y ago
    What the hell? That is incredibly discriminatory. Fuck off. I support those that counter those discriminatory mechanisms.
    - Anamon 1y ago
      Discriminatory against bots? That doesn't even make any sense.
      - bigbuppo 1y ago
        They probably have stock options.
  - deleted 1y ago
    [deleted]
  - 9rx 1y ago
    > as long as they are humans. I don't want it accessible to bots. A curious position. There isn't a secondary species using the internet. There is only humans. Unless you foresee some kind of alien invasion or earthworm uprising, nothing other than humans will ever access your content. Rejecting the tools humans use to bridge their biological gaps is rather nonsensical. > You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem! I suppose it would be pretty neat if humans were born with some kind of internet-like telepathy ability, but lacking that mechanism isn't any kind of real problem. Humans are well adept at using tools and have successfully used tools for millennia. The internet itself is a tool! Which, like before, makes rejecting the human use of tools nonsensical.
stray 1y ago
You require something the bot won't have that a human would. Anybody may watch the demo screen of an arcade game for free, but you have to insert a quarter to play — and you can have even greater access with a key. > and you’ve explicitly left a sign saying ‘you are not welcome here’ And the sign said "Long-haired freaky people Need not apply" So I tucked my hair up under my hat And I went in to ask him why He said, "You look like a fine upstandin' young man I think you'll do" So I took off my hat and said, "Imagine that Huh, me workin' for you"
- michaelt 1y ago
  > You require something the bot won't have that a human would. Is this why the “open web” is showing me a captcha or two, along with their cookie banner and newsletter pop up these days?
  - bigbuppo 1y ago
    Up until people started making a big stink about CAPTCHAs being used for unpaid labor at scale, uh, well they had two purposes.

bakql 1y ago

Stop your http server if you do not wish to receive http requests.

vkou 1y ago
Turn off your phone if you don't want to receive robo-dialed calls and unsolicited texts 300 times a day. Fence off your yard if you don't want people coming by and dumping a mountain of garbage on it every day. You can certainly choose to live in a society that thinks these are acceptable solutions. I think it's bullshit, and we'd all be better off if anyone doing these things would be breaking rocks with their teeth in a re-education camp, until they learn how to be a decent human being.
bigbuppo 1y ago
Ah yes, and unplug the mail server to stop all spam. Great idea!

Calavar 1y ago

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

yuliyp 1y ago
Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.
- halJordan 1y ago
  No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.
  - oytis 1y ago
    > door to door salesman regularly walk right past them. Oh, now I understand why Americans can't see a problem here.
  - ahtihn 1y ago
    If a company was sending hundreds of salesmen to knock at a door one after the other, I'm pretty sure they could successfully get sued for harassment.
    - hsbauauvhabzb 1y ago
      Can’t Americans literally shoot each other for trespassing?
      - dragonwriter 1y ago
        Generally, legally, no, not just for ignoring a “no soliciting” sign.
        hsbauauvhabzb 1y ago
        But they’re presumably trespassing.
munk-a 1y ago
I disagree strongly here - though not from a technical perspective. There's absolutely a legal concept of making your work available for viewing without making it available for copying and AI scraping (while we can technically phrase it as just viewing a bunch of times) is effectively copying. Lets say a large art hosting site realizes how damaging AI training on their data can be - should they respond by adding a paywall before any of their data is visible? If that paywall is added (let's just say $5/mo) can most of the artists currently on their site afford to stay there? Can they afford it if their potential future patrons are limited to just those folks who can pay $5/mo? Would the scraper be able to afford a one time cost of $5 to scrape all of that data? I think, as much they are a deeply flawed concept, this is a case where EULAs or an assumption of no-access for training unless explicitly granted that's actually enforced through the legal system is required. There are a lot of small businesses and side projects that are dying because of these models and I think that creative outlet has societal value we would benefit from preserving.
- jMyles 1y ago
  > There's absolutely a legal concept of making your work available for viewing without making it available for copying This "legal concept" is enforceable through legacy systems of police and violence. The internet does not recognize it. How much more obvious can this get? If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just come to completely dominate the entire economy of the internet? If I can spin up a server in copyleftistan with a complete copy of every album and film ever made, available for free download, why would users in copyrightistan use the locked down services of their domestic economy?
  - kelnos 1y ago
    > legacy systems of police and violence You use "legacy" as if these systems are obsolete and on their way out. They're not. They're here to stay, and will remain dominant, for better or worse. Calling them "legacy" feels a bit childish, as if you're trying to ignore reality and base arguments on your preferred vision of how things should be. > The internet does not recognize it. Sure it does. Not universally, but there are a lot of things governments and law enforcement can do to control what people see and do on the internet. > If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just come to completely dominate the entire economy of the internet? No, of course not, that's silly. That only really works on the margins. Any other country would immediately slap economic sanctions on that free-for-all jurisdiction and cripple them. If that fails, there's always a military response they can resort to. > If I can spin up a server in copyleftistan with a complete copy of every album and film ever made, available for free download, why would users in copyrightistan use the locked down services of their domestic economy? Because the governments of all the copyrightistans will block all traffic going in and out of copyleftistan. While this may not stop determined, technically-adept people, it will work for the most part. As I said, this sort of thing only really works on the margins.
andoando 1y ago
Well yes this is exactly what's happening as of now. But there SHOULD be a way to upload content without giving it access to scrapers.
kelnos 1y ago
> If you are serving web pages, you are soliciting GET requests So what's the solution? How do I host a website that welcomes human visitors, but rejects all scrapers? There is no mechanism! The best I can do is a cat-and-mouse arms race where I try to detect the traffic I don't want, and block it, while the people generating the traffic keep getting more sophisticated about hiding from my detection. No, putting up a paywall is not a reasonable response to this. > The question is are you expressing a preference on etiquette versus a hard rule that must be followed. Well, there really aren't any hard rules that must be followed, because there are no enforcement mechanisms outside of going nuclear (requiring login). Everything is etiquette. And I agree that robots.txt is also etiquette, and it is super messed up that we tolerate "AI" companies stomping all over that etiquette. Do we maybe want laws that say everyone must respect robots.txt? Maybe? But then people will just move their scrapers to a jurisdiction without those laws. And I'm sure someone could make the argument that robots.txt doesn't apply to them because they spoofed a browser user-agent (or another user-agent that a site explicitly allows). So perhaps we have a new mechanism, or new laws, or new... something. But this all just highlights the point I'm making here: there is no reasonable mechanism (no, login pages and http auth don't count) for site owners to restrict access to their site based on these sorts of criteria. And that's a problem.
davesque 1y ago
If I order a package from a company selling a good, am I inviting all that company's competitors to show up at my doorstep to try and outbid the delivery person from the original company when they arrive, and maybe they all show up at the same time and cause my porch to collapse? No, because my front porch is a limited resource for which I paid for an intended purpose. Is it illegal for those other people to show up? Maybe not by the letter of the law.
pluto_modadic 1y ago
ignoring a rate limit gets you blocked.
- hsbauauvhabzb 1y ago
  Scrapers actively bypass this by rotating IP addresses.

davsti4 1y ago

Its simple, and I'll quote myself - "robots.txt isn't the law".

ColinWright 1y ago
Quoting Cervisia : > robots.txt. This is not the law In Germany, it is the law. § 44b UrhG says (translated): (1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations. (2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for text and data mining. (3) Uses pursuant to paragraph 2, sentence 1, are only permitted if the rights holder has not reserved these rights. A reservation of rights for works accessible online is only effective if it is in machine-readable form. -- https://news.ycombinator.com/item?id=45776825 https://news.ycombinator.com/item?id=45776825
- davsti4 1y ago
  Its the law pertaining to copyright. https://www.gesetze-im-internet.de/englisch_urhg/englisch_urhg.html https://www.gesetze-im-internet.de/englisch_urhg/englisch_ur... ... and that's only in Germany. If what you're protecting with robots.txt isn't copyright-able, then you'll need to find another legal means.
bigbuppo 1y ago
Violating norms makes you an abusive jerk at best.
- davsti4 1y ago
  That's a slippery slope opinion. One person's norm can oppose another's norm. Who defines them? If my "norm" is to stop at a stop sign, and yours is to roll through it, are you the abusive jerk?
  - bigbuppo 1y ago

nkrisc 1y ago

Put your content behind authentication if you don’t want it to be requested by just anyone.

kelnos 1y ago
But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!
- 1gn15 1y ago
  What the hell? That is incredibly discriminatory. Fuck off. I support those that counter those discriminatory mechanisms.
  - Anamon 1y ago
    Discriminatory against bots? That doesn't even make any sense.
    - bigbuppo 1y ago
      They probably have stock options.
- deleted 1y ago
  [deleted]
- 9rx 1y ago
  > as long as they are humans. I don't want it accessible to bots. A curious position. There isn't a secondary species using the internet. There is only humans. Unless you foresee some kind of alien invasion or earthworm uprising, nothing other than humans will ever access your content. Rejecting the tools humans use to bridge their biological gaps is rather nonsensical. > You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem! I suppose it would be pretty neat if humans were born with some kind of internet-like telepathy ability, but lacking that mechanism isn't any kind of real problem. Humans are well adept at using tools and have successfully used tools for millennia. The internet itself is a tool! Which, like before, makes rejecting the human use of tools nonsensical.

stray 1y ago

You require something the bot won't have that a human would. Anybody may watch the demo screen of an arcade game for free, but you have to insert a quarter to play — and you can have even greater access with a key. > and you’ve explicitly left a sign saying ‘you are not welcome here’ And the sign said "Long-haired freaky people Need not apply" So I tucked my hair up under my hat And I went in to ask him why He said, "You look like a fine upstandin' young man I think you'll do" So I took off my hat and said, "Imagine that Huh, me workin' for you"

michaelt 1y ago
> You require something the bot won't have that a human would. Is this why the “open web” is showing me a captcha or two, along with their cookie banner and newsletter pop up these days?
- bigbuppo 1y ago
  Up until people started making a big stink about CAPTCHAs being used for unpaid labor at scale, uh, well they had two purposes.