4 ms·

If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Di

If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.

bayindirh 2y ago
What if the scraper is not respecting robots.txt to begin with? Aren't they unethical enough to warrant a stronger method to prevent scraping?
- skeledrew 2y ago
  Should there be a difference in treatment between a user going on a website and manually copying the content over to a bot to process vs giving the bot the URL so it does the fetching as well? I've done both (mainly to get summaries or translations) and I know which I generally prefer.
  - bayindirh 2y ago
    Ideally no, but there are established norms and unwritten rules. Plus, a mechanism was built to communicate the limits. These norms were working for decades. The fences were reasonable because the demands were reasonable and both sides understood why they are there and respected these borders. This peace has been broken, norms are thrown away and people who did this cheered for what they did. Now, the people are fighting back. People were silent because the system was working. It was akin to mark some doors "authorized personnel only" but leaving them unlocked. People and programs respected these stickers. Now there are people and programs who don't, so people started to reinforce these doors. It doesn't matter what you prefer. The apples are spoiled now. There's no turning back. The days of peace and harmony are over, thanks to "move fast break things. We're doing something amazing anyway, and we don't no permission!" people. If your use is benign but my filter is preventing that use, you should get mad at the parties who caused this fence to appear. It's not my fault to put a fence to protect myself. To see the current state of affairs, see this list [0]. I'm very sensitive to ethical issues about training your model with my data without my consent, and selling it to earn monies. I don't care about how you stretch fair-use. The moment you earn money from your model, it's not fair-use anymore [1]. [0]: https://notes.bayindirh.io/notes/Lists/Discussions+about+Artificial+Intelligence https://notes.bayindirh.io/notes/Lists/Discussions+about+Art... [1]: https://news.ycombinator.com/item?id=39188979 https://news.ycombinator.com/item?id=39188979
    - skeledrew 2y ago
      Well, what'll happen for the most part is not users being mad, but a general migration to fenceless areas. Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot. And there will always be authors who don't mind their content being monitized or utilized by AI.
      - bayindirh 2y ago
        > but a general migration to fenceless areas. This is the ultimate goal already. We want every netizen (human or machine) to obey the written and unwritten rules and be a good netizen. > Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot. Absolutely not. I don't want my content to end in an LLM, period. I don't license it that way, and I don't consent. Humans are always welcome to read it though. An LLM is an hallucinating parrot anyway, so I don't want my words to be used in that LSD fueled computing chaos. > And there will always be authors who don't mind their content being monitized or utilized by AI. Yes, and there will always be authors who do mind their content being monitized [sic] or utilized by AI. This is life.
        skeledrew 2y ago
        > good netizen The rules dictating what this means change over time as context changes. And this is definitely one or the largest shifts since the advent of the www. > Absolutely not. OK, but it seems in that case you'll have to delist from search engines and any other place where an LLM may get a preview of what a link that a human gave it is about. And there's likely a spiral here, as as you remove content references from more access points, fewer humans will come across them leading to fewer LLM-based queries, so you'll definitely get your wish. Just not the way you envisioned. The LLM may hallucinate at times, but many including myself find the pros outweigh the cons, and will increasingly gravitate toward using such services for an increasing variety of tasks. This is the evolving state of the internet.
bakugo 2y ago
There is nothing unethical about not wanting AI companies to steal your content and sell it for a profit.
croes 2y ago
>and you'd do it too if you wanted your program to work properly. If I know the creator of the page doesn't want his page used by my program I wouldn't do it. >Discriminating based on User-Agent string is the unethical part. Not being exploited by an AI company is unethical? Robots.txt is made for telling bot identified by user agent what they are allowed to read.
- lolinder 2y ago
  > Robots.txt is made for telling bot identified by user agent what they are allowed to read. Specifically it's meant for instructing "automatic clients known as crawlers" [0]. A crawler is defined by MDN as "a program, often called a bot or robot, which systematically browses the Web to collect data from webpages." [1] As generally understood, wget is not a crawler even though it may be used to build one. Neither is curl. A crawler is a program which systematically browses the web, usually to build a search index. I see no evidence that Perplexity's crawler is ignoring robots.txt, I only see evidence that when a user does a one-off request for a specific URL then Perplexity uses Chrome to access the site. Basically, OP is using the wrong tool for the job and complaining when it doesn't work. If he wants to be excluded from Perplexity for one-off requests (as distinct from crawling) he needs to reach out to them, there is no applicable RFC. [0] https://www.rfc-editor.org/rfc/rfc9309.html https://www.rfc-editor.org/rfc/rfc9309.html [1] https://developer.mozilla.org/en-US/docs/Glossary/Crawler https://developer.mozilla.org/en-US/docs/Glossary/Crawler
marcus0x62 2y ago
Please explain - in detail - why using information communicated by the client to change how my server operates is “unethical”. Keep in mind I pay money and expend time to provide free content for people to consume.
- tensor 2y ago
  Here is a simple example. If you made your website only work in say, Microsoft Edge, and blocked everyone else telling them to download Edge. I'd think you're an asshole. Whether or not being an ass is unethical I'll leave to the philosophers. Clearly there are many other scenarios, and many that are more muddy, but overall when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion. The extreme end result of this is no more open web, just force people to download your app to consume your content. This is happening too and it sucks.
  - marcus0x62 2y ago
    The entire premise of the parent posters comment was that this is specifically unethical, so you lost me at the part where you deliberately decided to not address that in your reply.
    - stevenhuang 2y ago
      > when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion. ?
      - marcus0x62 2y ago
        Read it again. Especially this part: > Whether or not being an ass is unethical I'll leave to the philosophers.
surfingdino 2y ago
I find your ethical standards perplexing...
rknightuk 2y ago
I wouldn’t because I have ethics.
- sebzim4500 2y ago
  Here's my user agent on chrome: >Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 There are at least five lies here. * It isn't made by Mozilla * It doesn't use WebKit * It doesn't use KHTML * It isn't safari * That isn't even my version of chrome, presumably it hides the minor/patch versions for privacy reasons. Lying in your user agent in order to make the internet work is a practice that is almost as old as user agents. Your browser is almost certainly doing it right now to look at this comment.
  - Findecanor 2y ago
    > Lying in your user agent in order to make the internet work is a practice that is almost as old as user agents. Twenty years ago, I set up a web proxy on my Linux PC at home to change the User Agent because I was tired of getting popups about my web browser (Opera) not being Mozilla or Internet Explorer. It even contained the text "Shut the F up and follow w3c standards!" at first, until I realised that sites could use that to track me.

bayindirh 2y ago

What if the scraper is not respecting robots.txt to begin with? Aren't they unethical enough to warrant a stronger method to prevent scraping?

skeledrew 2y ago
Should there be a difference in treatment between a user going on a website and manually copying the content over to a bot to process vs giving the bot the URL so it does the fetching as well? I've done both (mainly to get summaries or translations) and I know which I generally prefer.
- bayindirh 2y ago
  Ideally no, but there are established norms and unwritten rules. Plus, a mechanism was built to communicate the limits. These norms were working for decades. The fences were reasonable because the demands were reasonable and both sides understood why they are there and respected these borders. This peace has been broken, norms are thrown away and people who did this cheered for what they did. Now, the people are fighting back. People were silent because the system was working. It was akin to mark some doors "authorized personnel only" but leaving them unlocked. People and programs respected these stickers. Now there are people and programs who don't, so people started to reinforce these doors. It doesn't matter what you prefer. The apples are spoiled now. There's no turning back. The days of peace and harmony are over, thanks to "move fast break things. We're doing something amazing anyway, and we don't no permission!" people. If your use is benign but my filter is preventing that use, you should get mad at the parties who caused this fence to appear. It's not my fault to put a fence to protect myself. To see the current state of affairs, see this list [0]. I'm very sensitive to ethical issues about training your model with my data without my consent, and selling it to earn monies. I don't care about how you stretch fair-use. The moment you earn money from your model, it's not fair-use anymore [1]. [0]: https://notes.bayindirh.io/notes/Lists/Discussions+about+Artificial+Intelligence https://notes.bayindirh.io/notes/Lists/Discussions+about+Art... [1]: https://news.ycombinator.com/item?id=39188979 https://news.ycombinator.com/item?id=39188979
  - skeledrew 2y ago
    Well, what'll happen for the most part is not users being mad, but a general migration to fenceless areas. Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot. And there will always be authors who don't mind their content being monitized or utilized by AI.
    - bayindirh 2y ago
      > but a general migration to fenceless areas. This is the ultimate goal already. We want every netizen (human or machine) to obey the written and unwritten rules and be a good netizen. > Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot. Absolutely not. I don't want my content to end in an LLM, period. I don't license it that way, and I don't consent. Humans are always welcome to read it though. An LLM is an hallucinating parrot anyway, so I don't want my words to be used in that LSD fueled computing chaos. > And there will always be authors who don't mind their content being monitized or utilized by AI. Yes, and there will always be authors who do mind their content being monitized [sic] or utilized by AI. This is life.
      - skeledrew 2y ago
        > good netizen The rules dictating what this means change over time as context changes. And this is definitely one or the largest shifts since the advent of the www. > Absolutely not. OK, but it seems in that case you'll have to delist from search engines and any other place where an LLM may get a preview of what a link that a human gave it is about. And there's likely a spiral here, as as you remove content references from more access points, fewer humans will come across them leading to fewer LLM-based queries, so you'll definitely get your wish. Just not the way you envisioned. The LLM may hallucinate at times, but many including myself find the pros outweigh the cons, and will increasingly gravitate toward using such services for an increasing variety of tasks. This is the evolving state of the internet.

bakugo 2y ago

There is nothing unethical about not wanting AI companies to steal your content and sell it for a profit.

croes 2y ago

>and you'd do it too if you wanted your program to work properly. If I know the creator of the page doesn't want his page used by my program I wouldn't do it. >Discriminating based on User-Agent string is the unethical part. Not being exploited by an AI company is unethical? Robots.txt is made for telling bot identified by user agent what they are allowed to read.

lolinder 2y ago
> Robots.txt is made for telling bot identified by user agent what they are allowed to read. Specifically it's meant for instructing "automatic clients known as crawlers" [0]. A crawler is defined by MDN as "a program, often called a bot or robot, which systematically browses the Web to collect data from webpages." [1] As generally understood, wget is not a crawler even though it may be used to build one. Neither is curl. A crawler is a program which systematically browses the web, usually to build a search index. I see no evidence that Perplexity's crawler is ignoring robots.txt, I only see evidence that when a user does a one-off request for a specific URL then Perplexity uses Chrome to access the site. Basically, OP is using the wrong tool for the job and complaining when it doesn't work. If he wants to be excluded from Perplexity for one-off requests (as distinct from crawling) he needs to reach out to them, there is no applicable RFC. [0] https://www.rfc-editor.org/rfc/rfc9309.html https://www.rfc-editor.org/rfc/rfc9309.html [1] https://developer.mozilla.org/en-US/docs/Glossary/Crawler https://developer.mozilla.org/en-US/docs/Glossary/Crawler

marcus0x62 2y ago

Please explain - in detail - why using information communicated by the client to change how my server operates is “unethical”. Keep in mind I pay money and expend time to provide free content for people to consume.

tensor 2y ago
Here is a simple example. If you made your website only work in say, Microsoft Edge, and blocked everyone else telling them to download Edge. I'd think you're an asshole. Whether or not being an ass is unethical I'll leave to the philosophers. Clearly there are many other scenarios, and many that are more muddy, but overall when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion. The extreme end result of this is no more open web, just force people to download your app to consume your content. This is happening too and it sucks.
- marcus0x62 2y ago
  The entire premise of the parent posters comment was that this is specifically unethical, so you lost me at the part where you deliberately decided to not address that in your reply.
  - stevenhuang 2y ago
    > when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion. ?
    - marcus0x62 2y ago
      Read it again. Especially this part: > Whether or not being an ass is unethical I'll leave to the philosophers.

surfingdino 2y ago

I find your ethical standards perplexing...

rknightuk 2y ago

I wouldn’t because I have ethics.

sebzim4500 2y ago
Here's my user agent on chrome: >Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 There are at least five lies here. * It isn't made by Mozilla * It doesn't use WebKit * It doesn't use KHTML * It isn't safari * That isn't even my version of chrome, presumably it hides the minor/patch versions for privacy reasons. Lying in your user agent in order to make the internet work is a practice that is almost as old as user agents. Your browser is almost certainly doing it right now to look at this comment.
- Findecanor 2y ago
  > Lying in your user agent in order to make the internet work is a practice that is almost as old as user agents. Twenty years ago, I set up a web proxy on my Linux PC at home to change the User Agent because I was tired of getting popups about my web browser (Opera) not being Mozilla or Internet Explorer. It even contained the text "Shut the F up and follow w3c standards!" at first, until I realised that sites could use that to track me.