4 ms·

I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? It’s a different UI, sure, but there should be no dis

by nnx 1y ago

I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? It’s a different UI, sure, but there should be no discrimination towards it as there should be no discrimination towards, say, Links terminal browser, or some exotic Firefox derivative.

deleted 1y ago
[deleted]
ploynog 1y ago
Being daft on purpose? I haven't heard that using an alternative browser suddenly increases the traffic that a user generates by several orders of magnitude to the point where it can significantly increase hosting cost. A web scraper on the other hand easily can and they often account for the majority of traffic especially on smaller sites. So your comparison is at least naive assuming good intentions or malicious if not.
magicmicah85 1y ago
A crawler intends to scrape the content to reuse for its own purposes while a browser has a human being using it. There's different intents behind the tools.
- JimDabell 1y ago
  Cloudflare asked Perplexity this question: > Hello, would you be able to assist me in understanding this website? https:// https:// […] .com/ In this case, Perplexity had a human being using it. Perplexity wasn’t crawling the site, Perplexity was being operated by a human working for Cloudflare.
gruez 1y ago
>I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? AI broke the brains of many people. The internet isn't a monolith, but prior to the AI boom you'd be hard pressed to find people who were pro-copyright (except maybe a few who wanted to use it to force companies to comply with copyleft obligations), pro user-agent restrictions, or anti-scraping. Now such positions receive consistent representation in discussions, and are even the predominant position in some places (eg. reddit). In the past, people would invoke principled justifications for why they opposed those positions, like how copyright constituted an immoral monopoly and stifled innovation, or how scraping was so important to interoperability and the open web. Turns out for many, none of those principles really mattered and they only held those positions because they thought those positions would harm big evil publishing/media companies (ie. symbolic politics theory). When being anti-copyright or pro-scraping helped big evil AI companies, they took the opposite stance.
- Fraterkes 1y ago
  I think the intelligent conclusion would be that the people you are looking at have more nuanced beliefs than you initially thought. Talking about broken brains is often just mediocre projecting
  - gruez 1y ago
    >I think the intelligent conclusion would be that the people you are looking at have more nuanced beliefs than you initially thought. You don't seem to reject my claim that for many, principles took a backseat to "does this help or hurt evil corporations". If that's what passes as "nuance" to you, then sure. >Talking about broken brains is often just mediocre projecting To be clear, that part is metaphorical/hyperbolic and not meant to be taken literally. Obviously I'm not diagnosing people who switched sides with a psychiatric condition.
    - ipaddr 1y ago
      People never agreed DOSing a site to take copyright material was acceptable. Many people did not have a problem with taking copyright material in a respectful way that didn't kill the resource. LLMs are killing the resource. This isn't a corporation vs person issue. No issue with an llm having my content but big issue with my server being down because llms are hammering the same page over and over.
      - gruez 1y ago
        >People never agreed DOSing a site to take copyright material was acceptable. Many people did not have a problem with taking copyright material in a respectful way that didn't kill the resource. Has it be shown that perplexity engages in "DOSing"? I've heard of anecdotes of AI bots gone amuck, and maybe that's what's happening here, but cloudflare hasn't really shown that. All they did was set up a robots.txt and shown that perplexity bypassed it. There's probably archivers out there that's using youtube-dl to hit download from youtube at 1+Gbit/s, tens of times more than a typical viewer is downloading. Does that mean it's fair game to point to a random instance of someone using youtube-dl and characterizing that as "DOSing"?

by nnx 1y ago

deleted 1y ago

[deleted]

ploynog 1y ago

Being daft on purpose? I haven't heard that using an alternative browser suddenly increases the traffic that a user generates by several orders of magnitude to the point where it can significantly increase hosting cost. A web scraper on the other hand easily can and they often account for the majority of traffic especially on smaller sites. So your comparison is at least naive assuming good intentions or malicious if not.

magicmicah85 1y ago

A crawler intends to scrape the content to reuse for its own purposes while a browser has a human being using it. There's different intents behind the tools.

JimDabell 1y ago
Cloudflare asked Perplexity this question: > Hello, would you be able to assist me in understanding this website? https:// https:// […] .com/ In this case, Perplexity had a human being using it. Perplexity wasn’t crawling the site, Perplexity was being operated by a human working for Cloudflare.

gruez 1y ago

>I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? AI broke the brains of many people. The internet isn't a monolith, but prior to the AI boom you'd be hard pressed to find people who were pro-copyright (except maybe a few who wanted to use it to force companies to comply with copyleft obligations), pro user-agent restrictions, or anti-scraping. Now such positions receive consistent representation in discussions, and are even the predominant position in some places (eg. reddit). In the past, people would invoke principled justifications for why they opposed those positions, like how copyright constituted an immoral monopoly and stifled innovation, or how scraping was so important to interoperability and the open web. Turns out for many, none of those principles really mattered and they only held those positions because they thought those positions would harm big evil publishing/media companies (ie. symbolic politics theory). When being anti-copyright or pro-scraping helped big evil AI companies, they took the opposite stance.

Fraterkes 1y ago
I think the intelligent conclusion would be that the people you are looking at have more nuanced beliefs than you initially thought. Talking about broken brains is often just mediocre projecting
- gruez 1y ago
  >I think the intelligent conclusion would be that the people you are looking at have more nuanced beliefs than you initially thought. You don't seem to reject my claim that for many, principles took a backseat to "does this help or hurt evil corporations". If that's what passes as "nuance" to you, then sure. >Talking about broken brains is often just mediocre projecting To be clear, that part is metaphorical/hyperbolic and not meant to be taken literally. Obviously I'm not diagnosing people who switched sides with a psychiatric condition.
  - ipaddr 1y ago
    People never agreed DOSing a site to take copyright material was acceptable. Many people did not have a problem with taking copyright material in a respectful way that didn't kill the resource. LLMs are killing the resource. This isn't a corporation vs person issue. No issue with an llm having my content but big issue with my server being down because llms are hammering the same page over and over.
    - gruez 1y ago
      >People never agreed DOSing a site to take copyright material was acceptable. Many people did not have a problem with taking copyright material in a respectful way that didn't kill the resource. Has it be shown that perplexity engages in "DOSing"? I've heard of anecdotes of AI bots gone amuck, and maybe that's what's happening here, but cloudflare hasn't really shown that. All they did was set up a robots.txt and shown that perplexity bypassed it. There's probably archivers out there that's using youtube-dl to hit download from youtube at 1+Gbit/s, tens of times more than a typical viewer is downloading. Does that mean it's fair game to point to a random instance of someone using youtube-dl and characterizing that as "DOSing"?