6 ms·

> it could present unacceptable risks for application developers or be used as a method for malicious attacks (e.g. credential stuffing or fake account creation

by diggan 2y ago

> it could present unacceptable risks for application developers or be used as a method for malicious attacks (e.g. credential stuffing or fake account creation). The article seems to want to distinguish between "bad" and "good" bots, yet beyond the introduction, seems to treat them exactly the same. Why are website authors so adamant I need to use whatever client they want to consume their content? If you put up a blog online, available publicly, do you really care if I read it in my terminal or via Firefox with uBlock? Or via an AI agent that fetches the article for me and tags it for me for further categorization? It seems like suddenly half the internet forgot about the term "user-agent", which up until recently was almost always our browsers, but sometimes feed readers, which was acceptable it seems. But now we have a new user-agent available, "AI Agents", that somehow is unacceptable and should be blocked? I'm not sure I agree with the premise that certain user-agents should be blocked, and I'll probably continue to let everyone chose their own user-agent when using my websites, it's literally one of the reasons I use the web and internet in the first place.

hansonkd 2y ago
> It seems like suddenly half the internet forgot about the term "user-agent", which up until recently was almost always our browsers, but sometimes feed readers, which was acceptable it seems. Was it really "suddenly"? it seems like for the past decade there has been an ongoing push to make everyone use "chromium" based browsers. I remember 10-15 years ago you would get blocked for not using IE or whatever, even though the site worked fine and there was no technical reason for the block. It was over 12 years ago when google effectively killed RSS to prevent alternative methods of access.
- diggan 2y ago
  > I remember 10-15 years ago you would get blocked for not using IE or whatever, even though the site worked fine and there was no technical reason for the block Reminds me of when I discovered that Google Inbox worked in Firefox, even though Google decided to only allow Chrome to access it: https://news.ycombinator.com/item?id=8606879 https://news.ycombinator.com/item?id=8606879 - "Why Is Google Blocking Inbox on Firefox?" - 213 points | Nov 14, 2014 | 208 comments (correct link to the gist is https://gist.github.com/victorb/1d0f4ee6dc5ec0d6646e https://gist.github.com/victorb/1d0f4ee6dc5ec0d6646e today) I think that "ongoing" push you're talking about was/is accidental, because a lot of people use Chrome. What I'm seeing now seems to be intentional, because people disagree with the ethics/morals surrounding AI, or seeing a large impact on their servers because of resource consumption, so more philosophical and/or practical, rather than accidental. But who knows, I won't claim to have exact insights into exactly what caused "Chrome is the new IE", could be it was very intentional and they never stopped.
  - SoftTalker 2y ago
    I don't remember Google (search, at least) ever not working in any browser I tried, and I used some oddball browsers over the years. Maybe apps like gmail and docs, but they simply would not work in other browsers. Remember in its early years Chrome was a darling because it was supporting the "modern" web. That was the whole stated reason Google developed Chrome: to support modern, rich web applications, and force other browsers to do the same if they wanted to stay relevant. Nobody guessed that Chrome would eventually be the new IE.
    - int_19h 2y ago
      Google has a very long history of downgrading or even outright blocking off their services based on User-Agent. Them doing it to Opera in particular was a big story back in those "early days". And no, it's not because it wouldn't work - people literally made it work by spoofing the UA.
- Klonoar 2y ago
  …10-15 years ago? Try like 20(~+)
zb3 2y ago
They only care about ad revenue. I guess if bots were paying them they wouldn't need to detect humans anymore..
Etheryte 2y ago
The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked. If I didn't block them, I would need to pay for a beefier server. All the while this doesn't give me or my regular visitors any benefit, it's just large corporations driving up my hosting costs.
- diggan 2y ago
  > The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked What are they downloading, like heavy videos and stuff? Initiating heavy processes or similar?
  - radlad 2y ago
    It takes 125,000 4MB requests to use up 0.5 TB bandwidth, which is the lowest offered by Vultr. I could see this being an issue for personal sites that include photos.
mcstempel 2y ago
Hey there, I'm the author of the post. I'm actually pretty sympathetic to your viewpoint, and I wanted to clarify my stance. I actually spent years working at a "good bot" company (Plaid), which focused on making users' financial data portable. The main reason Plaid existed was that banks made it hard for users to permission their data to other apps -- typically not solely out of security concerns, but to also actively limit competition. So, I know how the "bot detection" argument can be weaponized in unideal ways. That said, I think it’s reasonable for app developers to decide how their services are consumed (there are real cost drivers many have to think about) -- which includes the ability to have monitoring & guardrails in place for riskier traffic. If an app couldn't detect good bots, that app also can't do things like 1) support necessary revocation mechanisms for end users if they want to clawback agent permissions or 2) require human-in-the-loop authorization for sensitive actions. Main thing I care about is that AI agent use remains safe and aligned with user intent. For your example of an anonymous read-only site (e.g. blog), I'm less worried about that than an AI agent with read-write access on behalf of a real human's account. My idealistic long-term view though is that supporting AI agent use cases will eventually become table stakes. Users will gravitate toward services that let them automate tedious tasks and integrate AI assistants into their workflows. Companies that resist this trend may find themselves at a competitive disadvantage. Ultimately, this has started to happen with banking & OAuth, though pretty slowly.
- do_not_redeem 2y ago
  It seems like cases (1) and (2) would both be better handled by letting the user give their user agent a separate security context if they choose, instead of trying to detect/guess what kind of browser made that http request. I'm thinking about things like oauth permissions, GitHub's sudo mode, etc. Otherwise your magic detection code will inevitably end up telling an ELinks user "sorry, you need to download chrome to view your payment info".
  - mcstempel 2y ago
    You read our mind! https://stytch.com/blog/the-age-of-agent-experience/ https://stytch.com/blog/the-age-of-agent-experience/ Very much agreed that's the long-term goal, but I think we'll live in a world where most apps don't support oauth for a while longer (though I'd love for all of them to -- we're actually announcing something next week that makes this easy for any app to do) But we're also envisioning an interim period where users are delegating to unsanctioned external agents (e.g. OpenAI Operator, Anthropic Computer Use API, etc.) prior to apps catching up and offering proper oauth
cess11 2y ago
As an almost fanatic VPN and Tor user I'm already used to being blocked and circumventing it, usually by exiting through a data center IPv4 but I sometimes check whether they let SeleniumBase access with a chromedriver and surprisingly often they do. The Internet is a rather hostile place, I don't think that'll change anytime soon.
reverendsteveii 2y ago
I came here to say exactly this. Of course bots can do things that are shitty but we need to detect and ban the behavior and resist the urge to also try to detect and ban the type of user that tends to exhibit that behavior. If a bot comes onto my page, hands off the captcha to a human, then does human things at human speed and human scale I shouldn't care that they're a bot and I should allow them to do what they're doing. If a human comes onto my page and starts trying to brute force directory structures, mass-downloading tons of huge files and otherwise causing a problem I shouldn't care that they're a human and I should block them. This whole idea of bot detection and blocking seems to be an inversion of what I think is the best design principle we've discovered in the history of software development: build things that do simple, useful things without regard to who is using them or for what, then let consumers surprise you with what they do with it. Banning non-abusive agents is just locking out potential upsides for your app. OTOH if you make your living serving ads a bot bypassing your monetization is a problem for you. Either you detect and block them or eventually the value of an ad impression in your app will approach zero. So in some cases I guess merely not being a human is the abusive behavior.
golergka 2y ago
I'm building a service which needs to extract rss feeds from pages (hntorss.com if you're interested). Nothing else. From any rational point of view, website owner would actively want this parser to work as easily as possible — the whole point is for users to see the content you publish! Alas, I still get rate-limited, 400-ed and others because of user agent and other bot-detection mechanisms.
- chii 2y ago
  > the whole point is for users to see the content you publish! no, the whole point (for most sites) is to make money off the users visiting said site (currently via advertising). Another third party service which slurps the data, and redirect the users to a different site to consume the data means the original site lost the revenue, but paid the bandwidth cost. So it's understandable that many sites want to block such agents.
  - strogonoff 2y ago
    Even if it is not for profit (or especially so), the point of any publication is not just to get people to know something—you at least want people to read what you wrote and appreciate you for this. Using Web normally, with search and all, is well-behaved in this regard, but using attribution-stripping technology isn’t. If your readers don’t know you exist and you don’t know who your readers are or if they even exist, you basically become a ghost writer, content producer for LLMs (and in many cases some commercial LLM operator also makes money off your work, too).
  - golergka 2y ago
    Then you wouldn't have RSS feeds in the first place. I'm talking about sites that decide to have them for one reason or another.
digitaltrees 2y ago
thank you!!!! This is the right answer.
1shooner 2y ago
>now we have a new user-agent available, "AI Agents", that somehow is unacceptable and should be blocked? Giving deference or even exclusive access to certain service clients is as old as the commercial web. The article specifically cites security or other risk as the reason. Of course commercial media on the web today put conditions on the consumption of what they publish: ad-blocker nag screens, paywalls, etc. Usually that's just a commercial interest, but what about other conditions, like a disclaimer for medical or legal advice? AI Agents will cite your content without necessarily the context or due diligence you may be legally or ethically obligated to provide with that content. Generally, I agree that it holds us back from what the 'Agent Experience' web will inevitably need to become, but there are valid reasons for the incumbent patterns that should be resolved in a mutually beneficial way.
rkagerer 2y ago
Many (most?) commercial websites have terms of service that ban you from using bots, scripting, etc. It's a similar travesty.
immibis 2y ago
Because they make their money from showing ads to human eyeballs and not to AIs.

by diggan 2y ago

hansonkd 2y ago

> It seems like suddenly half the internet forgot about the term "user-agent", which up until recently was almost always our browsers, but sometimes feed readers, which was acceptable it seems. Was it really "suddenly"? it seems like for the past decade there has been an ongoing push to make everyone use "chromium" based browsers. I remember 10-15 years ago you would get blocked for not using IE or whatever, even though the site worked fine and there was no technical reason for the block. It was over 12 years ago when google effectively killed RSS to prevent alternative methods of access.

diggan 2y ago
> I remember 10-15 years ago you would get blocked for not using IE or whatever, even though the site worked fine and there was no technical reason for the block Reminds me of when I discovered that Google Inbox worked in Firefox, even though Google decided to only allow Chrome to access it: https://news.ycombinator.com/item?id=8606879 https://news.ycombinator.com/item?id=8606879 - "Why Is Google Blocking Inbox on Firefox?" - 213 points | Nov 14, 2014 | 208 comments (correct link to the gist is https://gist.github.com/victorb/1d0f4ee6dc5ec0d6646e https://gist.github.com/victorb/1d0f4ee6dc5ec0d6646e today) I think that "ongoing" push you're talking about was/is accidental, because a lot of people use Chrome. What I'm seeing now seems to be intentional, because people disagree with the ethics/morals surrounding AI, or seeing a large impact on their servers because of resource consumption, so more philosophical and/or practical, rather than accidental. But who knows, I won't claim to have exact insights into exactly what caused "Chrome is the new IE", could be it was very intentional and they never stopped.
- SoftTalker 2y ago
  I don't remember Google (search, at least) ever not working in any browser I tried, and I used some oddball browsers over the years. Maybe apps like gmail and docs, but they simply would not work in other browsers. Remember in its early years Chrome was a darling because it was supporting the "modern" web. That was the whole stated reason Google developed Chrome: to support modern, rich web applications, and force other browsers to do the same if they wanted to stay relevant. Nobody guessed that Chrome would eventually be the new IE.
  - int_19h 2y ago
    Google has a very long history of downgrading or even outright blocking off their services based on User-Agent. Them doing it to Opera in particular was a big story back in those "early days". And no, it's not because it wouldn't work - people literally made it work by spoofing the UA.
Klonoar 2y ago
…10-15 years ago? Try like 20(~+)

zb3 2y ago

They only care about ad revenue. I guess if bots were paying them they wouldn't need to detect humans anymore..

Etheryte 2y ago

The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked. If I didn't block them, I would need to pay for a beefier server. All the while this doesn't give me or my regular visitors any benefit, it's just large corporations driving up my hosting costs.

diggan 2y ago
> The problem is resource consumption. On some of my servers, scrapers, bots etc make up a vast majority of both the bandwidth and CPU usage when left unchecked What are they downloading, like heavy videos and stuff? Initiating heavy processes or similar?
- radlad 2y ago
  It takes 125,000 4MB requests to use up 0.5 TB bandwidth, which is the lowest offered by Vultr. I could see this being an issue for personal sites that include photos.

mcstempel 2y ago

Hey there, I'm the author of the post. I'm actually pretty sympathetic to your viewpoint, and I wanted to clarify my stance. I actually spent years working at a "good bot" company (Plaid), which focused on making users' financial data portable. The main reason Plaid existed was that banks made it hard for users to permission their data to other apps -- typically not solely out of security concerns, but to also actively limit competition. So, I know how the "bot detection" argument can be weaponized in unideal ways. That said, I think it’s reasonable for app developers to decide how their services are consumed (there are real cost drivers many have to think about) -- which includes the ability to have monitoring & guardrails in place for riskier traffic. If an app couldn't detect good bots, that app also can't do things like 1) support necessary revocation mechanisms for end users if they want to clawback agent permissions or 2) require human-in-the-loop authorization for sensitive actions. Main thing I care about is that AI agent use remains safe and aligned with user intent. For your example of an anonymous read-only site (e.g. blog), I'm less worried about that than an AI agent with read-write access on behalf of a real human's account. My idealistic long-term view though is that supporting AI agent use cases will eventually become table stakes. Users will gravitate toward services that let them automate tedious tasks and integrate AI assistants into their workflows. Companies that resist this trend may find themselves at a competitive disadvantage. Ultimately, this has started to happen with banking & OAuth, though pretty slowly.

do_not_redeem 2y ago
It seems like cases (1) and (2) would both be better handled by letting the user give their user agent a separate security context if they choose, instead of trying to detect/guess what kind of browser made that http request. I'm thinking about things like oauth permissions, GitHub's sudo mode, etc. Otherwise your magic detection code will inevitably end up telling an ELinks user "sorry, you need to download chrome to view your payment info".
- mcstempel 2y ago
  You read our mind! https://stytch.com/blog/the-age-of-agent-experience/ https://stytch.com/blog/the-age-of-agent-experience/ Very much agreed that's the long-term goal, but I think we'll live in a world where most apps don't support oauth for a while longer (though I'd love for all of them to -- we're actually announcing something next week that makes this easy for any app to do) But we're also envisioning an interim period where users are delegating to unsanctioned external agents (e.g. OpenAI Operator, Anthropic Computer Use API, etc.) prior to apps catching up and offering proper oauth

cess11 2y ago

As an almost fanatic VPN and Tor user I'm already used to being blocked and circumventing it, usually by exiting through a data center IPv4 but I sometimes check whether they let SeleniumBase access with a chromedriver and surprisingly often they do. The Internet is a rather hostile place, I don't think that'll change anytime soon.

reverendsteveii 2y ago

I came here to say exactly this. Of course bots can do things that are shitty but we need to detect and ban the behavior and resist the urge to also try to detect and ban the type of user that tends to exhibit that behavior. If a bot comes onto my page, hands off the captcha to a human, then does human things at human speed and human scale I shouldn't care that they're a bot and I should allow them to do what they're doing. If a human comes onto my page and starts trying to brute force directory structures, mass-downloading tons of huge files and otherwise causing a problem I shouldn't care that they're a human and I should block them. This whole idea of bot detection and blocking seems to be an inversion of what I think is the best design principle we've discovered in the history of software development: build things that do simple, useful things without regard to who is using them or for what, then let consumers surprise you with what they do with it. Banning non-abusive agents is just locking out potential upsides for your app. OTOH if you make your living serving ads a bot bypassing your monetization is a problem for you. Either you detect and block them or eventually the value of an ad impression in your app will approach zero. So in some cases I guess merely not being a human is the abusive behavior.

golergka 2y ago

I'm building a service which needs to extract rss feeds from pages (hntorss.com if you're interested). Nothing else. From any rational point of view, website owner would actively want this parser to work as easily as possible — the whole point is for users to see the content you publish! Alas, I still get rate-limited, 400-ed and others because of user agent and other bot-detection mechanisms.

chii 2y ago
> the whole point is for users to see the content you publish! no, the whole point (for most sites) is to make money off the users visiting said site (currently via advertising). Another third party service which slurps the data, and redirect the users to a different site to consume the data means the original site lost the revenue, but paid the bandwidth cost. So it's understandable that many sites want to block such agents.
- strogonoff 2y ago
  Even if it is not for profit (or especially so), the point of any publication is not just to get people to know something—you at least want people to read what you wrote and appreciate you for this. Using Web normally, with search and all, is well-behaved in this regard, but using attribution-stripping technology isn’t. If your readers don’t know you exist and you don’t know who your readers are or if they even exist, you basically become a ghost writer, content producer for LLMs (and in many cases some commercial LLM operator also makes money off your work, too).
- golergka 2y ago
  Then you wouldn't have RSS feeds in the first place. I'm talking about sites that decide to have them for one reason or another.

digitaltrees 2y ago

thank you!!!! This is the right answer.

1shooner 2y ago

>now we have a new user-agent available, "AI Agents", that somehow is unacceptable and should be blocked? Giving deference or even exclusive access to certain service clients is as old as the commercial web. The article specifically cites security or other risk as the reason. Of course commercial media on the web today put conditions on the consumption of what they publish: ad-blocker nag screens, paywalls, etc. Usually that's just a commercial interest, but what about other conditions, like a disclaimer for medical or legal advice? AI Agents will cite your content without necessarily the context or due diligence you may be legally or ethically obligated to provide with that content. Generally, I agree that it holds us back from what the 'Agent Experience' web will inevitably need to become, but there are valid reasons for the incumbent patterns that should be resolved in a mutually beneficial way.

rkagerer 2y ago

Many (most?) commercial websites have terms of service that ban you from using bots, scripting, etc. It's a similar travesty.

immibis 2y ago

Because they make their money from showing ads to human eyeballs and not to AIs.