3 ms·
I disagree with the other top-level comments at the moment: I believe Web Bot Auth is a useful and non-centralized emerging standard for self-identifying bots a
by bobbiechen 1y ago
I disagree with the other top-level comments at the moment: I believe Web Bot Auth is a useful and non-centralized emerging standard for self-identifying bots and agents.
This press release today is a better statement of _why_ this feature exists (as opposed to the submission link, which is nuts-and-bolts of implementing): https://blog.cloudflare.com/signed-agents/ https://blog.cloudflare.com/signed-agents/
Web Bot Auth is a way for bots to self-identify cryptographically. Unlike the user agent header (which is trivially spoofed) or known IPs (painful to manage), Web Bot Auth uses HTTP Message Signatures using the bot's key, which should be published at some well-known location.
This is a good thing! We want bots to be able to self-identify in a way that can't be impersonated. This gives website operators the power to allow or deny well-behaved bots with precision. It doesn't change anything about bots who try to hide their identity, who are not going to self-identify anyways.
It's worth reading the proposal on the details: https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-... . Nothing about this is limited to Cloudflare.
I'm also working on support for Web Bot Auth for our Agent Identification project at Stytch https://www.isagent.dev https://www.isagent.dev . Well-behaved bots benefit from this self-identification because it enables a better Agent Experience: https://stytch.com/blog/introducing-is-agent/ https://stytch.com/blog/introducing-is-agent/
- binarymax 1y agoI agree in principle, but I disagree that it should be designed and mandated by a private gatekeeper
- jacobn 1y agoIsn't that how most web standards got their start? One of the interested parties pushed something, then things evolved through the standards process? (And then it can of course get derailed, but that's a separate story)
- jrochkind1 1y agoWhat's now at the top has links to IETF drafts in the first paragraph. What am I missing? A way to authenticate identity for crawlers so I can allow-list ones I want to get in, exempt them from turnstile/captcha, etc -- is something I need. I'm not following what makes this controversial. Cryptographic verification of identity for web requests, sounds right.
- binarymax 1y agoI think about failure modes. What happens if cloudflare decides you are a bot and you’re not. What recourse do you have? What are the formal mechanisms to ensure a person is not blocked from the majority of the web because cloudflare is a middleman and you are a false positive?
- justincormack 1y agoThis is not a spec sbout false positives, ir is about self identification as a bot.
- delroth 1y agoDon't use a user agent that sends signed headers identifying you as a bot? How are any of the failure modes you mention not /improved/ by the spec proposal this comment section is about?
- jrochkind1 1y agoI am not following what any of that has to do with the Web Bot Auth protocol? it seems like complaints about Cloudflare's anti-DOS protection services and how they have a monopoly on such, I get that. I'm not seeing the connection to a protocol for bots/crawlers voluntarily cryptographically signing their http requests, so sites (anyone implementing the protocol not just cloudflare) can use it to authenticate known actors? I am interested in using it to exempt bots/crawlers I trust/support/have an agreement with from the anti-bot measures I, like many, am being forced to implement to keep our sites up under an enormously increased wave of what is apparently AI-training-motivated repeat crawling. Right now these measures are keeping out bots I don't want to keep out too. I would like to be able to securely identify them to let them in.
- blackswatter 1y ago[dead]
- sneak 1y agoThe problem with this is that key generation is free, so being a well-behaved unknown bot is the same as being an unidentified bot, which means that you go in the block/captcha/throttle bucket. It is only useful for whitelisting bots, not for banning bad ones, as bad ones can rotate keys. Whitelisting clients by identity is the death of the open web, and means that nobody will ever be able to compete with capital on even footing.
- everfrustrated 1y agoIsn't this somewhat equilivent to ensuring cookies are required? Obviously this technology is different but the same sort of result. What's the end game here? All humans end up having to use a unique encryption key to prove their humanness also?
- pmontra 1y agoI understand your concern and we are probably headed into that direction, but that does not prove humanness any more than the subject of this post proves botness. They prove the knowledge of the value of a key.
- marginalia_nu 1y agoI generally agree it's a good thing. It stacks the incentives so that bots can meaningfully build a good reputation, and be rewarded for behaving well. That said, I do think it's the whole procedure is more than a bit overcomplicated to the degree where I doubt it will be widely implemented. You could likely achieve almost the full effect with a request signing alone.
- account42 1y ago> This is a good thing! We want bots to be able to self-identify in a way that can't be impersonated. Who is we? I absolutely don't want that.
- estearum 1y agoEarnest question: why not? I would think "option to prove who you are and guarantee not to be impersonated" is a pretty broadly appealing capability except to people trying to do the impersonating.
- skeezyboy 1y ago>"option to prove who you are and guarantee not to be impersonated" guaranteed as long as no attacker gets hold of the private key, which cannot be guaranteed