4 ms·

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without

by grg0 1y ago

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without payment? So they need some way to tell the crawler and Joe apart, and presumably they require the crawler to set certain request headers. The headers aren't just to issue the payment, it's to identify the crawler in the first place?

xg15 1y ago
The writeup doesn't talk about actively misbehaving crawlers a lot, but this bit implies for me that the headers are for the "happy path", I.e. crawlers wanting to pay: > Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. I don't see how it would make sense otherwise, as the requirements for crawlers include applying for a registration with Cloudflare. Who in their right mind would jump through registration hoops only so they can not access a site? This wouldn't even keep away the crawlers that are operating today. I agree there has to be some way to distinguish crawlers from regular users, but the only way I can see how this could be done is with bot detection algorithms. ...which are imperfect and will likely flag some legitimate human users as bots. So yes, this will probably leading to web browsing becoming even more unpleasant.
quixoticaxolotl 1y ago
Joe will be fine. Cloudflare is pretty good at differentiating humans from bot traffic - see how we do it here: https://developers.cloudflare.com/turnstile/ https://developers.cloudflare.com/turnstile/ The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words: - we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted, as attested by the signature headers - in which case we let them pay - and then they get to access the content (Disclosure: I wrote the web bot auth implementation Cloudflare uses for pay per crawl)
- grg0 1y ago
  Ok, well, thanks for the clarification.
- xg15 1y ago
  Thanks for replying! Do you have some provision for false positives as well, like sending a captcha in the body of the 402 response? (So in case the client was a human and not a bot, they could still try to solve the captcha)

by grg0 1y ago

xg15 1y ago

The writeup doesn't talk about actively misbehaving crawlers a lot, but this bit implies for me that the headers are for the "happy path", I.e. crawlers wanting to pay: > Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. I don't see how it would make sense otherwise, as the requirements for crawlers include applying for a registration with Cloudflare. Who in their right mind would jump through registration hoops only so they can not access a site? This wouldn't even keep away the crawlers that are operating today. I agree there has to be some way to distinguish crawlers from regular users, but the only way I can see how this could be done is with bot detection algorithms. ...which are imperfect and will likely flag some legitimate human users as bots. So yes, this will probably leading to web browsing becoming even more unpleasant.

quixoticaxolotl 1y ago

Joe will be fine. Cloudflare is pretty good at differentiating humans from bot traffic - see how we do it here: https://developers.cloudflare.com/turnstile/ https://developers.cloudflare.com/turnstile/ The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words: - we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted, as attested by the signature headers - in which case we let them pay - and then they get to access the content (Disclosure: I wrote the web bot auth implementation Cloudflare uses for pay per crawl)

grg0 1y ago
Ok, well, thanks for the clarification.
xg15 1y ago
Thanks for replying! Do you have some provision for false positives as well, like sending a captcha in the body of the 402 response? (So in case the client was a human and not a bot, they could still try to solve the captcha)