3 ms·

In this 'arms race', will this serve as an actual deterrent? Can anyone involved in scraping chime in?

by momojo 2y ago

yuvalr1 2y ago
I am not involved in scraping, but to me this sounds like simply another tool in the arsenal. They say it's hard for the scraper to realize it has been caught this way because it's not being blocked. However, I don't see anything preventing scrapers from implementing heuristics to realize that.
- pona-a 2y ago
  Detecting the actual AI generated content is not an easy problem. Not following deep links and recognizing the particular website template and structure is easier. I really feel a monoculture of anti-bot tools can defeat their effectiveness. When you have to optimize for Anubis, Nepenthes, Quixotic, and Cloudflare, each independently evolving and different in method and implementation, it might just be practical to give up.
dudus 2y ago
This won't stop the big ones, Google, Meta, OpenAI, Perplexity, or even the Chinese Govt. But it will make it harder for new entrants.
- skybrian 2y ago
  Not sure it’s targeted at them, either. Which of those entities have misbehaving bots? Seems like Google, at least, should be following robots.txt?
  - bigiain 2y ago
    Google 2025 is not the Google you remember and respect. "GoogleAssociationService bot was kind enough to ask 1,000,000+ times yesterday for the same file from 4000+ Google IP addresses. Answer was the same 404 - File Not Found. The User-Agent does not provide a support link unlike their other bots." -- https://en.osm.town/@osm_tech/114205536438977922 https://en.osm.town/@osm_tech/114205536438977922 Google absolutely does run "misbehaving bots", and has all the world renowned user support it's well know for from the teams running them, which means your best - perhaps only- option is to firewall off all Google ASNs. With Google search's decline in usefulness and it's plummeting referral traffic, combined with their unashamed AI-grifting copyright infringement and IP theft, the tradeoff in the old thinking of "I need to let Google crawl my site because I still naively believe SEO will make my business successful" is rapidly moving towards "Fuck you Google, you don't get anything I publish for free anymore."
    - skybrian 2y ago
      Thanks for the link. Apparently Google has more bots than I thought. But is that really a Google bot or is someone else using their name? I don't see 'GoogleAssociationService' listed in their documentation [1]. They do say it’s from Google IP addresses, but it might be someone running a bot in Google Cloud? Maybe they checked that, but we can’t tell from a tweet. Seems like a reasonable approach might be to whitelist the documented Google bots and block others. [1] https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers https://developers.google.com/search/docs/crawling-indexing/...
marginalia_nu 2y ago
I run an above board crawler, but in general crawler traps are relatively easy to work around, especially coming from a big target like Cloudflare, where it really pays off to build a specialized workaround that fingerprints and avoids the trap. Cloudflare's strength is arguably that they have enough traffic data they don't have to rely on stuff like this, they can gather statistics and identify bot patterns in ways smaller actors can not. It's trickier when you have 10,000 different webmasters inventing their own solutions to do sabotage crawlers, where the juice isn't worth the squeeze when it comes to implementing individual workarounds.
gwittel 2y ago
I work a product that involves a security crawler (phish, malware detection, etc). It’s just a new arms race. Crawlers will adapt. Cloudflare is already heavily abused by threat actors to host, and gate their malicious content. This means our crawler has to handle anti-bot and CAPTCHAs. It’s a pain. Cloudflare is no help. They have a “verified bot” program but it’s a joke for security. You must register a unique, identifiable user agent, and come from a set of self declared IPs. Cloudflare users can check a box to filter these bots out. And now you're easily fingerprintable so the bad guys can just filter you even without Cloudflare’s help. So now we have a choice. Operate above board and miss security threats. Or operate outside the rules (as opaquely defined by Cloudflare), and do right by our customers. All of this on CFs side is to solve a real problem. Unfortunately by not working with the industry in a productive manner, Cloudflare is just creating new problems for everyone else.
- disqard 2y ago
  "verified bot" really sounds like an oxymoron :)

In this 'arms race', will this serve as an actual deterrent? Can anyone involved in scraping chime in?

by momojo 2y ago

In this 'arms race', will this serve as an actual deterrent? Can anyone involved in scraping chime in?

I am not involved in scraping, but to me this sounds like simply another tool in the arsenal. They say it's hard for the scraper to realize it has been caught this way because it's not being blocked. However, I don't see anything preventing scrapers from implementing heuristics to realize that.

pona-a 2y ago
Detecting the actual AI generated content is not an easy problem. Not following deep links and recognizing the particular website template and structure is easier. I really feel a monoculture of anti-bot tools can defeat their effectiveness. When you have to optimize for Anubis, Nepenthes, Quixotic, and Cloudflare, each independently evolving and different in method and implementation, it might just be practical to give up.

dudus 2y ago

This won't stop the big ones, Google, Meta, OpenAI, Perplexity, or even the Chinese Govt. But it will make it harder for new entrants.

skybrian 2y ago
Not sure it’s targeted at them, either. Which of those entities have misbehaving bots? Seems like Google, at least, should be following robots.txt?
- bigiain 2y ago
  Google 2025 is not the Google you remember and respect. "GoogleAssociationService bot was kind enough to ask 1,000,000+ times yesterday for the same file from 4000+ Google IP addresses. Answer was the same 404 - File Not Found. The User-Agent does not provide a support link unlike their other bots." -- https://en.osm.town/@osm_tech/114205536438977922 https://en.osm.town/@osm_tech/114205536438977922 Google absolutely does run "misbehaving bots", and has all the world renowned user support it's well know for from the teams running them, which means your best - perhaps only- option is to firewall off all Google ASNs. With Google search's decline in usefulness and it's plummeting referral traffic, combined with their unashamed AI-grifting copyright infringement and IP theft, the tradeoff in the old thinking of "I need to let Google crawl my site because I still naively believe SEO will make my business successful" is rapidly moving towards "Fuck you Google, you don't get anything I publish for free anymore."
  - skybrian 2y ago
    Thanks for the link. Apparently Google has more bots than I thought. But is that really a Google bot or is someone else using their name? I don't see 'GoogleAssociationService' listed in their documentation [1]. They do say it’s from Google IP addresses, but it might be someone running a bot in Google Cloud? Maybe they checked that, but we can’t tell from a tweet. Seems like a reasonable approach might be to whitelist the documented Google bots and block others. [1] https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers https://developers.google.com/search/docs/crawling-indexing/...

marginalia_nu 2y ago

I run an above board crawler, but in general crawler traps are relatively easy to work around, especially coming from a big target like Cloudflare, where it really pays off to build a specialized workaround that fingerprints and avoids the trap. Cloudflare's strength is arguably that they have enough traffic data they don't have to rely on stuff like this, they can gather statistics and identify bot patterns in ways smaller actors can not. It's trickier when you have 10,000 different webmasters inventing their own solutions to do sabotage crawlers, where the juice isn't worth the squeeze when it comes to implementing individual workarounds.

gwittel 2y ago

I work a product that involves a security crawler (phish, malware detection, etc). It’s just a new arms race. Crawlers will adapt. Cloudflare is already heavily abused by threat actors to host, and gate their malicious content. This means our crawler has to handle anti-bot and CAPTCHAs. It’s a pain. Cloudflare is no help. They have a “verified bot” program but it’s a joke for security. You must register a unique, identifiable user agent, and come from a set of self declared IPs. Cloudflare users can check a box to filter these bots out. And now you're easily fingerprintable so the bad guys can just filter you even without Cloudflare’s help. So now we have a choice. Operate above board and miss security threats. Or operate outside the rules (as opaquely defined by Cloudflare), and do right by our customers. All of this on CFs side is to solve a real problem. Unfortunately by not working with the industry in a productive manner, Cloudflare is just creating new problems for everyone else.

disqard 2y ago
"verified bot" really sounds like an oxymoron :)