3 ms·

This could be OpenAI, or it could be another company using their header pattern. It has long been common for scrapers to adopt the header patterns of search en

by Aurornis 11mo ago

This could be OpenAI, or it could be another company using their header pattern. It has long been common for scrapers to adopt the header patterns of search engine crawlers to hide in logs and bypass simple filters. The logical next step is for smaller AI players to present themselves as the largest players in the space. Some search engines provide a list of their scraper IP ranges specifically so you can verify if scraper activity is really them or an imitator. EDIT: Thanks to the comment below for looking this up and confirming this IP matches OpenAI’s range.

jsheard 11mo ago
In this case it is actually OpenAI, the IP (74.7.175.182) is in one of their published ranges (74.7.175.128/25). https://openai.com/searchbot.json https://openai.com/searchbot.json I don't know if imitating a major crawler is really worth it, it may work against very naive filters, but it's easy to definitively check whether you're faking so it's just handing ammo to more advanced filters which do check. $ curl -I https://www.cloudflare.com HTTP/2 200 $ curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com HTTP/2 403
- Aurornis 11mo ago
  Thanks for looking it up!
- btown 11mo ago
  I don't have a statistic here, but I'm always surprised how many websites I come across that do limited user-agent and origin/referrer checks, but don't maintain any kind of active IP based tracking. If you're trying to build a site-specific scraper and are getting blocked, mimicking headers is an easy and often sufficient step.
  - xyzzy_plugh 11mo ago
    If you can't tell the difference between active tracking and inspecting request headers, then it's worth committing a bit of time to ponder. Particularly the costs associated with IP tracking at scale.
ccgreg 11mo ago
> Some search engines provide a list of their scraper IP ranges Common Crawl's CCBot has published IP ranges. We aren't a search engine (although there are search engines using our data) and we like to describe our crawler as a crawler, not a "scraper".
bobsmooth 11mo ago
>The logical next step is for smaller AI players to present themselves as the largest players in the space. We think we're so different from animals https://en.wikipedia.org/wiki/Mimicry https://en.wikipedia.org/wiki/Mimicry

jsheard 11mo ago

In this case it is actually OpenAI, the IP (74.7.175.182) is in one of their published ranges (74.7.175.128/25). https://openai.com/searchbot.json https://openai.com/searchbot.json I don't know if imitating a major crawler is really worth it, it may work against very naive filters, but it's easy to definitively check whether you're faking so it's just handing ammo to more advanced filters which do check. $ curl -I https://www.cloudflare.com HTTP/2 200 $ curl -I -H "User-Agent: Googlebot" https://www.cloudflare.com HTTP/2 403

Aurornis 11mo ago
Thanks for looking it up!
btown 11mo ago
I don't have a statistic here, but I'm always surprised how many websites I come across that do limited user-agent and origin/referrer checks, but don't maintain any kind of active IP based tracking. If you're trying to build a site-specific scraper and are getting blocked, mimicking headers is an easy and often sufficient step.
- xyzzy_plugh 11mo ago
  If you can't tell the difference between active tracking and inspecting request headers, then it's worth committing a bit of time to ponder. Particularly the costs associated with IP tracking at scale.

ccgreg 11mo ago

> Some search engines provide a list of their scraper IP ranges Common Crawl's CCBot has published IP ranges. We aren't a search engine (although there are search engines using our data) and we like to describe our crawler as a crawler, not a "scraper".

bobsmooth 11mo ago

>The logical next step is for smaller AI players to present themselves as the largest players in the space. We think we're so different from animals https://en.wikipedia.org/wiki/Mimicry https://en.wikipedia.org/wiki/Mimicry