3 ms·
> According to Google, it’s possible to verify Googlebot by matching the crawler’s IP against a list of published Googlebot IPs. This is rather technical and hi
by fastball 1y ago
> According to Google, it’s possible to verify Googlebot by matching the crawler’s IP against a list of published Googlebot IPs. This is rather technical and highly intensive
Wat. Blocklisting IPs is not very technical (for someone running a website that knows + cares about crawling) and is definitely not intensive. Fetch IP list, add to blocklist. Repeat daily with cronjob.
Would take an LLM (heh) 10 seconds to write you the necessary script.
- simonw 1y agoLooks like OpenAI publish the IPs of their training data crawler too: https://openai.com/gptbot.json https://openai.com/gptbot.json
- kinix 1y agoFrom how I read it, the author seems to be suggesting that this list of IPs be on an allowlist as they see Google as "less nefarious". As such, sure, allowing google IPs is as easy as allowing all IPs, but discerning who are "nefarious actors" is probably harder. A more tounge-in-cheek point: all scripts take an LLM ~10 seconds to write, doesn't mean it's right though.