4 ms·
How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation
by regularfry 2y ago
How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.
- what 2y agoGoogle publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.
- lacksconfidence 2y agoHow about if they claim to be google chrome running on windows 11, from a residential IP address? Is that a human or an AI bot?
- regularfry 2y agoAnd in that case both systems end up with a situation new entrants can't enter.
- usefulcat 2y agoI don't see how that helps the case where the UA looks like a normal browser and the source IP looks residential.