Hey, thanks for the reply! Specific scenarios require creative solutions. For a search engine, how do you differentiate between robots and legitimate users? It

Hey, thanks for the reply! Specific scenarios require creative solutions. For a search engine, how do you differentiate between robots and legitimate users? It seems a rate limiting step is likely the best solution. If query rate from the same IP exceeds a threshold, throttle them creatively. +100ms the next time, +250ms the next, etc. The upside is these bots will adjust their strategies to hit your site slower, which is the whole point, isn't it. If they spread requests across IPs, perhaps try fingerprinting. I'm not sure how effective that is on the backend though.