2 ms·
(Author) I'm running a search engine though. Do you propose I require users to register an account, and then not allow them to search? I think my backend is pl
by marginalia_nu 4y ago
(Author) I'm running a search engine though. Do you propose I require users to register an account, and then not allow them to search?
I think my backend is plenty fast given it's hosted on a PC off domestic broadband. Most searches complete sub-100ms.
- julianlam 4y agoHey, thanks for the reply! Specific scenarios require creative solutions. For a search engine, how do you differentiate between robots and legitimate users? It seems a rate limiting step is likely the best solution. If query rate from the same IP exceeds a threshold, throttle them creatively. +100ms the next time, +250ms the next, etc. The upside is these bots will adjust their strategies to hit your site slower, which is the whole point, isn't it. If they spread requests across IPs, perhaps try fingerprinting. I'm not sure how effective that is on the backend though.