7 ms·
Yet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death.
by uallo 3y ago
Yet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death. Luckily, I already block the IP address they're using as it has been used for (other?) malicious bots before.
[1] In my case, it relies on third-party APIs that are heavily rate limited. Any bot ignoring rate limitation measures will effectively (D)DOS my service.
- BigBalli 3y agoYet another reason why you should handle these scenarios on your own rather than hoping clients/users will.
- NegativeK 3y agoThere's absolutely nothing wrong with being furious at someone because you have to waste time dealing with their bad behavior.
- colechristensen 3y agoThere are plenty of ways you can (and should) rate limit requests on your end. It is a pretty basic security and reliability practice. Also if you're dealing with an actual malicious adversary real or automated rate limiting can be more effective than blocking. (logic to detect and overcome an even very significant rate limit is much more complex than to detect dropping, ignoring, or 4xx 5xx response blocking methods) For example, a method to rate limit based on IP with nginx http://nginx.org/en/docs/http/ngx_http_limit_req_module.html http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
- uallo 3y agoSure. I already use several rate limitation measures, return fake data for repeating offenders, and also outright block some others. It is still laughable that a somewhat "reputable" bot does not even know about basic HTTP headers.
- gumballindie 3y agoOne option is to completely ban openai’s crawler ip addresses. They steal content without credit anyway - as most ai companies do - so there’s no benefit in allowing them access.
- GaggiX 3y ago>so there’s no benefit in allowing them access. Well, you're helping improving the model.
- dpacmittal 3y agoWhich benefits the company.
- GaggiX 3y agoYeah but I don't think it's an inherently bad thing. +100M people use ChatGPT without paying anything, in this it benefits much more than the company.
- gumballindie 3y agoOh but they do pay. They pay their own time to gradually train the model and feed their data. There's no such thing as "free".
- GaggiX 3y agoThat's just a win-win situation, you're using their services for free because it helps you, they use your interaction to improve the model; the model is still free to use.
- gumballindie 3y agoThere's no win win situation. My content is stolen and given to others. I've lost. Google paid me for traffic via ads, therefore I allowed google to ingest my content. You as a person could read it. I've never given you permission to resell it, and if you did, I'd come after you to pay royalties. The same must apply to openai and other leeches.