3 ms·
Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving t
by techjamie 1y ago
Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.
- OutOfHere 1y agoIf we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.
- DamonHD 1y agoVirtually nothing pays attention to 429s that I have observed. More things pay attention to 500s and 503s. Some however use those as a trigger to repoll immediately.
- felurx 1y agoI suppose they're going for a trade-off between "cheaper" and "satisfying" (as in, the satiafaction of sticking it to unethical shitty companies)