3 ms·
Petal is one of the bots I block from accessing my sites. They were DoSed multiple times as Petal's spiders ignored the "429 Too Many Requests" response header.
by uallo 4y ago
Petal is one of the bots I block from accessing my sites. They were DoSed multiple times as Petal's spiders ignored the "429 Too Many Requests" response header. They just don't care and perform several requests per second anyway. I won't use a search engine that does not even support basic rate limitation measures.
- boesboes 4y agoI've blocked them in the past too for this, but it seems they are better behaved now. I don't have any longer term data, but over the past week they averaged 1.5 request per minute and that is divided among ~630 unique websites. Infact, the UAs I blocked are no longer in use it seems so they are no longer limited. EDIT: btw, not endorsing their search or anything, just mentioning thye may have learned and improved ;)