3 ms·
Yep, and this help mitigate abuse of the tool, e.g. someone continuously running it on every page of a website to slow the site down (I helped work on Blacklig
by simonft-mrkp 5y ago
Yep, and this help mitigate abuse of the tool, e.g. someone continuously running it on every page of a website to slow the site down
(I helped work on Blacklight)
- neurostimulant 5y agoIf that's the concern, can you put a rate limit on per-domain basis instead? Many other tools that scan websites allow you to scan arbitrary page url, not just root/index url.
- simonft-mrkp 5y agoWe could have, yeah. There are some UX questions there we'd have to answer, and we were developing this with a very small team and limited time, so opted for a simple solution. We did make the collection software available on Github if people want to check out arbitrary pages on their own: https://github.com/the-markup/blacklight-collector/ https://github.com/the-markup/blacklight-collector/