3 ms·
If that's the concern, can you put a rate limit on per-domain basis instead? Many other tools that scan websites allow you to scan arbitrary page url, not just
by neurostimulant 6y ago
If that's the concern, can you put a rate limit on per-domain basis instead? Many other tools that scan websites allow you to scan arbitrary page url, not just root/index url.
- simonft-mrkp 6y agoWe could have, yeah. There are some UX questions there we'd have to answer, and we were developing this with a very small team and limited time, so opted for a simple solution. We did make the collection software available on Github if people want to check out arbitrary pages on their own: https://github.com/the-markup/blacklight-collector/ https://github.com/the-markup/blacklight-collector/