3 ms·
There is no ability to run a random page though that tool. It can only look at the root index page of the domain. And this mostly makes sense. It means they ca
by georgyo 6y ago
There is no ability to run a random page though that tool. It can only look at the root index page of the domain.
And this mostly makes sense. It means they can cache results more easily, there is no way the tool could return so quickly unless it was cached.
- simonft-mrkp 6y agoYep, and this help mitigate abuse of the tool, e.g. someone continuously running it on every page of a website to slow the site down (I helped work on Blacklight)
- neurostimulant 6y agoIf that's the concern, can you put a rate limit on per-domain basis instead? Many other tools that scan websites allow you to scan arbitrary page url, not just root/index url.
- simonft-mrkp 6y agoWe could have, yeah. There are some UX questions there we'd have to answer, and we were developing this with a very small team and limited time, so opted for a simple solution. We did make the collection software available on Github if people want to check out arbitrary pages on their own: https://github.com/the-markup/blacklight-collector/ https://github.com/the-markup/blacklight-collector/