4 ms·
I thought that too but running a random repository through that same tool isn’t any different.
by KMnO4 6y ago
I thought that too but running a random repository through that same tool isn’t any different.
- deleted 6y ago[deleted]
- the_duke 6y agoI tried the same thing. The tool doesn't handle the direct link, it just gives information for about.gitlab.com again.
- georgyo 6y agoThere is no ability to run a random page though that tool. It can only look at the root index page of the domain. And this mostly makes sense. It means they can cache results more easily, there is no way the tool could return so quickly unless it was cached.
- simonft-mrkp 6y agoYep, and this help mitigate abuse of the tool, e.g. someone continuously running it on every page of a website to slow the site down (I helped work on Blacklight)
- neurostimulant 6y agoIf that's the concern, can you put a rate limit on per-domain basis instead? Many other tools that scan websites allow you to scan arbitrary page url, not just root/index url.
- simonft-mrkp 6y agoWe could have, yeah. There are some UX questions there we'd have to answer, and we were developing this with a very small team and limited time, so opted for a simple solution. We did make the collection software available on Github if people want to check out arbitrary pages on their own: https://github.com/the-markup/blacklight-collector/ https://github.com/the-markup/blacklight-collector/