4 ms·
Why a service and not a library? It looks like a great way for you to discover URLs but like a terribly slow way for people to avoid implementing robots.txt ru
by dscpls 7y ago
Why a service and not a library?
It looks like a great way for you to discover URLs but like a terribly slow way for people to avoid implementing robots.txt rules.
- fooock 7y agoThe project have multiple subprojects, and one of these is the parser. Any developer can compile or extend it without more effort and create a library. Just know to code in Kotlin / Java. The aim of this project is only check if a given web resource can be crawled by a user-agent, but using a API