3 ms·
The service is very nice and I understand your reason for developing it. I see this service to be having more value in helping companies find all the web pages,
by itsmefaz 7y ago
The service is very nice and I understand your reason for developing it. I see this service to be having more value in helping companies find all the web pages, rather than just the allowed ones.
I understand the unethical nature of the above method, however, I see it happening quite a lot in practice.
- fooock 7y agoYes, in the practice people sometimes don't want to be polite with webmasters, and choose not obey robots.txt rules. Thanks for the suggestion!
- itsmefaz 7y agoExactly, your service could definitely be used as an alternate to parsing robots.txt (which traditionally is in xml) to a more standard json parsing. Along with the advantages that comes with making it REST.