3 ms·
Sponge 0.22: A website crawler and links downloader command line tool
- zzo38computer 7y agoI might suggest to add options to configure the referrer and user-agent strings, and an option to specify whether or not to use robots.txt. Another possible option would be to refuse to access the local network.
- spypunk 7y agoThanks for the feedback!
- zzo38computer 7y agoActually, an option to refuse to access the local network is worthless if the program does not already have the ability to cross domains. But if such a thing is added in future, then for security purposes the user might not want to allow crossing the domain from the internet to the local network.