4 ms·
So that means it's fetching the content (or at least the HTML portion) of every eligible link? That seems like potentially a lot of network traffic for what it
by rcthompson 10y ago
So that means it's fetching the content (or at least the HTML portion) of every eligible link? That seems like potentially a lot of network traffic for what it does. Find if you have a fast connection, I suppose.
- fcanela 10y agoCan be even worse. I have seen some ugly corporate services using links with GET request to perform delete operations and having serious data loss caused by crawlers like this tool.
- softdev12 10y agoThis was definitely a consideration. It was why there is a blacklist feature and why only links of a certain length and certain type are analyzed. Any suggestions can be made on github via a pull request.
- softdev12 10y agoGood point. There is some consideration for building an option to limit the number of links analyzed. So if you set it to 5 or 10, then the tool would stop at that limit. Feel free to open an issue on github.