4 ms·
A way to submit an URL for (re)crawling would be good instead of having to wait until the spider somehow finds or revisits it. You could add a maximum of once
by Klasiaster 6y ago
A way to submit an URL for (re)crawling would be good instead of having to wait until the spider somehow finds or revisits it.
You could add a maximum of once per day per domain if you are worried about abuse.
- ColinHayhurst 6y agoThank you. This is a good idea and one we need to consider more than we have thus far.