3 ms·
I do a big recrawl around once every 10 weeks, but I have a RSS crawler that does daily visits and fetches new items. But that should be very low volume traffi
by marginalia_nu 2y ago
I do a big recrawl around once every 10 weeks, but I have a RSS crawler that does daily visits and fetches new items. But that should be very low volume traffic.
- stackghost 2y agoInteresting, does your RSS crawler auto-detect feeds? That's a neat solution I'd never considered.
- marginalia_nu 2y agoThe big crawler tries to find feeds, and the RSS crawler acts on what the big one has found.