6 ms·
Lately there's been some consternation in the small-web community about the aggressive crawling by OpenAI and friends. How often does yours (re)crawl its index
by stackghost 2y ago
Lately there's been some consternation in the small-web community about the aggressive crawling by OpenAI and friends.
How often does yours (re)crawl its indexed sites?
- marginalia_nu 2y agoI do a big recrawl around once every 10 weeks, but I have a RSS crawler that does daily visits and fetches new items. But that should be very low volume traffic.
- stackghost 2y agoInteresting, does your RSS crawler auto-detect feeds? That's a neat solution I'd never considered.
- marginalia_nu 2y agoThe big crawler tries to find feeds, and the RSS crawler acts on what the big one has found.