Maybe something worth exploring, run it like an inverted generational garbage collector. Separate the problematic domains and run them in a lower priority back

Maybe something worth exploring, run it like an inverted generational garbage collector. Separate the problematic domains and run them in a lower priority background crawl; then do the opposite as well, have a higher priority list for crawling sites that have regular updates more often, based on their last update gap. Basically like a generational garbage collector but in this case it's generational content collector, although arguably, still mostly garbage.