The Ahmia project publishes how they approach this problem, e.g. here: https://github.com/ahmia/ahmia-crawler https://github.com/ahmia/ahmia-crawler Their rep

The Ahmia project publishes how they approach this problem, e.g. here: https://github.com/ahmia/ahmia-crawler https://github.com/ahmia/ahmia-crawler Their repos also showcase how they're using blacklisting to keep from linking to CSAM.