It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently. If only that script worked

It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently. If only that script worked for every website. But, alas, it does not.