You can also use something that’s built already and extend it. I have used in the past linkchecker, https://github.com/linkchecker/linkchecker https://github.co

You can also use something that’s built already and extend it. I have used in the past linkchecker, https://github.com/linkchecker/linkchecker https://github.com/linkchecker/linkchecker to do crawling too. You can get the report and analyze it however you want.