4 ms·
I think it is as complex topic as solving captcha or fighting email spam. You can check screenshots, you can check for new CSS/JS files included in page. That
by appliku 6y ago
I think it is as complex topic as solving captcha or fighting email spam.
You can check screenshots, you can check for new CSS/JS files included in page.
That's a start.
Also you should track DNS/NS changes.
Only screenshots are not enough. What if they redirect users elsewhere? Open new windows (well, browsers fight it themselves, but still).
To summarise this:
- DNS changes
- changes in included JS/CSS files
- changes in new outgoing links on a given page.
This should be enough for starters
- Akcium 6y agoThen I'll need to make some crawler, which is also not that easy to implement. Maybe it's a good idea to have some kind of complex functionality, broken down to parts: - screenshots - links - DNS - ... so that user can enable/disable them as he needs too
- rovr138 6y agoYou can also use something that’s built already and extend it. I have used in the past linkchecker, https://github.com/linkchecker/linkchecker https://github.com/linkchecker/linkchecker to do crawling too. You can get the report and analyze it however you want.