I can vouch that a personal crawler is useful - if a link goes away, I put it in an archive and update the link to The Wayback Machine. It's also nice to have a

I can vouch that a personal crawler is useful - if a link goes away, I put it in an archive and update the link to The Wayback Machine. It's also nice to have a timestamp showing when the link was last available and the current "title" tag and meta description for the page (along with the original metadata on the date I linked the page.) It makes it easy to prune or correct broken links.