You can’t generalise that. Maybe I crawl to provide an annotated preview of their website, to make users of my application more likely to click the link and vis

You can’t generalise that. Maybe I crawl to provide an annotated preview of their website, to make users of my application more likely to click the link and visit it? There are lots of ways in which crawling benefits everyone, it just requires some mutual respect.