it's a stretchy argument, but one i'm going make anyway: before the crawl, there is discovery. discovery is the process of knowing all the URLs to pass to the c

it's a stretchy argument, but one i'm going make anyway: before the crawl, there is discovery. discovery is the process of knowing all the URLs to pass to the crawler. discovery is fueled by * links * sitemap.xml who creates links (and sitemaps)? humans (either editorially or programmatically, but at some point humans were involved in the link creation process) defining the pinterest fetching process as "not indexing" or "not crawling" seems a little bit unfair against your friendly neighborhood googlebot. also if pinterest is "currated by humans" then the pin-tag would be unnecessary. a simple plain text "DO NOT PIN THIS" message in the article/product/image/.. (whatever people pin these days) description would be sufficient.