Chris Siebenmann has shared many thoughts about crawlers, aka spiders. https://utcc.utoronto.ca/~cks/space/?search=spider https://utcc.utoronto.ca/~cks/space/?

Chris Siebenmann has shared many thoughts about crawlers, aka spiders. https://utcc.utoronto.ca/~cks/space/?search=spider https://utcc.utoronto.ca/~cks/space/?search=spider https://utcc.utoronto.ca/~cks/space/?search=crawlers https://utcc.utoronto.ca/~cks/space/?search=crawlers It's interesting how in 2005 he was advocating for playing nicely, because yes, search engines were hypothetically driving traffic to sites they crawled. And there don't seem to be a lot of complaints about performance hits. Though typically, his comments and observations are restricted to his blog's corner of his department.