4 ms·
IA might be large enough to earn consideration, but generally scrapers just don't care about being good citizens. I work in the GLAM space and we offer OAI-PMH
by katatue 16d ago
IA might be large enough to earn consideration, but generally scrapers just don't care about being good citizens. I work in the GLAM space and we offer OAI-PMH interfaces for the harvesting of our collections data - which doesn't stop companies from preferring to scrape our website for worse (less complete, less structured, less standardized) data instead.
- mitxela 16d agoDo they know it exists? On my site, some types of blocked bots are getting plain-text instructions saying why I'm blocking them and what they can do instead - and it seems to have worked in some cases.