My understanding is that scrape shield protects against certain abusive patterns of robot access, but not all URLs on the domain would ban robots. So archive ma

My understanding is that scrape shield protects against certain abusive patterns of robot access, but not all URLs on the domain would ban robots. So archive may have many contaminated pages that scrape shield was enabled for but did not ban all robots from crawling based on various criteria.