Sites allow search engines to index them (instead of telling them to go away with robots.txt) because the search traffic is worth it to them. Search engines do

Sites allow search engines to index them (instead of telling them to go away with robots.txt) because the search traffic is worth it to them. Search engines don't allow people to scrape them (resorting to blocking after scrapers ignore robots.txt) because they don't get anything similarly valuable in return. (Disclosure: I work for Google, though not on search.)