Probably not, but if they don't want to be scraped, the first place they should notify about this is robots.txt. I was just stating an example that you can allo

Probably not, but if they don't want to be scraped, the first place they should notify about this is robots.txt. I was just stating an example that you can allow some bots and not others if you like. Of course, forging your user agent, disobeying robots.txt or scraping after you were told not to, is wrong ethically.