scrapy for python is pretty good, check it out. In most cases getting banned is the big issue. The bigger the site, the more advanced their bot detection is. Y

scrapy for python is pretty good, check it out. In most cases getting banned is the big issue. The bigger the site, the more advanced their bot detection is. You can use luminato.io to get residential and mobile IP's, but it's pricey. Some sites will also obfuscate the DOM, ie. removing classnames and ID's, which complicates the data extraction. http://scrapinghub.com/ http://scrapinghub.com/ has a paid "do it for me" service, which may be an option depending on your budget.