For those who have been trying to scrape pages that make AJAX requests, something you can not do with BeautifulSoup alone - I would recommend using Selenium Ser

For those who have been trying to scrape pages that make AJAX requests, something you can not do with BeautifulSoup alone - I would recommend using Selenium Server (http://docs.seleniumhq.org/ http://docs.seleniumhq.org/). This allows you to automate a real browser (so your scraping requests look like real page requests, not robots). In addition, tools exist for Selenium that let you scale up easily. You can use Selenium Grid 2 (https://code.google.com/p/selenium/wiki/Grid2 https://code.google.com/p/selenium/wiki/Grid2) to run multiple browser instances in parallel. This is very beneficial for web scraping or automated UI testing.