3 ms·
python selenium bindings are also nice if you absolutely have to deal with information put together by javascript code.
by tomsthumb 12y ago
python selenium bindings are also nice if you absolutely have to deal with information put together by javascript code.
- gee_totes 12y agoHave you thought about using a headless web browser like casperjs?
- Jake232 12y agoPhantomJS can be run through Selenium.
- deleted 12y ago[deleted]
- Jake232 12y agoI actually cover this (very briefly) here: http://jakeaustwick.me/python-web-scraping-resource/#thesiteisshowingdifferentcontenttomyscraper http://jakeaustwick.me/python-web-scraping-resource/#thesite... I should dedicate a section to it though, will stick that on my to-do list.
- tomarr 12y agoI've always opted for Ghost.py rather than selenium as I've found it uses less memory and pretty capable. Admittedly my scrapes are normally pretty targeted and not over 10k+ sites. One query I had from your piece: "When I've found myself in the unfortunate place of getting my proxies banned before on certain sites, they have been more the happy to switch them out for new IP's for me." If you're getting banned from sites, is it not time to leave them alone? If they don't want your traffic (which the admin/system has judged as too much), should you really be circumventing it? With that and the robots.txt bit (which admittedly you justified), you've got to be careful not to slip into a bit of a grey area with scraping, which people regard suspiciously in the first place.
- mrfusion 12y agoCan't the Python code just call PhantomJS directly? That's what I did.