3 ms·
Coming from a Ruby background I've always been curious about Python's libraries for scraping. I've tried scrapy and beautiful soup, but somehow kept going back
by fareesh 8y ago
Coming from a Ruby background I've always been curious about Python's libraries for scraping. I've tried scrapy and beautiful soup, but somehow kept going back to Nokogiri and mechanize.
I found the CSS selector or xpath based syntax and the DSL to be a lot more convenient and less verbose to deal with.
Is selenium still the best bet for parsing JS powered pages these days? I was under the impression that headless chrome was more memory / performance efficient.
I do a lot of scraping work but my methods have not really evolved in the past 3-4 years, always on the lookout for something more elegant / quicker.
- onesmallcoin 8y agoLook into chromedriver, It maintained by the Chromium team and provides an executable that allows webdriver to control chrome. I've used this successfully in the past running chrome in headless mode, it seems pretty scalable. If I had to build the same product again, I'd still use chromedriver. Although I'd also consider Sikuli for image recognition / automation outside of the browser. Check out the chromedriver project here! http://chromedriver.chromium.org/getting-started http://chromedriver.chromium.org/getting-started
- theshadowknows 8y ago+1 for Sikuli! I only very rarely see it mentioned anywhere!
- AdamM12 8y agoIf it doesn't require JS parsing I used Parsel. It's the underlying selector library for scrapy.