4 ms·
I found a lot of use cases for web scraping is kinda ad-hoc and usually, occurs as part of another task (eg. a research project or enhancing a record). I ended
by laktek 9y ago
I found a lot of use cases for web scraping is kinda ad-hoc and usually, occurs as part of another task (eg. a research project or enhancing a record). I ended up releasing a simple hosted API service called Page.REST (https://page.rest https://page.rest) for people who would like to save that extra dev effort and infrastructure cost.
- gmac 9y agoI agree, and I find scripting a web browser via the developer console a really productive approach. First, it's completely interactive. Second, it's the browser, so absolutely everything works. It doesn't matter if the data you want is only loaded by an obscure JS function when a hidden form is submitted on a button click. Just find the button, .click() it, and wait for a mutation event. I have a write up on this[1], but I need to extend it with some more advanced examples. [1] https://github.com/jawj/web-scraping-for-researchers https://github.com/jawj/web-scraping-for-researchers
- dmn001 9y agoThat may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.
- newlyretired 9y agoI've done a lot of scraping in my day, and I've found that lxml/requests is 2-3 OOM more resource efficient than a Selenium based browser. That JS/rendering engine is HEAVY!
- akvadrako 9y agoYou can even control Chrome remotely via Python with a pretty simple web sockets api: https://github.com/marty90/PyChromeDevTools https://github.com/marty90/PyChromeDevTools That's really the best of both worlds.