4 ms·
My web scraping toolkit (Python): -Beautiful Soup -Requests Lib -JSON Lib -Selenium -Urllib2 -Cookielib Handles 99% of things I encounter. With dynamic s
by madenine 10y ago
My web scraping toolkit (Python):
-Beautiful Soup
-Requests Lib
-JSON Lib
-Selenium
-Urllib2
-Cookielib
Handles 99% of things I encounter. With dynamic sites, you're often better off simulating the request than controlling the browser.
Then you get to parse your way through someone's annoyingly formatted JSON.