4 ms·

Thanks for taking the time to reply with the limited information provided. >> why you're transferring the session to Requests at all Only for speed. Selenium

Thanks for taking the time to reply with the limited information provided. >> why you're transferring the session to Requests at all Only for speed. Selenium is very slow. Requests seems to return responses faster. With 10(0)s of URLS to scrape, this adds up. >> I'd probably class the whole thing, generate the full list of urls to scrape and setup your empty `holdsurl` list both in the init. Then I'd probably write a class method that just runs the whole thing and returns the `holdsurl` list This seems to be the way to g. One method is making it hard to read. Thanks for the other tips as well for adding checking the parsing etc. I'll be adding that in.

staticautomatic 9y ago
Sure thing. Yeah Requests is definitely faster. If you really want to get as much speed out of this as possible you'd go asynchronous but that's another can of worms. This level of optimization may be overkill but if you want a speed boost (not sure how much of one though) while still making the requests synchronously you could use urllib instead of Requests (since you're only fetching the page and not actually using Requests' abstractions, you're wasting the additional overhead Requests incurs). You could also just use lxml directly instead of soup. As with Requests, you're incurring the overhead associated with calling soup even though most of what you're using soup for is the optional lxml parser. I don't know what soup uses for locating stuff in the tree but if it's not lxml, it's probably slower than lxml.