Sure thing. Yeah Requests is definitely faster. If you really want to get as much speed out of this as possible you'd go asynchronous but that's another can of

Sure thing. Yeah Requests is definitely faster. If you really want to get as much speed out of this as possible you'd go asynchronous but that's another can of worms. This level of optimization may be overkill but if you want a speed boost (not sure how much of one though) while still making the requests synchronously you could use urllib instead of Requests (since you're only fetching the page and not actually using Requests' abstractions, you're wasting the additional overhead Requests incurs). You could also just use lxml directly instead of soup. As with Requests, you're incurring the overhead associated with calling soup even though most of what you're using soup for is the optional lxml parser. I don't know what soup uses for locating stuff in the tree but if it's not lxml, it's probably slower than lxml.