There's a lot that's unclear here but I'll try. I'd start by splitting out the url generation, request, and the parsing into separate functions. First write a

There's a lot that's unclear here but I'll try. I'd start by splitting out the url generation, request, and the parsing into separate functions. First write a function that generates a list of all the urls you want to scrape. It'll take your config and number of pages as args. If the list is going to be super long then have it return a generator. Then you'd have a function for the request that takes only the url as an arg and returns the response content. You'll want to catch session-related exceptions and http status code issues before parsing. The parsing function should just take the response content and `hold_no` as args. You should add some code to check the objects you're generating during parsing to make sure they actually contain what you're after. For example, your `rows` variable could return an empty list for a variety of reasons that aren't exceptions but are things you'd want to catch nonetheless. For example, maybe the element(s) can't be found because they haven't loaded yet, the returned html is malformed, or you simply made a typo in your own code. Under the hood lxml is always going to return an empty list when it can't find something. It's hard to say how to structure this given that it's part of some bigger thing we can't see. I'd probably class the whole thing, generate the full list of urls to scrape and setup your empty `holdsurl` list both in the init. Then I'd probably write a class method that just runs the whole thing and returns the `holdsurl` list, but there are other equally good options. One thing I don't understand is why you're transferring the session to Requests at all. I don't see anything you're doing with the Session() object that you wouldn't already be able to do with Selenium/soup/lxml.