Having written a few scrapers, including the browser kind. I found the trick was to specialize. Each website respond differently, some can just use requests, o

Having written a few scrapers, including the browser kind. I found the trick was to specialize. Each website respond differently, some can just use requests, others would need selenium with pre-existing profile. To develop a common utility is almost futile. I do have useful building blocks, but for each individual things I want to scrape I scale out using project specific code. It's never too slow either - the time it would take to fill in all of the required bits in a do it all tool would have been similar.