Thanks for this, I am very new to python, so I will try this out. But I think at the moment my bottleneck is the scraping part. The current program hits 365 pa

Thanks for this, I am very new to python, so I will try this out. But I think at the moment my bottleneck is the scraping part. The current program hits 365 pages (each day for a year) and managed to pull down the 20 million results. I am now trying to figure out how to pull more info from each day, and that's why I am thinking about trying to optimize. Does cProfile measure things like time on a webpage (via bs4)? Do you have any links on optimization "thinking", as in, I don't know if it is my approach that needs optimizing, vs my code.