I also had this idea, but I never executed it as I haven't thought of a way to solve the real vs. estimated times perfectly. Probably can get close w/some data

I also had this idea, but I never executed it as I haven't thought of a way to solve the real vs. estimated times perfectly. Probably can get close w/some data mining, but not sure if it's worth the effort. RE: scraping - instead of putting logic in your scraper, just download the entire section you need, store it in file format. Then parse and shove into database whenever you feel like it. You could rerun the parsing since you'll have all the historically scraped website data on disk.