This is super true. For my city’s portal as well. I’ve found one way around this by versioning the dataset - that is, committing the diffs in git. Credit to Sim

This is super true. For my city’s portal as well. I’ve found one way around this by versioning the dataset - that is, committing the diffs in git. Credit to Simon Willison’s git-scraping technique. I do this with my power company’s outage map: https://github.com/patricktrainer/entergy-outages https://github.com/patricktrainer/entergy-outages 67k commits! https://simonwillison.net/2020/Oct/9/git-scraping/ https://simonwillison.net/2020/Oct/9/git-scraping/