I tried this approach for scraping a REST API returning json (on a few tens of thousands of ids). Since it was a long running job with many failures (throttlin

I tried this approach for scraping a REST API returning json (on a few tens of thousands of ids). Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script. I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty. I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.