3 ms·
Given that you want to be good stewards of book data, have you considered publishing bulk snapshots to archive.org on a set cadence? It would strongly reduce an
by upon_drumhead 2y ago
Given that you want to be good stewards of book data, have you considered publishing bulk snapshots to archive.org on a set cadence? It would strongly reduce any needs to do any sort of bulk scraping and also ensure that should something happen to your service, the data isn't lost forever.
- dyogenez 2y agoI hadn't thought of that, but I love the idea! How's that work?
- upon_drumhead 2y agoRegister for an account and create a new item. You can replace files in the item , update the description to indicate what date the snapshot was made and what it contains. https://help.archive.org/help/managing-and-editing-your-items-a-basic-guide/ https://help.archive.org/help/managing-and-editing-your-item... It's a very open platform. Think up what the best format for your data is and upload a compressed zip file or tar.gz of the data. I'd likely do different archives for images and metadata, so people that want to just process metadata can download that specific data and work on it. Luckily as you can edit over time, you can experiment and adjust based upon user's feedback.
- dyogenez 2y agoNice! Yeah I like the idea of sharing our static book data here on some kind of cadence. I'll pencil this in by the end of the year.