4 ms·
>a 100TB raid10 setup, and a NAS at home doing the same How do you deal with these onerous constraints? Do you have a system for deciding how many copies of ar
by javajosh 4y ago
>a 100TB raid10 setup, and a NAS at home doing the same
How do you deal with these onerous constraints? Do you have a system for deciding how many copies of archive.org to keep locally?
- fishtacos 4y agoI know you're being facetious, but here's some fun stats from archive.org. A few highlights from the Petabox storage system: No Air Conditioning, instead use excess heat to help heat the building. Raw Numbers as of December 2021: 4 data centers, 745 nodes, 28,000 spinning disks Wayback Machine: 57 PetaBytes Books/Music/Video Collections: 42 PetaBytes Unique data: 99 PetaBytes Total used storage: 212 PetaBytes Considering this data is from a year ago, it's got to be substantially larger now. Link: https://archive.org/web/petabox.php https://archive.org/web/petabox.php
- deleted 4y ago[deleted]
- chaxor 4y agoI'm not sure what exactly you mean - but we typically store DNA sequencing data from people that want us to analyze it, databases of proteins and scientific articles, ML models, etc. Proper considerations of how to store the data and compress it correctly help a lot, but unfortunately academics are pretty terrible at choosing the right data structures for the job, and often several copies are needed for mirroring the raw data, then the processed versions are updated.