3 ms·
I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the sev
by npunt 6y ago
I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.
- vermilingua 6y agoAnd disk life.
- philjohn 6y agoperversely, wouldn't spinning drives up and down impact drive life?
- chemicalnovae 6y agoProbably yes, but only if it was happening a lot; I don’t know what the cross-over point would be though that you’re better off just keeping the drives spinning...
- bnewbold 6y agoThe disks are spinning all the time, and most disks are seeing fairly frequent reads to some content or another. A lot of content is very rarely accesses, but almost every disk has some content which gets accessed. If spinning disks had only frequently-accessed content, they would be unable to keep up with the read rate or read throughput, things balance out reasonably on average. Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request. Note: I work at IA but am not on core infrastructure team
- npunt 6y agoInteresting thanks for the insights! Then would the few second delay be more a matter of time it takes to decompress the contents, or that files are stored on disks which are being accessed a lot, or something else? Always been curious about it.