3 ms·

I'm probably explaining something most everyone here already knows, but: I don't know what Glacier uses, but the "offline" storage is much the same how mainfra

I'm probably explaining something most everyone here already knows, but: I don't know what Glacier uses, but the "offline" storage is much the same how mainframes use tape - disk based files (datasets in mainframe speak) are migrated to tape after a given amount of time to decrease storage costs. These datasets needs to be recalled back to disk to be used again. This is nearly identical to how Glacier operates with S3 as explained by Amazon. The reason tape isn't available immediately is because the tape drive, unlike a traditional harddrive, doesn't contain a reader. So multiple tapes typically sit in machines with only only a few readers that move back and forth based on requests made to read the tapes. If there are many requests, the amount of time to recall a dataset increases. Glacier may or may not be using tape, but the behavior is nearly identical to how tape has been used in past systems.

Someone 11y ago
You're referring to https://en.m.wikipedia.org/wiki/Hierarchical_storage_management https://en.m.wikipedia.org/wiki/Hierarchical_storage_managem.... AFAIK, most systems work at the file level, but if you address space is large enough, you could implement this as a virtual memory system that swaps RAM to disk, disk to slower disk, slower disk to tape, tapes to tapes stored offline, etc. I don't know what Amazon uses, either, but I expect Glacier data to be pushed out to some layer that doesn't need power while idle at all, heavily checksummed and also replicated across the globe. The relatively heavy prices tag for accessing it probably is there to keep contention on some relatively thin pipe low. For example, if they use blue ray, they may have only a few readers online for every 100 or 1000 customers, both to keep prices low and to differentiate Glacier more from their other products.
- icebraining 11y ago
  if you address space is large enough, you could implement this as a virtual memory system that swaps RAM to disk, disk to slower disk, slower disk to tape, tapes to tapes stored offline, etc. Funny to imagine, - My program has been stuck copying a variable for over half an hour! - Yeah, there was a page fault and the tape loading guy still hasn't come from lunch
  - Someone 11y ago
    Yes, but the other side of the coin is "why are those pages of the program still on disk? They haven't been touched in years". But such systems tend to keep parts under human control. You rarely ever need your fire extinguisher, but that doesn't mean the LRU replacement algorithm that keeps your house clean can move them to the back of your attic.
- JimmaDaRustla 11y ago
  True. Back in the day, before automated tape readers, tapes were stored in silos, like a library, and employees would find the cataloged tapes and manually retrieve and load them into tape readers. My company use to hire co-ops to do this back in the 70s or so.

Someone 11y ago

You're referring to https://en.m.wikipedia.org/wiki/Hierarchical_storage_management https://en.m.wikipedia.org/wiki/Hierarchical_storage_managem.... AFAIK, most systems work at the file level, but if you address space is large enough, you could implement this as a virtual memory system that swaps RAM to disk, disk to slower disk, slower disk to tape, tapes to tapes stored offline, etc. I don't know what Amazon uses, either, but I expect Glacier data to be pushed out to some layer that doesn't need power while idle at all, heavily checksummed and also replicated across the globe. The relatively heavy prices tag for accessing it probably is there to keep contention on some relatively thin pipe low. For example, if they use blue ray, they may have only a few readers online for every 100 or 1000 customers, both to keep prices low and to differentiate Glacier more from their other products.

icebraining 11y ago
if you address space is large enough, you could implement this as a virtual memory system that swaps RAM to disk, disk to slower disk, slower disk to tape, tapes to tapes stored offline, etc. Funny to imagine, - My program has been stuck copying a variable for over half an hour! - Yeah, there was a page fault and the tape loading guy still hasn't come from lunch
- Someone 11y ago
  Yes, but the other side of the coin is "why are those pages of the program still on disk? They haven't been touched in years". But such systems tend to keep parts under human control. You rarely ever need your fire extinguisher, but that doesn't mean the LRU replacement algorithm that keeps your house clean can move them to the back of your attic.
JimmaDaRustla 11y ago
True. Back in the day, before automated tape readers, tapes were stored in silos, like a library, and employees would find the cataloged tapes and manually retrieve and load them into tape readers. My company use to hire co-ops to do this back in the 70s or so.