4 ms·

While there's an explosion of data that we keep hearing about, the aspect I'm curious about is how much of that data will at any one point be "hot" or even "war

While there's an explosion of data that we keep hearing about, the aspect I'm curious about is how much of that data will at any one point be "hot" or even "warm," in that it needs to be processed and analyzed in short order (talking microseconds or nanoseconds, not in tens of seconds like tape (or more as the data then needs to be sent over a network to a server somewhere else)). Cold storage can be archived on tape or HDDs, sure, but as compute continues to grow, surely more and more data will need to be hot or warm at any given moment. Just whether that hot/warm data will grow its overall share of data in existence or simply grow at the same rate is what I wonder about.

CaliforniaKarl 8y ago
So, first I would quibble a bit with the time scales you're talking about, as tape access could easily be more than tens of seconds, depending on status: Even a loaded tape might take a minute to reposition, if the data you want is at the opposite end. In the world I support, everything has to be funded by something. We have our "cheap and deep" storage, for example (http://oak-storage.stanford.edu http://oak-storage.stanford.edu), but labs who use it have to pay for it. Unchecked growth is highly unlikely, as grants run out eventually, and then the data will have to go somewhere else. Often these days, that ends up being Google Drive, but there is demand for tape archival storage. Even the Stanford Digital Repository (https://library.stanford.edu/research/stanford-digital-repository https://library.stanford.edu/research/stanford-digital-repos...), who are extremely serious when they talk about "long-term", say they're developing pricing models.
- Rafuino 8y ago
  Well, I was keying off the article itself, which said the following: The long length of the tape held in a cartridge—normally hundreds of meters—results in average data-access times of 50 to 60 seconds It's tens of seconds, up to a minute, but that's just average, so I think we're on the same page here! Even when not in academia, everything has to be funded by someone at some point. Sounds like you have a pretty awesome job though, and I imagine Stanford is one of the least likely to get cut off from precious NSF grants!
dingaling 8y ago
When I started working on mainframes at a big financial company there were jokes about the 'tape monkeys' which loaded tapes when we requested old datasets. We laughed and dismissed that, surely everything was mechanised. It was only after a year or so that I was cleared to visit the data centre and discovered that the mechanical tape silos had failed years ago. Every tape had to be located and loaded by hand. I felt guilty about all those datasets I had ordered on whim... and was surprised that the DC guys didn't have a 'Hit List' of progammers.
- Rafuino 8y ago
  I'd be very paranoid if I were you :)

CaliforniaKarl 8y ago

So, first I would quibble a bit with the time scales you're talking about, as tape access could easily be more than tens of seconds, depending on status: Even a loaded tape might take a minute to reposition, if the data you want is at the opposite end. In the world I support, everything has to be funded by something. We have our "cheap and deep" storage, for example (http://oak-storage.stanford.edu http://oak-storage.stanford.edu), but labs who use it have to pay for it. Unchecked growth is highly unlikely, as grants run out eventually, and then the data will have to go somewhere else. Often these days, that ends up being Google Drive, but there is demand for tape archival storage. Even the Stanford Digital Repository (https://library.stanford.edu/research/stanford-digital-repository https://library.stanford.edu/research/stanford-digital-repos...), who are extremely serious when they talk about "long-term", say they're developing pricing models.

Rafuino 8y ago
Well, I was keying off the article itself, which said the following: The long length of the tape held in a cartridge—normally hundreds of meters—results in average data-access times of 50 to 60 seconds It's tens of seconds, up to a minute, but that's just average, so I think we're on the same page here! Even when not in academia, everything has to be funded by someone at some point. Sounds like you have a pretty awesome job though, and I imagine Stanford is one of the least likely to get cut off from precious NSF grants!

dingaling 8y ago

When I started working on mainframes at a big financial company there were jokes about the 'tape monkeys' which loaded tapes when we requested old datasets. We laughed and dismissed that, surely everything was mechanised. It was only after a year or so that I was cleared to visit the data centre and discovered that the mechanical tape silos had failed years ago. Every tape had to be located and loaded by hand. I felt guilty about all those datasets I had ordered on whim... and was surprised that the DC guys didn't have a 'Hit List' of progammers.

Rafuino 8y ago
I'd be very paranoid if I were you :)