4 ms·

the stream metaphore isn't appropriate for all types of data. memory allows a storage paradigm to be picked for the task at hand. Consider redis which is a gre

the stream metaphore isn't appropriate for all types of data. memory allows a storage paradigm to be picked for the task at hand. Consider redis which is a great example of this. How do you store redis' data efficiently? Well it turns that aof files are slow to start up and disk backed virtual memory is slow. The problem goes away instantly with NVM - the jib is done with no filesystem api used.

ajross 14y ago
I dare say that the stream metaphor is a better fit for more types of data than Redis is. To first approximation, all data in the modern world is video files. You really want to store those in a raw memory space?
- gouranga 14y ago
  That's ok until you need to read hundreds of multiple streams from a stream device. You end up with random access then at which point the stream paradigm breaks down and you have to use memory.
- Cushman 14y ago
  Filmmakers would kill for that ability.
  - ajross 14y ago
    Filmmakers don't work with compressed video files (which I picked precisely because they're inherently a stream, and because I'm not kidding when I say that they're basically all the data there is in this world). And seem to be doing quite well with DRAM anyway.
    - Cushman 14y ago
      Fair point. Although, I'd wonder if it's right that video is inherently a stream, and not that that's a limitation imposed by storage speeds. I might only want to watch video front-to-back, but (without knowing much about encoding) I could easily imagine that my experience might be improved by my computer having random access.
      - sp332 14y ago
        It's still effectively a stream even if you skip around in time. Each frame is so large that it is a stream itself. So even if you watch a few seconds here and a few there, your use case is still pretty much optimized for streaming data.
      - Tuna-Fish 14y ago
        Modern video encoding is based on interpolating from the last frame, that is, only storing a real frame occasionally, and most of the time just having diffs from the last frame. ^+ This means that the data is inherently a stream -- a decoder needs to have completed decoding the previous frame before it can start on the next. (^+ and B-frames, but let's not overcomplicate things)

ajross 14y ago

I dare say that the stream metaphor is a better fit for more types of data than Redis is. To first approximation, all data in the modern world is video files. You really want to store those in a raw memory space?

gouranga 14y ago
That's ok until you need to read hundreds of multiple streams from a stream device. You end up with random access then at which point the stream paradigm breaks down and you have to use memory.
Cushman 14y ago
Filmmakers would kill for that ability.
- ajross 14y ago
  Filmmakers don't work with compressed video files (which I picked precisely because they're inherently a stream, and because I'm not kidding when I say that they're basically all the data there is in this world). And seem to be doing quite well with DRAM anyway.
  - Cushman 14y ago
    Fair point. Although, I'd wonder if it's right that video is inherently a stream, and not that that's a limitation imposed by storage speeds. I might only want to watch video front-to-back, but (without knowing much about encoding) I could easily imagine that my experience might be improved by my computer having random access.
    - sp332 14y ago
      It's still effectively a stream even if you skip around in time. Each frame is so large that it is a stream itself. So even if you watch a few seconds here and a few there, your use case is still pretty much optimized for streaming data.
    - Tuna-Fish 14y ago
      Modern video encoding is based on interpolating from the last frame, that is, only storing a real frame occasionally, and most of the time just having diffs from the last frame. ^+ This means that the data is inherently a stream -- a decoder needs to have completed decoding the previous frame before it can start on the next. (^+ and B-frames, but let's not overcomplicate things)