5 ms·

It's not a complicated version of the same solution - it's cheaper version of the same solution. As expensive as flash is, it's still cheaper than the equivalen

by russss 17y ago

It's not a complicated version of the same solution - it's cheaper version of the same solution. As expensive as flash is, it's still cheaper than the equivalent amount of RAM.

gojomo 17y ago
You wouldn't necessarily need a RAM buffer of the same size as source storage; it depends a lot on what X% of all song-seconds will be played in the next N buffered minutes. If X is 100%, the approaches converge... but I think lots of things (popularity of songs, discretion over next song played, choice of N, etc.) could make buffers much smaller than total catalog size helpful in seek-reduction. (The buffer could also be read-write SSD instead of RAM. Or perhaps just for the most popular items or skip-prone early ranges of files.)
- russss 17y ago
  These SSD streamers only hold the most popular 10-20% of Last.fm's content, so they are effectively just holding the bytes which are most likely to be streamed.
ajross 17y ago
Cheaper to buy as a part, not cheaper to implement. Given that last.fm has a bunch of boxes with the data already on them, replacing them all with SSD versions sounds like a more expensive solution than a bunch of buffering proxies to me. And in any case, I said "complicated", not "cheaper", and I'll stand by that. Complexity has costs all by itself.
- russss 17y ago
  Run-of-the-mill caching proxies are no good in front of a distributed progressive streaming system such as this, because they would have to be modified to stream progressively and also to authenticate requests, which would increase the complexity far more than the comparatively minor changes Last.fm had to make to MogileFS. So I say it's cheaper and less complex.
  - ajross 17y ago
    Huh? I'm all but certain that a default squid install would work fine. "Progressive streaming" at the server side generally doesn't require any configuration at all in TCP -- the client reads what it wants, the buffers fill up, and the transmission stalls on a missing ACK. And the authentication layer is almost certainly downstream of the backend storage anyway, so I'm not sure how that would matter. Swapping hardware is a really expensive expensive change in the IT world, significantly more so than deploying new hardware on an existing infrastructure. That you somehow think otherwise is surprising to me.
InclinedPlane 17y ago
Moreover, in order to build out to 320, 640, or 1280 GB with SSD you need only off the shelf technologies (SATA/SAS raid), it's not even that expensive (even a 1TB top of the line SSD array in RAID 10 would cost less than $50k). To match that amount of storage in RAM would require multiple machines, rare motherboards or other hardware, and a huge outlay for RAM. Not to mention it would require pretty significant rewrites of their applications in order to use the RAM properly.

by russss 17y ago

It's not a complicated version of the same solution - it's cheaper version of the same solution. As expensive as flash is, it's still cheaper than the equivalent amount of RAM.

gojomo 17y ago

You wouldn't necessarily need a RAM buffer of the same size as source storage; it depends a lot on what X% of all song-seconds will be played in the next N buffered minutes. If X is 100%, the approaches converge... but I think lots of things (popularity of songs, discretion over next song played, choice of N, etc.) could make buffers much smaller than total catalog size helpful in seek-reduction. (The buffer could also be read-write SSD instead of RAM. Or perhaps just for the most popular items or skip-prone early ranges of files.)

russss 17y ago
These SSD streamers only hold the most popular 10-20% of Last.fm's content, so they are effectively just holding the bytes which are most likely to be streamed.

ajross 17y ago

Cheaper to buy as a part, not cheaper to implement. Given that last.fm has a bunch of boxes with the data already on them, replacing them all with SSD versions sounds like a more expensive solution than a bunch of buffering proxies to me. And in any case, I said "complicated", not "cheaper", and I'll stand by that. Complexity has costs all by itself.

russss 17y ago
Run-of-the-mill caching proxies are no good in front of a distributed progressive streaming system such as this, because they would have to be modified to stream progressively and also to authenticate requests, which would increase the complexity far more than the comparatively minor changes Last.fm had to make to MogileFS. So I say it's cheaper and less complex.
- ajross 17y ago
  Huh? I'm all but certain that a default squid install would work fine. "Progressive streaming" at the server side generally doesn't require any configuration at all in TCP -- the client reads what it wants, the buffers fill up, and the transmission stalls on a missing ACK. And the authentication layer is almost certainly downstream of the backend storage anyway, so I'm not sure how that would matter. Swapping hardware is a really expensive expensive change in the IT world, significantly more so than deploying new hardware on an existing infrastructure. That you somehow think otherwise is surprising to me.

InclinedPlane 17y ago

Moreover, in order to build out to 320, 640, or 1280 GB with SSD you need only off the shelf technologies (SATA/SAS raid), it's not even that expensive (even a 1TB top of the line SSD array in RAID 10 would cost less than $50k). To match that amount of storage in RAM would require multiple machines, rare motherboards or other hardware, and a huge outlay for RAM. Not to mention it would require pretty significant rewrites of their applications in order to use the RAM properly.