5 ms·

I'm a little confused as to the actual problem being solved here. For SSDs there is essentially no seek penalty, so random read and sequential read performance

by sadiq 17y ago

I'm a little confused as to the actual problem being solved here. For SSDs there is essentially no seek penalty, so random read and sequential read performance are essentially the same. Spinning hard drives, because of the seek penalty, have pretty poor random read performance but fairly good sequential read. Surely, for a streaming application, it should be sequential read that is the key requirement? You can predict with a very high degree of accuracy what piece of data you'll be requiring in the near future, at which point the seek can be scheduled well in advance. This, of course, assumes your data is laid out in a sane fashion and you're not blindly jumping all over the disks. Strikes me that last.fm could potentially do with looking at approaches like Facebook's Haystack where they put considerable time in to minimising the amount of IOPS required to pull an image off of the disk.

russss 17y ago
Last.fm does progressive streaming, not HTTP downloads. i.e. it reads 16KB of one track, sends it to one user, then 16KB of the next track, sends it to the next user, etc. Then after 1 second it goes through and does the whole thing again. This is a massively seek-heavy operation.
- sadiq 17y ago
  Hmmm yes, you're right. Even with knowledge of the accesses required in 1s time, you're still bounded by the speed of the movements of the heads. That said. I wonder if there's any trickery you can do with pre-generating the interleaved data for a group of users. As in: 1) Read the next 5 (or any amount really) tracks for each of the users in a group. 2) Interleave the 16KB blocks in the same pattern you'd be streaming. Step 1 is just a whole set of sequential reads. Step 2 is one big sequential write. Then your streaming operation is a single sequential read. Of course, there's a good chance the overhead on doing the above outweighs the gain.
  - russss 17y ago
    But you've got the additional bonus problem that users can skip Last.fm tracks at any time, so there's a pretty high probability that you won't need all that data anyway. (I used to work for Last.fm and we spent a fair amount of time working on this issue without a great amount of success.)
    - sadiq 17y ago
      D'oh, I hadn't considered that. I'm also wondering what moved them towards the SLC-based Intel drives as opposed to the MLC-based ones (that offer a much lower price-capacity), since in terms of read performance IOPS they're almost identical.
      - kogir 17y ago
        Probably reliability. From Intel's docs: X25-M: 1,200,000 Hours X25-E: 2,000,000 Hours I'd be willing to replace them more regularly for the added capacity though.
        chrisbolt 17y ago
        X25-M 80GB: $270 X25-E 64GB: $890 At that price you can buy 3 X25-Ms and still have more space. The real difference is in write cycles though: the X25-E can do ten times as many as the X25-M.
      - lozzd2 17y ago
        We were going to go for the larger sized (space wise) MLC based ones except that they're in the smaller form factor, not easy to fit in the server chassis. The connector on them is also non standard, unlike the MLC models which slide right in. Makes a big difference when you're buying a lot of them. Also, if we ever decide to re-commission these machines later, the SLC based ones will be more useful for us.
        chrisbolt 17y ago
        I've been using both the MLC and SLC Intels and they both have the same standard SATA connector. Also, server chassis are moving to 2.5" SATA bays, our latest servers have 2 dual processor motherboards in 1U (Supermicro calls them 1U-twin), with 4 2.5" bays for each motherboard.
- ajross 17y ago
  Only when implemented naively. Why not transfer the whole 5-10MB file at once into a buffer somewhere (on-box, in a proxy, whatever) and then stream from there? DRAM is vastly cheaper than trying to get a seek-heavy architecture to scale on seek-limited devices. But then, moving to flash is sort of a complicated version of the same solution...
  - russss 17y ago
    It's not a complicated version of the same solution - it's cheaper version of the same solution. As expensive as flash is, it's still cheaper than the equivalent amount of RAM.
    - gojomo 17y ago
      You wouldn't necessarily need a RAM buffer of the same size as source storage; it depends a lot on what X% of all song-seconds will be played in the next N buffered minutes. If X is 100%, the approaches converge... but I think lots of things (popularity of songs, discretion over next song played, choice of N, etc.) could make buffers much smaller than total catalog size helpful in seek-reduction. (The buffer could also be read-write SSD instead of RAM. Or perhaps just for the most popular items or skip-prone early ranges of files.)
      - russss 17y ago
        These SSD streamers only hold the most popular 10-20% of Last.fm's content, so they are effectively just holding the bytes which are most likely to be streamed.
    - ajross 17y ago
      Cheaper to buy as a part, not cheaper to implement. Given that last.fm has a bunch of boxes with the data already on them, replacing them all with SSD versions sounds like a more expensive solution than a bunch of buffering proxies to me. And in any case, I said "complicated", not "cheaper", and I'll stand by that. Complexity has costs all by itself.
- gojomo 17y ago
  Even with that explanation, and the risk of user-triggered track-skips, the seek-heaviness doesn't seem inherent to the problem. Much of the time, and oftentimes for runs of many minutes/megabytes at a time, you can predict exactly what data will be needed at what future time. Perhaps another approach would be to add more buffering in front of spinning disks -- perhaps RAM, perhaps SSD -- and tune the disk IO scheduler to be extreme in the other direction: have the app request everything at least N minutes in advance, and let the scheduler take up to N minutes to deliver it. Then it can take advantage of any seek-minimizing physically-near blocks over a much larger window of time. (For those occasional skip exceptions, data could be pulled from a smaller lower-latency system, or from a smaller set of 'whatever happens to be ready' fill-ins, etc.)
lanaer 17y ago
SSDs still have substantially higher sequential read speeds. The specific one they are using (the Intel X-25E) has over two or three times the sequential read speed of 2 of the fastest standard drives. http://anandtech.com/storage/showdoc.aspx?i=3631&p=21 http://anandtech.com/storage/showdoc.aspx?i=3631&p=21

by sadiq 17y ago

russss 17y ago

Last.fm does progressive streaming, not HTTP downloads. i.e. it reads 16KB of one track, sends it to one user, then 16KB of the next track, sends it to the next user, etc. Then after 1 second it goes through and does the whole thing again. This is a massively seek-heavy operation.

sadiq 17y ago
Hmmm yes, you're right. Even with knowledge of the accesses required in 1s time, you're still bounded by the speed of the movements of the heads. That said. I wonder if there's any trickery you can do with pre-generating the interleaved data for a group of users. As in: 1) Read the next 5 (or any amount really) tracks for each of the users in a group. 2) Interleave the 16KB blocks in the same pattern you'd be streaming. Step 1 is just a whole set of sequential reads. Step 2 is one big sequential write. Then your streaming operation is a single sequential read. Of course, there's a good chance the overhead on doing the above outweighs the gain.
- russss 17y ago
  But you've got the additional bonus problem that users can skip Last.fm tracks at any time, so there's a pretty high probability that you won't need all that data anyway. (I used to work for Last.fm and we spent a fair amount of time working on this issue without a great amount of success.)
  - sadiq 17y ago
    D'oh, I hadn't considered that. I'm also wondering what moved them towards the SLC-based Intel drives as opposed to the MLC-based ones (that offer a much lower price-capacity), since in terms of read performance IOPS they're almost identical.
    - kogir 17y ago
      Probably reliability. From Intel's docs: X25-M: 1,200,000 Hours X25-E: 2,000,000 Hours I'd be willing to replace them more regularly for the added capacity though.
      - chrisbolt 17y ago
        X25-M 80GB: $270 X25-E 64GB: $890 At that price you can buy 3 X25-Ms and still have more space. The real difference is in write cycles though: the X25-E can do ten times as many as the X25-M.
    - lozzd2 17y ago
      We were going to go for the larger sized (space wise) MLC based ones except that they're in the smaller form factor, not easy to fit in the server chassis. The connector on them is also non standard, unlike the MLC models which slide right in. Makes a big difference when you're buying a lot of them. Also, if we ever decide to re-commission these machines later, the SLC based ones will be more useful for us.
      - chrisbolt 17y ago
        I've been using both the MLC and SLC Intels and they both have the same standard SATA connector. Also, server chassis are moving to 2.5" SATA bays, our latest servers have 2 dual processor motherboards in 1U (Supermicro calls them 1U-twin), with 4 2.5" bays for each motherboard.
ajross 17y ago
Only when implemented naively. Why not transfer the whole 5-10MB file at once into a buffer somewhere (on-box, in a proxy, whatever) and then stream from there? DRAM is vastly cheaper than trying to get a seek-heavy architecture to scale on seek-limited devices. But then, moving to flash is sort of a complicated version of the same solution...
- russss 17y ago
  It's not a complicated version of the same solution - it's cheaper version of the same solution. As expensive as flash is, it's still cheaper than the equivalent amount of RAM.
  - gojomo 17y ago
    You wouldn't necessarily need a RAM buffer of the same size as source storage; it depends a lot on what X% of all song-seconds will be played in the next N buffered minutes. If X is 100%, the approaches converge... but I think lots of things (popularity of songs, discretion over next song played, choice of N, etc.) could make buffers much smaller than total catalog size helpful in seek-reduction. (The buffer could also be read-write SSD instead of RAM. Or perhaps just for the most popular items or skip-prone early ranges of files.)
    - russss 17y ago
      These SSD streamers only hold the most popular 10-20% of Last.fm's content, so they are effectively just holding the bytes which are most likely to be streamed.
  - ajross 17y ago
    Cheaper to buy as a part, not cheaper to implement. Given that last.fm has a bunch of boxes with the data already on them, replacing them all with SSD versions sounds like a more expensive solution than a bunch of buffering proxies to me. And in any case, I said "complicated", not "cheaper", and I'll stand by that. Complexity has costs all by itself.
gojomo 17y ago
Even with that explanation, and the risk of user-triggered track-skips, the seek-heaviness doesn't seem inherent to the problem. Much of the time, and oftentimes for runs of many minutes/megabytes at a time, you can predict exactly what data will be needed at what future time. Perhaps another approach would be to add more buffering in front of spinning disks -- perhaps RAM, perhaps SSD -- and tune the disk IO scheduler to be extreme in the other direction: have the app request everything at least N minutes in advance, and let the scheduler take up to N minutes to deliver it. Then it can take advantage of any seek-minimizing physically-near blocks over a much larger window of time. (For those occasional skip exceptions, data could be pulled from a smaller lower-latency system, or from a smaller set of 'whatever happens to be ready' fill-ins, etc.)

lanaer 17y ago

SSDs still have substantially higher sequential read speeds. The specific one they are using (the Intel X-25E) has over two or three times the sequential read speed of 2 of the fastest standard drives. http://anandtech.com/storage/showdoc.aspx?i=3631&p=21 http://anandtech.com/storage/showdoc.aspx?i=3631&p=21