4 ms·

I'm sure they wear out quick with heavy writes. On the other hand, the killer app for these has to be media streaming. How many more movies can Netflix suppor

I'm sure they wear out quick with heavy writes. On the other hand, the killer app for these has to be media streaming. How many more movies can Netflix support with these new drives? I wonder if they are space limited or iops limited. "The drive is aimed at read-intensive applications, such as data warehousing, media streaming and web servers. The typical workload envisioned for the 4TB drive is 90% read and 10% write, SanDisk stated."

lomnakkus 13y ago
"I'm sure they wear out quick with heavy writes." Do you have something to back that assertion up? Seriously, I'm interested, so if you have links to further information, I'm sure everyone would appreciate it.
- jhallenworld 13y ago
  Well here is one, but this is really for the device level: http://nepp.nasa.gov/workshops/etw2012/talks/Tuesday/T04_Heidecker_Flash_Qualification.pdf http://nepp.nasa.gov/workshops/etw2012/talks/Tuesday/T04_Hei... If I find a good system-level paper, I'll post it. Anyway, SSD drive wear tends to be rated like this: 5 years for x number of full drive writes per day. For good drives, x is 1. For really good SLC NAND based drives x could be as much as 10, but they are very expensive. For cheap consumer drives x could be 1/10. Based on vendor sales meetings, the aggressively large drives also have low write endurance (but then they say they are not optimized for heavy writes). Better drives should have a built-in write counter so you can track the remaining life. Another interesting parameter is their power-off "data retention" time. At end of life, this tends to be specified to be just 90 days.
- tim333 13y ago
  There seems a lot of contradictory data out there but Techreport did an experiment with 6 consumer grade 256GB drives endlessly reading and writing: http://techreport.com/review/24841/introducing-the-ssd-endurance-experiment http://techreport.com/review/24841/introducing-the-ssd-endur... They were still going after 600TB of read writes though with some degradation: http://techreport.com/review/26058/the-ssd-endurance-experiment-data-retention-after-600tb http://techreport.com/review/26058/the-ssd-endurance-experim...
  - jhallenworld 13y ago
    Interesting, my one quibble is that I don't see anything about the temperature. Flash memory is very sensitive to it.
colechristensen 13y ago
>How many more movies can Netflix support with these new drives? Netflix doesn't do their own hardware, they use AWS. One can also assume that the cost of the content far outweighs the cost of delivery.
- kordless 13y ago
  AWS' monthly costs approach cost of ownership. Netflix's costs in providing service are a substantial amount of their spend. AWS works well for Netflix because they have a workload that varies on a daily and weekly basis - by using AWS' excess capacity during peak times, Netflix is able to save money over owning their infrastructure outright. Given AWS is one big Tetris game, jamming instances in where they fit, this has worked out well for both parties. Until now, with 'now' being defined as the dawn of distributed commodity compute.
- latch 13y ago
  The AWS marketing folks don't get paid enough. https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware
  - adventured 13y ago
    They do, however, use AWS to run most of their service: http://gigaom.com/2013/12/02/netflix-is-balancing-its-streaming-traffic-across-amazons-cloud/ http://gigaom.com/2013/12/02/netflix-is-balancing-its-stream...
- insaneirish 13y ago
  This is untrue. AWS is used for some things, but not all. I've heard certain details I'm unwilling to share, but some of what they do is public (https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware). Also, they pay developers to work on optimizing network performance of FreeBSD for this custom hardware.
jedberg 13y ago
> I wonder if [Netflix is] space limited or iops limited. I don't think you're asking the right question. The more iops, the less space and the less cost. I'd say Netflix is always optimizing for both. The right question to ask is what is the break-even point between space and cost where these drives make sense, and that would depend mostly on how popular the most popular content is. If everyone were watching the same few things, but enough that it can't just all fit in RAM, it would make a lot of sense to have a lot of these drives. But if the watching is spread out across a lot of content, then not so much.
- jhallenworld 13y ago
  Hmm.. I read that a 2 hour Netflix movie is about 2 GB. So you need ~2.2 Mbps per user or 183 iops / client in terms of 1.5K packets. One drive can fit 2000 movies. 75K iops/drive / 183 iops/client = 400 clients/drive (this is 900 Mbps, but the drive says up to 400 MB/s, so not bandwidth limited). Put 16 in a 2U server for 6400 clients. This is 14 Gbps, so you need two 10 G NICs to complete our Netflix appliance. But you are right, if caching works we don't need all these drives. With 128 GB of RAM we can fit 64 movies in RAM. I could easily believe that most people only watch the popular movies.
  - justincormack 13y ago
    Here is the link to what they actually use https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware
- reitzensteinm 13y ago
  Some quick figures to put it in perspective: According to instantwatcher.com, there are 6843 movies on Netflix streaming currently. If we conservatively say that the 1080p stream is 2gb, and the other formats add up to another 2gb, that's a total of ~27 TB of space required. There are 3657 seasons of TV, let's say an average of 2gb per episode, and an average of 14 episodes a season. That's ~102 TB. So napkin math suggests ~129TB of storage to hold all of Netflix, or 33 of these drives. Since they need redundancy and to have servers distributed around the country for peering, and these drives will probably cost ~$10k each (5x premium on consumer), I doubt they'll be putting their long tail content on them. How many people in Boston are currently watching Mission Impossible, the TV series from 1966? Or content like it? A standard hard drive can serve 500 people HD streams so long as you cache ahead into memory intelligently in big enough chunks. Within a few generations, SSDs will probably be cheap enough to not even bother with mechanical, for this and many other use cases. [1]http://instantwatcher.com/titles/all?tv_filter=movies+only http://instantwatcher.com/titles/all?tv_filter=movies+only
zatkin 13y ago
It's important to note that most companies these days that have 'big data' typically follow the WORM (write once, read many) pattern. They utilize technologies that write their data to the disk once, but never delete.
virtuallynathan 13y ago
Netflix has 28TB of Flash storage in a 1U box (e3-12xxv2, 125w system power usage) with 40Gbit/s of Network capacity, which gets fully saturated. They run BSD, and Nginx, I believe. This would double their density.

lomnakkus 13y ago

"I'm sure they wear out quick with heavy writes." Do you have something to back that assertion up? Seriously, I'm interested, so if you have links to further information, I'm sure everyone would appreciate it.

jhallenworld 13y ago
Well here is one, but this is really for the device level: http://nepp.nasa.gov/workshops/etw2012/talks/Tuesday/T04_Heidecker_Flash_Qualification.pdf http://nepp.nasa.gov/workshops/etw2012/talks/Tuesday/T04_Hei... If I find a good system-level paper, I'll post it. Anyway, SSD drive wear tends to be rated like this: 5 years for x number of full drive writes per day. For good drives, x is 1. For really good SLC NAND based drives x could be as much as 10, but they are very expensive. For cheap consumer drives x could be 1/10. Based on vendor sales meetings, the aggressively large drives also have low write endurance (but then they say they are not optimized for heavy writes). Better drives should have a built-in write counter so you can track the remaining life. Another interesting parameter is their power-off "data retention" time. At end of life, this tends to be specified to be just 90 days.
tim333 13y ago
There seems a lot of contradictory data out there but Techreport did an experiment with 6 consumer grade 256GB drives endlessly reading and writing: http://techreport.com/review/24841/introducing-the-ssd-endurance-experiment http://techreport.com/review/24841/introducing-the-ssd-endur... They were still going after 600TB of read writes though with some degradation: http://techreport.com/review/26058/the-ssd-endurance-experiment-data-retention-after-600tb http://techreport.com/review/26058/the-ssd-endurance-experim...
- jhallenworld 13y ago
  Interesting, my one quibble is that I don't see anything about the temperature. Flash memory is very sensitive to it.

colechristensen 13y ago

>How many more movies can Netflix support with these new drives? Netflix doesn't do their own hardware, they use AWS. One can also assume that the cost of the content far outweighs the cost of delivery.

kordless 13y ago
AWS' monthly costs approach cost of ownership. Netflix's costs in providing service are a substantial amount of their spend. AWS works well for Netflix because they have a workload that varies on a daily and weekly basis - by using AWS' excess capacity during peak times, Netflix is able to save money over owning their infrastructure outright. Given AWS is one big Tetris game, jamming instances in where they fit, this has worked out well for both parties. Until now, with 'now' being defined as the dawn of distributed commodity compute.
latch 13y ago
The AWS marketing folks don't get paid enough. https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware
- adventured 13y ago
  They do, however, use AWS to run most of their service: http://gigaom.com/2013/12/02/netflix-is-balancing-its-streaming-traffic-across-amazons-cloud/ http://gigaom.com/2013/12/02/netflix-is-balancing-its-stream...
insaneirish 13y ago
This is untrue. AWS is used for some things, but not all. I've heard certain details I'm unwilling to share, but some of what they do is public (https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware). Also, they pay developers to work on optimizing network performance of FreeBSD for this custom hardware.

jedberg 13y ago

> I wonder if [Netflix is] space limited or iops limited. I don't think you're asking the right question. The more iops, the less space and the less cost. I'd say Netflix is always optimizing for both. The right question to ask is what is the break-even point between space and cost where these drives make sense, and that would depend mostly on how popular the most popular content is. If everyone were watching the same few things, but enough that it can't just all fit in RAM, it would make a lot of sense to have a lot of these drives. But if the watching is spread out across a lot of content, then not so much.

jhallenworld 13y ago
Hmm.. I read that a 2 hour Netflix movie is about 2 GB. So you need ~2.2 Mbps per user or 183 iops / client in terms of 1.5K packets. One drive can fit 2000 movies. 75K iops/drive / 183 iops/client = 400 clients/drive (this is 900 Mbps, but the drive says up to 400 MB/s, so not bandwidth limited). Put 16 in a 2U server for 6400 clients. This is 14 Gbps, so you need two 10 G NICs to complete our Netflix appliance. But you are right, if caching works we don't need all these drives. With 128 GB of RAM we can fit 64 movies in RAM. I could easily believe that most people only watch the popular movies.
- justincormack 13y ago
  Here is the link to what they actually use https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware
reitzensteinm 13y ago
Some quick figures to put it in perspective: According to instantwatcher.com, there are 6843 movies on Netflix streaming currently. If we conservatively say that the 1080p stream is 2gb, and the other formats add up to another 2gb, that's a total of ~27 TB of space required. There are 3657 seasons of TV, let's say an average of 2gb per episode, and an average of 14 episodes a season. That's ~102 TB. So napkin math suggests ~129TB of storage to hold all of Netflix, or 33 of these drives. Since they need redundancy and to have servers distributed around the country for peering, and these drives will probably cost ~$10k each (5x premium on consumer), I doubt they'll be putting their long tail content on them. How many people in Boston are currently watching Mission Impossible, the TV series from 1966? Or content like it? A standard hard drive can serve 500 people HD streams so long as you cache ahead into memory intelligently in big enough chunks. Within a few generations, SSDs will probably be cheap enough to not even bother with mechanical, for this and many other use cases. [1]http://instantwatcher.com/titles/all?tv_filter=movies+only http://instantwatcher.com/titles/all?tv_filter=movies+only

zatkin 13y ago

It's important to note that most companies these days that have 'big data' typically follow the WORM (write once, read many) pattern. They utilize technologies that write their data to the disk once, but never delete.

virtuallynathan 13y ago

Netflix has 28TB of Flash storage in a 1U box (e3-12xxv2, 125w system power usage) with 40Gbit/s of Network capacity, which gets fully saturated. They run BSD, and Nginx, I believe. This would double their density.