4 ms·

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size

by moe 14y ago

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix. You rarely hear of Google outages, or iTunes, or Youtube, or [insert six dozen other popular brands], do you? Yes it can happen to the best of them, but the EC2 outages are really piling up lately.

pixl97 14y ago
You rarely hear of Google outages http://techcrunch.com/2012/12/10/gmail-experiences-a-widespread-outage-most-users-affected/ http://techcrunch.com/2012/12/10/gmail-experiences-a-widespr... ,or iTunes http://appleinsider.com/articles/12/11/19/itunes-match-down-as-apples-icloud-issues-continue http://appleinsider.com/articles/12/11/19/itunes-match-down-... ,or Youtube http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-down-not-googles-best-day/ http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-... Everyone goes down, but EC2 has been buggy as hell from what I can tell. And before everyone says Netflix should `just` move or add more bandwidth... http://www.nbcnews.com/technology/technolog/netflix-uses-32-7-percent-internet-bandwidth-119517 http://www.nbcnews.com/technology/technolog/netflix-uses-32-... They represent a huuuge amount of bandwidth usage!
- toomuchtodo 14y ago
  So work with eyeball networks and move your gear to the edge; dumping it all in AWS and hoping for the best is failing (at best). Do all the auth/billing/recommendations/transcoding in AWS, and push all the video content out to boxes at the ISPs.
- mh- 14y ago
  That bandwidth for delivering video is being streamed from a CDN outside AWS.
res0nat0r 14y ago
>Especially one as trivial as Netflix which is largely read-only. I stopped reading after this, since you obviously dont understand how complex their architecture really is.
- sbov 14y ago
  I'm never one to trivialize another company's problems so would hesitate to say Netflix's case is easy, but you cannot counter this statement by saying their architecture is complex. I can build a complex architecture for "Hello World" if I wanted to. Complex architecture != complex problem.
- moe 14y ago
  I work for a media streaming site (rank <30k on alexa) and I assure you the architecture doesn't change much when you scale it up to rank 99 (netflix). "Trivial" may sound harsh but in terms of large scale systems it really doesn't get much simpler than serving static files. Obviously their auxiliary services (billing, content ingestion, social etc.) are nowhere near trivial. But there's no reason for the core services that are required to get the catalog and video to most[1] devices to not be damn near 100% available. I'm not talking down netflix for having an outage anyway. Shit happens and afaik their overall track record is not bad at all. I only replied to jrockway's claim that multi-datacenter redundancy would require inconceivable engineering effort or amounts of money - neither is true. [1] Excluding those that need realtime transcoding, absurd DRM schemes or similar.
  - brown9-2 14y ago
    How can you assure us of what the #99 system is like when you admit to only working on the ~30k system? That's not what "assure" means.
    - moe 14y ago
      We actually talk to other people in the industry sometimes, imagine that...
jrockway 14y ago
You rarely hear of Google outages, or iTunes, or Youtube, or [insert six dozen other popular brands], do you? Yes it can happen to the best of them, but the EC2 outages are really piling up lately. To me, this makes perfect sense. Google products are almost always designed to tolerate data center failures, and so you don't hear about data center failures affecting Google products. On the other hand, Joe's Random EC2 app is not designed for the same high availability, and so it's down whenever Amazon is. (This, incidentally, is a reasonable trade-off for most people. A few hours of downtime a year is often much cheaper than paying engineers to ensure that those few hours are five minutes or less.) So the flaw here is not Amazon. The flaw is relying on one cluster of computers to handle all your computing needs. Amazon could make their product high-replication by default, but it would be so slow and expensive that nobody would use it. So the task is kicked to the application developers instead of the platform developers, and you should consider holding them responsible for the downtime. (Incidentally, you can compare EC2 to AppEngine here. How often is your favorite AppEngine app down? Less than your favorite EC2 app, probably, because AppEngine pretty much forces the high-replication datastore, even though the semantics are strange for developers used to traditional single-homed or master/slave architectures. And if you read the AppEngine discussion groups, you'll see that users raise most of the concerns that I describe here: "Why is AppEngine more expensive than EC2?" "Why is AppEgnine slower than EC2?", and so on.) Now, why does Netflix's video serving go down when an EC2 cell does? I have absolutely no idea. I do know that they do more than just stream bytes, though; they have to authenticate users, track what they're watching, and apply DRM to the streams. So there's that. If you want fault-tolerant video distribution, look no further than the Pirate Bay. Although their servers are frequently seized, the video bits keep flowing. That's by design, not by accident.

by moe 14y ago

pixl97 14y ago

You rarely hear of Google outages http://techcrunch.com/2012/12/10/gmail-experiences-a-widespread-outage-most-users-affected/ http://techcrunch.com/2012/12/10/gmail-experiences-a-widespr... ,or iTunes http://appleinsider.com/articles/12/11/19/itunes-match-down-as-apples-icloud-issues-continue http://appleinsider.com/articles/12/11/19/itunes-match-down-... ,or Youtube http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-down-not-googles-best-day/ http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-... Everyone goes down, but EC2 has been buggy as hell from what I can tell. And before everyone says Netflix should `just` move or add more bandwidth... http://www.nbcnews.com/technology/technolog/netflix-uses-32-7-percent-internet-bandwidth-119517 http://www.nbcnews.com/technology/technolog/netflix-uses-32-... They represent a huuuge amount of bandwidth usage!

toomuchtodo 14y ago
So work with eyeball networks and move your gear to the edge; dumping it all in AWS and hoping for the best is failing (at best). Do all the auth/billing/recommendations/transcoding in AWS, and push all the video content out to boxes at the ISPs.
mh- 14y ago
That bandwidth for delivering video is being streamed from a CDN outside AWS.

res0nat0r 14y ago

>Especially one as trivial as Netflix which is largely read-only. I stopped reading after this, since you obviously dont understand how complex their architecture really is.

sbov 14y ago
I'm never one to trivialize another company's problems so would hesitate to say Netflix's case is easy, but you cannot counter this statement by saying their architecture is complex. I can build a complex architecture for "Hello World" if I wanted to. Complex architecture != complex problem.
moe 14y ago
I work for a media streaming site (rank <30k on alexa) and I assure you the architecture doesn't change much when you scale it up to rank 99 (netflix). "Trivial" may sound harsh but in terms of large scale systems it really doesn't get much simpler than serving static files. Obviously their auxiliary services (billing, content ingestion, social etc.) are nowhere near trivial. But there's no reason for the core services that are required to get the catalog and video to most[1] devices to not be damn near 100% available. I'm not talking down netflix for having an outage anyway. Shit happens and afaik their overall track record is not bad at all. I only replied to jrockway's claim that multi-datacenter redundancy would require inconceivable engineering effort or amounts of money - neither is true. [1] Excluding those that need realtime transcoding, absurd DRM schemes or similar.
- brown9-2 14y ago
  How can you assure us of what the #99 system is like when you admit to only working on the ~30k system? That's not what "assure" means.
  - moe 14y ago
    We actually talk to other people in the industry sometimes, imagine that...

jrockway 14y ago

You rarely hear of Google outages, or iTunes, or Youtube, or [insert six dozen other popular brands], do you? Yes it can happen to the best of them, but the EC2 outages are really piling up lately. To me, this makes perfect sense. Google products are almost always designed to tolerate data center failures, and so you don't hear about data center failures affecting Google products. On the other hand, Joe's Random EC2 app is not designed for the same high availability, and so it's down whenever Amazon is. (This, incidentally, is a reasonable trade-off for most people. A few hours of downtime a year is often much cheaper than paying engineers to ensure that those few hours are five minutes or less.) So the flaw here is not Amazon. The flaw is relying on one cluster of computers to handle all your computing needs. Amazon could make their product high-replication by default, but it would be so slow and expensive that nobody would use it. So the task is kicked to the application developers instead of the platform developers, and you should consider holding them responsible for the downtime. (Incidentally, you can compare EC2 to AppEngine here. How often is your favorite AppEngine app down? Less than your favorite EC2 app, probably, because AppEngine pretty much forces the high-replication datastore, even though the semantics are strange for developers used to traditional single-homed or master/slave architectures. And if you read the AppEngine discussion groups, you'll see that users raise most of the concerns that I describe here: "Why is AppEngine more expensive than EC2?" "Why is AppEgnine slower than EC2?", and so on.) Now, why does Netflix's video serving go down when an EC2 cell does? I have absolutely no idea. I do know that they do more than just stream bytes, though; they have to authenticate users, track what they're watching, and apply DRM to the streams. So there's that. If you want fault-tolerant video distribution, look no further than the Pirate Bay. Although their servers are frequently seized, the video bits keep flowing. That's by design, not by accident.