4 ms·

Try doing faster than realtime transcoding of 2000 simultaneous streams. Not that trivial.

by kvogt 17y ago

DarkShikari 17y ago
Let's do some math then. Let's assume the videos are 320x240, for example. Since we have so many streams at once, we can assume perfect scaling across multiple cores and systems. A 320x240 video encodes at about 195 FPS on my Core 2 Duo Conroe 2Ghz with one encoding thread, using x264 and preset "veryfast". Now, if we go up to a Quad at 3Ghz and use all four cores, we're up to 1170fps. A Penryn is about 10% faster per-clock than a Conroe on x264. We're up to 1287fps. A Core i7 is about 40% faster per-clock than a Conroe on x264, including the effects of Hyperthreading. We're up to 1802 FPS, or 60 streams, and we haven't even left our single processor. For all 2000 streams, we'll need 33 CPUs, or maybe a few more if we allow for a bit of overhead, just in case. Not that hard now, is it? Bonus points: you save bandwidth, since H.264 compresses a lot better than Sorenson H.263 (FLV1). (Also, preset "ultrafast" is another factor of two faster than that, though the quality is low enough that nobody should be using it...)
- jacquesm 17y ago
  In practice you are not going to be doing nearly as good as that. your co-workers hack notwithstanding. Assuming 'perfect scaling' and discounting such things as networking overhead are going to upset that picture by a considerable amount. You'll be lucky to get to 500 Mbit out per physical machine using 16 cores and streaming to about 2000 users with 75 incoming streams. So that's roughly 5 streams per core. To just multiplex 2000 streams to 40 K viewers with adaptive frame rate you'll need roughly 30 sixteen core machines, note that that does NOT include transcoding yet.
  - DarkShikari 17y ago
    You'll be lucky to get to 500 Mbit out per physical machine using 16 cores and streaming to about 2000 users with 75 incoming streams. So that's roughly 5 streams per core. Why do you have the same computers handling encoding as distribution? That's stupid. To just multiplex 2000 streams to 40 K viewers with adaptive frame rate you'll need roughly 30 sixteen core machines, note that that does NOT include transcoding yet. Then your multiplexer is unimaginably inefficient. Last summer I worked for a company that multiplexes live streams for millions of viewers; it is not as hard as you think, especially since you only have to do the work once per channel, not once per user, since all users watching a given channel receive the same stream.
    - jacquesm 17y ago
      I think you are missing what adaptive frame rate is all about...
      - DarkShikari 17y ago
        You have yet to explain what "adaptive frame rate" actually is in this context. Do you mean that each user receives a different framerate based on the download bandwidth that he has? You really only need a couple framerates to cover all users--and I suspect you mean "bitrate", not "framerate", in this case (though changing the framerate is one effective way to improve the quality of an adaptive bitrate scheme).
        jacquesm 17y ago
        Adaptive means that based on the connection quality every user receives the maximum quality that their link is capable of while guaranteeing audio delivery with a minimum lag to stay as close to realtime as possible. This means that every user has an individual stream because the connection quality can change very rapidly over short periods of time, they might switch a secondary stream on (or off) for a moment and whatever else can change the quality of the connection. You can't use very large windows because that would mean that if there was a hickup in the line it would take a long time to recover, which would degrade perceived quality. Video is relatively forgiving by the way, the audio is the real kicker, a single packet delivered a little bit too late will instantly cause annoyance whereas with the video you can be a little early or late and it doesn't seem to matter too much. Initial encoding happens on different machines (obviously) but a stream multiplexer that maximizes the end user experience is a lot more complicated than just encoding to a preset selection of rates, after all every keyframe and subsequent updates have their encoding scheme determined by the throughput measured during the transmission of the keyframe(s)+updates just preceding it. For non-live consumption the situation is a lot easier, you can pre-buffer and that takes care of most problems but in live transmissions that is not possible unless you want horrible lag (which does not work when the connection is two way, it screws up the interaction between people).
jacquesm 17y ago
We're doing about 1500 right now, and no, it's not trivial. 16 core machines make life a bit easier though, and the hosting provider has been good to their word of providing free machines as long as we use this much bandwidth :) Output on a typical box is about 300-400 Mbit/sec.

Try doing faster than realtime transcoding of 2000 simultaneous streams. Not that trivial.

by kvogt 17y ago

Try doing faster than realtime transcoding of 2000 simultaneous streams. Not that trivial.

Let's do some math then. Let's assume the videos are 320x240, for example. Since we have so many streams at once, we can assume perfect scaling across multiple cores and systems. A 320x240 video encodes at about 195 FPS on my Core 2 Duo Conroe 2Ghz with one encoding thread, using x264 and preset "veryfast". Now, if we go up to a Quad at 3Ghz and use all four cores, we're up to 1170fps. A Penryn is about 10% faster per-clock than a Conroe on x264. We're up to 1287fps. A Core i7 is about 40% faster per-clock than a Conroe on x264, including the effects of Hyperthreading. We're up to 1802 FPS, or 60 streams, and we haven't even left our single processor. For all 2000 streams, we'll need 33 CPUs, or maybe a few more if we allow for a bit of overhead, just in case. Not that hard now, is it? Bonus points: you save bandwidth, since H.264 compresses a lot better than Sorenson H.263 (FLV1). (Also, preset "ultrafast" is another factor of two faster than that, though the quality is low enough that nobody should be using it...)

jacquesm 17y ago
In practice you are not going to be doing nearly as good as that. your co-workers hack notwithstanding. Assuming 'perfect scaling' and discounting such things as networking overhead are going to upset that picture by a considerable amount. You'll be lucky to get to 500 Mbit out per physical machine using 16 cores and streaming to about 2000 users with 75 incoming streams. So that's roughly 5 streams per core. To just multiplex 2000 streams to 40 K viewers with adaptive frame rate you'll need roughly 30 sixteen core machines, note that that does NOT include transcoding yet.
- DarkShikari 17y ago
  You'll be lucky to get to 500 Mbit out per physical machine using 16 cores and streaming to about 2000 users with 75 incoming streams. So that's roughly 5 streams per core. Why do you have the same computers handling encoding as distribution? That's stupid. To just multiplex 2000 streams to 40 K viewers with adaptive frame rate you'll need roughly 30 sixteen core machines, note that that does NOT include transcoding yet. Then your multiplexer is unimaginably inefficient. Last summer I worked for a company that multiplexes live streams for millions of viewers; it is not as hard as you think, especially since you only have to do the work once per channel, not once per user, since all users watching a given channel receive the same stream.
  - jacquesm 17y ago
    I think you are missing what adaptive frame rate is all about...
    - DarkShikari 17y ago
      You have yet to explain what "adaptive frame rate" actually is in this context. Do you mean that each user receives a different framerate based on the download bandwidth that he has? You really only need a couple framerates to cover all users--and I suspect you mean "bitrate", not "framerate", in this case (though changing the framerate is one effective way to improve the quality of an adaptive bitrate scheme).
      - jacquesm 17y ago
        Adaptive means that based on the connection quality every user receives the maximum quality that their link is capable of while guaranteeing audio delivery with a minimum lag to stay as close to realtime as possible. This means that every user has an individual stream because the connection quality can change very rapidly over short periods of time, they might switch a secondary stream on (or off) for a moment and whatever else can change the quality of the connection. You can't use very large windows because that would mean that if there was a hickup in the line it would take a long time to recover, which would degrade perceived quality. Video is relatively forgiving by the way, the audio is the real kicker, a single packet delivered a little bit too late will instantly cause annoyance whereas with the video you can be a little early or late and it doesn't seem to matter too much. Initial encoding happens on different machines (obviously) but a stream multiplexer that maximizes the end user experience is a lot more complicated than just encoding to a preset selection of rates, after all every keyframe and subsequent updates have their encoding scheme determined by the throughput measured during the transmission of the keyframe(s)+updates just preceding it. For non-live consumption the situation is a lot easier, you can pre-buffer and that takes care of most problems but in live transmissions that is not possible unless you want horrible lag (which does not work when the connection is two way, it screws up the interaction between people).

jacquesm 17y ago

We're doing about 1500 right now, and no, it's not trivial. 16 core machines make life a bit easier though, and the hosting provider has been good to their word of providing free machines as long as we use this much bandwidth :) Output on a typical box is about 300-400 Mbit/sec.