3 ms·

This is done particularly when you are implementing adaptive bitrates (the thing that Netflix uses where it automatically sends you a higher or lower quality pi

This is done particularly when you are implementing adaptive bitrates (the thing that Netflix uses where it automatically sends you a higher or lower quality picture depending on your Internet connection). In adaptive bitrate world, you split a video up into fragments, say 2-10 seconds large, and encode each segment in multiple bitrates, so that every say 5 seconds the video player can make a decision to download a different quality for the next 5 seconds. Ok, but why not split the file up for standard encoding? Well, you can't just concatenate two .mp4 together without re-encoding and have it make sense to most media players (as far as I am aware), and moreover, it's inefficient from a RAM perspective when doing that. 1 second of RAW uncompressed 4k (24 fps) video is about 600MB. Source content for a single episode/movie at Netflix (I don't work there, just something I read once) can reach into the terabytes easily.

dylan604 3y ago
okay, i don't agree with anything in your reply. segmenting a video file for HLS/DASH delivery is not at all the same thing I'm suggesting. Just for the sake of round numbers, i'm saying to take a 90 minute feature into nine 10-minute segments. fire up 10 separate instances to encode each 10-minute segment. you've just increased the encode time 10x. also, DASH/HLS does not require segmented files. you can have a single contiguous file like an fmp4 as a DASH/HLS source. >Ok, but why not split the file up for standard encoding?<snip> at this point, you would be better served by just writing an elementary stream rather than a muxed mp4 file since it's just a segment anyways so why waste the cycles on muxing? you then absolutely 100% can concat those streams (even if you did mux them into a container). if you think you can't, you clearly have not tried very hard. >I don't work there, just something I read once I don't work there either, but do have 30+ years of experience with this subject. Sadly, you're not as well informed as you might think. People don't tend to encode to AV1 from RAW. They instead are dealing with a deliverable file most typically a ProRes in today's world after the post process has been completed. No where near terabytes for a feature. More closely to a couple hundred gigabytes for UHD HDR content. You seem to be unnecessarily exaggerating. Edit: it's a 10x increase in encode speed, not time. that would be opposite effect.
- bick_nyers 3y ago
  Why did the encode time increase by 10x in that instance? Can't you just seek in the video to the I-frame before the cut point and start your encode there? I've never tried merging streams across computers so was naively just thinking that your output from each computer would be an MP4 but that makes sense. I pulled that info. from a Netflix talk, perhaps video cameras back from when that talk occured didn't compress the video for you? Besides, isn't IMAX all intra-encoded? It was my understanding that IMAX films are actually just a series of J2K images, so I would imagine that the video cameras used there would also be intra-encoded.
  - dylan604 3y ago
    s/increase/decrease/ i was thinking increased the encode speed 9x, but typed increased encode time. i also swapped the number of segments by segment duration. 9 segments of 10mins = 9x increase in performance. Sounds like you are confusing Netflix' recommended formats for acquisition vs delivery. Cameras capture RAW formats (rarely is it uncompressed though), and the post houses use that as sources. The post house/color correction will the create the delivery formats which is typically ProRes. RAW is not a friendly format for distribution in the slightest. The full workflow from camera original to what ends up being streamed to the end viewer changes formats multiple times through the process.
    - bick_nyers 3y ago
      Gotcha, that makes much more sense
matsemann 3y ago
For certain containers / codecs you can concatenate files without re-encoding. Do it quite often with ffmpef using -c:copy and it's basically at the speed of the disk.
- bick_nyers 3y ago
  The parent comment I was responding to was discussing splitting encodes across multiple computers then re-combining which is what I was referring to. Still sounds like it is possible and I was incorrect.
scottlamb 3y ago
> Ok, but why not split the file up for standard encoding? Well, you can't just concatenate two .mp4 together without re-encoding and have it make sense to most media players (as far as I am aware) You can't just literally `cat foo-[123].mp4 > foo.mp4` with old-school non-fragmented .mp4 files, but you just have to shuffle the container stuff around a bit. You don't need to re-encode. One downside is if you decide ahead of time that you're going to divide the video into fixed 5-second fragments/segments/chunks to encode independently, you're going to end up with that-length closed GOPs that don't match scene transitions or the like. IDR frame every 5 seconds. So no B/P frames that reference stuff 10 seconds ago, no periodic incremental refresh, nothing fancy.

dylan604 3y ago

okay, i don't agree with anything in your reply. segmenting a video file for HLS/DASH delivery is not at all the same thing I'm suggesting. Just for the sake of round numbers, i'm saying to take a 90 minute feature into nine 10-minute segments. fire up 10 separate instances to encode each 10-minute segment. you've just increased the encode time 10x. also, DASH/HLS does not require segmented files. you can have a single contiguous file like an fmp4 as a DASH/HLS source. >Ok, but why not split the file up for standard encoding?<snip> at this point, you would be better served by just writing an elementary stream rather than a muxed mp4 file since it's just a segment anyways so why waste the cycles on muxing? you then absolutely 100% can concat those streams (even if you did mux them into a container). if you think you can't, you clearly have not tried very hard. >I don't work there, just something I read once I don't work there either, but do have 30+ years of experience with this subject. Sadly, you're not as well informed as you might think. People don't tend to encode to AV1 from RAW. They instead are dealing with a deliverable file most typically a ProRes in today's world after the post process has been completed. No where near terabytes for a feature. More closely to a couple hundred gigabytes for UHD HDR content. You seem to be unnecessarily exaggerating. Edit: it's a 10x increase in encode speed, not time. that would be opposite effect.

bick_nyers 3y ago
Why did the encode time increase by 10x in that instance? Can't you just seek in the video to the I-frame before the cut point and start your encode there? I've never tried merging streams across computers so was naively just thinking that your output from each computer would be an MP4 but that makes sense. I pulled that info. from a Netflix talk, perhaps video cameras back from when that talk occured didn't compress the video for you? Besides, isn't IMAX all intra-encoded? It was my understanding that IMAX films are actually just a series of J2K images, so I would imagine that the video cameras used there would also be intra-encoded.
- dylan604 3y ago
  s/increase/decrease/ i was thinking increased the encode speed 9x, but typed increased encode time. i also swapped the number of segments by segment duration. 9 segments of 10mins = 9x increase in performance. Sounds like you are confusing Netflix' recommended formats for acquisition vs delivery. Cameras capture RAW formats (rarely is it uncompressed though), and the post houses use that as sources. The post house/color correction will the create the delivery formats which is typically ProRes. RAW is not a friendly format for distribution in the slightest. The full workflow from camera original to what ends up being streamed to the end viewer changes formats multiple times through the process.
  - bick_nyers 3y ago
    Gotcha, that makes much more sense

matsemann 3y ago

For certain containers / codecs you can concatenate files without re-encoding. Do it quite often with ffmpef using -c:copy and it's basically at the speed of the disk.

bick_nyers 3y ago
The parent comment I was responding to was discussing splitting encodes across multiple computers then re-combining which is what I was referring to. Still sounds like it is possible and I was incorrect.

scottlamb 3y ago

> Ok, but why not split the file up for standard encoding? Well, you can't just concatenate two .mp4 together without re-encoding and have it make sense to most media players (as far as I am aware) You can't just literally `cat foo-[123].mp4 > foo.mp4` with old-school non-fragmented .mp4 files, but you just have to shuffle the container stuff around a bit. You don't need to re-encode. One downside is if you decide ahead of time that you're going to divide the video into fixed 5-second fragments/segments/chunks to encode independently, you're going to end up with that-length closed GOPs that don't match scene transitions or the like. IDR frame every 5 seconds. So no B/P frames that reference stuff 10 seconds ago, no periodic incremental refresh, nothing fancy.