4 ms·

I had forgotten about the film-grain extraction, which is a clever approach to a huge problem for compression. But... did I miss it, or was there no mention of

I had forgotten about the film-grain extraction, which is a clever approach to a huge problem for compression. But... did I miss it, or was there no mention of any tool to specify grain parameters up front? If you're shooting "clean" digital footage and you decide in post that you want to add grain, how do you convey the grain parameters to the encoder? It would degrade your work and defeat some of the purpose of this clever scheme if you had to add fake grain to your original footage, feed the grainy footage to the encoder to have it analyzed for its characteristics and stripped out (inevitably degrading real image details at least a bit), and then have the grain re-added on delivery. So you need a way to specify grain characteristics to the encoder directly, so clean footage can be delivered without degradation and grain applied to it upon rendering at the client.

crazygringo 10mo ago
You just add it to your original footage, and accept whatever quality degradation that grain inherently provides. Any movie or TV show is ultimately going to be streamed in lots of different formats. And when grain is added, it's often on a per-shot basis, not uniformly. E.g. flashback scenes will have more grain. Or darker scenes will have more grain added to emulate film. Trying to tie it to the particular codec would be a crazy headache. For a solo project it could be doable but I can't ever imagine a streamer building a source material pipeline that would handle that.
- VerifiedReports 10mo ago
  Mmmm, no, because if the delivery conduit uses AV1, you can optimize for it and realize better quality by avoiding the whole degrading round of grain analysis and stripping. "I can't ever imagine a streamer building a source material pipeline that would handle that." That's exactly what the article describes, though. It's already built, and Netflix is championing this delivery mechanism. Netflix is also famous for dictating technical requirements for source material. Why would they not want the director to be able to provide a delivery-ready master that skips the whole grain-analysis/grain-removal step and provides the best possible image quality? Presumably the grain extraction/re-adding mechanism described here handles variable grain throughout the program. I don't know why you'd assume that it doesn't. If it didn't, you'd wind up with a single grain level for the entire movie; an entirely unacceptable result for the very reason you mention. This scheme loses a major opportunity for new productions unless the director can provide a clean master and an accompanying "grain track." Call it a GDL: grain decision list. This would also be future-proof; if a new codec is devised that also supports this grain layer, the parameters could be translated from the previous master into the new codec. I wish Netflix could go back and remove the hideous soft-focus filtration from The West Wing, but nope; that's baked into the footage forever.
  - irae 10mo ago
    I believe you are speculating on digital mastering and not codec conversion. From the creator's PoV their intention and quality is defined in post-production and mastering, color grading and other stuff I am not expert on. But I know a bit more from music mastering and you might be thinking of a workflow similar to Apple, which allows creators to master for their codec with "Mastered for iTuenes" flow, where the creators opt-in to an extra step to increase quality of the encoding and can hear in their studio the final quality after Apple encodes and DRMs the content on their servers. In video I would assume that is much more complicated, since there are many quality the video is encoded to allow for slower connections and buffering without interruptions. So I assume the best strategy is the one you mentioned yourself, where AV1 obviously detects on a per scene or keyframe interval the grain level/type/characteristics and encode as to be accurate to the source material at this scene. In other words: The artist/director preference for grain is already per scene and expressed in the high bitrate/low-compression format they provide to Netflix and competitors. I find it unlikely that any encoder flags would specifically benefit the encoding workflow in the way you suggested it might.
    - VerifiedReports 10mo ago
      "I believe you are speculating on digital mastering and not codec conversion." That's good, since that's what I said. "The artist/director preference for grain is already per scene and expressed in the high bitrate/low-compression format they provide to Netflix and competitors. I find it unlikely that any encoder flags would specifically benefit the encoding workflow in the way you suggested it might." I'm not sure you absorbed the process described in the article. Netflix is analyzing the "preference for grain" as expressed by the grain detected in the footage, and then they're preparing a "grain track," as a stream of metadata that controls a grain "generator" upon delivery to the viewer. So I don't know why you think this pipeline wouldn't benefit from having the creator provide perfectly accurate grain metadata to the delivery network along with already-clean footage up front; this would eliminate the steps of analyzing the footage and (potentially lossily) removing fake grain... only to re-add an approximation of it later. All I'm proposing is a mastering tool that lets the DIRECTOR (not an automated process) do the "grain analysis" deliberately and provide the result to the distributor.
bob1029 10mo ago
Actual film grain (i.e., photochemical) is arguably a valid source of information. You can frame it as noise, but does provide additional information content that our visual system can work with. Removing real film grain from content and then recreating it parametrically on the other side is not the same thing as directly encoding it. You are killing a lot of information. It is really hard to quantify exactly how we perceive this sort of information so it's easy to evade the consequences of screwing with it. Selling the Netflix board on an extra X megabits/s per streamer to keep genuine film grain that only 1% of the customers will notice is a non-starter.
- VerifiedReports 10mo ago
  Exactly. In the case of stuff shot on film, there's little to be done except increase bitrate if you want maximal fidelity. In the case of fake grain that's added to modern footage, I'm calling out the absurdity of adding it, analyzing it, removing it, and putting yet another approximation of it back in.
  - breve 10mo ago
    > I'm calling out the absurdity of adding it, analyzing it, removing it, and putting yet another approximation of it back in Why is it absurd? The entire encoding process is an approximation of the original image. Lossy compression inevitably throws away information to make the file size smaller. And the creation of the original video is entirely separated from the distribution of it. It'll be stored losslessly for one thing. The only question that matters is does the image look good enough after encoding. If it doesn't look good enough then no one will watch it. If it does look good enough then you've got a viable video distribution service that minimizes its bandwidth costs.
    - VerifiedReports 10mo ago
      "Lossy compression inevitably throws away information to make the file size smaller." So? That fact only emphasizes the absurdity and loss potential here: 1. Acquire "clean" digital footage. 2. Add fake grain to said footage 3. Compress the grainy footage with lossy compression, wasting a bunch of the data on fake detail that you just added. 4. Analyze the footage to determine the character of the fake grain, calculate parameters to approximate it later with other fake grain 5. Strip out the fake grain, with potential for loss of some original image details 6. Re-add fake grain with the calculated parameters If you don't see the absurdity there, I don't know what to tell you.

crazygringo 10mo ago

You just add it to your original footage, and accept whatever quality degradation that grain inherently provides. Any movie or TV show is ultimately going to be streamed in lots of different formats. And when grain is added, it's often on a per-shot basis, not uniformly. E.g. flashback scenes will have more grain. Or darker scenes will have more grain added to emulate film. Trying to tie it to the particular codec would be a crazy headache. For a solo project it could be doable but I can't ever imagine a streamer building a source material pipeline that would handle that.

VerifiedReports 10mo ago
Mmmm, no, because if the delivery conduit uses AV1, you can optimize for it and realize better quality by avoiding the whole degrading round of grain analysis and stripping. "I can't ever imagine a streamer building a source material pipeline that would handle that." That's exactly what the article describes, though. It's already built, and Netflix is championing this delivery mechanism. Netflix is also famous for dictating technical requirements for source material. Why would they not want the director to be able to provide a delivery-ready master that skips the whole grain-analysis/grain-removal step and provides the best possible image quality? Presumably the grain extraction/re-adding mechanism described here handles variable grain throughout the program. I don't know why you'd assume that it doesn't. If it didn't, you'd wind up with a single grain level for the entire movie; an entirely unacceptable result for the very reason you mention. This scheme loses a major opportunity for new productions unless the director can provide a clean master and an accompanying "grain track." Call it a GDL: grain decision list. This would also be future-proof; if a new codec is devised that also supports this grain layer, the parameters could be translated from the previous master into the new codec. I wish Netflix could go back and remove the hideous soft-focus filtration from The West Wing, but nope; that's baked into the footage forever.
- irae 10mo ago
  I believe you are speculating on digital mastering and not codec conversion. From the creator's PoV their intention and quality is defined in post-production and mastering, color grading and other stuff I am not expert on. But I know a bit more from music mastering and you might be thinking of a workflow similar to Apple, which allows creators to master for their codec with "Mastered for iTuenes" flow, where the creators opt-in to an extra step to increase quality of the encoding and can hear in their studio the final quality after Apple encodes and DRMs the content on their servers. In video I would assume that is much more complicated, since there are many quality the video is encoded to allow for slower connections and buffering without interruptions. So I assume the best strategy is the one you mentioned yourself, where AV1 obviously detects on a per scene or keyframe interval the grain level/type/characteristics and encode as to be accurate to the source material at this scene. In other words: The artist/director preference for grain is already per scene and expressed in the high bitrate/low-compression format they provide to Netflix and competitors. I find it unlikely that any encoder flags would specifically benefit the encoding workflow in the way you suggested it might.
  - VerifiedReports 10mo ago
    "I believe you are speculating on digital mastering and not codec conversion." That's good, since that's what I said. "The artist/director preference for grain is already per scene and expressed in the high bitrate/low-compression format they provide to Netflix and competitors. I find it unlikely that any encoder flags would specifically benefit the encoding workflow in the way you suggested it might." I'm not sure you absorbed the process described in the article. Netflix is analyzing the "preference for grain" as expressed by the grain detected in the footage, and then they're preparing a "grain track," as a stream of metadata that controls a grain "generator" upon delivery to the viewer. So I don't know why you think this pipeline wouldn't benefit from having the creator provide perfectly accurate grain metadata to the delivery network along with already-clean footage up front; this would eliminate the steps of analyzing the footage and (potentially lossily) removing fake grain... only to re-add an approximation of it later. All I'm proposing is a mastering tool that lets the DIRECTOR (not an automated process) do the "grain analysis" deliberately and provide the result to the distributor.

bob1029 10mo ago

Actual film grain (i.e., photochemical) is arguably a valid source of information. You can frame it as noise, but does provide additional information content that our visual system can work with. Removing real film grain from content and then recreating it parametrically on the other side is not the same thing as directly encoding it. You are killing a lot of information. It is really hard to quantify exactly how we perceive this sort of information so it's easy to evade the consequences of screwing with it. Selling the Netflix board on an extra X megabits/s per streamer to keep genuine film grain that only 1% of the customers will notice is a non-starter.

VerifiedReports 10mo ago
Exactly. In the case of stuff shot on film, there's little to be done except increase bitrate if you want maximal fidelity. In the case of fake grain that's added to modern footage, I'm calling out the absurdity of adding it, analyzing it, removing it, and putting yet another approximation of it back in.
- breve 10mo ago
  > I'm calling out the absurdity of adding it, analyzing it, removing it, and putting yet another approximation of it back in Why is it absurd? The entire encoding process is an approximation of the original image. Lossy compression inevitably throws away information to make the file size smaller. And the creation of the original video is entirely separated from the distribution of it. It'll be stored losslessly for one thing. The only question that matters is does the image look good enough after encoding. If it doesn't look good enough then no one will watch it. If it does look good enough then you've got a viable video distribution service that minimizes its bandwidth costs.
  - VerifiedReports 10mo ago
    "Lossy compression inevitably throws away information to make the file size smaller." So? That fact only emphasizes the absurdity and loss potential here: 1. Acquire "clean" digital footage. 2. Add fake grain to said footage 3. Compress the grainy footage with lossy compression, wasting a bunch of the data on fake detail that you just added. 4. Analyze the footage to determine the character of the fake grain, calculate parameters to approximate it later with other fake grain 5. Strip out the fake grain, with potential for loss of some original image details 6. Re-add fake grain with the calculated parameters If you don't see the absurdity there, I don't know what to tell you.