9 ms·

How does a video codec work?

561 points by dreampeppers99 7y ago

setheron 7y ago
Playing videos is so complicated.
vkaku 7y ago
Good. Want more people to understand how compression works. Things like motion estimation, encoding the differences in frames, all that stuff is magic to the uninitiated. :) But this stuff is absolutely worth learning and every engineer should show interest :)
- RivieraKid 7y ago
  Why is it worth learning? I have a long list of stuff that I want to learn but codecs don't seem useful enough to be included there.
  - vkaku 7y ago
    Codecs teach you if the high fidelity of the data you store may not be as important as you'd think it is, and sometimes you'd be able to process all of that with a fraction of the compute power and cost involved.
unlinked_dll 7y ago
One thing that I wish more folks did with DSP/media tech was to start with concepts instead of diving into details first. Like, "how does a video codec work?" should start with: what is the problem? (reducing the bits per second required for a video stream, since it's big) and how? (don't send superfluous detail, since it's either redundant or imperceptible). Than dive into the details of how color is represented, how images are structured, how 2d signal transforms work, the principle behind the DCT as a method of representing the same data with better energy compaction by decorrelating different components, and how that can be used advantageously to reduce the number of bits for a still image, then talk about shared data between images, etc etc. I've noticed as a DSP guy that when I talk about things without concepts first that everyone's eyes glaze over. Although it is nice that everyone thinks it's black magic, good job security.
- irrational 7y ago
  I wish this about so many things. Instead of telling me that I should use some technology, tell me what problem it solves. How does the technology solve that problem? You should use React. Why? What problem does it solve? How does it solve that problem? What did people do before React? When is React not the appropriate solution, or an overkill solution?
- deburo 7y ago
  That was great. Do you have a blog series sonewhere?
- unethical_ban 7y ago
  Yeah, this article had an odd progression. With the premise "How does a codec work?", it goes (Make a distinction between codec and container) -> (Discuss several codecs and their brief history and royalty scheme) -> (Use heavy industry jargon to discuss details that only someone who already gets compression could possibly understand).
- ofrzeta 7y ago
  To be fair the article does that. The above URL just links to an anchor in the middle where it talks about codec implementation. For instance this quote from the article "We learned that it's not feasible to use video without any compression; a single one hour video at 720p resolution with 30fps would require 278GB*. Since using solely lossless data compression algorithms like DEFLATE (used in PKZIP, Gzip, and PNG), won't decrease the required bandwidth sufficiently we need to find other ways to compress the video. ... In order to do this, we can exploit how our vision works. We're better at distinguishing brightness than colors, the repetitions in time, a video contains a lot of images with few changes, and the repetitions within the image, each frame also contains many areas using the same or similar color." https://github.com/leandromoreira/digital_video_introduction#redundancy-removal https://github.com/leandromoreira/digital_video_introduction...
  - lightedman 7y ago
    278GB is a bit off. The real number is higher than that, but not by too much more, but they definitely messed up on their math somewhere. 1280 horizontal pixels times 720 vertical pixels times 8 bits per pixel color channel times 3 color channels (RGB) times 30 FPS times 60 SPM (sec per min) times 60 MPH (min per hour) divided by 8 bits to convert bits to bytes and finally divided by 1,024 bytes per kilobyte gives 291,600,000KB, exactly 291.6GB. And now days, that kind of uncompressed video is more than feasible, at least in countries with developed gigabit internet capability, as you'd need just 634mbit of bandwidth to handle an uncompressed 720p stream at 30 FPS. Local storage on even first-gen SATA drives can hit double+ that. Storage itself? Well, we've got the cloud, right? /s
killface 7y ago
magic. got it.
userbinator 7y ago
If you're interested in more "hands on" or practical video codec learning, I recommend writing a H.261 decoder. Only two supported frame sizes, no B-frames, and no intra prediction (effectively a subset of JPEG) make for a simple yet possibly quite rewarding weekend project that can be completed in around 700 lines of C (my attempt). Unfortunately not much existing media is available in 261, but I think watching a video being decoded entirely by code you wrote is a pretty fun experience, including all the weird and amusing distortions you can see when debugging; and from there you can move on to MPEG-1 with variable frame sizes and B frames (another weekend, assuming you reuse much of the 261 exercise --- I ended up with 1k lines total to decode MPEG-1), and that has somewhat more existing media you'll be able to watch. Then you can try H.262/MPEG-2 and enjoy the intricacies of handling interlacing as well as being able to decode DVDs and a lot of existing content; and then there's H.263 which has intra prediction... I haven't gotten past the first two largely for reasons of time and other things to play with, but IMHO getting a basic implementation of a video decoder is not that hard especially when you're working from a standard.
- sparklingriver 7y ago
  Very cool idea! Is your code for this online anywhere?
  - userbinator 7y ago
    Sorry, no. But a search of GitHub reveals some others have.
- nayuki 7y ago
  I'm sold on this idea! Are there readily available documents that specify the bitstream and decoding semantics for H.261?
  - userbinator 7y ago
    www.itu.int has H.261 and the rest of the H.* standards, they're a free download. This is H.261: https://www.itu.int/rec/T-REC-H.261/en https://www.itu.int/rec/T-REC-H.261/en Direct link to the spec itself is here: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.261-199303-I!!PDF-E&type=items https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.26...
deleted 7y ago
[deleted]
alexirae 7y ago
A similar tutorial like this but for audio?
- Tomte 7y ago
  https://xiph.org/video/ https://xiph.org/video/
greggman2 7y ago
As much as I like Apple it seems like they are in the back pocket of big media when it comes to this topic. AFAICT Apple only supports h.264 and h.265 meaning 1.3 billion iOS devices have browsers that can't access open standards for video. Yet one more reason why Apple should be required to allow alternate browser engines IMO. Some will claim it's a battery issue but Apple has the resources to add hardware support for other codecs and I'm only guessing could already handle it just fine with their current "Bionic" chips. I'm not sure what other executes can be dreamt up. I'm sure they'll follow in the comments below.
- savoytruffle 7y ago
  It sounds like you don't like Apple at all
  - Bnshsysjab 7y ago
    Irrespective of love or hate, support for open video standards would be better for everyone.
    - CharlesW 7y ago
      Apple does support (and contribute to) open video standards.
      - danbmil99 7y ago
        sure, if by support you mean "embrace, extend, extinguish". Today, Apple is every bit as monopolistic and anti-competitive as Microsoft was during Peak Ballmer.
        CharlesW 7y ago
        If that's the narrative you're focused on, you'll want to point to something else to support that. Apple not only uses standards-based compressed media formats exclusively, but the MPEG-4 file format is the QuickTime Movie file format (which Apple contributed to the standard).
hamilyon2 7y ago
I wish there was an article on how hardware decoding works. Detailed but approachable. I guess reading source code of drivers and popular open-source video players will give me an idea, but it is too much work. Does anyone have a technical article highlighting differences between hardware decoders and important details.
- rasz 7y ago
  Afaik nowadays hardware decoding is a magic black box you shove bytestream into affair. Its all hidden away in binary driver/firmware blobs.
- spbaar 7y ago
  Seconded. I tried to dig up info on this a few times but found zilch.
lambeteja 7y ago
lambeteja
DavidVoid 7y ago
I can also recommend this article on just how impressive H.264 at compressing a file while retaining good quality. https://sidbala.com/h-264-is-magic/ https://sidbala.com/h-264-is-magic/
sabujp 7y ago
this is good, then cringed when I saw the diagram about DRM that had M$, for msft. i guess the other companies aren't about making money at all costs?
doublerabbit 7y ago
This is all very cool, but what annoys me the most is that, let’s say you create video authoring software, it creates a piece of unique content and you decide to use a codec that’s licensed; you’ve got to pay royalties. If you use another it’s not supported by default. .FLAC for example or .oggv To not be liable to lawsuits and to support your own you need to come up with your own codec, resulting in your own platform and the circle starts once again. Why in this age can we just not all work together and make something that’s usable by everything for everyone without any restrictions.
peter_d_sherman 7y ago
Great article + amazingly extensive collection of links!

Playing videos is so complicated.

Good. Want more people to understand how compression works. Things like motion estimation, encoding the differences in frames, all that stuff is magic to the uninitiated. :) But this stuff is absolutely worth learning and every engineer should show interest :)

RivieraKid 7y ago
Why is it worth learning? I have a long list of stuff that I want to learn but codecs don't seem useful enough to be included there.
- vkaku 7y ago
  Codecs teach you if the high fidelity of the data you store may not be as important as you'd think it is, and sometimes you'd be able to process all of that with a fraction of the compute power and cost involved.

unlinked_dll 7y ago

One thing that I wish more folks did with DSP/media tech was to start with concepts instead of diving into details first. Like, "how does a video codec work?" should start with: what is the problem? (reducing the bits per second required for a video stream, since it's big) and how? (don't send superfluous detail, since it's either redundant or imperceptible). Than dive into the details of how color is represented, how images are structured, how 2d signal transforms work, the principle behind the DCT as a method of representing the same data with better energy compaction by decorrelating different components, and how that can be used advantageously to reduce the number of bits for a still image, then talk about shared data between images, etc etc. I've noticed as a DSP guy that when I talk about things without concepts first that everyone's eyes glaze over. Although it is nice that everyone thinks it's black magic, good job security.

irrational 7y ago
I wish this about so many things. Instead of telling me that I should use some technology, tell me what problem it solves. How does the technology solve that problem? You should use React. Why? What problem does it solve? How does it solve that problem? What did people do before React? When is React not the appropriate solution, or an overkill solution?
deburo 7y ago
That was great. Do you have a blog series sonewhere?
unethical_ban 7y ago
Yeah, this article had an odd progression. With the premise "How does a codec work?", it goes (Make a distinction between codec and container) -> (Discuss several codecs and their brief history and royalty scheme) -> (Use heavy industry jargon to discuss details that only someone who already gets compression could possibly understand).
ofrzeta 7y ago
To be fair the article does that. The above URL just links to an anchor in the middle where it talks about codec implementation. For instance this quote from the article "We learned that it's not feasible to use video without any compression; a single one hour video at 720p resolution with 30fps would require 278GB*. Since using solely lossless data compression algorithms like DEFLATE (used in PKZIP, Gzip, and PNG), won't decrease the required bandwidth sufficiently we need to find other ways to compress the video. ... In order to do this, we can exploit how our vision works. We're better at distinguishing brightness than colors, the repetitions in time, a video contains a lot of images with few changes, and the repetitions within the image, each frame also contains many areas using the same or similar color." https://github.com/leandromoreira/digital_video_introduction#redundancy-removal https://github.com/leandromoreira/digital_video_introduction...
- lightedman 7y ago
  278GB is a bit off. The real number is higher than that, but not by too much more, but they definitely messed up on their math somewhere. 1280 horizontal pixels times 720 vertical pixels times 8 bits per pixel color channel times 3 color channels (RGB) times 30 FPS times 60 SPM (sec per min) times 60 MPH (min per hour) divided by 8 bits to convert bits to bytes and finally divided by 1,024 bytes per kilobyte gives 291,600,000KB, exactly 291.6GB. And now days, that kind of uncompressed video is more than feasible, at least in countries with developed gigabit internet capability, as you'd need just 634mbit of bandwidth to handle an uncompressed 720p stream at 30 FPS. Local storage on even first-gen SATA drives can hit double+ that. Storage itself? Well, we've got the cloud, right? /s

killface 7y ago

magic. got it.

userbinator 7y ago

If you're interested in more "hands on" or practical video codec learning, I recommend writing a H.261 decoder. Only two supported frame sizes, no B-frames, and no intra prediction (effectively a subset of JPEG) make for a simple yet possibly quite rewarding weekend project that can be completed in around 700 lines of C (my attempt). Unfortunately not much existing media is available in 261, but I think watching a video being decoded entirely by code you wrote is a pretty fun experience, including all the weird and amusing distortions you can see when debugging; and from there you can move on to MPEG-1 with variable frame sizes and B frames (another weekend, assuming you reuse much of the 261 exercise --- I ended up with 1k lines total to decode MPEG-1), and that has somewhat more existing media you'll be able to watch. Then you can try H.262/MPEG-2 and enjoy the intricacies of handling interlacing as well as being able to decode DVDs and a lot of existing content; and then there's H.263 which has intra prediction... I haven't gotten past the first two largely for reasons of time and other things to play with, but IMHO getting a basic implementation of a video decoder is not that hard especially when you're working from a standard.

sparklingriver 7y ago
Very cool idea! Is your code for this online anywhere?
- userbinator 7y ago
  Sorry, no. But a search of GitHub reveals some others have.
nayuki 7y ago
I'm sold on this idea! Are there readily available documents that specify the bitstream and decoding semantics for H.261?
- userbinator 7y ago
  www.itu.int has H.261 and the rest of the H.* standards, they're a free download. This is H.261: https://www.itu.int/rec/T-REC-H.261/en https://www.itu.int/rec/T-REC-H.261/en Direct link to the spec itself is here: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.261-199303-I!!PDF-E&type=items https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.26...

deleted 7y ago

[deleted]

alexirae 7y ago

A similar tutorial like this but for audio?

Tomte 7y ago
https://xiph.org/video/ https://xiph.org/video/

greggman2 7y ago

As much as I like Apple it seems like they are in the back pocket of big media when it comes to this topic. AFAICT Apple only supports h.264 and h.265 meaning 1.3 billion iOS devices have browsers that can't access open standards for video. Yet one more reason why Apple should be required to allow alternate browser engines IMO. Some will claim it's a battery issue but Apple has the resources to add hardware support for other codecs and I'm only guessing could already handle it just fine with their current "Bionic" chips. I'm not sure what other executes can be dreamt up. I'm sure they'll follow in the comments below.

savoytruffle 7y ago
It sounds like you don't like Apple at all
- Bnshsysjab 7y ago
  Irrespective of love or hate, support for open video standards would be better for everyone.
  - CharlesW 7y ago
    Apple does support (and contribute to) open video standards.
    - danbmil99 7y ago
      sure, if by support you mean "embrace, extend, extinguish". Today, Apple is every bit as monopolistic and anti-competitive as Microsoft was during Peak Ballmer.
      - CharlesW 7y ago
        If that's the narrative you're focused on, you'll want to point to something else to support that. Apple not only uses standards-based compressed media formats exclusively, but the MPEG-4 file format is the QuickTime Movie file format (which Apple contributed to the standard).

hamilyon2 7y ago

I wish there was an article on how hardware decoding works. Detailed but approachable. I guess reading source code of drivers and popular open-source video players will give me an idea, but it is too much work. Does anyone have a technical article highlighting differences between hardware decoders and important details.

rasz 7y ago
Afaik nowadays hardware decoding is a magic black box you shove bytestream into affair. Its all hidden away in binary driver/firmware blobs.
spbaar 7y ago
Seconded. I tried to dig up info on this a few times but found zilch.

lambeteja 7y ago

lambeteja

DavidVoid 7y ago

I can also recommend this article on just how impressive H.264 at compressing a file while retaining good quality. https://sidbala.com/h-264-is-magic/ https://sidbala.com/h-264-is-magic/

sabujp 7y ago

this is good, then cringed when I saw the diagram about DRM that had M$, for msft. i guess the other companies aren't about making money at all costs?

doublerabbit 7y ago

This is all very cool, but what annoys me the most is that, let’s say you create video authoring software, it creates a piece of unique content and you decide to use a codec that’s licensed; you’ve got to pay royalties. If you use another it’s not supported by default. .FLAC for example or .oggv To not be liable to lawsuits and to support your own you need to come up with your own codec, resulting in your own platform and the circle starts once again. Why in this age can we just not all work together and make something that’s usable by everything for everyone without any restrictions.

peter_d_sherman 7y ago

Great article + amazingly extensive collection of links!