18 ms·

H.264 is magic (2016)

680 points by dpeck 7y ago

thomascgalvin 7y ago
I've seen this article before, and this part always stuck with me: > I captured the screen of this home page and produced two files: > PNG screenshot of the Apple homepage 1015KB > 5 Second 60fps H.264 video of the same Apple homepage 175KB > Eh. What? Those file sizes look switched. > No, they're right. The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. The article does go into how and why this is possible (tl;dr H.264 is lossey, png is not), but for the difference in human-detectable quality, H.264 is astonishingly more efficient.
- ChrisSD 7y ago
  It's a good example but the difference doesn't quite have to be as dramatic as the article makes out. Screenshot tools rarely produce optimised PNGs. A quick test of the image in the article shows that optipng (lossless) can bring the size down to ~570kb or pngquant (lossy) to ~240kb. Zopfil could further compress the images produced by optipng or pngquant, but it's a case of diminishing returns at that point. To be clear the article makes a good comparison but I think it perhaps overstates the case to make a point.
viraptor 7y ago
I like this high-level summary. One thing I'd change though is that the photo loses details in the frequency domain only. In real view it actually gains unnecessary details/noise - it's not that the grill details disappeared, it's the whole image that gained an anti-grill layer. (You can now see it on the smooth part of the laptop)
deleted 7y ago
[deleted]
- andyfleming 7y ago
  Not sure why you were downvoted. It is literally in the article.
  - Dylan16807 7y ago
    When someone is explaining what part of the article left the biggest impression on them and why, it's weird and unhelpful to go "that's literally in the article!"
    - andyfleming 7y ago
      Oh, maybe they edited their comment (or maybe I misread it). Originally, I believe it said that it _didn't_ explain why at all.
      - viraptor 7y ago
        Same here. I'd be happy to delete my comment in either case, but it's too late now.
bscphil 7y ago
Interesting introduction. Too bad it doesn't really get into some of the more complicated things an advanced H.264 encoder like x264 is doing, e.g. adaptive quantization methods. There's also a mistake or two, for example > P-frames are frames that will encode a motion vector for each of the macro-blocks from the previous frame. Actually, P-frames can use data from multiple previous frames, not just the last one. I think it's worth pointing out, as well, that even though we've technically surpassed H.264 with newer codecs like H.265, VP9, and AV1, all of which are roughly 20%-50% more efficient, this has required tremendous increases in encoding complexity. H.264 is special - it seems to occupy a kind of inflection point on the complexity / efficiency curve. It's far more efficient than previous codecs like Xvid, WMV, and so on, but at the same time even a lot of underpowered devices from over a decade ago can easily play it. We're not likely to see tradeoffs that good again in the video codec space.
- clouddrover 7y ago
  When you control for image quality VP9 outperforms H.264 in single-threaded decoding: https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecoding-performance-vs-hevch-264/ https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecod... You get the same image quality at a lower bitrate and faster decoding. VP9 encoding is much slower, but Intel's SVT-VP9 encoder is getting over 300 frames per second on the right hardware: https://www.phoronix.com/scan.php?page=news_item&px=SVT-VP9-Open-Source https://www.phoronix.com/scan.php?page=news_item&px=SVT-VP9-...
  - bscphil 7y ago
    That's an interesting claim. I have to say it doesn't match my experience at all, e.g. when playing VP9 videos on Youtube. Several thoughts: * I'm not sure what the "ffh264" decoder they're talking about is. My understanding is that pretty much everyone is using libx264 for both encoding and decoding. Maybe the latter is more performant? * Using SSIM to pick "equal quality" files will probably not give very accurate results. * SVT-VP9 is, if I'm not mistaken, a hardware encoder. One of the issues with these is that they require some significant tradeoffs to get better speed, so they'll have worse quality than a software encoder at the same bitrate. If you go with software you really pay the price when it comes to encoding time.
dpeck 7y ago
Previously discussed: [HN 2016] - https://news.ycombinator.com/item?id=12871403 https://news.ycombinator.com/item?id=12871403
- userbinator 7y ago
  I wrote a comment there about frame differencing and wanting to try writing a video decoder after having written a JPEG one. Now over 2.5 years later, I can say that I did write a toy H.261 decoder, along with one for MPEG-1 and the beginnings of another for H.262/MPEG-2 in my spare time, and it wasn't all that difficult. In fact, because of the need to not only decode an image for each frame, but to do it quickly enough to keep up with the framerate on the limited hardware of the time, these early video codecs are in some ways simpler than JPEG --- e.g. all the Huffman tables are static, the colourspace and subsampling are either fixed or have only a small number of variations, etc. As evidence in support of this, my JPEG decoder is around 750 LoC (in C), while the H.261 one is slightly smaller at just under 700 LoC. The MPEG-1 decoder is a bit more complex, at close to 1kLoC. I haven't finished the H.262 one and it is more complex (interlacing is a pain...), but it's probably doable in less than 2kLoC total for a decoder that understands both MPEG-1 and 2/H.262 --- the latter is somewhat a superset of the former. I wasn't really optimising for size, nor did I have a good idea of how much code it'd take before I finished, so these should be quite "realistic" estimates of complexity. I've seen and given recommendations on here to write a JPEG codec as a learning exercise, and a search of GitHub reveals plenty of others who've done it; but the same situation doesn't seem to hold for video. Nonetheless, having done it and realised it's not that much work (i.e. should be doable in a weekend or two), I now recommend trying to write codecs for H.261 and MPEG-1 too. There's not that much media now in those two codecs, but if you get to MPEG-2, you can experience watching DVDs using your own code.
  - jwr 7y ago
    The complexity in H.264 gets you later on, when you try to build a complete (e.g. fully compliant) decoder. There are some killer features which seemed like a good idea at the time, but which make the complexity staggering. MBAFF is a good example (Macroblock-Adaptive Frame/Field encoding). What this means is that the encoder can choose for every block whether it wants to encode it as a single frame or two interlaced fields. Combined with B-frames (and remember that B-frames reference frames both in the past and in the future) this makes for really interesting memory access patterns, which then kills your cache locality and murders performance.
mistercow 7y ago
Overall, this article is a pretty good overview of how lossy image/video compression works. However, what the author describes as "quantization" is actually a low pass filter. With quantization, you do not simply zero out the high frequency components. You "snap" them to specific intervals. For example, if you have some data varying between 0 and 100, like [7, 39, 97, 42, 13], quantizing that by a factor of 5 would give you [5, 40, 95, 40, 15]. This gives you an approximation of the fine details, rather than simply throwing them away.
- deleted 7y ago
  [deleted]
Iv 7y ago
FFT is the real magic there and has been since the 90s. Convince me otherwise.
- boapnuaput 7y ago
  Very well. I would like to correct two misconceptions that you carry. Not only is the Fast Fourier Transform not magical, but it comes out of a straightforward elaboration of the question of how multiplication of two numbers might be done quickly. The technique was discovered before the (19)90s, too; Gauss discovered it over two centuries ago.
  - sjwright 7y ago
    I think you are intentionally misunderstanding the OP's point. He is talking about FFT as applied to lossy video compression. In direct response to the OP, I would suggest that motion estimation and entropy coding are up there on equal footing with FFT.
- ChrisLomont 7y ago
  The FFT is lossless, so there’s no compression as a result of using it. The compression is gained by perceptual modeling and dumping parts, and by powerful entropy modeling, like CABAC and such. Also more advanced things are usually used in modern compression: DCTs, sliding window stuff, wavelets, CABAC stuff, motion prediction schemes, etc., none of which were used in the 90s (except DCT and rudimentary prediction).
  - acchow 7y ago
    "FFT is lossless" - is this true without infinite precision values?
    - hoseja 7y ago
      https://en.wikipedia.org/wiki/Nyquist_rate https://en.wikipedia.org/wiki/Nyquist_rate
PorterDuff 7y ago
Thinking back on the different dead ends, I figure that H.264 is just a reasonable evolutionary stage. Now, Pixelon...that was probably magic.
- viraptor 7y ago
  Somehow I missed this dot-com crash story. The number of things wrong with it is amazing: https://www.wired.com/2000/05/perilous-fall-of-pixelon/ https://www.wired.com/2000/05/perilous-fall-of-pixelon/
  - doomrobo 7y ago
    If stories like this interest you, I highly recommend Nat Geo's Valley of the Boom. I don't even like "history shows" but it's a funny and self-aware take on SV in the 90s with solid acting and a pretty low budget.
etaioinshrdlu 7y ago
We have the tech today for even wildly better compression, at least for natural looking images. Using neural nets. There is a big computational cost to them, but if history is any guide, that won't be an issue forever.
- flud 7y ago
  Got any links to that?
  - sandos 7y ago
    Its sort of obvious if you see stuff like this: https://www.pcgamesn.com/nvidia/ai-gaugan-app-painting-neural-network https://www.pcgamesn.com/nvidia/ai-gaugan-app-painting-neura... The neural network basically has a huge lookup table of "natural" looking stuff. Just point into that table and get what you want out of it: grass, trees, a crowd, etc and the NN will produce it to your specifications.
alanh 7y ago
Don’t use blockquotes instead of section headings… please. This makes my skin crawl, but more importantly, it messes up assistive technology such as rotor and text-to-speech.
Wowfunhappy 7y ago
> Let's say you've been playing a video on YouTube. You missed the last few seconds of dialog, so you scrub back a few seconds. Have you noticed that it doesn't instantly start playing from that timecode you just selected. It pauses for a few moments and then plays. It's already buffered those frames from the network, since you just played it, so why that pause? Because you've asked the decoder to jump to some arbitrary frame, the decoder has to redo all the calculations - starting from the nearest I-frames and adding up the motion vector deltas to the frame you're on - and this is computationally expensive, and hence the brief pause. Uh, is that actually what's going on there? Because for some reason, seeking is basically instant when I'm playing locally-saved videos, including ones I downloaded directly from Youtube and didn't re-encode. Actually, this is one of the reasons why I download videos before watching them, instead of using the normal Youtube player.
- sand500 7y ago
  It might also be because youtube uses DASH?
- sp332 7y ago
  Yeah. VLC for example doesn't have a frame-by-frame backward seek, because they argue it would blow up the memory consumption to keep all those frames around just in case you want to jump backward. And since videos are mostly hardware-decoded on the GPU these days, you'd be wasting all that memory in your VRAM. Here's one of many threads where Jean-Baptiste Kempf addresses the feature request. https://forum.videolan.org/viewtopic.php?p=390778&sid=1a571d0ef612114307fdd41229a6278f#p390778 https://forum.videolan.org/viewtopic.php?p=390778&sid=1a571d... The amount of pausing is going to vary a lot by video. If you have an I-frame followed by 250 P-frames, there will probably be a noticeable pause.
  - Wowfunhappy 7y ago
    It has been a long time since I used VLC—I generally use mpv on Windows and QuickTime X on Mac because I like their minimal UIs—but I remember it being instant too! Side note, using VRAM to make seeking instantaneous seems to me like a perfectly good use of resources, at least if my device has the VRAM to spare. On my desktop, I have a 1080 Ti, and it mostly sits idle when I'm watching a video...
deleted 7y ago
[deleted]
zeroimpl 7y ago
Chroma subsampling is a terrible idea in the digital era. You should just assign a different compression factor to the chroma channels instead. I don't understand why we keep using this system of discarding 75% of the color data BEFORE applying the lossy compression algorithm. (Well I do know of one argument - apparently it reduces the amount of CPU time required to compress and decompress the data, but I think that's a pretty lame reason)
- kevin_thibedeau 7y ago
  There is a physiological reason that you can't resolve the chroma at the same resolution as the luminance.
  - blt 7y ago
    But even after you decide to allocate << 50% of your information bandwidth to chroma, there may still be ways to use that allocation than are more effective (perceptually) than 2x downsampling.
    - pingucrimson 7y ago
      This. Subsampling is kind-of the most primitive method of "lossy compression" there is.
- blt 7y ago
  Good point. When compressing a full video to a size target, I've always felt the 1080p, squashed version was still better than the 720p, less-squashed version.
- plorkyeran 7y ago
  x264 supports encoding 4:4:4 (i.e. no chroma subsampling) video. There's some very obvious cases where it produces better results, but it's typically just slower with no upside.
  - ttoinou 7y ago
_bxg1 7y ago
Fascinating. I love these accessible dives.
jwr 7y ago
The interesting thing about H.264 is that it achieves excellent compression rates not by a single "miracle trick", but by combining multiple small-gain techniques. Every technique lets you gain on the order of 1-3%, but when you combine them all, you get fantastic results. Another tidbit of (potentially) interesting information: it took many years for H.264 to achieve its full potential, for two reasons: 1) Implementing a decoder is really, really complicated, especially if you want it to run fast, and if you think a miracle "hardware decoder" solves the problem, you need to remember that H.264 places heavy requirements on memory bandwidth and caching (B frames), so it's not like you can just feed a stream through a chip and get video frames on the other side. 2) Implementing an encoder is really, really complicated :-) — and there are many ways to encode a video stream. H.264 delivers the tools, but does not specify how you should use them. It took many years for decoders to get to a state where most of H.264 is used efficiently.
- tgtweak 7y ago
  It's a tremendous effort to simply ensure standards. There are so many knobs and dials on h264 encoders, and features in each profile, is hard to appreciate the compatability we have today across devices - especially considering how much is in silicon. I remember sponsoring a feature in x264 (the best software encoder) and the level of complexity in that code is mind bending. The patch was to allow better CPU utilization when running with over 16 cores - something the authors hadn't run into at the time and had no systems to test on. I did a simple profile during encode, sent them the trace, in 2 days the patch was tested and working and it hit main in a week's time. Boom, all 48 cores used. Made me realize just how much talent is behind these technologies and I still shake my head at how much money is generated with this and how little of that makes it's way back to the developers at the core of it.
  - ahartmetz 7y ago
    x264 is really amazing because it has an amazing team of knowledgeable people who are personally interested in its success. x265 seems to be just some commercially run open source project and I'm not sure if it will ever make as good use of the spec as x264 does. For a time x265 was only about as good as x264 regarding subjective quality per bit.
Invictus0 7y ago
Neat! I'd love to read another post like this about H.265.
mrmondo 7y ago
I’d be interested to see this same analysis with H.265 given that its the way forward and the standard on the for many devices and media.
nurettin 7y ago
> You wouldn't say HHHHHHHHH. You would just say "10 tosses, all heads" - bam! You've just compressed some data! Easy. I saved you hours of mindfuck lectures. This is obviously an oversimplification Oversimplification and exaggeration. Understanding Huffman encoding and deflate algorithm are just two five minute articles away.
- dTal 7y ago
  It's also not compressed at all - the english prose is 21 characters, over twice as long. Also the English prose requires at least ASCII, while the string of H can be encoded with a single bit each. 168 bits > 10 bits. To be fair, it is faster to say, out loud, because our verbal "character set" is so much richer. A unit of pronunciation is roughly a syllable. "10 tosses, all heads" has only 5 syllables, while "ach ach ach..." takes 10. But this insight is a bit much to ask of someone new to the concept of compression.
  - ljcn 7y ago
    It's difficult to describe but I read it more like converting the raw data (a list of results) to a generator. Akin to the "compression" described in this xkcd: https://www.xkcd.com/1155/ https://www.xkcd.com/1155/
alfonsodev 7y ago
Still what got into all MacBooks > mid 2015 is a hardware HVEC encoder(if I’m not mistaken), anyone knows why HEVC and not H.264 ? EDIT: *HEVC not HVEC NOTE: I found out HEVC is H.265 so my question might not make sense :)
- zapzupnz 7y ago
  Fairly sure hardware H.264 decoding has been included with every Mac, usually on the GPU, for years and years now. A couple years after the iPhone, I think.
  - alfonsodev 7y ago
    Decoding sure, but I was saying about encoding, my Macbook mid 2015 comes with "Intel Iris Pro 1536 MB" and takes 25% cpu to capture screen with Quicktime, while newer MacBooks >=2016 come with HEVC, which after some searches I found out is also called H.256[1][2], and is not just in Macs[3] [1] https://support.apple.com/en-us/HT208238 https://support.apple.com/en-us/HT208238 [2] https://www.macrumors.com/guide/hevc-video-macos-high-sierra-ios-11/ https://www.macrumors.com/guide/hevc-video-macos-high-sierra... [3]https://www.techspot.com/article/1131-hevc-h256-enconding-playback/ https://www.techspot.com/article/1131-hevc-h256-enconding-pl... EDIT: I see now the label 2016, so opening question didn't make sense :) sorry.
    - kalleboo 7y ago
      They also have encoding. The software might not be using it properly though. https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video I think screen recording is slow not due to the video encoding but since it defeats standard GPU acceleration of the GUI in order to be able to capture the contents
      - namibj 7y ago
        It really shouldn't. You just feed the framebuffer both into the DMA output, and in the H.26x encoder. If needed, use another framebuffer space if the encoder is slow, so you don't _have_ to recycle the buffer after this frame, and risk the encoder not being done yet.
carlsborg 7y ago
But it isn’t free so it’s a pity it’s taking over the world. Chrome plays h.264 / mp4 because they are licenced. Firefox counts on OS support. (The Cisco plugin only supports webrtc calls) so you can’t legally watch MP4 videos on Linux. Webm seems to be as good and free. Can some comment if I am mistaken about this?
- suhail 7y ago
  It’s all capped at a max payment and not terribly expensive. VP9 and its new versions will take over but it will be a while until it’s implemented in hw fully.
  - carlsborg 7y ago
    Non-free standards are bad for open source and innovation in general.
    - tntn 7y ago
      The good thing about standards is everyone can have their own.
- nicoburns 7y ago
  It's not, but it actually won't be too long until the patents expire for it (a quick google says 2027). The same used to be true of mp3. It was ubiquitous, but annoyingly not open, and now it's both.
- mkl 7y ago
  A bit tangential, but you are mistaken in that you are confusing container formats and codecs. MP4 is a container format, which can contain, among other codecs, h.264 video. WebM is another container format, which can contain, among other codecs, VP9 video. MKV and AVI are other container formats, and h.265, VP8, and AV1 are other codecs. There are lots of codecs compared to container formats, and the same codecs can often go in multiple different containers (you can even convert encoded video losslessly to a different container format). Licensing issues are almost all around codecs, not container formats.
gingabriska 7y ago
I bought a laptop camera module and attached to a USB cable and it pegged my raspberry CPU at 90% and soon I started receiving temperature warnings through that thermometer icon on the screen. I wonder why webcams don't directly use the onboard graphic card. Even the cheap webcams all suffer from the very same problem. The reason I bought laptop card, I thought it would be higher quality and lower price and would be good for monitoring my 3d prints. I've no found any Raspberry Pi camera with autofocus with IR filter intact ( I need it for full light use and better colours during day time)
- Jaruzel 7y ago
  Try the Microsoft LifeCam range. AutoFocus, Supported by Linux, Really good image.
mojuba 7y ago
Judging from the comments, the article mostly resonates with people who already know how the codec works, but it doesn't seem to be very useful to engineers who, like me, want to understand it in a bit more technical detail. The author could have removed the casual conversation noise in the text (very annoying in technical articles!) and filled it with some more useful bits instead.
- jannemann 7y ago
  I enjoyed the reading and think that it is not a technical article rather entertainment. Maybe that is wrong too and it is both. Why should every technical article by definition be written in a boring language? For technical details use the terms and search them in your favourite search engine. There are literally tons of articles about every aspect of H.265.
  - mojuba 7y ago
    > For technical details use the terms and search them in your favourite search engine. Thanks for lecturing me, but the article in question is neither too entertaining nor too informative. That was my point. Other than that, of course I can find better materials on the net.
mrob 7y ago
One under-appreciated advantage of H.264: it's the last video codec where the artifacts look obviously artificial. Modern codecs have artifacts that look too much like real objects, so they take more brain power to ignore. After watching low bitrate H.264 for a few minutes I stop noticing the artifacts, which isn't the case with modern codecs.
- reacweb 7y ago
  Very interesting. Do you have links on this subject (or could you write an article) ?
  - mrob 7y ago
    It's just personal observation. I noticed the same thing in audio artifacts with the RNNoise demo from xiph.org: https://people.xiph.org/~jm/demo/rnnoise/ https://people.xiph.org/~jm/demo/rnnoise/
- jeffhuys 7y ago
  What codecs are you talking about? I'm interested to see these artifacts for myself.
  - mrob 7y ago
    Comparison videos from: http://video.1ko.ch/codec-comparison/ http://video.1ko.ch/codec-comparison/ http://video.1ko.ch/codec-comparison/videos/vp9_562.webm http://video.1ko.ch/codec-comparison/videos/vp9_562.webm http://video.1ko.ch/codec-comparison/videos/x264_562.mp4 http://video.1ko.ch/codec-comparison/videos/x264_562.mp4 At 15 seconds, in the VP9 version, it looks like smoke is pouring off the top-left metal pipe of the machine. It immediately draws my attention because it's so out of place. But in the H.264 version, the whole image is noisy, so nothing distracts me.
    - PuffinBlue 7y ago
      I'm not really sure which is which but the webm version looks way way way better to my eye. Not even any comparison really.
montenegrohugo 7y ago
What a nice article! Love finding these on hn
mehrdadn 7y ago
General question about video codecs: why are codecs tied to resolutions? Why can't some codecs be sued for resolutions larger than some maximum? This is probably the one thing I never understood about codecs.
- cesarb 7y ago
  Not a codec expert, but I can see two reasons. The first is that some fields in the encoding are not big enough, for instance a 12-bit field for the resolution in some header can represent only up to 4095 pixels (or 4096 if "zero pixels" is not allowed). The second is that all video codecs I know of subdivide the image into small blocks and compress each block individually (mostly; there's some prediction from nearby blocks, and some mixing of the blocks at the end to smooth the edges between them), and older codecs only have blocks which are too small for higher resolutions (imagine for instance a 8x8 block in a 320x240 video, and then in the same video at 1280x960; the block clearly covers much less of the later video, so there's less variation for it to work with. Increase the resolution too much, and small blocks would start acting more like individual pixels, and you'd lose nearly all the compression.)
  - mehrdadn 7y ago
    The first reason seems possible I guess, but not likely as being the only reasons. The second reason though should only imply that you lose the compression, not that you're unable to produce the file in the given format in the first place? What I'm referring is that I recall encoder tools outright rejecting high resolutions for older formats.
- opencl 7y ago
  Mostly it's just because the standards say so. There's nothing about how MPEG-2 works that inherently makes it impossible to encode 4K video with it, the standard just says that MPEG-2 video shall not exceed 80Mbit/s, 1920 horizontal pixels, 1152 vertical pixels, or 62,668,800 luminance samples per second. The limits are just there for interoperability. As is, if you encode a video that complies with MPEG-2 High Level, then you can be pretty confident that any MPEG-2 High Level decoder will decode it. Without the limit everything becomes a mess. You're writing a decoder, what max res should you support? 4K? 8K? The people writing decoders don't agree and then people trying to distribute these high resolution files find that they work in some decoders but not others.
eternalban 7y ago
I believe the "magic" is happening in the visual cortex.
social_quotient 7y ago
In the article near the top it says “1080p @ 60 Hz = 1920x1080x60x3 => ~370 MB/sec of raw data.” Can someone help me understand the 60 Hz part of this? Was it meant to be fps or does it really mean Hz? And why?
- proaralyst 7y ago
  The author means 60 frames per second, which has the units of Hertz. You sometimes see this unit used for screen refresh rates in your system settings.
- syn0byte 7y ago
  The average refresh rate of a computer display is 60Hz. In this context it's interchangeable with framerate but we think of it in terms of Hz because of the monitors.
IronWolve 7y ago
The chroma subsampling is why 4k video played on a 1080p/1440p monitor looks so much better/clear/detailed on monitors. Typical 1080P native encoding has less chroma detail for size saving.
arendtio 7y ago
Nice summary! I just wish he wouldn't have used pounds for comparison... I mean there are 3 countries left on this planet that don't use the metric system officially. Even the scientists in the US use it. Reminds me of http://www.joeydevilla.com/wordpress/wp-content/uploads/2008/08/king-henrys-foot.jpg http://www.joeydevilla.com/wordpress/wp-content/uploads/2008... But given how he managed to simplify such a complex technology to an easy-to-read article I don't really want to criticize him for using the wrong unit system.
- austinshea 7y ago
  And, yet, you did.
nojvek 7y ago
I have to say I really love sid's style of writing. Humurous and entertaining yet brief and to the point. Hope there's more like this to come.
- davidcollantes 7y ago
  Written in 2016, it is the only entry on the blog. Odds are slim, at least coming from Sid.
jimbo1qaz 7y ago
Chroma subsampling is a terrible idea for non-photographic images. People are not "terrible" at color perception. I can clearly see its harmful effects to on-screen captures, line art, anime, 2d video games, or even Super Mario 64's hat. Youtube's chroma subsampling makes colors bleed, Mario's hat turn into chunky red blocks approximately filling Mario's hat, screen captures discolored and grainy around text, and sharp colored anti-aliased lines turn into a discolored mess. (Mario's hat was pixelated on pannenkoek2012's emulated SM64 videos at native 640x480. Maybe newer video decoders antialias the chroma channel, so Mario's hat is not pixelated but "merely" blurry.) I upload oscilloscope videos with colored lines on black backgrounds, and stopped using brightly colored lines partly because color was being blurred and discolored by Youtube chroma subsampling.
vagab0nd 7y ago
I worked on this project where we needed to analyze the motion of objects from the camera in real time. We were trying to find an algorithm that was fast enough to run on the hardware, but to no avail. One day, it dawned on me that since our hardware encodes the video in h264, we should be able to get some of that info from the video itself. So we tried extracting the motion vectors from the video and sure enough, it was good enough for our use and we got it basically for free.
sytelus 7y ago
It is pretty amazing that H.264 is not one algorithm that just popped out and changed the world, instead it is accumulation of collection of tricks developed over several decades. I distinctly remember few folks saying that you would need to violate laws of physics to transmit 1080p @ 60 fps over wifi. Now we are all doing it many times over and billions of dollars are being made every year. This is one of the unsung achievement worthy of Nobel-prize level awards but no one seems to know who these people were. I wonder if there is any detailed history of how various components of H.264 came together, who led this effort, how projects were funded for such a long time?

680 points by dpeck 7y ago

I've seen this article before, and this part always stuck with me: > I captured the screen of this home page and produced two files: > PNG screenshot of the Apple homepage 1015KB > 5 Second 60fps H.264 video of the same Apple homepage 175KB > Eh. What? Those file sizes look switched. > No, they're right. The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. The article does go into how and why this is possible (tl;dr H.264 is lossey, png is not), but for the difference in human-detectable quality, H.264 is astonishingly more efficient.

ChrisSD 7y ago
It's a good example but the difference doesn't quite have to be as dramatic as the article makes out. Screenshot tools rarely produce optimised PNGs. A quick test of the image in the article shows that optipng (lossless) can bring the size down to ~570kb or pngquant (lossy) to ~240kb. Zopfil could further compress the images produced by optipng or pngquant, but it's a case of diminishing returns at that point. To be clear the article makes a good comparison but I think it perhaps overstates the case to make a point.

viraptor 7y ago

I like this high-level summary. One thing I'd change though is that the photo loses details in the frequency domain only. In real view it actually gains unnecessary details/noise - it's not that the grill details disappeared, it's the whole image that gained an anti-grill layer. (You can now see it on the smooth part of the laptop)

deleted 7y ago

[deleted]

andyfleming 7y ago
Not sure why you were downvoted. It is literally in the article.
- Dylan16807 7y ago
  When someone is explaining what part of the article left the biggest impression on them and why, it's weird and unhelpful to go "that's literally in the article!"
  - andyfleming 7y ago
    Oh, maybe they edited their comment (or maybe I misread it). Originally, I believe it said that it _didn't_ explain why at all.
    - viraptor 7y ago
      Same here. I'd be happy to delete my comment in either case, but it's too late now.

bscphil 7y ago

Interesting introduction. Too bad it doesn't really get into some of the more complicated things an advanced H.264 encoder like x264 is doing, e.g. adaptive quantization methods. There's also a mistake or two, for example > P-frames are frames that will encode a motion vector for each of the macro-blocks from the previous frame. Actually, P-frames can use data from multiple previous frames, not just the last one. I think it's worth pointing out, as well, that even though we've technically surpassed H.264 with newer codecs like H.265, VP9, and AV1, all of which are roughly 20%-50% more efficient, this has required tremendous increases in encoding complexity. H.264 is special - it seems to occupy a kind of inflection point on the complexity / efficiency curve. It's far more efficient than previous codecs like Xvid, WMV, and so on, but at the same time even a lot of underpowered devices from over a decade ago can easily play it. We're not likely to see tradeoffs that good again in the video codec space.

clouddrover 7y ago
When you control for image quality VP9 outperforms H.264 in single-threaded decoding: https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecoding-performance-vs-hevch-264/ https://blogs.gnome.org/rbultje/2015/09/28/vp9-encodingdecod... You get the same image quality at a lower bitrate and faster decoding. VP9 encoding is much slower, but Intel's SVT-VP9 encoder is getting over 300 frames per second on the right hardware: https://www.phoronix.com/scan.php?page=news_item&px=SVT-VP9-Open-Source https://www.phoronix.com/scan.php?page=news_item&px=SVT-VP9-...
- bscphil 7y ago
  That's an interesting claim. I have to say it doesn't match my experience at all, e.g. when playing VP9 videos on Youtube. Several thoughts: * I'm not sure what the "ffh264" decoder they're talking about is. My understanding is that pretty much everyone is using libx264 for both encoding and decoding. Maybe the latter is more performant? * Using SSIM to pick "equal quality" files will probably not give very accurate results. * SVT-VP9 is, if I'm not mistaken, a hardware encoder. One of the issues with these is that they require some significant tradeoffs to get better speed, so they'll have worse quality than a software encoder at the same bitrate. If you go with software you really pay the price when it comes to encoding time.

dpeck 7y ago

Previously discussed: [HN 2016] - https://news.ycombinator.com/item?id=12871403 https://news.ycombinator.com/item?id=12871403

userbinator 7y ago
I wrote a comment there about frame differencing and wanting to try writing a video decoder after having written a JPEG one. Now over 2.5 years later, I can say that I did write a toy H.261 decoder, along with one for MPEG-1 and the beginnings of another for H.262/MPEG-2 in my spare time, and it wasn't all that difficult. In fact, because of the need to not only decode an image for each frame, but to do it quickly enough to keep up with the framerate on the limited hardware of the time, these early video codecs are in some ways simpler than JPEG --- e.g. all the Huffman tables are static, the colourspace and subsampling are either fixed or have only a small number of variations, etc. As evidence in support of this, my JPEG decoder is around 750 LoC (in C), while the H.261 one is slightly smaller at just under 700 LoC. The MPEG-1 decoder is a bit more complex, at close to 1kLoC. I haven't finished the H.262 one and it is more complex (interlacing is a pain...), but it's probably doable in less than 2kLoC total for a decoder that understands both MPEG-1 and 2/H.262 --- the latter is somewhat a superset of the former. I wasn't really optimising for size, nor did I have a good idea of how much code it'd take before I finished, so these should be quite "realistic" estimates of complexity. I've seen and given recommendations on here to write a JPEG codec as a learning exercise, and a search of GitHub reveals plenty of others who've done it; but the same situation doesn't seem to hold for video. Nonetheless, having done it and realised it's not that much work (i.e. should be doable in a weekend or two), I now recommend trying to write codecs for H.261 and MPEG-1 too. There's not that much media now in those two codecs, but if you get to MPEG-2, you can experience watching DVDs using your own code.
- jwr 7y ago
  The complexity in H.264 gets you later on, when you try to build a complete (e.g. fully compliant) decoder. There are some killer features which seemed like a good idea at the time, but which make the complexity staggering. MBAFF is a good example (Macroblock-Adaptive Frame/Field encoding). What this means is that the encoder can choose for every block whether it wants to encode it as a single frame or two interlaced fields. Combined with B-frames (and remember that B-frames reference frames both in the past and in the future) this makes for really interesting memory access patterns, which then kills your cache locality and murders performance.

mistercow 7y ago

Overall, this article is a pretty good overview of how lossy image/video compression works. However, what the author describes as "quantization" is actually a low pass filter. With quantization, you do not simply zero out the high frequency components. You "snap" them to specific intervals. For example, if you have some data varying between 0 and 100, like [7, 39, 97, 42, 13], quantizing that by a factor of 5 would give you [5, 40, 95, 40, 15]. This gives you an approximation of the fine details, rather than simply throwing them away.

deleted 7y ago
[deleted]

Iv 7y ago

FFT is the real magic there and has been since the 90s. Convince me otherwise.

boapnuaput 7y ago
Very well. I would like to correct two misconceptions that you carry. Not only is the Fast Fourier Transform not magical, but it comes out of a straightforward elaboration of the question of how multiplication of two numbers might be done quickly. The technique was discovered before the (19)90s, too; Gauss discovered it over two centuries ago.
- sjwright 7y ago
  I think you are intentionally misunderstanding the OP's point. He is talking about FFT as applied to lossy video compression. In direct response to the OP, I would suggest that motion estimation and entropy coding are up there on equal footing with FFT.
ChrisLomont 7y ago
The FFT is lossless, so there’s no compression as a result of using it. The compression is gained by perceptual modeling and dumping parts, and by powerful entropy modeling, like CABAC and such. Also more advanced things are usually used in modern compression: DCTs, sliding window stuff, wavelets, CABAC stuff, motion prediction schemes, etc., none of which were used in the 90s (except DCT and rudimentary prediction).
- acchow 7y ago
  "FFT is lossless" - is this true without infinite precision values?
  - hoseja 7y ago
    https://en.wikipedia.org/wiki/Nyquist_rate https://en.wikipedia.org/wiki/Nyquist_rate

PorterDuff 7y ago

Thinking back on the different dead ends, I figure that H.264 is just a reasonable evolutionary stage. Now, Pixelon...that was probably magic.

viraptor 7y ago
Somehow I missed this dot-com crash story. The number of things wrong with it is amazing: https://www.wired.com/2000/05/perilous-fall-of-pixelon/ https://www.wired.com/2000/05/perilous-fall-of-pixelon/
- doomrobo 7y ago
  If stories like this interest you, I highly recommend Nat Geo's Valley of the Boom. I don't even like "history shows" but it's a funny and self-aware take on SV in the 90s with solid acting and a pretty low budget.

etaioinshrdlu 7y ago

We have the tech today for even wildly better compression, at least for natural looking images. Using neural nets. There is a big computational cost to them, but if history is any guide, that won't be an issue forever.

flud 7y ago
Got any links to that?
- sandos 7y ago
  Its sort of obvious if you see stuff like this: https://www.pcgamesn.com/nvidia/ai-gaugan-app-painting-neural-network https://www.pcgamesn.com/nvidia/ai-gaugan-app-painting-neura... The neural network basically has a huge lookup table of "natural" looking stuff. Just point into that table and get what you want out of it: grass, trees, a crowd, etc and the NN will produce it to your specifications.

alanh 7y ago

Don’t use blockquotes instead of section headings… please. This makes my skin crawl, but more importantly, it messes up assistive technology such as rotor and text-to-speech.

Wowfunhappy 7y ago

> Let's say you've been playing a video on YouTube. You missed the last few seconds of dialog, so you scrub back a few seconds. Have you noticed that it doesn't instantly start playing from that timecode you just selected. It pauses for a few moments and then plays. It's already buffered those frames from the network, since you just played it, so why that pause? Because you've asked the decoder to jump to some arbitrary frame, the decoder has to redo all the calculations - starting from the nearest I-frames and adding up the motion vector deltas to the frame you're on - and this is computationally expensive, and hence the brief pause. Uh, is that actually what's going on there? Because for some reason, seeking is basically instant when I'm playing locally-saved videos, including ones I downloaded directly from Youtube and didn't re-encode. Actually, this is one of the reasons why I download videos before watching them, instead of using the normal Youtube player.

sand500 7y ago
It might also be because youtube uses DASH?
sp332 7y ago
Yeah. VLC for example doesn't have a frame-by-frame backward seek, because they argue it would blow up the memory consumption to keep all those frames around just in case you want to jump backward. And since videos are mostly hardware-decoded on the GPU these days, you'd be wasting all that memory in your VRAM. Here's one of many threads where Jean-Baptiste Kempf addresses the feature request. https://forum.videolan.org/viewtopic.php?p=390778&sid=1a571d0ef612114307fdd41229a6278f#p390778 https://forum.videolan.org/viewtopic.php?p=390778&sid=1a571d... The amount of pausing is going to vary a lot by video. If you have an I-frame followed by 250 P-frames, there will probably be a noticeable pause.
- Wowfunhappy 7y ago
  It has been a long time since I used VLC—I generally use mpv on Windows and QuickTime X on Mac because I like their minimal UIs—but I remember it being instant too! Side note, using VRAM to make seeking instantaneous seems to me like a perfectly good use of resources, at least if my device has the VRAM to spare. On my desktop, I have a 1080 Ti, and it mostly sits idle when I'm watching a video...

deleted 7y ago

[deleted]

zeroimpl 7y ago

Chroma subsampling is a terrible idea in the digital era. You should just assign a different compression factor to the chroma channels instead. I don't understand why we keep using this system of discarding 75% of the color data BEFORE applying the lossy compression algorithm. (Well I do know of one argument - apparently it reduces the amount of CPU time required to compress and decompress the data, but I think that's a pretty lame reason)

kevin_thibedeau 7y ago
There is a physiological reason that you can't resolve the chroma at the same resolution as the luminance.
- blt 7y ago
  But even after you decide to allocate << 50% of your information bandwidth to chroma, there may still be ways to use that allocation than are more effective (perceptually) than 2x downsampling.
  - pingucrimson 7y ago
    This. Subsampling is kind-of the most primitive method of "lossy compression" there is.
blt 7y ago
Good point. When compressing a full video to a size target, I've always felt the 1080p, squashed version was still better than the 720p, less-squashed version.
plorkyeran 7y ago
x264 supports encoding 4:4:4 (i.e. no chroma subsampling) video. There's some very obvious cases where it produces better results, but it's typically just slower with no upside.
- ttoinou 7y ago

_bxg1 7y ago

Fascinating. I love these accessible dives.

jwr 7y ago

The interesting thing about H.264 is that it achieves excellent compression rates not by a single "miracle trick", but by combining multiple small-gain techniques. Every technique lets you gain on the order of 1-3%, but when you combine them all, you get fantastic results. Another tidbit of (potentially) interesting information: it took many years for H.264 to achieve its full potential, for two reasons: 1) Implementing a decoder is really, really complicated, especially if you want it to run fast, and if you think a miracle "hardware decoder" solves the problem, you need to remember that H.264 places heavy requirements on memory bandwidth and caching (B frames), so it's not like you can just feed a stream through a chip and get video frames on the other side. 2) Implementing an encoder is really, really complicated :-) — and there are many ways to encode a video stream. H.264 delivers the tools, but does not specify how you should use them. It took many years for decoders to get to a state where most of H.264 is used efficiently.

tgtweak 7y ago
It's a tremendous effort to simply ensure standards. There are so many knobs and dials on h264 encoders, and features in each profile, is hard to appreciate the compatability we have today across devices - especially considering how much is in silicon. I remember sponsoring a feature in x264 (the best software encoder) and the level of complexity in that code is mind bending. The patch was to allow better CPU utilization when running with over 16 cores - something the authors hadn't run into at the time and had no systems to test on. I did a simple profile during encode, sent them the trace, in 2 days the patch was tested and working and it hit main in a week's time. Boom, all 48 cores used. Made me realize just how much talent is behind these technologies and I still shake my head at how much money is generated with this and how little of that makes it's way back to the developers at the core of it.
- ahartmetz 7y ago
  x264 is really amazing because it has an amazing team of knowledgeable people who are personally interested in its success. x265 seems to be just some commercially run open source project and I'm not sure if it will ever make as good use of the spec as x264 does. For a time x265 was only about as good as x264 regarding subjective quality per bit.

Invictus0 7y ago

Neat! I'd love to read another post like this about H.265.

mrmondo 7y ago

I’d be interested to see this same analysis with H.265 given that its the way forward and the standard on the for many devices and media.

nurettin 7y ago

> You wouldn't say HHHHHHHHH. You would just say "10 tosses, all heads" - bam! You've just compressed some data! Easy. I saved you hours of mindfuck lectures. This is obviously an oversimplification Oversimplification and exaggeration. Understanding Huffman encoding and deflate algorithm are just two five minute articles away.

dTal 7y ago
It's also not compressed at all - the english prose is 21 characters, over twice as long. Also the English prose requires at least ASCII, while the string of H can be encoded with a single bit each. 168 bits > 10 bits. To be fair, it is faster to say, out loud, because our verbal "character set" is so much richer. A unit of pronunciation is roughly a syllable. "10 tosses, all heads" has only 5 syllables, while "ach ach ach..." takes 10. But this insight is a bit much to ask of someone new to the concept of compression.
- ljcn 7y ago
  It's difficult to describe but I read it more like converting the raw data (a list of results) to a generator. Akin to the "compression" described in this xkcd: https://www.xkcd.com/1155/ https://www.xkcd.com/1155/

alfonsodev 7y ago

Still what got into all MacBooks > mid 2015 is a hardware HVEC encoder(if I’m not mistaken), anyone knows why HEVC and not H.264 ? EDIT: *HEVC not HVEC NOTE: I found out HEVC is H.265 so my question might not make sense :)

zapzupnz 7y ago
Fairly sure hardware H.264 decoding has been included with every Mac, usually on the GPU, for years and years now. A couple years after the iPhone, I think.
- alfonsodev 7y ago
  Decoding sure, but I was saying about encoding, my Macbook mid 2015 comes with "Intel Iris Pro 1536 MB" and takes 25% cpu to capture screen with Quicktime, while newer MacBooks >=2016 come with HEVC, which after some searches I found out is also called H.256[1][2], and is not just in Macs[3] [1] https://support.apple.com/en-us/HT208238 https://support.apple.com/en-us/HT208238 [2] https://www.macrumors.com/guide/hevc-video-macos-high-sierra-ios-11/ https://www.macrumors.com/guide/hevc-video-macos-high-sierra... [3]https://www.techspot.com/article/1131-hevc-h256-enconding-playback/ https://www.techspot.com/article/1131-hevc-h256-enconding-pl... EDIT: I see now the label 2016, so opening question didn't make sense :) sorry.
  - kalleboo 7y ago
    They also have encoding. The software might not be using it properly though. https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video I think screen recording is slow not due to the video encoding but since it defeats standard GPU acceleration of the GUI in order to be able to capture the contents
    - namibj 7y ago
      It really shouldn't. You just feed the framebuffer both into the DMA output, and in the H.26x encoder. If needed, use another framebuffer space if the encoder is slow, so you don't _have_ to recycle the buffer after this frame, and risk the encoder not being done yet.

carlsborg 7y ago

But it isn’t free so it’s a pity it’s taking over the world. Chrome plays h.264 / mp4 because they are licenced. Firefox counts on OS support. (The Cisco plugin only supports webrtc calls) so you can’t legally watch MP4 videos on Linux. Webm seems to be as good and free. Can some comment if I am mistaken about this?

suhail 7y ago
It’s all capped at a max payment and not terribly expensive. VP9 and its new versions will take over but it will be a while until it’s implemented in hw fully.
- carlsborg 7y ago
  Non-free standards are bad for open source and innovation in general.
  - tntn 7y ago
    The good thing about standards is everyone can have their own.
nicoburns 7y ago
It's not, but it actually won't be too long until the patents expire for it (a quick google says 2027). The same used to be true of mp3. It was ubiquitous, but annoyingly not open, and now it's both.
mkl 7y ago
A bit tangential, but you are mistaken in that you are confusing container formats and codecs. MP4 is a container format, which can contain, among other codecs, h.264 video. WebM is another container format, which can contain, among other codecs, VP9 video. MKV and AVI are other container formats, and h.265, VP8, and AV1 are other codecs. There are lots of codecs compared to container formats, and the same codecs can often go in multiple different containers (you can even convert encoded video losslessly to a different container format). Licensing issues are almost all around codecs, not container formats.

gingabriska 7y ago

I bought a laptop camera module and attached to a USB cable and it pegged my raspberry CPU at 90% and soon I started receiving temperature warnings through that thermometer icon on the screen. I wonder why webcams don't directly use the onboard graphic card. Even the cheap webcams all suffer from the very same problem. The reason I bought laptop card, I thought it would be higher quality and lower price and would be good for monitoring my 3d prints. I've no found any Raspberry Pi camera with autofocus with IR filter intact ( I need it for full light use and better colours during day time)

Jaruzel 7y ago
Try the Microsoft LifeCam range. AutoFocus, Supported by Linux, Really good image.

mojuba 7y ago

Judging from the comments, the article mostly resonates with people who already know how the codec works, but it doesn't seem to be very useful to engineers who, like me, want to understand it in a bit more technical detail. The author could have removed the casual conversation noise in the text (very annoying in technical articles!) and filled it with some more useful bits instead.

jannemann 7y ago
I enjoyed the reading and think that it is not a technical article rather entertainment. Maybe that is wrong too and it is both. Why should every technical article by definition be written in a boring language? For technical details use the terms and search them in your favourite search engine. There are literally tons of articles about every aspect of H.265.
- mojuba 7y ago
  > For technical details use the terms and search them in your favourite search engine. Thanks for lecturing me, but the article in question is neither too entertaining nor too informative. That was my point. Other than that, of course I can find better materials on the net.

mrob 7y ago

One under-appreciated advantage of H.264: it's the last video codec where the artifacts look obviously artificial. Modern codecs have artifacts that look too much like real objects, so they take more brain power to ignore. After watching low bitrate H.264 for a few minutes I stop noticing the artifacts, which isn't the case with modern codecs.

reacweb 7y ago
Very interesting. Do you have links on this subject (or could you write an article) ?
- mrob 7y ago
  It's just personal observation. I noticed the same thing in audio artifacts with the RNNoise demo from xiph.org: https://people.xiph.org/~jm/demo/rnnoise/ https://people.xiph.org/~jm/demo/rnnoise/
jeffhuys 7y ago
What codecs are you talking about? I'm interested to see these artifacts for myself.
- mrob 7y ago
  Comparison videos from: http://video.1ko.ch/codec-comparison/ http://video.1ko.ch/codec-comparison/ http://video.1ko.ch/codec-comparison/videos/vp9_562.webm http://video.1ko.ch/codec-comparison/videos/vp9_562.webm http://video.1ko.ch/codec-comparison/videos/x264_562.mp4 http://video.1ko.ch/codec-comparison/videos/x264_562.mp4 At 15 seconds, in the VP9 version, it looks like smoke is pouring off the top-left metal pipe of the machine. It immediately draws my attention because it's so out of place. But in the H.264 version, the whole image is noisy, so nothing distracts me.
  - PuffinBlue 7y ago
    I'm not really sure which is which but the webm version looks way way way better to my eye. Not even any comparison really.

montenegrohugo 7y ago

What a nice article! Love finding these on hn

mehrdadn 7y ago

General question about video codecs: why are codecs tied to resolutions? Why can't some codecs be sued for resolutions larger than some maximum? This is probably the one thing I never understood about codecs.

cesarb 7y ago
Not a codec expert, but I can see two reasons. The first is that some fields in the encoding are not big enough, for instance a 12-bit field for the resolution in some header can represent only up to 4095 pixels (or 4096 if "zero pixels" is not allowed). The second is that all video codecs I know of subdivide the image into small blocks and compress each block individually (mostly; there's some prediction from nearby blocks, and some mixing of the blocks at the end to smooth the edges between them), and older codecs only have blocks which are too small for higher resolutions (imagine for instance a 8x8 block in a 320x240 video, and then in the same video at 1280x960; the block clearly covers much less of the later video, so there's less variation for it to work with. Increase the resolution too much, and small blocks would start acting more like individual pixels, and you'd lose nearly all the compression.)
- mehrdadn 7y ago
  The first reason seems possible I guess, but not likely as being the only reasons. The second reason though should only imply that you lose the compression, not that you're unable to produce the file in the given format in the first place? What I'm referring is that I recall encoder tools outright rejecting high resolutions for older formats.
opencl 7y ago
Mostly it's just because the standards say so. There's nothing about how MPEG-2 works that inherently makes it impossible to encode 4K video with it, the standard just says that MPEG-2 video shall not exceed 80Mbit/s, 1920 horizontal pixels, 1152 vertical pixels, or 62,668,800 luminance samples per second. The limits are just there for interoperability. As is, if you encode a video that complies with MPEG-2 High Level, then you can be pretty confident that any MPEG-2 High Level decoder will decode it. Without the limit everything becomes a mess. You're writing a decoder, what max res should you support? 4K? 8K? The people writing decoders don't agree and then people trying to distribute these high resolution files find that they work in some decoders but not others.

eternalban 7y ago

I believe the "magic" is happening in the visual cortex.

social_quotient 7y ago

In the article near the top it says “1080p @ 60 Hz = 1920x1080x60x3 => ~370 MB/sec of raw data.” Can someone help me understand the 60 Hz part of this? Was it meant to be fps or does it really mean Hz? And why?

proaralyst 7y ago
The author means 60 frames per second, which has the units of Hertz. You sometimes see this unit used for screen refresh rates in your system settings.
syn0byte 7y ago
The average refresh rate of a computer display is 60Hz. In this context it's interchangeable with framerate but we think of it in terms of Hz because of the monitors.

IronWolve 7y ago

The chroma subsampling is why 4k video played on a 1080p/1440p monitor looks so much better/clear/detailed on monitors. Typical 1080P native encoding has less chroma detail for size saving.

arendtio 7y ago

Nice summary! I just wish he wouldn't have used pounds for comparison... I mean there are 3 countries left on this planet that don't use the metric system officially. Even the scientists in the US use it. Reminds me of http://www.joeydevilla.com/wordpress/wp-content/uploads/2008/08/king-henrys-foot.jpg http://www.joeydevilla.com/wordpress/wp-content/uploads/2008... But given how he managed to simplify such a complex technology to an easy-to-read article I don't really want to criticize him for using the wrong unit system.

austinshea 7y ago
And, yet, you did.

nojvek 7y ago

I have to say I really love sid's style of writing. Humurous and entertaining yet brief and to the point. Hope there's more like this to come.

davidcollantes 7y ago
Written in 2016, it is the only entry on the blog. Odds are slim, at least coming from Sid.

jimbo1qaz 7y ago

Chroma subsampling is a terrible idea for non-photographic images. People are not "terrible" at color perception. I can clearly see its harmful effects to on-screen captures, line art, anime, 2d video games, or even Super Mario 64's hat. Youtube's chroma subsampling makes colors bleed, Mario's hat turn into chunky red blocks approximately filling Mario's hat, screen captures discolored and grainy around text, and sharp colored anti-aliased lines turn into a discolored mess. (Mario's hat was pixelated on pannenkoek2012's emulated SM64 videos at native 640x480. Maybe newer video decoders antialias the chroma channel, so Mario's hat is not pixelated but "merely" blurry.) I upload oscilloscope videos with colored lines on black backgrounds, and stopped using brightly colored lines partly because color was being blurred and discolored by Youtube chroma subsampling.

vagab0nd 7y ago

I worked on this project where we needed to analyze the motion of objects from the camera in real time. We were trying to find an algorithm that was fast enough to run on the hardware, but to no avail. One day, it dawned on me that since our hardware encodes the video in h264, we should be able to get some of that info from the video itself. So we tried extracting the motion vectors from the video and sure enough, it was good enough for our use and we got it basically for free.

sytelus 7y ago

It is pretty amazing that H.264 is not one algorithm that just popped out and changed the world, instead it is accumulation of collection of tricks developed over several decades. I distinctly remember few folks saying that you would need to violate laws of physics to transmit 1080p @ 60 fps over wifi. Now we are all doing it many times over and billions of dollars are being made every year. This is one of the unsung achievement worthy of Nobel-prize level awards but no one seems to know who these people were. I wonder if there is any detailed history of how various components of H.264 came together, who led this effort, how projects were funded for such a long time?