21 ms·

It's interesting that they are able to continue improving video compression. You'd think that it would have all been figured out by now. Is this continued impr

It's interesting that they are able to continue improving video compression. You'd think that it would have all been figured out by now. Is this continued improvement related to the improvement of technology? Or just coincidental? Like, why couldn't have H.266 been invented 30 years ago? Is it because the computers back in the day wouldn't have been fast enough to realistically use it? Do we have algorithms today that can compress way better but would be too slow to encode/decode?

aey 6y ago
Compression is AI. It’s never going to be “all” figured out.
- 0-_-0 6y ago
  Another way if saying it is that compression is understanding.
  - jacobush 6y ago
    Lossy compression is, I feel compelled to add.
    - aey 6y ago
      Actually both! Arithmetic coding works over any kind of predictor.
  - milansuk 6y ago
    End-credits are just text. So it should be possible to put it through OCR and save only text, positions, and fonts. And the text is also possible to compress with a dictionary.
    - gsich 6y ago
      True, but end credits take very little space compared to the rest of the movie.
    - jaywalk 6y ago
      The text, positions and fonts could very well take up more space than the compressed video. And then with fonts, you have licensing issues as well.
    - lukifer 6y ago
      Credits also contain logos/symbols (near the end), and often have stylistic flairs as well. Video compression is based on making predictions and then adding information (per Shannon's definition) for the deltas from those predictions. The pattern of credits statically sliding at a consistent rate is exactly the sort of prediction codecs are optimized for; for instance, the same algorithms will save space by predicting repeated pixel patterns during a slow camera pan. Still, I've often thought it would be nice if text were a more first-class citizen within video codecs. I think it's more a toolchain/workflow problem than a shortcoming in video compression technology as such. Whoever is mastering a Blu-Ray or prepping a Hollywood film for Netflix is usually not the same person cutting and assembling the original content. For innumerable reasons (access to raw sources, low return on time spent, chicken-egg playback compatibility), it just doesn't make sense to (for instance) extract the burned-in stylized subtitles and bake them into the codec as text+font data, as opposed to just merging them into the film as pixels and calling it a day. Fun fact: nearly every Pixar Blu-Ray is split into multiple forking playback paths for different languages, such that if you watch it in French, any scenes with diegetic text (newspapers, signs on buildings) are re-rendered in French. Obviously that's hugely inefficient; yet at 50GB, there's storage to spare, so why not? The end result is a nice touch and a seamless experience.
      - dfox 6y ago
        MPEG 4 (taken as a the whole body of standards, not as two particular video codecs) actually has provisions for text content, vector video layers and even rudimentary 3D objects. On the other hand I'm almost sure that there are no practical implementations of any of that.
        giantrobot 6y ago
        Don't forget Java bytecode tracks!
        duskwuff 6y ago
        Oh, and that's only the beginning. The MPEG-4 standard also includes some pretty wacky kitchen-sink features like animated human faces and bodies (defined in MPEG-4 part 2 as "FBA objects"), and an XML format for representing musical notation (MPEG-4 part 23, SMR).
    - blattimwind 6y ago
      x264 is kinda absurdly good at compressing screencasts, even a nearly lossless 1440p screencast will only have about 1 Mbit/s on average. The only artifacts I can see are due to 4:2:0 chroma subsampling (i.e. color bleed on single-pixel borders and such), but that has nothing to do with the encoder, and would almost certainly not happen in 4:4:4, which is supported by essentially nothing as far as distribution goes.
    - adrianmonk 6y ago
      Recognizing text and using it to increase compression ratios is possible. I believe that's what this 1974 paper is about: https://www.semanticscholar.org/paper/A-Means-for-Achieving-a-High-Degree-of-Compaction-Ascher-Nagy/83409db15d39f2d45e8ec16ff2a74e35ceabe433#extracted https://www.semanticscholar.org/paper/A-Means-for-Achieving-...
    - imhoguy 6y ago
      Why not use deep learning to recognize actor face patterns in scenes and build entire movies from AI models?
- adrianmonk 6y ago
  I'm not super strong on theory, but if I'm not mistaken, doesn't Kolmogorov complexity (https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity) say we can't even know if it is all figured out? The way I understand it is that one way to compress a document would be to store a computer program and, at the decompression stage, interpret the program so that running it outputs the original data. So suppose you have a program of some size that produces the correct output, and you want to know if a smaller-sized program can also. You examine one of the possible smaller-sized programs, and you observe that it is running a long time. Is it going to halt, or is it going to produce the desired output? To answer that (generally), you have to solve the halting problem. (This applies to lossless compression, but maybe the idea could be extended to lossy as well.)
  - corty 6y ago
    I really ain't a theorist either, but: If you are looking at Kolmogorov complexity you are right, we can't ever know. But Kolmogorov complexity is about single points in the space of possible outputs. It basically says "there might be possible outputs that do look random, but are actually produced by a very short encoding". One example would be the digits of pi. But if you look at the overall statistics of possible output streams, and at their averages, there is a lower bound for compression on average. As soon as the bitlength in the compressed stream matches the entropy in the uncompressed stream in bits, you reached maximum compression. There will be some streams that don't conform to those statistics, but their averages will. However, we are somewhat far away from matched entropy equilibrium for video compression. And even then, improvements can be made, not in compression ratio but in time, ops and energy needed for de/encoding.
sp332 6y ago
I don't know much about H.266, but some of the advances in H.265 depended on players having enough RAM to hold a bunch of previous decoded frames, so they could be referred to by later compressed data. Newer codecs tend to have a lot more options for the encoder to tune, so they need a combination of faster CPUs and smarter heuristics to explore the space of possible encodings quickly.
- pedrovhb 6y ago
  I wonder if instead of heuristics, machine learning could be used to figure out the best parameters.
  - DavidVoid 6y ago
    That seems likely at least. You could actually use ML for all of the video decoding, but that research is still in it's early stages. It has been done rather well with still images [1], so I'm sure it'll eventually be done with video too. Those ML techniques are still a little slow and require large networks (the one in [1] decodes to PNG at 0.7 megapixels/s and its network is 726MB) so more optimizations will be needed before they can see any real-world use. [1] https://hific.github.io/ https://hific.github.io/ HN thread: https://news.ycombinator.com/item?id=23652753 https://news.ycombinator.com/item?id=23652753
  - giantrobot 6y ago
    That's already done today. Most modern codecs support variable bitrate encoding so more of the data budget can be given to high complexity scenes. Source video can also be processed multiple times with varied parameters and that output then compared structurally to the source to find the parameters that best encode the scene. This is beyond the more typical multi pass encoding where a first pass over the source just provides some encoding hints for the second pass. It takes several times longer to encode (though is embarrassingly parallel) but the output ends up higher quality for a given data rate than a more naïve approach.
    - social_quotient 6y ago
dahfizz 6y ago
Here is a hint: > Because H.266/VVC was developed with ultra-high-resolution video content in mind, the new standard is particularly beneficial when streaming 4K or 8K videos on a flat screen TV. Compressing video is very different from gzipping a file. It's more about human perception than algorithms, really. The question is "what data can we delete without people noticing?", and it makes sense that answer is different for an 8k video than a 480p video.
- sparkling 6y ago
  So compressing a 1080p video with H266 will not result in similar file size/quality improvements as a 4k video? How much are we looking at for 1080p, 10%?
  - hexmiles 6y ago
    not an expert, but from what i understand is more like that they "extend" the codec with technique more effective on higher resolution content, or new "profile" (parameter) more effective for higher resolution content (a bit like you can have different parameter when you zip file). This new techniques can also be used for 1080p video (for example) but with lower gain. Also the "old" algorithm/system are generally still used but they may be improved/extend.
  - RBURBURXEIVG3KO 6y ago
    I think that for 1080p there wasn't much of an improvement between h.264 and h.265. This will likely also be true between h.265 and h.266.
  - jitendrac 6y ago
    yup, from the example I remember(I read it through link on HN but cannot find it in quick search, I wish I could link it), if you film(still film no movement) the macbook pro top to bottom in h264/MP4 at 1024 p resolution and again you take a picture from your camera. the results will be shocking, the video of 5-10 seconds will have lower storage size than the size of single Image. but when you inspect video carefully you will see the tiny details are missing, like the edges are not as sharp for metal body, the gloss of metal is bit different, the tiny speaker holes on top of keyboard are clear in Images and can be individually examined while in video they are fuzzy pixelated and so on. so, The end results: A 5 second video with 10s of frames per second is smaller in size than single Image taken from same camera.
k_bx 6y ago
From what I've heard (would love to hear more expertise on this), it's incredibly hard to invent a new video compression algorithm without breaking an existing set of patents, and there's also no easy way to even know whether you're breaking anything as you develop the algo. Thus the situation we're in is not that it's too hard to develop better codecs, but that you've very disincentivized to do so.
- zanny 6y ago
  Which then begs the question - why are video compression standards developed in the US at all? MPEG is obviously US based but Xiph is also a US nonprofit. The software patents should be hugely crippling the ability for Americans to develop competitive video codecs when every other nation doesn't have such nonsense. Why hasn't Europe invested in and developed better codecs that combine the techniques impossible to mix in the states? Is it just basically the same mechanism that leads to so much drug development happening in the US despite how backwards its medical system is, because those regressive institutions create profit incentives not available elsewhere (to develop drugs or video codecs for profit) and thus the US already has capitalists throwing money at what could be profitable whereas everyone else would look at it as an investment cost for basically research infrastructure.
  - not2b 6y ago
    MPEG is not US-based. https://en.wikipedia.org/wiki/Moving_Picture_Experts_Group https://en.wikipedia.org/wiki/Moving_Picture_Experts_Group The article we are all commenting on is by a German research organization that has been a major contributor to video coding standards. Perhaps you're confused by the patent issue? European companies are happy to file for US patents and collect the money.
  - k_bx 6y ago
    This sounds like a somewhat obvious way to side-step the patent mechanism, so I would assume patents prevent this kind of a thing, when you develop patent-breaking technology abroad and then "just use" it wherever you want. You're probably not allowed to use the patented technology in any of the products you're building.
sly010 6y ago
It's about the assumptions made during the standardization. Compared to 30 years ago, we now have better knowledge and statistics about what low level primitives are useful in a codec. E.g. jpeg operates on fixed 8×8 blocks independently, which makes it less efficient for very large images than a codec with variable block size. But variable block size adds overhead for very small images. An other reason can be common hardware. As hardware evolves, different hardware accelerated encoding/decoding techniques become feasible that gets folded into the new standards.
- pault 6y ago
  Something that I learned about 10 years ago when bandwidth was still expensive is that you can make a very large version of an image, set the jpeg compression to a ridiculously high value, then scale the image down in the browser. The artifacts aren't as noticeable when the image is scaled down, and the file size is actually smaller than what it would be if you encoded a smaller image with less compression.
  - MaxBarraclough 6y ago
    Interesting. I've never heard of this, but it makes some sense: the point of lossy image compression is to provide a better quality/size ratio than downscaling.
  - tatersolid 6y ago
    This “huge JPEG at low quality” technique has been widely known for years. But it is typically avoided by larger sites and CDNs, as it requires a lot more memory and processing on the client. Depending on the client or the number of images on the site the huge JPEG could be a crippling performance issue, or even a “site doesn’t work at all” issue.
- Dylan16807 6y ago
  > But variable block size adds overhead for very small images. What kind of overhead? The extra code shouldn't make a big difference if nothing is running it. And space-wise, it should only cost a few bits to say that the entire image is using the smallest block size. Is the worry just about complicating hardware decoders?
linuxftw 6y ago
The way I read the release was that it's not a lossless compression, it reads like it's downscaling 4k+ video to a lower format with 'no perceptible loss of quality.' Since this is also seemingly targeted at mobile, I'm guessing the lack of perceptible loss of quality is a direct function of screen size and pixel density on a smaller mobile devices. For me, this is another pointless advance in video technology. 720p or 1080p is fantastic video resolution, especially on a mobile phone. Less than 1% of the population cares or wants higher resolution. What new technologies are doing now is re-setting the patent expiration clock. As long as new video tech comes out every 5-10 years, HW manufacturers get to sell new chips, phone manufacturers get to sell new phones, TV manufacturers get to sell new TVs, rinse, repeat.
- Sohcahtoa82 6y ago
  > 720p or 1080p is fantastic video resolution, especially on a mobile phone. 720p is far from fantastic. It's noticeably blurry, even on mobile. 1080p is minimally acceptable, and is now over 10 years old. > Less than 1% of the population cares or wants higher resolution. That's a very bold claim. Have any studies or polls to back that up?
  - linuxftw 6y ago
    > That's a very bold claim. Have any studies or polls to back that up? From 1: > In Japan, only 14 percent of households will own a 4K TV in 2019 because most households already have relatively new TVs, IHS said. Let's dissect the reasoning. Most households already have a relatively new TV, thus a low adoption rate. Implying that needing a new TV generally, not the desire to upgrade resolution, is the primary motivating factor in purchasing a TV. In fact, most 4K TVs are already as cheap as the rest of the market. I truly believe that almost everyone does not care about 4K whatsoever. In fact, even if they do 'care' it's not because they know what they're talking about. Most of the enhancements that 4K TVs bring are an artifact of better display technology, rather than increased resolution. See 2. Streaming 4K+ video is a waste of resources with no tangible benefit to anyone other than marketing purposes. Netflix streams '4K' because everyone has a '4K' TV now and they demand it. [1]: https://www.twice.com/research/us-4k-tv-penetration-hit-35-2019-59741 https://www.twice.com/research/us-4k-tv-penetration-hit-35-2... [2]: https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stupid/ https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stup...
masklinn 6y ago
> It's interesting that they are able to continue improving video compression. You'd think that it would have all been figured out by now. Would you? AV1 was only officially released 2 years ago, h.265 7, h.264 14, …
ksec 6y ago
>Like, why couldn't have H.266 been invented 30 years ago? It is all a matter of trade offs and engineering. For MPEG / H.26x codec, the committees start the project by asking or defining the encoding and decoding complexities. And if you only read Reddit or HN, most of the comment's world view would be Video codec are only for Internet Video and completely disregard other video delivering platform. Which all have their own trade off and limitations. There is also cost in decoding silicon die size and power usage. If more video are being consumed on Mobile and battery is a limitation, can you expect hardware decoding energy usage to be within the previous codec? Does it scale with adding more transistors, are there Amdahl's law somewhere. etc It is easy to just say adding more transistor, but ultimately there is a cost to hardware vendors. Vast majority of the Internet seems to think most people working on MPEG Video Codec are patents trolls and idiots and paid little to no respect to its engineering. When as a matter of fact Video Codec are thousands of small tools within the spec, and pretty much insane amount of trial and error. It may not be as complicated as 3GPP / 5G level of complexity, but it is still lot of work. Getting something to compress better while doing it efficiently is hard. And as Moore's Law is slowing down. No one can continue to just throwing transistors at the problem.
giantrobot 6y ago
Video compression is a calculus of IO capacity, memory, and algorithmic complexity. Take the MPEG-1 codec for instance, it was new about 30 years ago. While today most people think of MPEG-1 videos as low quality the spec provides the ability to handle bit rates up to 100Mb/s and resolutions up to 4095x4095. That was way higher than the hardware of the time supported. One of MPEG-1's design goals was to get VHS-quality video at a bitrate that could stream over T1/E1 lines or 1x CD-ROMs. The limit on bitrate led to increased algorithmic complexity. It was well into the Pentium/PowerPC era until desktop systems could play back VCD quality MPEG-1 video in software. Later MPEG codecs increased their algorithmic complexity to squeeze better quality video into low bit rates. A lot of those features existed on paper 20-30 years ago but weren't practical on hardware of the time, even custom ASICs. Even within a spec features are bound to profiles so a file/stream can be handled by less capable decoders/hardware. There's plenty of video codecs or settings for them that can choke modern hardware. It also depends on what you mean by "modern hardware". There's codecs/configurations a Threadripper with 64GB of RAM in a mains powered jet engine sounding desktop could handle in software that would kill a Snapdragon with 6GB of RAM in a phone. There's also codecs/configurations the Snapdragon in the phone could play using hardware acceleration that would choke a low powered Celeron or Atom decoding in software.
- cellularmitosis 6y ago
  I wonder why all of the MPEG1 encoders of the day enforced a maximum of 320x240?
  - giantrobot 6y ago
    While the spec allowed for outrageous settings playback wouldn't have been possible. Most hardware decoders were meant for (or derived from) VCD playback. The VCD spec covered CIF video which mean QVGA would fall into the supported macroblock rate for hardware decoders. In MPEG-1's heyday there would haven't been a lot of point in encoding presets producing content common hardware decoders couldn't handle. There were several other video codecs in the same era that didn't have hardware decode requirements. Cinepak was widely used and could be readily played on a 68030, 486, and even CD-ROM game consoles. As I recall Cinepak encoders had more knobs and dials since the output didn't need to hit a hardware decoder limitation.
Ace17 6y ago
Ten-year software video compression engineer here: TL;DR: it's partly because we're using higher video resolutions. A non-negligible part of the improvement stems from adapting existing algorithms to the now-doubled-resolution. Almost all video compression standards split the input frame into fixed-size square blocks, aka "macroblocks". To put it simply, the macroblock is the coarsest granularity level at which compression happens. - H.264 and MPEG-2 Video use 16x16 macroblocks (ignoring MBAFF). - H.265 use configurable quad-tree-like macroblocks, with a frame-level configurable size up to 64x64. - AV1 makes this block-size configurable up to 128x128. Which means: Compression to H.264 a SD video (720x576, used by DVDs) results in 1620 macroblocks/frame. Compressing to H.265 a HD video (1920x1080) results in at least 506 macroblocks/frame. Compressing to AV1 a 4K video (3840x2160) results in at least 506 macroblocks/frame. But compressing to H.264 a 4K video (3840x2160) will result in 32400 macroblocks/frame. The problem is, there are constant bitcosts per-macroblock ((mostly) regardless of the input picture). So using H.264 to compress 4K video will be inefficient. When you take an old compression standard to encode recent-resolution content, you're using the compression standard outside of the resolution domain for which it was optimized. > Is this continued improvement related to the improvement of technology? Or just coincidental? Of course, there also "real" improvements (in the sense "qualitative improvements that would have benefited to compression old video resolutions, if only we had invented them sooner"). For example: - the context-adaptive arithmetic coding from H.264, which is a net improvement over classic variable-length huffman coding used by MPEG-2 (and H.264 baseline profile). - the entropy coding used by AV1, which is a net improvement over H.264's CABAC. - integer DCT (introduced by H.264), which allow bit-accuracy checking and way lot easier and smaller hardware implementations (compared to floating point DCT that is used by MPEG2). - loop filters: H.264 pioneered the idea of a normative post-processing step, whose output could be used to predict next frames. H.264 had 1 loop filter ("deblocking"). HEVC had 2 loop filters: "deblocking" and "SAO". AV1 has 4 loop filters. All of these a real improvements, brought to us by time, and extremely clever and dedicated people. However, the compression gains of these improvements are nowhere near the "50% less bitrate" that is used to sell each new advanced-high-efficiency-versatile-nextgen video codec. Without increasing - a lot - the frame resolution, selling a new video compression standard will be a lot harder. Besides, now that the resolutions seems to have settled up around 4K/8K (and that "high definition" has become the lowest resolution we might have to deal with :D), things are going to get interesting ... provided that we don't start playing the same game with framerates!
nwallin 6y ago
Computers wouldn't have been fast enough. Moore's law is a hell of a drug. In the mid '90s, PCs often weren't fast enough to decode DVDs, which were typically 720x480 24FPS MPEG2. DVD drives were often shipped with accelerator cards that decoded MPEG2 in hardware. I had one. My netbook is many orders of magnitude faster than my old Pentium Pro. But it's not fast enough to decode 1080p 30fps H.265 or VP9 in software. It must decode VP9/H.265 on the GPU or not at all. MPEG2 is trivial to decode by comparison. I would expect a typical desktop PC of the mid '90s to take seconds to decode a frame of H.265, if it even had enough RAM to be able to do it at all. It's an engineering tradeoff between compression efficiency of the codec and the price of the hardware which is required to execute it. If a chip which is capable of decoding the old standard costs $8, and a chip which is capable of decoding the new standard costs $9, sure, the new standard will get lots of adoption. But if a chip which is capable of decoding the new standard costs $90, lots of vendors will balk.
- savoytruffle 6y ago
  Indeed. The brand-new fancy Blue & White Power Mac G3's from early 1999 were the first Macs that shipped with a DVD drive, and they could play video DVD's but they had an obvious (and strange) additional decoder mezzanine card on the already unusual Rage128 PCI video card. By the end of that year the G4 Power Macs were just barely fast enough to play DVD's with software decoding and assistance from the PCI or later AGP video card. And after a while (perhaps ~ 2002?), even the Blue G3's could do it in software even if you got a different video card, as long as you also upgraded to a G4 CPU (they were all in ZIF sockets). It was very taxing on computers at y2k! Later autumn 2000 G3 iMacs could also play DVD's but I think they needed special help from a video co-processor.
joveian 6y ago
Good compression is quite complex and can go wrong in an unimaginable variety of ways. Remember when Xerox copiers would copy numbers incorrectly due to compression? The numbers would look clear, they just wouldn't always be the same numbers that you started with. https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
- rasz 6y ago
  Xerox problem stemmed from simple replacement of "recognized" numbers with learned dictionary. Good implementation would use learned alphabet atlas as a supplement, encoding difference between guess and source image. That way even predicted 0 instead of 8 wouldnt be catastrophic with encoder filling in missing detail.
johnvanommen 6y ago
> Like, why couldn't have H.266 been invented 30 years ago? Is it because the computers back in the day wouldn't have been fast enough to realistically use it? Here's something to consider: In 1995, a typical video stream was 640 x 480 x 24fps. That's 7,372,800 pixels per second. In 2020, we have some content that's 7680 x 4320 x 120fps. That's 3,981,312,000 pixels per second, or a 540 fold increase in 25 years. The massive increase in image size actually makes it easier to use high compression ratios. I found this out the hard way recently, when I was trying to compress and email a powerpoint presentation that a coworker had presented on video. In a nutshell, the powerpoint doc with it's sharp edges and it's low resolution made it difficult to compress. Increase framerate plays a factor too; due to decades of research on motion interpolation, algorithms have become quite good at guessing what content can be eliminated from a moving stream.
marta_morena_25 6y ago
"You'd think that it would have all been figured out by now." Would you? Video compression is one of the few things that we will work on for the next 1000 years and still be nowhere near finished. The best video compression would be to know the state of the universe at the big bang, have a timestamp of the beginning and end of your clip and spatial coordinates defining your viewport. Then some futuristic quantum computer would just simulate the content of your clip... So yeah, sure we are done with video compression :). This is of course an extreme example of constant time compression that may or may not be ever feasible (if we live in a computer simulation of an alien race, then it is already happening). But the gist is the same. Video compression is mostly about inferring the world and computing movement not by storing the content of the image. For instance by taking a snapshot of the world, decomposing it into geometric shapes (pretty much the opposite of 3D rendering) and then computing the next frames by morphing these shapes + some diff data that snaps these approximations back in line with the actual data. We are all but in the very infancy of video compression. What should surprise you is why it takes us so long to get anywhere.
Slamidan 6y ago
I still remember when my pc would take 24 hours to compress a dvd to a high quality h264 mkv, sure you could squeeze it down with fast presets in handbrake but the point was transparency. Now I'm sure for most normal pc's the time to compress at the same quality with h.265 is the same 24 hours, in 4k, even longer, I'm sure h266 would take more than twice as long easily. Early pc's had separate and very expensive mpeg decode boards just to decode dvd, creative sold a set, the cpu simply couldn't even handle mpeg 2. I know its hard to believe but there was a time when playing back an mp3 was a big ask, all these algorithms could be made long ago, but they would have been impractical fantasy. Only now are we seeing real partial cheated resolution ray tracing in modern high end gaming hardware now which is a good comparison, ray tracing has been with us for a long time, only hardware advancement over decades has made it viable. It amused me that they claimed 4k uhd h265 is now 10GB for a movie, that's garbage bitrate, they always ask too much of these codecs.
- samoa42 6y ago
  > I know its hard to believe but there was a time when playing back an mp3 was a big ask can confirm. audio playback would stutter on my 486dx if one dared to multitask.

aey 6y ago

Compression is AI. It’s never going to be “all” figured out.

0-_-0 6y ago
Another way if saying it is that compression is understanding.
- jacobush 6y ago
  Lossy compression is, I feel compelled to add.
  - aey 6y ago
    Actually both! Arithmetic coding works over any kind of predictor.
- milansuk 6y ago
  End-credits are just text. So it should be possible to put it through OCR and save only text, positions, and fonts. And the text is also possible to compress with a dictionary.
  - gsich 6y ago
    True, but end credits take very little space compared to the rest of the movie.
  - jaywalk 6y ago
    The text, positions and fonts could very well take up more space than the compressed video. And then with fonts, you have licensing issues as well.
  - lukifer 6y ago
    Credits also contain logos/symbols (near the end), and often have stylistic flairs as well. Video compression is based on making predictions and then adding information (per Shannon's definition) for the deltas from those predictions. The pattern of credits statically sliding at a consistent rate is exactly the sort of prediction codecs are optimized for; for instance, the same algorithms will save space by predicting repeated pixel patterns during a slow camera pan. Still, I've often thought it would be nice if text were a more first-class citizen within video codecs. I think it's more a toolchain/workflow problem than a shortcoming in video compression technology as such. Whoever is mastering a Blu-Ray or prepping a Hollywood film for Netflix is usually not the same person cutting and assembling the original content. For innumerable reasons (access to raw sources, low return on time spent, chicken-egg playback compatibility), it just doesn't make sense to (for instance) extract the burned-in stylized subtitles and bake them into the codec as text+font data, as opposed to just merging them into the film as pixels and calling it a day. Fun fact: nearly every Pixar Blu-Ray is split into multiple forking playback paths for different languages, such that if you watch it in French, any scenes with diegetic text (newspapers, signs on buildings) are re-rendered in French. Obviously that's hugely inefficient; yet at 50GB, there's storage to spare, so why not? The end result is a nice touch and a seamless experience.
    - dfox 6y ago
      MPEG 4 (taken as a the whole body of standards, not as two particular video codecs) actually has provisions for text content, vector video layers and even rudimentary 3D objects. On the other hand I'm almost sure that there are no practical implementations of any of that.
      - giantrobot 6y ago
        Don't forget Java bytecode tracks!
      - duskwuff 6y ago
        Oh, and that's only the beginning. The MPEG-4 standard also includes some pretty wacky kitchen-sink features like animated human faces and bodies (defined in MPEG-4 part 2 as "FBA objects"), and an XML format for representing musical notation (MPEG-4 part 23, SMR).
  - blattimwind 6y ago
    x264 is kinda absurdly good at compressing screencasts, even a nearly lossless 1440p screencast will only have about 1 Mbit/s on average. The only artifacts I can see are due to 4:2:0 chroma subsampling (i.e. color bleed on single-pixel borders and such), but that has nothing to do with the encoder, and would almost certainly not happen in 4:4:4, which is supported by essentially nothing as far as distribution goes.
  - adrianmonk 6y ago
    Recognizing text and using it to increase compression ratios is possible. I believe that's what this 1974 paper is about: https://www.semanticscholar.org/paper/A-Means-for-Achieving-a-High-Degree-of-Compaction-Ascher-Nagy/83409db15d39f2d45e8ec16ff2a74e35ceabe433#extracted https://www.semanticscholar.org/paper/A-Means-for-Achieving-...
  - imhoguy 6y ago
    Why not use deep learning to recognize actor face patterns in scenes and build entire movies from AI models?
adrianmonk 6y ago
I'm not super strong on theory, but if I'm not mistaken, doesn't Kolmogorov complexity (https://en.wikipedia.org/wiki/Kolmogorov_complexity https://en.wikipedia.org/wiki/Kolmogorov_complexity) say we can't even know if it is all figured out? The way I understand it is that one way to compress a document would be to store a computer program and, at the decompression stage, interpret the program so that running it outputs the original data. So suppose you have a program of some size that produces the correct output, and you want to know if a smaller-sized program can also. You examine one of the possible smaller-sized programs, and you observe that it is running a long time. Is it going to halt, or is it going to produce the desired output? To answer that (generally), you have to solve the halting problem. (This applies to lossless compression, but maybe the idea could be extended to lossy as well.)
- corty 6y ago
  I really ain't a theorist either, but: If you are looking at Kolmogorov complexity you are right, we can't ever know. But Kolmogorov complexity is about single points in the space of possible outputs. It basically says "there might be possible outputs that do look random, but are actually produced by a very short encoding". One example would be the digits of pi. But if you look at the overall statistics of possible output streams, and at their averages, there is a lower bound for compression on average. As soon as the bitlength in the compressed stream matches the entropy in the uncompressed stream in bits, you reached maximum compression. There will be some streams that don't conform to those statistics, but their averages will. However, we are somewhat far away from matched entropy equilibrium for video compression. And even then, improvements can be made, not in compression ratio but in time, ops and energy needed for de/encoding.

sp332 6y ago

I don't know much about H.266, but some of the advances in H.265 depended on players having enough RAM to hold a bunch of previous decoded frames, so they could be referred to by later compressed data. Newer codecs tend to have a lot more options for the encoder to tune, so they need a combination of faster CPUs and smarter heuristics to explore the space of possible encodings quickly.

pedrovhb 6y ago
I wonder if instead of heuristics, machine learning could be used to figure out the best parameters.
- DavidVoid 6y ago
  That seems likely at least. You could actually use ML for all of the video decoding, but that research is still in it's early stages. It has been done rather well with still images [1], so I'm sure it'll eventually be done with video too. Those ML techniques are still a little slow and require large networks (the one in [1] decodes to PNG at 0.7 megapixels/s and its network is 726MB) so more optimizations will be needed before they can see any real-world use. [1] https://hific.github.io/ https://hific.github.io/ HN thread: https://news.ycombinator.com/item?id=23652753 https://news.ycombinator.com/item?id=23652753
- giantrobot 6y ago
  That's already done today. Most modern codecs support variable bitrate encoding so more of the data budget can be given to high complexity scenes. Source video can also be processed multiple times with varied parameters and that output then compared structurally to the source to find the parameters that best encode the scene. This is beyond the more typical multi pass encoding where a first pass over the source just provides some encoding hints for the second pass. It takes several times longer to encode (though is embarrassingly parallel) but the output ends up higher quality for a given data rate than a more naïve approach.
  - social_quotient 6y ago

dahfizz 6y ago

Here is a hint: > Because H.266/VVC was developed with ultra-high-resolution video content in mind, the new standard is particularly beneficial when streaming 4K or 8K videos on a flat screen TV. Compressing video is very different from gzipping a file. It's more about human perception than algorithms, really. The question is "what data can we delete without people noticing?", and it makes sense that answer is different for an 8k video than a 480p video.

sparkling 6y ago
So compressing a 1080p video with H266 will not result in similar file size/quality improvements as a 4k video? How much are we looking at for 1080p, 10%?
- hexmiles 6y ago
  not an expert, but from what i understand is more like that they "extend" the codec with technique more effective on higher resolution content, or new "profile" (parameter) more effective for higher resolution content (a bit like you can have different parameter when you zip file). This new techniques can also be used for 1080p video (for example) but with lower gain. Also the "old" algorithm/system are generally still used but they may be improved/extend.
- RBURBURXEIVG3KO 6y ago
  I think that for 1080p there wasn't much of an improvement between h.264 and h.265. This will likely also be true between h.265 and h.266.
- jitendrac 6y ago
  yup, from the example I remember(I read it through link on HN but cannot find it in quick search, I wish I could link it), if you film(still film no movement) the macbook pro top to bottom in h264/MP4 at 1024 p resolution and again you take a picture from your camera. the results will be shocking, the video of 5-10 seconds will have lower storage size than the size of single Image. but when you inspect video carefully you will see the tiny details are missing, like the edges are not as sharp for metal body, the gloss of metal is bit different, the tiny speaker holes on top of keyboard are clear in Images and can be individually examined while in video they are fuzzy pixelated and so on. so, The end results: A 5 second video with 10s of frames per second is smaller in size than single Image taken from same camera.

k_bx 6y ago

From what I've heard (would love to hear more expertise on this), it's incredibly hard to invent a new video compression algorithm without breaking an existing set of patents, and there's also no easy way to even know whether you're breaking anything as you develop the algo. Thus the situation we're in is not that it's too hard to develop better codecs, but that you've very disincentivized to do so.

zanny 6y ago
Which then begs the question - why are video compression standards developed in the US at all? MPEG is obviously US based but Xiph is also a US nonprofit. The software patents should be hugely crippling the ability for Americans to develop competitive video codecs when every other nation doesn't have such nonsense. Why hasn't Europe invested in and developed better codecs that combine the techniques impossible to mix in the states? Is it just basically the same mechanism that leads to so much drug development happening in the US despite how backwards its medical system is, because those regressive institutions create profit incentives not available elsewhere (to develop drugs or video codecs for profit) and thus the US already has capitalists throwing money at what could be profitable whereas everyone else would look at it as an investment cost for basically research infrastructure.
- not2b 6y ago
  MPEG is not US-based. https://en.wikipedia.org/wiki/Moving_Picture_Experts_Group https://en.wikipedia.org/wiki/Moving_Picture_Experts_Group The article we are all commenting on is by a German research organization that has been a major contributor to video coding standards. Perhaps you're confused by the patent issue? European companies are happy to file for US patents and collect the money.
- k_bx 6y ago
  This sounds like a somewhat obvious way to side-step the patent mechanism, so I would assume patents prevent this kind of a thing, when you develop patent-breaking technology abroad and then "just use" it wherever you want. You're probably not allowed to use the patented technology in any of the products you're building.

sly010 6y ago

It's about the assumptions made during the standardization. Compared to 30 years ago, we now have better knowledge and statistics about what low level primitives are useful in a codec. E.g. jpeg operates on fixed 8×8 blocks independently, which makes it less efficient for very large images than a codec with variable block size. But variable block size adds overhead for very small images. An other reason can be common hardware. As hardware evolves, different hardware accelerated encoding/decoding techniques become feasible that gets folded into the new standards.

pault 6y ago
Something that I learned about 10 years ago when bandwidth was still expensive is that you can make a very large version of an image, set the jpeg compression to a ridiculously high value, then scale the image down in the browser. The artifacts aren't as noticeable when the image is scaled down, and the file size is actually smaller than what it would be if you encoded a smaller image with less compression.
- MaxBarraclough 6y ago
  Interesting. I've never heard of this, but it makes some sense: the point of lossy image compression is to provide a better quality/size ratio than downscaling.
- tatersolid 6y ago
  This “huge JPEG at low quality” technique has been widely known for years. But it is typically avoided by larger sites and CDNs, as it requires a lot more memory and processing on the client. Depending on the client or the number of images on the site the huge JPEG could be a crippling performance issue, or even a “site doesn’t work at all” issue.
Dylan16807 6y ago
> But variable block size adds overhead for very small images. What kind of overhead? The extra code shouldn't make a big difference if nothing is running it. And space-wise, it should only cost a few bits to say that the entire image is using the smallest block size. Is the worry just about complicating hardware decoders?

linuxftw 6y ago

The way I read the release was that it's not a lossless compression, it reads like it's downscaling 4k+ video to a lower format with 'no perceptible loss of quality.' Since this is also seemingly targeted at mobile, I'm guessing the lack of perceptible loss of quality is a direct function of screen size and pixel density on a smaller mobile devices. For me, this is another pointless advance in video technology. 720p or 1080p is fantastic video resolution, especially on a mobile phone. Less than 1% of the population cares or wants higher resolution. What new technologies are doing now is re-setting the patent expiration clock. As long as new video tech comes out every 5-10 years, HW manufacturers get to sell new chips, phone manufacturers get to sell new phones, TV manufacturers get to sell new TVs, rinse, repeat.

Sohcahtoa82 6y ago
> 720p or 1080p is fantastic video resolution, especially on a mobile phone. 720p is far from fantastic. It's noticeably blurry, even on mobile. 1080p is minimally acceptable, and is now over 10 years old. > Less than 1% of the population cares or wants higher resolution. That's a very bold claim. Have any studies or polls to back that up?
- linuxftw 6y ago
  > That's a very bold claim. Have any studies or polls to back that up? From 1: > In Japan, only 14 percent of households will own a 4K TV in 2019 because most households already have relatively new TVs, IHS said. Let's dissect the reasoning. Most households already have a relatively new TV, thus a low adoption rate. Implying that needing a new TV generally, not the desire to upgrade resolution, is the primary motivating factor in purchasing a TV. In fact, most 4K TVs are already as cheap as the rest of the market. I truly believe that almost everyone does not care about 4K whatsoever. In fact, even if they do 'care' it's not because they know what they're talking about. Most of the enhancements that 4K TVs bring are an artifact of better display technology, rather than increased resolution. See 2. Streaming 4K+ video is a waste of resources with no tangible benefit to anyone other than marketing purposes. Netflix streams '4K' because everyone has a '4K' TV now and they demand it. [1]: https://www.twice.com/research/us-4k-tv-penetration-hit-35-2019-59741 https://www.twice.com/research/us-4k-tv-penetration-hit-35-2... [2]: https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stupid/ https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stup...

masklinn 6y ago

> It's interesting that they are able to continue improving video compression. You'd think that it would have all been figured out by now. Would you? AV1 was only officially released 2 years ago, h.265 7, h.264 14, …

ksec 6y ago

>Like, why couldn't have H.266 been invented 30 years ago? It is all a matter of trade offs and engineering. For MPEG / H.26x codec, the committees start the project by asking or defining the encoding and decoding complexities. And if you only read Reddit or HN, most of the comment's world view would be Video codec are only for Internet Video and completely disregard other video delivering platform. Which all have their own trade off and limitations. There is also cost in decoding silicon die size and power usage. If more video are being consumed on Mobile and battery is a limitation, can you expect hardware decoding energy usage to be within the previous codec? Does it scale with adding more transistors, are there Amdahl's law somewhere. etc It is easy to just say adding more transistor, but ultimately there is a cost to hardware vendors. Vast majority of the Internet seems to think most people working on MPEG Video Codec are patents trolls and idiots and paid little to no respect to its engineering. When as a matter of fact Video Codec are thousands of small tools within the spec, and pretty much insane amount of trial and error. It may not be as complicated as 3GPP / 5G level of complexity, but it is still lot of work. Getting something to compress better while doing it efficiently is hard. And as Moore's Law is slowing down. No one can continue to just throwing transistors at the problem.

giantrobot 6y ago

Video compression is a calculus of IO capacity, memory, and algorithmic complexity. Take the MPEG-1 codec for instance, it was new about 30 years ago. While today most people think of MPEG-1 videos as low quality the spec provides the ability to handle bit rates up to 100Mb/s and resolutions up to 4095x4095. That was way higher than the hardware of the time supported. One of MPEG-1's design goals was to get VHS-quality video at a bitrate that could stream over T1/E1 lines or 1x CD-ROMs. The limit on bitrate led to increased algorithmic complexity. It was well into the Pentium/PowerPC era until desktop systems could play back VCD quality MPEG-1 video in software. Later MPEG codecs increased their algorithmic complexity to squeeze better quality video into low bit rates. A lot of those features existed on paper 20-30 years ago but weren't practical on hardware of the time, even custom ASICs. Even within a spec features are bound to profiles so a file/stream can be handled by less capable decoders/hardware. There's plenty of video codecs or settings for them that can choke modern hardware. It also depends on what you mean by "modern hardware". There's codecs/configurations a Threadripper with 64GB of RAM in a mains powered jet engine sounding desktop could handle in software that would kill a Snapdragon with 6GB of RAM in a phone. There's also codecs/configurations the Snapdragon in the phone could play using hardware acceleration that would choke a low powered Celeron or Atom decoding in software.

cellularmitosis 6y ago
I wonder why all of the MPEG1 encoders of the day enforced a maximum of 320x240?
- giantrobot 6y ago
  While the spec allowed for outrageous settings playback wouldn't have been possible. Most hardware decoders were meant for (or derived from) VCD playback. The VCD spec covered CIF video which mean QVGA would fall into the supported macroblock rate for hardware decoders. In MPEG-1's heyday there would haven't been a lot of point in encoding presets producing content common hardware decoders couldn't handle. There were several other video codecs in the same era that didn't have hardware decode requirements. Cinepak was widely used and could be readily played on a 68030, 486, and even CD-ROM game consoles. As I recall Cinepak encoders had more knobs and dials since the output didn't need to hit a hardware decoder limitation.

Ace17 6y ago

Ten-year software video compression engineer here: TL;DR: it's partly because we're using higher video resolutions. A non-negligible part of the improvement stems from adapting existing algorithms to the now-doubled-resolution. Almost all video compression standards split the input frame into fixed-size square blocks, aka "macroblocks". To put it simply, the macroblock is the coarsest granularity level at which compression happens. - H.264 and MPEG-2 Video use 16x16 macroblocks (ignoring MBAFF). - H.265 use configurable quad-tree-like macroblocks, with a frame-level configurable size up to 64x64. - AV1 makes this block-size configurable up to 128x128. Which means: Compression to H.264 a SD video (720x576, used by DVDs) results in 1620 macroblocks/frame. Compressing to H.265 a HD video (1920x1080) results in at least 506 macroblocks/frame. Compressing to AV1 a 4K video (3840x2160) results in at least 506 macroblocks/frame. But compressing to H.264 a 4K video (3840x2160) will result in 32400 macroblocks/frame. The problem is, there are constant bitcosts per-macroblock ((mostly) regardless of the input picture). So using H.264 to compress 4K video will be inefficient. When you take an old compression standard to encode recent-resolution content, you're using the compression standard outside of the resolution domain for which it was optimized. > Is this continued improvement related to the improvement of technology? Or just coincidental? Of course, there also "real" improvements (in the sense "qualitative improvements that would have benefited to compression old video resolutions, if only we had invented them sooner"). For example: - the context-adaptive arithmetic coding from H.264, which is a net improvement over classic variable-length huffman coding used by MPEG-2 (and H.264 baseline profile). - the entropy coding used by AV1, which is a net improvement over H.264's CABAC. - integer DCT (introduced by H.264), which allow bit-accuracy checking and way lot easier and smaller hardware implementations (compared to floating point DCT that is used by MPEG2). - loop filters: H.264 pioneered the idea of a normative post-processing step, whose output could be used to predict next frames. H.264 had 1 loop filter ("deblocking"). HEVC had 2 loop filters: "deblocking" and "SAO". AV1 has 4 loop filters. All of these a real improvements, brought to us by time, and extremely clever and dedicated people. However, the compression gains of these improvements are nowhere near the "50% less bitrate" that is used to sell each new advanced-high-efficiency-versatile-nextgen video codec. Without increasing - a lot - the frame resolution, selling a new video compression standard will be a lot harder. Besides, now that the resolutions seems to have settled up around 4K/8K (and that "high definition" has become the lowest resolution we might have to deal with :D), things are going to get interesting ... provided that we don't start playing the same game with framerates!

nwallin 6y ago

Computers wouldn't have been fast enough. Moore's law is a hell of a drug. In the mid '90s, PCs often weren't fast enough to decode DVDs, which were typically 720x480 24FPS MPEG2. DVD drives were often shipped with accelerator cards that decoded MPEG2 in hardware. I had one. My netbook is many orders of magnitude faster than my old Pentium Pro. But it's not fast enough to decode 1080p 30fps H.265 or VP9 in software. It must decode VP9/H.265 on the GPU or not at all. MPEG2 is trivial to decode by comparison. I would expect a typical desktop PC of the mid '90s to take seconds to decode a frame of H.265, if it even had enough RAM to be able to do it at all. It's an engineering tradeoff between compression efficiency of the codec and the price of the hardware which is required to execute it. If a chip which is capable of decoding the old standard costs $8, and a chip which is capable of decoding the new standard costs $9, sure, the new standard will get lots of adoption. But if a chip which is capable of decoding the new standard costs $90, lots of vendors will balk.

savoytruffle 6y ago
Indeed. The brand-new fancy Blue & White Power Mac G3's from early 1999 were the first Macs that shipped with a DVD drive, and they could play video DVD's but they had an obvious (and strange) additional decoder mezzanine card on the already unusual Rage128 PCI video card. By the end of that year the G4 Power Macs were just barely fast enough to play DVD's with software decoding and assistance from the PCI or later AGP video card. And after a while (perhaps ~ 2002?), even the Blue G3's could do it in software even if you got a different video card, as long as you also upgraded to a G4 CPU (they were all in ZIF sockets). It was very taxing on computers at y2k! Later autumn 2000 G3 iMacs could also play DVD's but I think they needed special help from a video co-processor.

joveian 6y ago

Good compression is quite complex and can go wrong in an unimaginable variety of ways. Remember when Xerox copiers would copy numbers incorrectly due to compression? The numbers would look clear, they just wouldn't always be the same numbers that you started with. https://www.theregister.com/2013/08/06/xerox_copier_flaw_means_dodgy_numbers_and_dangerous_designs/ https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...

rasz 6y ago
Xerox problem stemmed from simple replacement of "recognized" numbers with learned dictionary. Good implementation would use learned alphabet atlas as a supplement, encoding difference between guess and source image. That way even predicted 0 instead of 8 wouldnt be catastrophic with encoder filling in missing detail.

johnvanommen 6y ago

> Like, why couldn't have H.266 been invented 30 years ago? Is it because the computers back in the day wouldn't have been fast enough to realistically use it? Here's something to consider: In 1995, a typical video stream was 640 x 480 x 24fps. That's 7,372,800 pixels per second. In 2020, we have some content that's 7680 x 4320 x 120fps. That's 3,981,312,000 pixels per second, or a 540 fold increase in 25 years. The massive increase in image size actually makes it easier to use high compression ratios. I found this out the hard way recently, when I was trying to compress and email a powerpoint presentation that a coworker had presented on video. In a nutshell, the powerpoint doc with it's sharp edges and it's low resolution made it difficult to compress. Increase framerate plays a factor too; due to decades of research on motion interpolation, algorithms have become quite good at guessing what content can be eliminated from a moving stream.

marta_morena_25 6y ago

"You'd think that it would have all been figured out by now." Would you? Video compression is one of the few things that we will work on for the next 1000 years and still be nowhere near finished. The best video compression would be to know the state of the universe at the big bang, have a timestamp of the beginning and end of your clip and spatial coordinates defining your viewport. Then some futuristic quantum computer would just simulate the content of your clip... So yeah, sure we are done with video compression :). This is of course an extreme example of constant time compression that may or may not be ever feasible (if we live in a computer simulation of an alien race, then it is already happening). But the gist is the same. Video compression is mostly about inferring the world and computing movement not by storing the content of the image. For instance by taking a snapshot of the world, decomposing it into geometric shapes (pretty much the opposite of 3D rendering) and then computing the next frames by morphing these shapes + some diff data that snaps these approximations back in line with the actual data. We are all but in the very infancy of video compression. What should surprise you is why it takes us so long to get anywhere.

Slamidan 6y ago

I still remember when my pc would take 24 hours to compress a dvd to a high quality h264 mkv, sure you could squeeze it down with fast presets in handbrake but the point was transparency. Now I'm sure for most normal pc's the time to compress at the same quality with h.265 is the same 24 hours, in 4k, even longer, I'm sure h266 would take more than twice as long easily. Early pc's had separate and very expensive mpeg decode boards just to decode dvd, creative sold a set, the cpu simply couldn't even handle mpeg 2. I know its hard to believe but there was a time when playing back an mp3 was a big ask, all these algorithms could be made long ago, but they would have been impractical fantasy. Only now are we seeing real partial cheated resolution ray tracing in modern high end gaming hardware now which is a good comparison, ray tracing has been with us for a long time, only hardware advancement over decades has made it viable. It amused me that they claimed 4k uhd h265 is now 10GB for a movie, that's garbage bitrate, they always ask too much of these codecs.

samoa42 6y ago
> I know its hard to believe but there was a time when playing back an mp3 was a big ask can confirm. audio playback would stutter on my 486dx if one dared to multitask.