8 ms·
A hands-on introduction to video technology: image, video, codec and more
- metaphor 9y agoGot really excited for a second thinking this was discussion on transport technology as opposed to encoding.
- minipci1321 9y agoWhat aspects of the transport technology are you specifically interested in?
- samstave 9y agoUh...assume I know absolutely nothing; can you start with a simple ELI5, and then elaborate/point me off in the right direction with a seed foundation of knowledge?
- minipci1321 9y agoLOL, I'll try. Two major things to grasp about transport technology are the a) "entry point" and b) the notion of time. a) multiple types of media data are encoded independently and then bundled together in what essentially looks like an endless file (called a stream file). So when given a chunk of such a file, the decoder needs to quickly identify the nearest offset in it where it can begin decoding simultaneously all the individual media it needs. This is called "access point". Decoding cannot be started at any random place in the stream, as it generally requires context (so an access point allows to start decoding with the context being empty for all required media -- audio, video, graphics, subtitles etc). Stream file formats (called containers) are designed to solve this, provide access points to the decoder, as easily and frequently as possible. b) a decoder, when driven by a running presentation device -- video screen, audio amplifier etc -- is essentially a pump. The encoder can be looked at like a pump too, when they are separated by network. If decoder runs faster than the source feeds it, it will drain the pipe and will make the presentation device run idle (which will be noticeable to the consumer). If it runs slower, at some point it will be drowned in data from the source. So the pumping rhythm needs to be maintained identical between both ends. The most practical way to synchronize the "piping clocksource" is via the stream file itself (which has to carry time sample data for that). Again, different containers solve this differently (some not at all). EDIT: I didn't mention (should go into b)) the effort to make constant the throughput of the pumping -- "constant bit rate", as I believe with the advent of transport schemes which require point-to-point connections (as opposed to multicast streaming), the importance of this goes lower now.
- profpandit 9y agoRe: b) For video, the encoder contains a model of the decoder, including the amount of buffering available to the decoder. The bit-rate controller at the encoder uses this model to ensure that the decoder always has the right amount of data in its input buffers. It also ensures that the information rate of the channel is matched with that of the compressed stream in a live transmission setting. The transport scheme which operates at a layer below the codec, therefore only needs to take care of delay and packet delivery/loss related issues over the channel. Media is typically transmitted over UDP.
- profpandit 9y agoIt's interesting to note that the architecture of the first ISO codec MPEG (1) is almost identical to the one we have today H.265 That codec was standardised in the late 90s So this design has carried through for about 20 years. Most of the changes relate to the targeted parameters such as frame size, frame rate and bitrate. Only the last step 264 --> 265 seems to have added new features. This is a very well written introduction
- bluedino 9y agoWhere can one read about all the also-ran or proprietary codecs used in the 90's? It truly was the dark ages of digital video with low frame rates and postage stamp sized windows.
- profpandit 9y agoTry the FAQ for comp.compression
- innocenat 9y agoIt is not like it is not tried. Wavelet compression never took off. I don't know if it is because the format of is just better, or there was never enough investment into those formats.
- profpandit 9y agoThe problem with Wavelet based compression was since wavelet transforms were applied globally to the whole frame at a time, while they were suitable for still image compression they couldn't really take advantage of motion compensation so their applicability for video was low. Same with fractal based techniques. Besides, as resolutions got higher and higher the blockiness of the 8x8 DCT became less and less a factor
- userbinator 9y agoYou can still use wavelet compression to encode the residual from MC, but I think the biggest problem is performance: DCTs have been optimised far more than wavelet transforms. Even in still-image compression, the difference is noticeable --- I have some high-resolution PDFs containing JPEG2000 scanned images, and they take significantly longer to render than the equivalent containing JPEG images.
- alextooter 9y agoAmazing work.
- ilzmastr 9y agoThis was also food for whiteboards in the show silicon valley. Compare: https://github.com/leandromoreira/digital_video_introduction/blob/master/i/thor_codec_block_diagram.png https://github.com/leandromoreira/digital_video_introduction... with: http://imgur.com/a/Sne89 http://imgur.com/a/Sne89
- bpicolo 9y agoThose don't really seem similar. Diagrams with boxes and arrows aren't uncommon.
- heydenberk 9y ago"Q" = Quantizer, "EC" = Entropy Coding, etc. It's definitely similar if you look closer.
- microcolonel 9y ago"LZW" -> Lempel–Ziv–Welch on the entropy codes doesn't really make sense though (maybe that's why the word "stupid" is next to it).
- ccommsxx 9y agolooks like this contains a bunch of creative commons (CC-BY-SA) content ripped from wikipedia without proper attribution. please add the missing attribution https://it.wikipedia.org/wiki/File:Pixel_geometry_01_Pengo.jpg https://it.wikipedia.org/wiki/File:Pixel_geometry_01_Pengo.j... https://en.wikipedia.org/wiki/Chroma_subsampling#/media/File:Colorcomp.jpg https://en.wikipedia.org/wiki/Chroma_subsampling#/media/File... etc
- dreampeppers99 9y agoI'm so sorry, I didn't mean to make it wrong, I tried really hard to put all the references in a list https://github.com/leandromoreira/digital_video_introduction#references https://github.com/leandromoreira/digital_video_introduction... ! I'll fix these attributions but also feel free to point me more or even PR.
- heydenberk 9y agoXiph.org wrote fascinating stuff about video compression when working on their next-generation codec, Daala https://xiph.org/daala/ https://xiph.org/daala/
- 0xelectron 9y agoThis is really great. We seriously lacked a good introduction to video technology.
- barrkel 9y agoVideo compression is not understood well enough throughout the whole stack yet. I recently got a 1080p projector for home use, so now movies / TV series in my home are viewed on a 100" screen. Content is mostly from Netflix and Amazon Prime Video. Netflix does a really good job with encoding. I cannot say the same for Amazon Prime Video; even with their exclusive (in UK) offerings, like American Gods or Mr Robot, the quality of the encode is quite poor when viewed on a big screen. Banding, shimmering blocky artifacts on subtle gradients, insufficient bit budget for dimly lit scenes - once you become aware of the issues, it becomes really distracting. OTOH a really big screen is a fantastic ad for high quality high bitrate content. Anything less than 2GB/hour is noticeably poor.
- branchless 9y agoCould you share a link to the projector you bought? And what do you project it onto?
- jrimbault 9y agoA nice comparison I often do: I take a plain black image full #000, and put it side by side with a video.
- jsmthrowaway 9y agoKeep in mind that for historical reasons related to the way analog TVs used to work, black in video is almost never "#000". We call that setup, or superblack. In the analog days NTSC black was 7.5 IRE, not 0, which is why you could always tell when a television was turned on even with nothing on the screen. More than one digital standard kept this custom, setting black at (roughly, in your terms) "#111". So if you're seeing encoding artifacts or grayness on black video, that's more a legacy of video itself and not the encoding work done by whoever made the media. Say I sit down at a switcher and push the Black button. You'd think nothing would come out, right? Not quite: that switcher will almost always emit broadcast black, not "null," whether it's analog or digital. Sure, you can probably get Photoshop to spit out 0% then encode in some codec that lets you keep 0% and successfully display it on a modern digital television, but the more equipment and encodes you add to a video production the more likely the signal will be pulled up to broadcast black, so it's better to just think of that as black. There's a similar exercise with white on the upper end. Remember when analog TVs would buzz when you wired up your computer and put RGB white on the screen? That's superwhite.
- kozak 9y agoThe frequency of 60/1001 Hz and the situation where we are stuck with it basically forever is a shame upon the entire profession of video engineers.
- TeMPOraL 9y agoAny more background on the 60/1001Hz thing?
- kozak 9y agohttps://en.wikipedia.org/wiki/NTSC https://en.wikipedia.org/wiki/NTSC
- Houshalter 9y agohttps://m.youtube.com/watch?v=3GJUM6pCpew https://m.youtube.com/watch?v=3GJUM6pCpew
- city41 9y agoI liked that the green channel in Mario's picture was titled "Luigi". Nice touch :)
- callesgg 9y agoI believe the next step in video compression will be more on smarts like object tracking och object recognition. Machine learning becoming more and more popular will probably help :)
- profpandit 9y agoThey started along that direction with MPEG-4. But it didn't go very far then.
- alfg 9y agoThis is awesome. I work in the VOD space, specifically in content protection and this is great reference guide. I've been meaning to write a similar guide for DRM.
- thomastjeffery 9y agoRemember when we did this ugly interlacing thing, so that we could get a higher (50/60fps) framerate? When did we decide that 24/25/30fps was good enough? Now we have a Blu-Ray standard that cannot handle greater than 30fps, and media corporations that are unwilling to release content via any other medium. Put that together with ever-increasing resolutions, and the amount of pixels something moves across from one frame to the next becomes greater, and video looks more and more choppy. Franky, this is a much bigger problem than NTSC ever was. Even with content (The Hobbit, Billy Lynn's Halftime Walk) being created at higher framerates, users have no way to get the content outside of a specialized theater because the Blu-Ray standard cannot handle it, and because people seem to honestly believe that higher framerates look bad. I suppose we can only hope that creators take better advantage of digital mediums that do not have such moronic, and frankly harmful, arbitrary limitations.
- DesiLurker 9y agoMan that was horrible, hated the deinterlacing artifacts. at codec level it was nasty if you ever wanted debug issues with MBAFF. good thing hevc did away with that crap.
- Houshalter 9y agoHigh frame rates are beautiful. Gamers have known that for a long time and are obsessive about fps. The guy that did that amazing full motion video on a 1981 PC talks about doing experiments. And found that high frame rates were much more important than the image resolution. And had some nice examples of high res but choppy video next to high frame rate video with crap resolution. He somehow got 60 fps video on that ancient hardware and it looked acceptable.
- zimmund 9y ago> High frame rates are beautiful Yes, but it all depends on what you are seeing. Games look fantastic, VR probably looks great. For movies, I'd stick with 24 fps, as it keeps the "style" of film. Subtle things like the motion blur produced at that framerate and how we see/expect the movement in the screen are altered when the framerate changes. As with a lot of things, it depends on what you want to achieve!
- m1el 9y agoHow much of that repo is protected by patents and cannot be reused?
- rasz 9y agofirst example interlacing image is wrong, shows running dogo with simulated division into scan lines, but does not take into account timing difference - that was one of the mayor sources of deinterlacing artifacts. Alternating fields are 1/60 second apart in time. http://www.onlinevideo.net/2011/05/learn-the-basics-of-deinterlacing-your-online-videos/ http://www.onlinevideo.net/2011/05/learn-the-basics-of-deint...