37 ms·
The End of Video Coding?
- carbonatedmilk 8y agoPull quote: We run compute on the cloud and have no real-time requirements. I was asked by the Chair, “How much complexity increase is acceptable?” I was not prepared for the question, so did some quick math in my mind estimating an upper bound and said “At the worst case 100X.” The room of about a hundred video standardization experts burst out laughing. I looked at the Chair perplexed, and he says, “Don’t worry they are happy that they can try-out new things. People typically say 3X.” We were all immersed in the video codec space and yet my views surprised them and vice versa.
- carbonatedmilk 8y agoThis makes entire sense to me: When you're going to stream the episode of Stranger things to 15million people, who really cares what the one-time cost to encode is? Surely you'd take algorithmic complexity that was 100x or even 1000x the baseline if it provided you with even a few kB of bandwidth savings?
- rayiner 8y agoNetflix is probably at the extreme end. Remember: the same codecs are used to compress the videos your cell phone takes.
- haikuginger 8y agoHardware implementations of codecs can be made that perform encode at low power if a certain level of quality/efficiency can be given up. This is already the case for AVC/HEVC on mobile devices where storage and power considerations overwhelm the possible quality and coding efficiency advances that could be available from a highly-intensive, highly-tunable CPU-based encode. In comparison, if you look at digital cinema, video is dumped as raw, unencoded pixels onto high-speed, high-power SSDs without any compression; this way, the original image can be manipulated losslessly before going through an intensive encode process for an optimal quality/space balance for end-user media delivery.
- rayiner 8y agoRight, but designing a codec that is scalable in complexity to accommodate real-time hardware encoding on one end, and 100x Netflix encoding on the other end probably isn't easy.
- Twirrim 8y agoWhy does it have to be the same codec? Use the right tool for the job.
- Slartie 8y agoThe encoding part may well differ, but having the same bitstream format and thus being able to use the same decoders for a very wide range of efficiency/time trade-offs has huge advantages. The necessity of using hardware implementations of video decoders in many scenarios requires a big initial investment into a new codecs' decoding part before even the first frame of video may be displayed by it. This is why pretty much anyone has standardized on h264, and now on h265, as the video codec of choice, despite using very different encoder software or hardware to realize hugely differing trade-offs on the encoding side of things.
- tatersolid 8y agoThe very popular RED digital cinema cameras have always shot in compressed form using a low-latency wavelet-based codec (presumably with some ASIC or GPU acceleration on board). http://www.red.com/learn/red-101/redcode-file-format http://www.red.com/learn/red-101/redcode-file-format http://www.theblackandblue.com/2012/05/02/shooting-red-epic-7/ http://www.theblackandblue.com/2012/05/02/shooting-red-epic-... What cameras shoot uncompressed 8K HDR video? That doesn’t even seem possible given current SSD bus speeds. Note that many high-quality lossless formats are actually still compressed (they subtract current frame data from the previous frame and then apply LZ4 for example).
- vardump 8y ago> What cameras shoot uncompressed 8K HDR video? That doesn’t even seem possible given current SSD bus speeds. 12-bit bayer RGB matrix 8k @24 fps: 7680 * 4320 * 12bit * 24/s = 1.2 GB/s uncompressed. Currently SSDs go up to 3 GB/s. And you could have a RAID 0 array made out of multiple SSDs. So it is possible. Entirely another matter whether it makes sense. One hour of video at this bitrate requires 4.3 TB.
- lostcolony 8y agoYou still care, since at some point the cost of the extra encoder-hours used is going to outpace the bandwidth savings. I'm guessing she mentally assumed 'at no bandwidth savings, how much margin do we have?', since the amount saved wasn't ever quantified.
- martincmartin 8y agoIt's not bandwidth savings. It's growing the user base. There are lots of people on low bandwidth connections. Maybe now they don't find Netflix watchable, because the video quality is to crappy. But if you can fit higher quality video into the same bandwidth, they'd become users. I agree with your overall point though, that at some point the extra cost for encode isn't worth it.
- alephnil 8y agoNetflix is in fact working very hard to optimize the quality per bandwidth at low bitrates. Megha Manohara from Netflix had a conference talk about it last year: https://www.youtube.com/watch?v=S9E9xUbcYAM https://www.youtube.com/watch?v=S9E9xUbcYAM Their goal is to have video of acceptable quality at 250 kbit/s, which makes Netflix an option for a lot more people worldwide.
- spenczar5 8y agoThat's part of it, but I think you'd be astonished at how much IP transit can end up costing when you're distributing high-quality video. It's pretty staggering how much bandwidth can cost. I would not be surprised in the slightest if at least 1/4 of Netflix's total engineering spend is on network transit, so any savings that are per-video can have a huge impact. Source: I work at a company that distributes a lot of high-quality video :)
- derf_ 8y agoThere are a couple of things that make Netflix somewhat unique. 1) They have a fairly limited catalog (contrast with the constant ingestion of new content by Youtube, for example) 2) The cheapest way for them to ensure they have sufficient capacity to handle peak loads leaves them with a lot of extra compute during non-peak times. That excess compute is essentially free.
- bufferoverflow 8y agoBandwidth is extremely cheap, so 15 million by a few kB is just 15+ GB, which costs pennies in terms of bandwidth. That doesn't buy you that much encoding compute time.
- Yetanfou 8y agoBandwidth can be cheap, all depending on the connection you're using. It can also be extremely expensive: [ Dutch ] https://www.telegraaf.nl/vrij/2167192/zoontje-duitsers-kijkt-films-tijdens-cruise-12-000-euro https://www.telegraaf.nl/vrij/2167192/zoontje-duitsers-kijkt... If your Dutch isn't up to this let me summarise: A German family took a boat trip from Kiel to Oslo. Their 12yo son watched a few movies (or clips, no idea) while on board. Total data consumption was 470 MB. They got a bill for €12.000,- which comes down to just over €25 per MB. Bandwidth can be very expensive as well. Of course this is an extreme case but any byte shaved off an encoded stream can end up saving some people a lot of money.
- phlakaton 8y agoFor you, perhaps, it's cheap. For the use cases they are talking about, it's expensive and limited. Agreed that, if all they can do is shave a "few kB" off a video, it's probably not worth the investment. But what if they can buy a 5–10% bandwidth improvement that actually looks better on the device they're targeting?
- kemitche 8y agoNetflix isn't necessarily trying to reduce the bandwidth on _their_ side. They want to be able to deliver higher quality to users on lower bandwidth connections. The better that goes, the larger their prospective customer base.
- dick_sucker2 8y agoThat's a big issue in rural areas with microwave links. Everyone who has one of these links is limited by bandwidth and they all want to watch Netflix.
- maherbeg 8y agoNetflix also encodes to hundreds of combinations of bit rates and resolutions. If improving the quality for a given bitrate lets them reduce some of the complexity (by shrinking the combinations needed to serve users), then there are pretty nice advantages there.
- nine_k 8y agoSuddenly they're in the same space as 3D-rendered movies: they can spend a lot of time once to make the result good for a million of spectators. It, of course, only works if you render widely replicated material. Even an average Youtube video is likely not like that, to say nothing of the encoding done in a consumer-grade camera / phone.
- lev99 8y ago> encoding done in a consumer-grade camera / phone. Just brainstorming here, but encoding local photos/videos might be an acceptable task for a cellphone to do when it is fully charged and plugged in.
- admax88q 8y agoThat would severely limit the amount of photos and video you can capture between charging sessions. Most phones don't have a ton of storage, and raw video/photos are huge.
- icebraining 8y agoPlus storage might not be fast enough to store raw video in realtime. And you'd kill it much sooner with all the writing (larger raw files + reencoded files).
- jascenso 8y agoNote that they are speaking about encoding complexity. The point is that decoder complexity increases are also needed (although, not as much !) and every time that encoder complexity has increased in the past, decoder complexity also grew. The really interesting question is how much decoder complexity increase is acceptable.
- da_chicken 8y agoI'm someone with an intermediate level of coding experience, but I have no idea what exactly is meant here by "encoding complexity". What's the difference between 3X and 100X? What's the encoding complexity of h.264 vs MPEG-2? Is it just saying that they're OK with the encoding algorithm taking 100 times as much processing power or taking 100 times as many processor cycles to compress a given data stream?
- dingo_bat 8y agoIt seems to me that despite the tech, Netflix video quality is really horrible. Youtube is consistently much higher quality, even though Netflix's own "fast.com" tells me my connection is capable of 75Mbps downstream. Which should be enough for crisp 1080p. Multiple times I've been so frustrated with Netflix's quality that I've started a movie, stopped it because it looked like something out of a VCR, torrented the true 1080p version and watched that.
- SirHound 8y agoCould it be your ISP is targeting and throttling Netflix? There's probably a way to verify this. I tend to find the quality of HD Netflix streams pretty great over my PS4. Certainly never a cause to download a torrent instead.
- dingo_bat 8y ago> Could it be your ISP is targeting and throttling Netflix? There's probably a way to verify this. I believe fast.com is supposed to test against actual Netflix video delivery servers, just to detect this kind of ISP fuckery.
- deleted 8y ago[deleted]
- dooglius 8y agoYou can do a pretty direct test with https://www.netflix.com/title/70136810 https://www.netflix.com/title/70136810
- dingo_bat 8y agoThat just takes me to the netflix homepage for some reason.
- gempir 8y agoCould it be you are using an incompatible Browser? E.g. Chrome on Windows works not very good the native Netflix windows app delivers a way better quality
- shmerl 8y agoA post about innovation and moving beyond block-based encoding and no mention of Daala's idea of lapped transforms?
- cjensen 8y agoThere is no shortage of new ideas for video encoding. The post is welcoming both new ideas and encouraging more new ideas rather than being any kind of summary of current tech.
- shmerl 8y agoI guess there might be no shortage in general, but there are few which are publicly developed with the aim to make free codecs. Which other efforts are there besides Daala?
- kylophone 8y agoAV1 https://en.wikipedia.org/wiki/AV1 https://en.wikipedia.org/wiki/AV1
- shmerl 8y agoIt's still block based. I was talking about new approaches which the blog post is referencing.
- ksec 8y agoAll the wavelet based codec shown good results in lab or test, but in practice they are extremely hard to tune and the engineering hours that needs to put in before it being variable is far beyond any one single company are willing to invest. I think Daala will likely live in AV2. But let wait for AV1 to go out first before they work on that. As the promised bit stream frozen still hasn't happened.
- zellyn 8y agoI see this: In the ITU-T VCEG and ISO/IEC MPEG standardization world, the Joint Video Experts Team (JVET) was formed in October 2017 to develop a new video standard that has capabilities beyond HEVC. The recently-concluded Call for Proposals attracted an impressive number of 32 institutions from industry and academia, with a combined 22 submissions. The new standard, which will be called Versatile Video Coding (VVC), is expected to be finalized by October 2020. Can anyone with more industry knowledge chime in here? To me, that sounds a lot like the kind of group that created the patent- and royalty-encumbered stuff the AOM was created to avoid.
- simonbh 8y agoAt the bottom of page 3, Patent and Copyright is covered. I am not a lawyer and it is not clear to me how this mixing of code from different groups will work from a licensing standpoint. https://www.itu.int/en/ITU-T/studygroups/2017-2020/16/Documents/video/JVET-ToR-TD-PLEN-0155A1.pdf https://www.itu.int/en/ITU-T/studygroups/2017-2020/16/Docume...
- derf_ 8y agoI have some industry knowledge. You are exactly correct.
- ksec 8y agoWell, they are the same group, or same organisation that created the group for HEVC.
- ksec 8y ago>To address the disconnect between researchers in academia and standardization and the industry users of video coding technology, This annoys me quite a bit. Because then it list out the so called large-scale video encoding from Facebook, Twitter, and Youtube. As if Video Encoding are only done by OTT providers. And the Video Encode from iPhone ( Consumers ) TV broadcast ( the good old content distributor ), livestream of events from Sports to Olympics.. etc doesn't matter. It is a very Silicon Valley mentality, and shown in AOM / AV1. After all they are creating their own codec for their own use. While other industry codec organisation will have to take care many "edge" use case. >So how will we get to deliver HD quality Stranger Things at 100 kbps for the mobile user in rural Philippines? How will we stream a perfectly crisp 4K-HDR-WCG episode of Chef’s Table without requiring a 25 Mbps broadband connection? It is interesting this 100kbps bitrate and rural Philippines comes up. Because this is the exact same quote from Amazon's video specialist Ben Waggoner mentioned on Doom9. Shouldn't we be a little more realistic with the bitrate? We have 20 years of experience and research and yet we still don't have a single Audio codec ( two dimensions ) that could perform better then MP3 128Kbps at half the bitrate. Opus only manage to slightly edge it out at 96kbps, and that is with selected samples. There is only so far we can go, 100Kbps is barely enough for Audio. And we have Massive MIMO, and 5G, both will bring immense capacity increase to current Network. There is so much in the pipeline to further increase efficiency, capacity, lower latency, cost, and power. It is a little hard to think designing a 100Kbps for Video. Currently Youtube streams 1080p AVC @ ~2.2Mbps. Which seems to be fine with most people already, especially on Computer / Tablet or Smartphone Screen Size. HEVC can probably do similar quality with 1.5Mbps. VVC should be aiming at below 1Mbps. Netflix is doing 15Mbps 4K Streaming with HEVC ( And people are complaining about quality already, I have no idea why I don't watch Netflix) VVC should really be aiming at better quality with 8Mbps. We should aim we specific bitrate and resolution with Real World Encoder as anchor, and specific quality to achieve.
- hcnews 8y agoIn most of your reply you have failed to account for the third world (and frankly a decent chunk of first world). It seems to me you don't have an understanding of your user base. Please dig into how many people are actually on HD-capable bandwidths. I am glad that Silicon Valley is driving it since they have users all over the world, not just fiber-enabled users. If and when other people have a better aim, vision and understanding of the problem, I think you should find a good collaborative atmosphere. In the meantime, let's not design for 100m users when we should be thinking about 2-3b users.
- vidanay 8y agoI want to see an encoding that is basically RLE using offsets and lengths of π.
- lopmotr 8y ago"new techniques are evaluated against the state-of-the-art codec, for which the coding tools have been refined from decades of investment. It is then easy to drop the new technology as “not at-par.” This seems to be quite a general technology problem that looks like it applies to car engines, batteries, computer memory and any other highly optimized mature technology. I wonder if there's some way to change incentives so others get a chance. It happens in evolution in nature too and it's sometimes solved by mass extinctions. Hopefully that's not the only way.
- pishpash 8y agoThat's what research programs were supposed to do. The problem occurs when research is also evaluated against such specific things as wanted by industry that there's no room for new ideas to be fully funded for the long maturing process.
- MayeulC 8y agoAre there transmission schemes that can transfer multiple independent streams that add up for better quality? It could be streams containing information about smaller subblocs, etc. If you take a fourrier transform, the more coefficients you include, the more faithful the reproduction is. Splitting the quality level into multiple independent streams could have multiple advantages : - better use of Multicast, as viewers for different quality settings get the same data, so you use approximately the bandwidth for one full quality video instead of full quality + medium + low... - save on storage space for the same reason - save on transcoding time, as only one pass is needed - better suited for distributed storage/transmission, on a platform like PeerTube, where every p2p client could contribute back instead of clustering by quality. I am not working in this field, but I know a fair bit about compression, and this seems a no-brainer to me. Is it already done? Where? Or did I oversaw some issues?
- 0x09 8y agoThis is known as scalable coding and has been included as an extension or profile in at least every ISO and ITU standard since MPEG-2, see for example https://en.wikipedia.org/wiki/Scalable_Video_Coding https://en.wikipedia.org/wiki/Scalable_Video_Coding (this article seems to be specific to H.264)
- MayeulC 8y agoThank you a lot for the proper technical term. Do you happen to know if it is widely used in web-based streaming applications (I would be surprised, as I don't think I encountered it before), if ffmpeg supports it (in the case of H.264),and if AV1 includes such an extension?