10 ms·
How Facebook encodes videos
- BlueTemplar 6y agoHmm, not even a single comment mentioning AV1 ?
- londons_explore 6y agoI'm surprised Facebook still uses software to do video encoding. Most big companies with millions of hours of video uploaded each day have realised it's cheaper to stick a bunch of hardware video encoding chips onto an accelerator board and be able to transcode 100 HD streams simultaneously into all the formats and resolutions you need to host. The power savings on CPU's pay for the custom hardware in a matter of months. It does reduce flexibility when new video formats get released though.
- PicassoCTs 6y ago? Xilinx for the rescue?
- meteorfox 6y agoFacebook does have hardware for video transcoding. [1] https://youtu.be/0yk4SPRisPA https://youtu.be/0yk4SPRisPA
- deleted 6y ago[deleted]
- retrac 6y agoHardware encoders for newer codecs tend to be simply less bit-efficient. They can be more limited in features. My desktop's GPU can handle quite a few 1080p H.264 streams in parallel. But it can't do average-bitrate-capped CRF, which I think is the preferred type of encoding for streaming services.
- deleted 6y ago[deleted]
- prvc 6y agoWhat was mentioned in the article seem like very basic, obvious considerations. Meanwhile, Youtube has custom ASICs which efficiently produce multiple formats simultaneously.
- jeffbee 6y agoAnd yet, despite YouTube's abundantly clear motivation for doing so it took them years and years to develop the hardware and it only recently hit production. Could it be that acceleration for video encoding isn't as easy as people are making it sound?
- dylan604 6y agoI'm waiting for the "meh, i could roll it out in a weekend" responses.
- yjftsjthsd-h 6y agoOf course I could;) It would just be hacky, ugly, unreliable, scale badly, use either GPUs or inadequate FPGAs, and fall over once it exceeded 10 users. My resume doesn't need to know that, though:)
- dylan604 6y agoYou now know 1 way of not to do it.
- webmobdev 6y agoI would be suprised if FB doesn't use hardware encoders. That said, hardware encoders are fast, but they don't match the quality of the software (CPU) encoding. Slower CPU encoding still provide the best quality.
- thefourthchime 6y agoI can’t speak for all of the encoding options. But I can tell you that Dolby Hybrik, one of the main competitors and encoding space uses relatively cheap ec2 instances. Based on the speed I see from Telestream and Elemental. I think they’re software based as well
- cobookman 6y agoYoutube and Facebook have a much different encoding problem then say HBO/Netflix/Hulu...etc YT & FB have a few popular videos, and a ginormous long-tail. That long-tail needs to be encoded, and cheaply as it might not get many views. The TV/Movie VOD over IP industry stands to benefit optimizing the encoding to have the smallest filesize with highest quality. Spending considerable CPU cycles to find that best quality path. For those like Netflix, why would you invest with ASICs. You likely don't have a huge infra bill (relatively) when it comes to encoding. And CPU vs ASICs, the ASICs mean you loose flexibility to really fine-tune the quality and use the latest codecs. For those like YT/FB. You just gotta encode that long tail. Getting something cheap that's 80% the quality of CPU-based encoding is good enough for 99% of the content.
- mnt16 6y agoElemental is a mix of software and hardware encoding. They utilize GPU’s for some portion on the encode.
- astrange 6y agoHardware encoders are not as good because they have worse algorithms in order to make them faster. I don’t think you could make an x265 ASIC. Also, good video codec engineers can’t be hired for any predictable amount of money, and ones who speak Verilog barely exist.
- yaur 6y agoThis used to be true but modern NVIDIA cards beat x264 on both quality and energy cost.
- Wowfunhappy 6y agoThey do?! Even on the slow presets?
- paulmd 6y agoNVENC beats x264 medium. Not quite up to the level of the "slow" presets yet but you have to throw a huge amount of hardware at it to match them let alone beat them. Basically the number I came up with a few weeks ago from playing around with x264 settings on ffmpeg was between 6 and 12 cores to keep up with NVENC at 720p and 1080p, depending on framerate and the quality preset.
- ksec 6y agoWhat sort of bitrate are we talking about? I haven't tried using NVENC for years and last time I check it was clearly missing many details that x264 tries to preserve. NVENC is good at cleaning the noise and fast encoding. ( Or basically Game Streaming ). Which isn't something you want to do if you want to do movies encoding.
- paulmd 6y agohttps://developer.nvidia.com/blog/turing-h264-video-encoding-speed-and-quality/ https://developer.nvidia.com/blog/turing-h264-video-encoding...
- manca 6y agoThe problem is that most of the big tech thinks they are smarter than 1000's of codec engineers who worked on libx264 and they try to reinvent the wheel. They realize too late that all the money they threw into building their own stuff (or variants) do far worse in general case than the baseline libx264. They just don't want to admit it.
- LdSGSgvupDV 6y agoBidTechEngineer: Ya, I know, but I need that wasted money.
- manca 6y agoAbsolutely. In fact, I interviewed a guy last week that comes from very big tech and he basically himself admitted that they're basically wasting time and money chasing unknown "ideal" codec while almost always those projects hit dead end. He loves the money, I am sure, but he wants out anyway. :)
- Tade0 6y agoWho wouldn't eventually? I've been to a project where, because the development and "hardening" budgets were separate, we knowingly pushed out buggy code so that we could take advantage of the latter. Had I known that half of the job would be to game the system, I wouldn't have joined.
- nowherebeen 6y ago> half of the job would be to game the system, I wouldn't have joined. Bureaucracy is inevitable once a certain number of people join a group.
- ClumsyPilot 6y ago"Had I known that half of the job would be to game the system" You should be happy it's only half!
- 6y ago
- jimmySixDOF 6y agoLike someone already pointed out this whole announcement could be in response to Google announcing this week it's new Argos chipset approach to transcoding [1] https://arstechnica.com/gadgets/2021/04/youtube-is-now-building-its-own-video-transcoding-chips/ https://arstechnica.com/gadgets/2021/04/youtube-is-now-build...
- dheera 6y agoI could see lots of reasons to do software encoding at a company like Facebook which are probably not mentioned in the article, including e.g. extracting the optical flow which can be reused as part of various machine learning workflows. For example, if you want to only annotate 1 in 100 frames and interpolate with optical flow, you can get the flow in the process of doing video compression. I don't have inside info but there are various other things like this that I can think of for wanting to do it in software. Also, hardware encoders suck if the company making the hardware decides to suddenly EOL the product. With software encoders you can easily scale the backend whenever you want, at any time in the future, and without succumbing to supply chain issues.
- tinus_hn 6y agoIf you’re Facebook, the company making the hardware doesn’t suddenly decide to EOL the product, or they are sued for millions for breach of contract.
- ksec 6y agoGoogle released the paper [1] "Warehouse-scale video acceleration: co-design and deployment in the wild " a few days ago. https://news.ycombinator.com/item?id=26925307 https://news.ycombinator.com/item?id=26925307 >It does reduce flexibility when new video formats get released though. A new video codec is released and adopted once every ten years. So I dont think it would be a problem. Hardware encoding also trade compression quality for speed. And they are compensating it with slightly higher bitrate. It will also reduce their incentives for improving open source encoder. Although I think even Netflix switched to BEAMR ( Cant really blame them though ) I think the next frontier, for both Audio and Video will be Codec designed with LiveStreaming / Low Latency in mind. ( Or pretty much everything computing, I wish we could focus on latency, from Hardware Input, Display, Network, Disk, etc. Apple is certainly moving in that direction without talking about it. )
- cpeterso 6y agoI was talking with a Netflix product manager a few years ago and, IIUC, he said Netflix re-encodes their entire catalog monthly to take advantage of new encoder optimizations. Disk is cheap. They want to optimize network transfers to both save money and improve user experience.
- teruakohatu 6y agoNetflix has about 36,000 hours of content. On Youtube that much content is uploaded every 1 hour and 20 minutes! Or to put it another way, Netflix re-encodes every month what YouTube encodes in just over an hour.
- ynx 6y agocounterpoint: if you're big enough to have your own bona-fide cloud, then you might have more spare CPU capacity that can be flexibly allocated than available PCs with dedicated GPU
- zaphirplane 6y agoIf you are in the cloud business you are worried about heating power, with powering down disks a real thing
- ZeroGravitas 6y agoI've seen this contrasted with the Google stuff but it seemed to me the two approaches were entirely complementary. Facebook, which videos will we transcode into what versions. Google, with this list of videos to transcode how can we do it cheaply on a massive scale. I didn't spot anything that pitted approaches against each other.
- mappu 6y ago> An encoding family requires a minimum set of resolutions to be made available before we can deliver a video. [...] For example, having one video with all of its VP9 lanes adds more value than 10 videos with incomplete (and therefore, undeliverable) VP9 lanes. I don't see why this constraint is in place, you can absolutely serve video for certain-res users only with certain codecs (youtube certainly does this).
- vbsteven 6y agoI assume Facebook has this requirement for usability reasons. They don’t want a user to receive a video through one of its many share features and then not be able to view it.
- mappu 6y agoYou need baseline codecs for baseline visibility, sure. But the article is about selectively choosing which videos to encode with more advanced parameters, and there's no reason to hold back delivering your 1080p VP9 just because the 720p VP9 isn't ready yet - just serve something else.
- aledalgrande 6y agoWhat if you were streaming 1080 VP9 and then your player decides to downgrade to 720 because your connection became slower? At that point the missing 720 would make you unable to watch the video.
- yaur 6y agoIn MPEG-DASH codec is specified at the representation level so with a compliant packager and a compliant player the player should be able to switch down to the "fast h.264" rendition that exists before they start trying to create VP9 renditions.
- aarongolliver 6y agoThis is (was?) a big problem with google photos. If you upload a video and immediately share it with someone they'll just see a black screen. A few minutes later they might be able to watch a super-low res encoded version, and only much later does a HD version become available. I started waiting 10 minutes between generating a "share" link before sending it to my family because of all the confusion it was causing.
- stefan_ 6y agoMaybe the people at Twitter can read this and learn something.
- calebio 6y agoThey still haven't figured out how to give users control of their poor image cropping :(
- zeeshanqureshi 6y agoTheir "race-based" image cropping? https://youtu.be/Ok5sKLXqynQ?t=103 https://youtu.be/Ok5sKLXqynQ?t=103
- wongarsu 6y agoThat looks like cropping on whatever has the highest confidence value of being a face. Evidently not a good idea, it does look pretty racist.
- 1cvmask 6y agoThe one thing I find fascinating is how big a difference there is in “AI” based background removal tools. The discrepancy in quality is huge. This is the new one I found: removebackground.app
- juliendorra 6y agoI have found Twitter videos to have improved in quality recently, but there’s a disconcerting effect where when you upload your own video you see first a very low quality version, and the much better version replace it. So you are lead to believe that your video is indeed horribly recompressed, especially if you don’t wait a bit, when it seems that there’s actually several level of quality available.
- slig 6y agoAnd reddit.
- blowski 6y agoThese are the kinds of articles that show why Facebook has different problems to almost every other tech company. The complexity of these kinds of solutions is mind-boggling. Just imagine if all that ingenuity was focused on solving humanity’s problems, instead of sharing conspiracy theories and advertising.
- bob33212 6y agoI wish people spent more time on Facebook. Is that what FB employees think? Or are they like the rest of us and think fuck this guy, but I'll take his money.
- echelon 6y agoOther firms pay just as much, so there's plenty of choice in employment. Rather, working at Facebook just isn't an ethical problem for many of its employees. My engineering colleagues that voraciously use and consume Messenger and Instagram don't see a problem either. There a lot of "bad" companies. Certain advertising, defense, pharma, law, etc. firms are shady or morally bankrupt. It doesn't stop them from finding people to do the work.
- beervirus 6y agoDon’t forget spying on everyone on the internet!
- deleted 6y ago[deleted]
- saagarjha 6y agoBeing able to efficiently serve video is one of humanity's problems. You may not like the host or video content, but that doesn't mean the problem is not a useful one to solve.
- esjeon 6y agoI don't think this is difficult to engineer. It's a good example of ML/AI being used for optimization, which I believe is the most legit usecase that everyone should be aiming for. Still, the problem is how do we get to the scale where we desperately need stuffs like this...
- Const-me 6y agoAt least here, the video quality is not good on FB. I have 120 mbit/sec connection, youtube plays good quality 4k just fine. I don’t think I ever saw 1080p served by facebook, both resolution and bitrate are very low.
- ericbarrett 6y agoYour ISP might be throttling Facebook video
- gjs278 6y agodoubt it. facebook videos just absolutely suck in quality.
- gruez 6y agoThere's other explanations as well: * ISP has poor peering to facebook servers * ISP has cache/CDN node installed, but that's overloaded * the content that OP viewed isn't popular so it has to be pulled from origin, which adds another layer of complications
- ericbarrett 6y agoAll possible. But FB video (meaning also Instagram) is a popular target for cheap ISPs that don't rate a peer to constrict since it consumes such a large portion of users' total bandwidth, especially on mobile
- mrtksn 6y agoNothing is bad as reddit video that would load random segments instead of consequential ones or Twitter video that would be 3 seconds on 4K 20 seconds 144p 5 more seconds in buffering or Vimeo that’s buffering all the times on 100Mbps connection. Moral of the story: Video is hard, apart from Youtube, Twitch, Netflix and Amazon Prime I don’t know any service that plays video flowlessly. O.K. maybe some ad networks too.
- 6y ago
- the_duke 6y agoRelated question: is there any decent OS software that can intelligently pick encoding parameters that preserve quality while minimizing size? I recently tried to implement video uploading for an open source project, but naively choosing ffmpeg parameters can often result in noticeable quality loss / large output file size / long encoding time. And easily all three of those at the same time.
- dmitryminkovsky 6y agoI second this question. I also recently had this experience [0] and the results are disheartening. [0] https://stackoverflow.com/q/61784204/741970 https://stackoverflow.com/q/61784204/741970
- martinald 6y agoThe problem really is encoding from compressed video to compressed video again. It really isn't great whatever you do (Facebook/Twitter definitely haven't solved this problem either). The quality from going from source -> 15mbit/sec h265 -> 2mbit/sec h264 (like on a classic social media or whatever site) is absolutely terrible compared to going from source -> 2mbit/sec h264.
- dylan604 6y agoFacebook/YouTube don't care about how their compressed video looks. They just need it in the format so that they can get in front of their millions of user's eyballs. I'd be willing to bet that >95% of users don't "care" about compression quality. They just want the content, hence their decisions. There's just no way to properly encode that much content with a "we care about compression" thoughts.
- StavrosK 6y agoWhy is that? Isn't 15 mbit to 2 mbit a big enough difference in quality that you'd expect to retain all the good parts?
- martinald 6y ago
- neximo64 6y agoIn a way for playback of course that makes safari consume too much resources for the tab and restart the tab mid video so you never see the ending..
- teclordphrack2 6y agofuck facebook.
- dang 6y agoCan you please not do this here? You may not owe $BigCo better, but you do owe this community better if you're participating in it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- deleted 6y ago[deleted]
- don-code 6y agoIt seems like Facebook does a thing that doesn't scale, in the interest of time: feed a minimally processed video until other streams are available. That makes sense, but I wonder if it also means that video quality on Facebook generally suffers when high-profile events are taking place. Context on that wonder: I've always noticed that news outlets seem to carry downright horrible quality user-generated video clips of rallies, protests, and the like. Where everybody's carrying around stellar-quality video gear in their pockets these days, I've never figured out why that is.
- avidiax 6y agoI suspect two factors: 1) Livestreaming can have very poor quality when bandwidth constrained (e.g. at an overloaded cell site at a rally) 2) Viral videos get reencoded many times in many formats. The cumulative encoding errors are not only limited by the lowest quality reencode, but also by the defects in all previous encodes with various codecs.
- don-code 6y agoI bet you just answered my question - we aren't talking about a 5mbps video uploaded after the fact on home Wi-Fi, but a 384kbps video shoved down a 512kbps, TCP-unfriendly pipe as it's being shot. Thank you for that!
- ascorbic 6y agoMinimally-processed doesn't need to mean lower quality. In this case, the expensive advanced codecs are for mainly for smaller filesize.
- xiphias2 6y agoIs it too early to think about using the AI inferencing chips provided on mobile phones for decoding video? There are lots of papers that use deep CNNs for compression/decompression, and Facebook / Google would be the best companies to productionize it first.
- thefourthchime 6y agoI believe the next generation of encoders will use this. But the standardization process takes years. So it might be a little while
- astrange 6y agoWhat’s wrong with using the video decoding chips on the phones to decode video?
- Diggsey 6y agoIf you break videos up into short chunks, then you could simply encode those chunks on-demand into the perfect encoding for the requester. The advantages are: - You don't waste CPU encoding video into formats that won't be used. - You can use a standard caching solution to reuse those chunks. - Everyone gets the perfect encoding, always. - If most people watch the first 2 minutes and then give up on a 20 minute video, you don't waste CPU encoding the other 18 minutes. - You can introduce new encodings instantly for all videos, without going back and re-encoding historical videos. - You don't waste storage on video chunks in formats that will never be used. - It's really simple. The disadvantages would be: - You have more unpredictable load if a lot of people start watching (different) videos at once (although the common case of a lot of people watching the same video is still fine) and you could "cap" the load by switching to a fallback format to avoid becoming overloaded. - There might be an initial delay when playing or sweeping a video whilst the first chunk is encoded. On the other hand, it can't get much worse than it is already, and you could make sure these initial chunks are prioritised, or else serve the initial chunks via a fallback format.
- BlueTemplar 6y agoSounds like "real" (old-school) streaming ?
- maxgashkov 6y agoIt seems that this process will be incredibly stateful in the encoder component. Most of codecs targeted at low bandwidth for mobile streaming track scene changes and if you make chunks in a naive way (split via I-frame borders and encode 'em independent of each other), when reassembled final video will look choppy due to broken scene change relations. So after encoding each chunk you will have to carefully save relevant parts of encoder state and reuse it for the next chunk. Seems doable, but tricky to get right?
- craigc 6y ago> It’s really simple Have you ever written a stateless transcoder like this? Of course it can be done, but saying “you could simply encode those chunks” and “It’s really simple” is pretty misleading especially if you are changing frame rates or sample rates or audio codecs during the encoding process. That said, if there is someone that could do this at scale it would be Facebook. Also this would mess up ABR streaming at least for the first people to watch the video which would not really guarantee “the perfect encoding always”.
- lousken 6y agocan anyone show me decent looking video on facebook? whenever i see anything from my friends it's just a horrible pixelated mess, even the gaming channels i saw
- jakear 6y ago> A relatively small percentage (roughly one-third) of all videos on Facebook generate the majority of overall watch time. Surprising to me that it’s so large, I’d expect something like 5% or less account for half of watch time. I wonder how it compares across platforms (YouTube for instance would almost certainly be in the <5% bucket, I’d imagine)
- rahimnathwani 6y agoMaybe you and the author are considering different periods, e.g. A. what % of all videos generated the majority of overall watch time over the last week? B. what % of all videos generated the majority of overall watch time over the last year? If the watch time is growing by a high percentage month-over-month, perhaps the numbers would be similar. But, if overall watch time is flat, then you'd expect A to be much lower than B.
- TedShiller 6y agonot well
- chx 6y agoThis post does not, at all, try to steal Google's thunder after revealing Google Argos.
- bootcat 6y agoi think argos seems like a better techy project than this,
- astrange 6y agoI find it odd that this is a kind of mathematical article but doesn’t use the existing terms in codec literature - why does it say “cost/benefit” instead of “rate/distortion”?
- not2b 6y agoBecause FB doesn't care about rate or distortion, they care about cost to them and whether the result meets some minimum standard of acceptability.
- jaimex2 6y agoFacebook does a garbage job of encoding, I wouldn't be using them as a reference. Half the videos people send me on messenger are either terrible quality or just a frozen frame with audio. Both when played on my phone or via the web page. Videos sent by the same people on other services like Signal, Slack or Messages are fine.
- ashconnor 6y agoTwitter's video encoding is far worse.
- billjings 6y agoVideos over messaging are a different problem space from the one outlined in this article. The number of people watching a messaged video is low; latency might be the topline metric the team optimizes for, since it has a big effect on whether people stay in the conversation or leave. In fact, prioritizing latency might be the reason for the garbage quality you're seeing: reduce the bitrate, and people will receive the video more quickly and more reliably. Whether that's a smart decision is another question.
- mdoms 6y agoI feel the same way about their terrible UIs. Messenger is almost unusable on both web and android and new bugs are introduced frequently, but somehow their shit house technology (React) has become the industry standards.
- yaur 6y agoIs there an example of a video that can be encoded at 5MB with H.264 that can be encoded at an equivalent quality with VP9 at 3MB? There are a lot of things to say about this article, but that seems most glaring.
- loeg 6y agoI think the figure is intended to be illustrative, rather than literal.
- yaur 6y agoThat may be what its intended. But 5 -> 3 is a world apart from 5 -> 5.95. One is consumer friendly (and groundbreaking information theory) and the other is just about reducing margins while pretending to be consumer friendly.
- deleted 6y ago[deleted]
- ksec 6y agoThe first draft of H.264 is over over 20 years old. And in most of these comparison they could use anywhere from Baseline to High Profile. Modern VP9 could compete with HEVC, 40% reduction of Bitrate is very reasonable between generations as long as it is not some insanely low bitrate which doesn't compress well.
- jimmySixDOF 6y agoA Related Tangent is how FB handles different photo requests by saving 4 copies with different resolutions at the time of upload. It's even worse because according to [1]: " The number of Haystack photos written is 12 times the number of photos uploaded since the application scales each image to 4 sizes and saves each size in 3 different locations. " [1] https://bigdata.devcodenote.com/2015/04/haystack-facebooks-photo-storage.html https://bigdata.devcodenote.com/2015/04/haystack-facebooks-p...
- nbm 6y agoKeep in mind that the referenced paper that this page is based on is over a decade old now. Many things have changed since then, as you can imagine. Look for later papers and engineering blog posts for more about what’s changed since then.
- bootcat 6y agoSo they are using feedback to see how many people are watching a video to use more cpu for advanced compression - There is nothing innovative here - we want to make sure we provide resources to something thats viral and not to all videos, whats innovative here ? I think in cloud and all companies are using metrics to optimize their resources,
- sharkjacobs 6y agoI'd never heard of the practice, is it common with other video hosting services?
- bootcat 6y agoSo I was commenting on the particular technique, To Collect feedback metrics to judge usage and throw resources to optimize rather than blind optimizing every single video. These techniques are very simple but makes sense to use when you have such a high demand. But I would expect more from facebook.
- c8g 6y agoExcept facebook web always show me 1080p on mobile data connection. I have reported many times but I guess they don't care about user feedback. If I select 480p on youtube, It will never show me 1080p, unless I switch back to 1080p. On facebook, every video is by default 1080p. Even if I don't watch video, scrolling news feed also pre-load 1080p. This is a huge problem with limited mobile data plan.
- mdoms 6y agoWhy would I read this? Every video I share on Messenger gets crushed beyond any usefulness, I don't even bother anymore. Whatever FB does is only useful as a case study in failure.
- ogurechny 6y agoDon't mind me, I'm just admiring the well trained corporate lingo. While they mention “happiness”, it all really is about making user spend more time on Facebook EVEN IF there are external limits like data cap or an old device. Conveniently measured in minutes of life taken.