6 ms·
Video streaming at scale with Kubernetes and RabbitMQ
- adityapatadia 3y agoShameless plug here. In case someone does not want to build any of this and still stream video, you can check https://www.gumlet.com https://www.gumlet.com
- titan-node 3y agoJust use Livepeer for your encoding and decoding and you'll save like 95% of your cost for VOD.
- andrewstuart 3y agoMust be expensive to run on Google Cloud. Also looks pretty complex. The stabilization step presumably does a video encode …. that’s extremely expensive in terms of time, compute and money I wonder why it’s necessary.
- ericxtang 3y agoWe've been working on an alternative infrastructure and saves up to 80% on transcoding & delivery costmore affordable solution at Livepeer Studio (https://livepeer.studio/ https://livepeer.studio/). It uses un-utilized infrastructure around the world and incentivizes independent network operators to join the network (kind of like a 2-sided marketplace for video-specific compute). Please sign up for a free account and check it out! We'd love to get your feedback
- tehlike 3y agoI was thinking the same. CF on the front would improve on it but still. Hetzner or other bare metal providers would probably be a better idea.
- hotnfresh 3y agoCF meaning Cloudflare? If you’re serving video through them, then you’re in “enterprise plan” territory. You can’t do that on the free or “self-serve” paid plans. $5k+/m depending on bandwidth needs (and if you just need a cdn to push bits, CF won’t be competitive on price—their enterprise prices are tailored for companies that want all sorts of managed services and private networking stuff)
- mikeryan 3y agoUm. Cloudflare stream starts at $5 per month and you don’t pay for encoding only storage and bandwidth. You can serve a decent video library for $500 per month. https://www.cloudflare.com/products/cloudflare-stream/ https://www.cloudflare.com/products/cloudflare-stream/
- hotnfresh 3y agoAh, must’ve changed up their billing structure to provide more add-ons for the self-serve plans since last time I was dealing them them. That’s good.
- latchkey 3y agoNot necessarily. GCP, when used correctly, can be super cheap. You also don't know the contractual deals they have with GCP.
- klaussilveira 3y agoI wonder if it wouldn't be cheaper to run an on-prem farm of BestBuy-grade "gamer PC" for smaller scale networks like that.
- andrewstuart 3y agoSlap one of these puppies in…. AMD Alveo MA35D Media Accelerator https://www.xilinx.com/applications/data-center/video-imaging/alveo-ma35d.html https://www.xilinx.com/applications/data-center/video-imagin...
- goeiedaggoeie 3y agoIve used xilinx a fair bit for encoding. once you get past the pain of compiling your tooling for it it does speed up VOD encode significantly.
- dilyevsky 3y agoHow’s the quality? I heard it was so-so and i think you can’t close your own presets
- goeiedaggoeie 3y agoFor what I have down (2k and down, main) the quality has been OK. I have also read some complaints about quality at high res, but I am a happy customer
- dilyevsky 3y agoTwitch used to use cheap cores with Intel quicksync and maybe still does
- deleted 3y ago[deleted]
- alexandreolive 3y agoHello, I'm the writer of the article. Our solution gets videos from random people who present products we sent them. We get dodgy videos filmed on bad devices, and the process of contacting the user and getting him to re-upload another video in better quality is time-consuming for our team. We'd rather spend a little bit more in computing to try and save time overall. I hope this answers your question.
- andrewstuart 3y agoI have to ask, why bother with Kubernetes and all the associated config and pain? Why not just start a new spot instance? I can’t see any reason for Kubernetes in this architecture even though it’s the title of the post. Also personally I wouldn’t use rabbitmq … it’s pretty heavyweight… there’s lots of lightweight queues out there. Overall this architecture looks like it could be simplified. Also, the post doesn’t mention if the video encoding uses GPU hardware acceleration. Makes a big difference especially if using spot instances …. ffmpeg in CPU is extremely computationally expensive. Presumably all input videos need reencoding to convert them to HLS.
- malux85 3y agoThis is what I was wondering, in the article it looks like kubernetes is just used to launch the node containers - why is the database and rabbitmq outside of kubernetes? This architecture looks like it’s been cobbled together by a junior
- robertlagrant 3y agoKubernetes loves stateless services. Zero wrong with moving RabbitMQ or a database outside of it.
- malux85 3y agoExcept kubernetes has a whole storage provisioning system that gives you redundancy and automatic failover, if you’re going to the trouble of running kubernetes why not just run your whole infra on it? I run https://atomictessellator.com https://atomictessellator.com solo, using kubernetes, and my database, Minio object store, application servers, quantum workers, everything is all on kubernetes, it’s self healing and much simpler to run all the infrastructure the same. Recently I had a node failure while I was sleeping and the whole system healed itself while I slept, the monitoring system didn’t even alarm me because the small blip of increased latency while the pods rebalanced wasn’t above the alert threshold so it didn’t even wake me up. What happens in the article infra when the rabbitmq or database nodes fail? The whole system goes offline, which seems very silly setup when you have kubernetes sitting right there, who’s primary function is to handle all of this.
- schott12521 3y agoI thoroughly appreciated this article as I've been building a short-form video content streaming service and the performance hasn't been what I expected. Granted, I knew that my service needs to be able to scale at different bottlenecks, but a lot of "build your own video service!" tutorials start with: - Build a backend, return a video file - Build a frontend, embed the video And that leaves a lot to be desired in terms of performance. I think the actual steps should be: - Build a backend that consists of: - Video Ingestion service - Video Upload / Processing Service that saves the video into chunks - Build a streaming service that returns video chunks - Build a frontend that consists of: - Build or use a video streaming library that can play video chunks as a stream Edit: From the author's links, I found this website which is very informative: https://howvideo.works/ https://howvideo.works/
- andrewstuart 3y ago>> I've been building a short-form video content streaming service What does it do?
- schott12521 3y agoRight now I'm basically trying to just re-create the TikTok / Youtube Shorts / Instagram Reels experience of infinitely scrolling videos. Mostly just building for fun though.
- mmcclure 3y agoI helped work on howvideo.works, fun to see it helping people! The world of video is, I'd argue, one of those technical spaces that is extremely iceberg-y. You can get decently far enough using S3 + the HTML5 video tag, which I think creates a perception among some that video is just images but a little bigger, but that couldn't be further from the truth. You can really pick just about any step along the video pipeline from production to playback and go as deep for as many years as you'd like. This is both a semi-shameless plug and probably a few levels deeper than what you're looking for, but I organize a conference for video developers called Demuxed. The YouTube channel[1] has 8 years worth of conference videos about streaming video (and the 9th year is happening in a couple of weeks). The bullet points you mentioned are definitely covered across a few talks, but it's certainly not in any kind of "how to" format. [1]: https://youtube.com/demuxed https://youtube.com/demuxed
- com2kid 3y agoThis is nice if you only have to deliver in one format, but as soon as you want to show up on TVs you are stuck delivering in a lot of formats, and life gets complicated quickly. Throw subtitles in multiple languages, and different audio tracks, into the mix, and all of a sudden streaming video becomes a nightmare. Finally, if you are dealing with copyrighted materials, you have to be aware as to what country your user is physically residing in while accessing the videos, as you likely don't have a license to stream all your videos in every country all at once. Throw this all into a blender and what is needed is a very fancy asset catalog management system, and that part right there ends up being annoyingly complicated.
- dbrueck 3y agoOh, this is just the tip of the iceberg. Many parts of on-demand video streaming are largely commoditized at this point. Add in support for linear (live) streaming and ad insertion and things start to get really interesting. :)
- chunkymilk 3y agoMy background is in linear (title VI and OTT) and ad-insertion. A big chunk of my job is explaining to folks that just because you solved VOD that doesn't mean you've solved linear. It's almost best to think of them as two distinct problems.
- jackcviers3 3y agoArticle made me nostalgic. My very first job was writing streaming media services for radio. Integrating the ad services over events embedded into the stream identifying content was a pretty nifty solution. You send the metadata to your ad server, it selects the ads based on your user and media metadata, you swap out the player's media url to play the ad(s), and keep the stream going in the background. When the commercial break is up, you kill the ad stream and swap the media stream player back in. Presto changeo, seamless ads in 2007. You also track media playback through the stream embedded events. The hard part back then was live video/audio encoding. These cards lived at the stations, and streamed to our cdn. Geocoding for American service bases in foreign countries was interesting too, the streaming rights for stuff limits the countries you can stream in. Monitoring the live encoders was an interesting problem - I setup an event-based solution so I could capture the audio waveform and analyze the frequency buckets on dozens of streams at once. Could tell if a stream was silent pretty quickly, and alert the station engineer via email. Back then, a lot of this stuff wasn't off the shelf like it was today. Even on the frontend,working with Flash, window's media player sdk, and the QuickTime sdk and integrating it with Javascript was challenging. And the backends were all in php and Java 1.4, and mysql, and a whole lot of memcache. Was a really great experience for a new professional programmer to write all that.
- totallyunknown 3y agoWhile the article provides guidance on utilizing standard software and services to construct a basic video upload platform, it lacks deeper insights into advanced scaling techniques.
- dvliman 3y agoI built a similar video pipeline, not on Kubernetes but using EC2 instances for those hungry FFMPEG encoder. The system differ in that it was not user generated video content. It was coming from the cameras in our fitness studio. Here is the article if anyone intereste to read about: https://dev.to/dvliman/building-a-live-streaming-app-in-clojure-329m https://dev.to/dvliman/building-a-live-streaming-app-in-cloj...
- devgoth 3y agoawesome article! curious -- why clojure?
- robinduckett 3y agoBecause CS Degree
- dvliman 3y agoNo specific reason. It could have been built in any language. It was just the language we were using and enjoyed at that time.
- mannyv 3y agoFuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep forgetting the acronym for). If you want to get fancy you can do rendition audio, which not everything supports. Rendition audio means sharing one audio stream amongst N number of video streams. You can use FFMPEG to transcode, but I'd suggest using AWS MediaConvert. It's cheap, fast, and probably does everything you want. Using FFmpeg directly works, but why bother. You will get an option wrong and screw everything up. You don't want your video to not work on some random device that 50k people are using in some country you didn't think about. He's using RabbitMQ but you should use SQS, because SQS can trigger lambdas...which means no polling required. But use whatever queue you want. You can kick the process off by attaching a Lambda to S3, which will start the process when the file is uploaded. You can kick your "availability activation" off by attaching a Lambda to the S3 output bucket. Background: I help run a streaming service and built the backend pipeline. This omits the entire "metadata management and analytics" side as well. That's left as an exercise for the user.
- mannyv 3y agoOne thing people pointed out is minimizing egress fees. You do that by using CloudFlare, fastly, or another CDN. You can get bandwidth costs down to < .001/gb by committing to $1500/mo in bandwidth. The CDNs will pull from S3 once, then cache it forever (assuming you do it right).
- mannyv 3y agoBut even if you'd don't use AWS use something else. Any video encoder service will be cheaper and more reliable than using k8 + your own hacked together docker instance. It's literally not worth it. Do you really want to spend your time messing around with ffmpeg and setting the correct GOP values? Or trying to create your own b-frame track? At some point you actually need to do what all these settings are for. Until that time you should use other services.
- 3y ago
- FaisalMahmoud 3y agoFor general video streaming, Mux.com has greatly decreased my development time. Getting playback working is straightforward. And for advanced use cases, like real time editing and preview in a web browser, it works as expected and doesn’t get in the way.
- thomasjudge 3y agoOP mentions that "I would love to be a little mouse and peek at YouTube’s complete architecture to see how far we are from them." You can occasionally find posts -often linked here- from another player in streaming video which you might have heard of, discussing technical architecture. For example, this might be a little lower level that you may be interested in as it relates to kernel optimizations to jack bit throughput rates, but I dig this sort of thing - https://www.youtube.com/watch?v=36qZYL5RlgY https://www.youtube.com/watch?v=36qZYL5RlgY
- cedws 3y agoI've heard somewhere that YouTube has their own transcoding hardware.
- EricBurnett 3y agoIndeed. https://blog.youtube/inside-youtube/new-era-video-infrastructure/ https://blog.youtube/inside-youtube/new-era-video-infrastruc... , https://research.google/pubs/pub50300/ https://research.google/pubs/pub50300/ . (Search the paper title and you should be able to find the pdf itself elsewhere). I'm not actually sure on balance how much transcode gets done in hardware vs software, since it's also very amenable to using batch compute that's otherwise idle. I'll guess that most or all live transcoding - streams, on-the-fly transcode into formats not pregenerated - are done in hardware, and transcoding new formats for the back catalog are probably done on a mixture of mechanisms where and when capacity is available. (Source: Googler, not on YouTube though.)
- jspizziri 3y agoWe’ve built a similar pipeline architecture for our product. One key thing I’ll mention is that we’re using Shaka-streamer which is a python wrapper around Shaka-packager (which in turn is a wrapper around ffmpeg). We queue our transcode jobs into a redis queue, and use k8s to scale the transcode workers based on queue volume. Lastly, as a few folks have mentioned, we have an experimental on-prem transcoding cluster with consumer grade HW that is pretty cheap. If you’re interested in working on transcoding I’d highly recommend taking a look at Shaka-packager/streamer.
- dilyevsky 3y agoI used to work on system like this and even built the logic to use preemptible pool effectively just like OP. If I had to design it from scratch today I would use Temporal for job scheduling - their durable compute concept is perfect fit for this and we had a lot of trouble maturing the equivalent scheduling system trying to keep up with rapidly growing scale
- akbansa 3y agoI am sure it's difficult for someone to build and scale video infrastructure. A few companies are doing it for you; plug in the APIs, and you're done. Gumlet (https://www.gumlet.com https://www.gumlet.com): Per-title encoding (Netflix's approach) to optimize and transcode your videos to boost engagement rates. Moreover, securing your videos is easy with digital rights management solutions paired with Widevine and Fairplay. Made for developers, by developers. Mux: Developer-friendly video infrastructure for your on-demand & live video needs. I love Gumlet because of their pricing and support.
- sonof0dn 3y agoGreat article. The other day in my local dev group I gave a talk about something similar but much less sophisticated.
- Maxou44 3y ago[dead]