7 ms·
FFmpeg merges WebRTC support
- karlkloss 1y ago"Sadly, you must enable JavaScript to get past this challenge." Nope. Get lost. Running random code from websites you don't know is asking for desaster.
- qwertox 1y agoWhat does this mean? That websites could connect directly to an FFmpeg instance and receive an audio- and/or video-stream? Phoronix has a somewhat more informative page: https://www.phoronix.com/news/FFmpeg-Lands-WHIP-Muxer https://www.phoronix.com/news/FFmpeg-Lands-WHIP-Muxer
- bigfishrunning 1y agoIt means that programs that use the FFmpeg libraries (looks like libavformat specifically) can consume webrtc streams
- qwertox 1y agoSo it's only the receiving part of WebRTC, now being able to use WHIP in order to ask a server for a stream?
- Sean-Der 1y agoCurrently only the sending part! WHIP is 'pushing media via WebRTC' - https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html WHEP is 'pulling media via WebRTC' - https://github.com/wish-wg/webrtc-http-egress-protocol/blob/main/draft-ietf-wish-whep.md https://github.com/wish-wg/webrtc-http-egress-protocol/blob/... WHEP isn't standardized/still changing a bit. After it lands I will try and get it into OBS/FFmpeg (and more)
- jcelerier 1y agoHmm, what does that mean for instance for workloads that use gstreamer's whepsrc? Is there a risk of incompatibility of a whep server running today with next year webrtc?
- okdood64 1y agoI still don't understand any practical use cases. Can you give some examples? (I'm not being obtuse here I'm genuinely curious what this can enable now.)
- ninkendo 1y agoMy guess is you could more easily build an open source client for whatever video conferencing system you want that uses WebRTC (most services like teams, discord, zoom, etc seem to use WebRTC as a fallback for browsers, if not using it wholesale for everything, although there may be countermeasures to block unofficial clients.)
- lmm 1y agoMy first thought is a nice way to save a stream in whatever format you want (e.g. transcode for watching on an old phone or something on your commute), just ffmpeg -i <stream> and then all your usual video format options, instead of having to download it and then convert it afterwards. ffmpeg also has some processing abilities of its own, so you could e.g. greenscreen (chroma key) from a stream onto an existing video background. ffmpeg is a pretty low-level building block and as others have said, it's mostly used as a library - a lot of video players or processing tools can now add support for stream inputs easily, and that's probably where the biggest impact is.
- jcelerier 1y agoI want my desktop app https://ossia.io https://ossia.io which uses ffmpeg to be able to send & receive video to another computer over internet without having to fiddle with opening ports on each other's routers. This combined with a server like vdo.ninja solves that.
- darkvertex 1y agoWebRTC excels at sub-second latency peer to peer, so you can do near-realtime video, so anywhere that is useful. Say you wanted to do a virtual portal installation connecting views from two different cities with live audio, you could have ffmpeg feed off a professional cinema or DSLR camera device with a clean audio feed and stream that over WebRTC into a webpage-based live viewer. Or say you wanna do a webpage that remote controls a drone or rover robot, it would be great for that.
- dark-star 1y agoAre there any popular/well-known WebRTC senders (or servers)? I'm pretty sure this is not for YouTube etc., right? So what would I watch through WebRTC?
- Sean-Der 1y agoTwitch supports WHIP today. Lots of WebRTC services support WHIP (Cloudflare, LiveKit, Dolby...) webrtcHacks has an article on it[0] kind of old, but captures the spirit of it! [0] https://webrtchacks.com/tag/simulcast/ https://webrtchacks.com/tag/simulcast/
- tyre 1y agoInteresting I keep getting blocked by the bot detection on iOS safari, both from our work WiFi and cellular data. Anubis let me go
- deleted 1y ago[deleted]
- jsheard 1y agoAre you getting the "access denied" page, or an infinite challenge loop?
- kairosisme 1y agoFWIW I also can’t pass the Anubis pass on iOS Safari, even though I can on any other site. I see the Anubis success screen for a moment before it switches to the “invalid response” screen. edit: Trying again a few minutes later worked
- rafram 1y ago[flagged]
- __turbobrew__ 1y agoI got stuck on access denied. Canada IPv4. Safari on iOS.
- xena 1y agoDo you happen to have a dual-stack network?
- MrThoughtful 1y agoI know there are JavaScript ports of FFmpeg and I would love to use them. But so far, I never got it working. I tried it with AI and this prompt: Make a simple example of speeding up an mp4 video in the browser using a version of ffmpeg that runs in the browser. Don't use any server side tech like node. Make it a single html file. But so far every LLM I tried failed to come up with a working solution.
- simlevesque 1y agoDon't try to do cutting edge stuff with a brain that doesn't know anything past a certian date.
- minimaxir 1y agoThe JS ports of FFmpeg (or WASM port if you want the in-browser approach) are very old and would be more than present in modern LLM training datasets, albeit likely not enough of a proportion for LLMs to understand it well. https://github.com/Kagami/ffmpeg.js/ https://github.com/Kagami/ffmpeg.js/ https://github.com/ffmpegwasm/ffmpeg.wasm https://github.com/ffmpegwasm/ffmpeg.wasm
- colechristensen 1y agoTrying to do things off the beaten path with LLMs is rarely successful, especially if there's a related much more popular option. I'm convinced that programmers' bias towards LLMs is strongly correlated with the weirdness of their work. Very often my strange ideas pushed to LLMs look like solutions but are rather broken and hallucinated attempts which only vaguely represent what needs to be done.
- bigfishrunning 1y ago> I'm convinced that programmers' bias towards LLMs is strongly correlated with the weirdness of their work. This is an extremely astute observation; my work has always been somewhat weird and I've never found LLMs to be more then an interesting party-trick
- 1y ago
- throwpoaster 1y agoWhat’s ffmpeg security auditing like? Seems reactive from their site.
- bigfishrunning 1y agoUsing Pion no less! very cool!
- Xeoncross 1y agoI assume you mean https://github.com/pion/webrtc https://github.com/pion/webrtc, I don't see any Go, I thought they just fixed a bug with compatibility with Pion.
- autoexec 1y agoHopefully this doesn't make it more dangerous to keep ffmpeg on our systems. WebRTC security flaws are responsible for a lot of compromises. It's one of the first things I disable after installing a browser
- Sean-Der 1y agoWhat security flaws? This implementation is very small. I feel 100% confident we are giving users the best thing possible.
- globie 1y agoI assume autoexec is referring to the plethora of WebRTC vulnerabilities which have affected browsers, messengers, and any other software which implements WebRTC for client use. Its full implementation is seemingly difficult to get right. Of course, you're right that this implementation is very small. It's very different than a typical client implementation, I don't share the same concerns. It's also only the WHIP portion of WebRTC, and anyone processing user input through ffmpeg is hopefully compiling a version enabling only the features they use, or at least "--disable-muxer=whip" and others at configure time. Or, you know, you could specify everything explicitly at runtime so ffmpeg won't load features based on variable user input.
- gruez 1y ago>I assume autoexec is referring to the plethora of WebRTC vulnerabilities which have affected browsers, messengers, and any other software which implements WebRTC for client use. Its full implementation is seemingly difficult to get right. Like what? I did a quick search and most seem to be stuff like ip leaks and fingerprinting, which isn't relevant in ffmpeg.
- lpln3452 1y agoThis is exactly the question I have. While WebRTC causes fingerprinting risks in browsers, isn’t that unrelated to running ffmpeg?
- jauntywundrkind 1y agoNot the SCTP parts! It's implementing WebRTC-HTTP Ingestion Protocol (WHIP), a commonly used low-latency HTTP protocol for talking to a gateway that talks actual WebRTC to peers over WebRTC's SCTP-based protocol. https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html I hope some day we can switch to a QUIC or WebTransport based p2p protocol, rather than use SCTP. QUIC does the SCTP job very well atop existing UDP, rather than add such wild complexity & variance. One candidate, Media-over-Quic ?MoQ), but the browser doesn't have a p2p quic & progress on that stalled out years ago. https://quic.video/ https://quic.video/ https://datatracker.ietf.org/group/moq/about/ https://datatracker.ietf.org/group/moq/about/
- Sean-Der 1y agoHow would you like to see/use the SCTP parts? I am not sure how to expose them since the WHIP IETF draft makes no mention/suggestion of it. Most 'WHIP Providers' also support DataChannel. But it isn't a standardized thing yet
- jauntywundrkind 1y agoWebRTC actual's complezity is very high. WHIP seems to be the standard path for most apps to integrate, but it does rely on an exterior service to actually do anything. Hypothetically ffmpeg could be an ICE server for peer-connecting, do SDP for stream negotiation possibly with a side of WHEP (egress protocol) as well, could do SCTP for actual stream transfer. Such that it could sort of act as a standalone peer, rather than offload that work to a gateway service. Worth noting that gstreamer & OBS also are WHIP based, rely on an external gateway for their WebRTC support. There's not one clear way to do a bunch of the WebRTC layer cake (albeit WHEP is fairly popular I think at this point?), so WHIP is a good way to support sending videos, without having to make a bunch of other decisions that may or may not jive with how someone wants to implement WebRTC in their system; those decisions are all in the WHIP gateway. It may be better to decouple, not try to do it all, which would require specific opinionative approaches.
- msgodel 1y agoThat should make self hosting streams/streaming CDNs way easier. If you know how to use it ffmpeg is such an amazing stand alone/plug and play piece of media software.
- Sean-Der 1y agoIt's so exciting. Especially with Simulcast it will make it SO cheap/easy for people. I made https://github.com/Glimesh/broadcast-box https://github.com/Glimesh/broadcast-box in a hope to make self-hosting + WebRTC a lot easier :)
- eigenvalue 1y agoLLMs really know how to use it incredibly well. You can ask them to do just about any video related task and they can give you an ffmpeg one liner to do it.
- rietta 1y agoWow, you are not wrong. I just asked Gemini "how can I use ffmpeg to apply a lower third image to a video?" and it gave a very detailed explanation of using an overlay filter. Have not tested its answer yet but on its face it looks legit.
- Ajedi32 1y agoIt could very well be legit, but if you "have not tested its answer yet" the fact that it can generate something that looks plausible doesn't really tell you much. Generating plausible-sounding but incorrect answers is like the #1 most common failure mode for LLMs.
- asadm 1y agoin recent usage, that only happens 10% of the time for me. Usually the results are grounded and so work usually fine.
- 1y ago
- Sean-Der 1y agoI am so incredibly excited for WebRTC broadcasting. I wrote up some reasons in the Broadcast Box[0] README and the OBS PR [1] Now that GStreamer, OBS and FFmpeg all have WHIP support we finally have a ubiquitous protocol for video broadcasting for all platforms (Mobile, Web, Embedded, Broadcasting Software etc...) I have been working on Open Source + WebRTC Broadcasting for years now. This is a huge milestone :) [0] https://github.com/Glimesh/broadcast-box?tab=readme-ov-file#what-is-broadcast-box https://github.com/Glimesh/broadcast-box?tab=readme-ov-file#... [1] https://github.com/obsproject/obs-studio/pull/7926 https://github.com/obsproject/obs-studio/pull/7926
- maxmcd 1y agoThanks for all your work Sean! It's been a delight to use your webrtc libs and see your impact across a broad range of technical efforts.
- Sean-Der 1y agoThank you :) When are you coming back to the WebRTC space, lots more cool stuff you could b doing :) I really loved [0] it's so cool that a user can access a server behind a firewall/NAT without setting up a VPN or having SSH constantly listening. [0] https://github.com/maxmcd/webtty https://github.com/maxmcd/webtty
- WhyNotHugo 1y agoAre there any video players which can play a webrtc stream? Last I checked, VLC and other popular tools still don’t support it.
- RedShift1 1y agoThat's impossible, VLC supports everything. If VLC doesn't support it, it doesn't exist.
- byteknight 1y agoAmen.
- SeriousM 1y agoI can't wait to see this in Jellyfin implemented!
- colordrops 1y agoWhat would this provide?
- theobr 1y agoAbsolutely huge
- alexfromapex 1y agoIt would be cool to have a chat too
- pkz 1y agoDoes this mean that ffmpeg now can record a Jitsi video meeting audio stream?
- shmerl 1y agoDoes it allow more realtime streaming than SRT on LAN? I'm still waiting for ffmpeg CLI tool to merge pipewire + xdg-desktop-portal support. You still can't record a screen or window on Wayland with it.
- Sean-Der 1y agoWith WebRTC you can expect ~100ms with zero optimizations on your LAN. With bitwhip[0] I got it way lower then that even. [0] https://github.com/bitwhip/bitwhip https://github.com/bitwhip/bitwhip
- shmerl 1y agoThat's nice. I had hard time getting low latency with SRT, but managed to get within the range of roughly slightly less than one second using gpu-screen-recorder on one end and ffplay on the other end with flags for low latency.
- matt3210 1y agoI love seeing the Anubis graphics unexpectedly. I’ve seen it at ffmpeg and gnu so far (among others)
- crabmusket 1y agoI do too, but this time it won't let me in :/
- leland-takamine 1y agoAnyone been able to successfully build ffmpeg from source to include whip support? Struggling to figure out the right ./configure options
- _Manch 1y agoYou need --enable-muxer=whip and --enable-openssl
- deleted 1y ago[deleted]
- leland-takamine 1y agoSeems work but fails with 500 when streaming to cloudflare stream though: https://gist.github.com/Leland-Takamine/1a13f31c7521d02236242af2a643e1f4 https://gist.github.com/Leland-Takamine/1a13f31c7521d0223624...
- leland-takamine 1y agoResolved by adding an audio stream
- dyl000 1y agoAnubis isn’t letting me through ;(
- Mofpofjis 1y agoA commit that was "co-authored-by" 6+ people and has three thousand lines of code: this is a total wreck of a development workflow. This feature should have been implemented with a series of about 20 patches. Awful.
- jpk 1y agoI mean, it probably was a branch that several people contributed commits to that was squashed prior to merge into mainline. Folks sometimes have thoughts about whether there's value in squashing or not, but it's a pretty common and sensible workflow.
- fc417fc802 1y ago> common and sensible Perhaps "common and technically works" would be a better way to put that (similarly for rebase). I suspect people would stop squashing if git gained the ability to tag groups of commits with topics in either a nested or overlapping manner.
- Daemon404 1y ago(long time FFmpeg dev here) You are being downvoted, but you are entirely correct. This is also explicitly not allowed in FFmpeg, but this was pushed after many months, with no heads up on the list, no final review sign off, and with some developers expressing (and continuing to express) reservations about its quality on the list and IRC.
- bigfishrunning 1y agoThat's really unfortunate to hear. I'm a huge fan of Webrtc and Pion, and was very excited to get some ffmpeg integration -- hopefully some of the quality issues will be ironed out before the next ffmpeg release
- Daemon404 1y agoThere's quite some time until the next release, I believe, so it should be. The biggest thing missing right now is NACK support, and one of the authors has said they intend to do this (along with fixing old OpenSSL version support, and supporting other libraries). Until that is done, it isn't really "prod ready", so to speak. For some context, there has been a history of half-supported things being pushed to FFmpeg by companies or people who just need some subset of $thing, in the past, and vendors using that to sell their products with "FFmpeg isn't good enough" marketing, while the feature is either brought up to standard, or in some cases, removed, as the original authors vanish, so it's perhaps a touchy subject for us :) (and why my post was perhaps unnecessarily grumpy). As for the git / premature push stuff, I strongly believe it is a knock-on effect of mailing list based development - the team working on this support did it elsewhere, and had a designated person send it to the list, meaning every bit of communication is garbled. But that is a whole different can of worms :D.
- quantadev 1y agoPublic Service Announcement: There's a reddit topic for WebRTC, that doesn't get enough action imo! Get in there ya'll... https://www.reddit.com/r/WebRTC https://www.reddit.com/r/WebRTC
- spartanatreyu 1y agoNo. Reddit lost their own community's trust when the CEO ejected the community's moderators. Information posted there is now far less likely to be qualitative compared to other places, so what's the point of going there?
- quantadev 1y agoWhere's the best place that people are talking about WebRTC then? Hacker News doesn't have rooms by topic does it?
- Sean-Der 1y agoBroadcast Box has a discord https://discord.gg/An5jjhNUE3 https://discord.gg/An5jjhNUE3 You could also join the Pion one https://pion.ly/discord https://pion.ly/discord Other place is video-dev Slack https://www.video-dev.org/ https://www.video-dev.org/ X also has a great community of hackers!
- quantadev 1y agoThanks for the links! Yeah, I'm on X as my primary news source for tech, but I mainly only follow AI-related people and orgs. I should search for WebRTC.
- _flux 1y agoAre now Discord, Slack and X now better options than Reddit? Too bad nobody is using open forums (e.g. Discourse, Matrix, Mastodon).
- 1y ago
- cranberryturkey 1y agoCan someone ELI5 what this means? i've been using ffmpeg for over a decade.
- esbeeb 1y agoWebRTC is very, very hard to code for. But if FFmpeg abstracts that complexity away, then WebRTC becomes much easier to add to a software project wishing to benfit from that which WebRTC offers.
- cranberryturkey 1y agoI guess I still don't understand. You don't really "code" with ffmpeg. It just is used to transform media formats or publish to a public streaming endpoint.
- marxisttemp 1y agoAll of ffmpeg’s functionality is accessible from C (and transitively most other programming languages) via libavformat, libavcodec etc. FFmpeg supporting WebRTC means that projects using these libraries gain support for WebRTC in code.
- mrheosuper 1y agoso RTC is real time communication, not real time clock...
- wang_zuo 1y agoThe author seems to be an undergraduate from china. very impressive!
- esbeeb 1y agoGajim, the XMPP client, has been awaiting this for a long time! Their Audio/Video calling features fell into deprecation, and they've been patiently waiting for FFmpeg to make it much easier for them to add Audio/Video calling features back again.
- dedosk 1y agoGajim and XMPP is still used out there? I miss the days when I could use pidgin for chat apps. Now it is all wallet garden/app-per-service.
- NicuCalcea 1y agoI'm quite happy with Beeper, it still has some bugs and isn't open source, but it saves me from remembering where different contacts live.
- rw_grim 1y agoThere's plugins for most of the modern stuff at https://pidgin.im/plugins https://pidgin.im/plugins
- ec109685 1y agoWhy doesn’t a PR of that magnitude come with tests?
- chompychop 1y agoI have a beginner question - Can WebRTC be used as an alternative to sending base64-encoded images to a backend server for image processing? Is this approach recommended?
- actionfromafar 1y agoNow I have the question - when does one send base64-encoded images to a backend server?
- Spivak 1y agoThe OpenAI API is a pretty high-profile example of this existing in the real world. You use it in their conversations interface when you want to include images in the conversation. Discord also uses it for attachments https://discord.com/developers/docs/reference#image-data https://discord.com/developers/docs/reference#image-data. More generally it's when you want to send image data as part of a larger JSON blob.
- Sean-Der 1y agoDepends on your needs! https://github.com/pion/webrtc/tree/master/examples/save-to-disk https://github.com/pion/webrtc/tree/master/examples/save-to-... here is an example of a server that does what you need on the backend
- sylware 1y agoOMG, this is not completely brain damaged c++ code lost in the middle of one of the web engines from the whatng cartel??? or C code with one billion dependencies with absurd SDKs??? Quick! Quick! I need to find something bad about it... wait... AH! Does it compile with the latest libressl? Hopefully not (like python _ssl.c) and I can start talking bad about it. ;P Ofc, that was irony. We all know the main issue with webRTC is not its implementations, but webRTC itlself. All that said, it is exactly at this very time twitch.tv chose to break ffmpeg HLS (its current beta HLS streams are completely breaking ffmpeg HLS support...).
- FrostKiwi 1y agoOMG YEEEEES. I'm building web based remote control and if this allows me to do ffmpeg gdigrab, have that become a WebRTC stream and be consumed by a client without the ExpressJS gymnastics I do right now, I'll be over the moon.