9 ms·
I designed my own fast game streaming video codec – PyroWave
- Fidelix 1y agoUnbelievable... Good job mate. Can't wait until one day this gets into Moonlight or something like it.
- cpeth 1y agoExactly what I was thinking. Wish I had the time and expertise to give adding support for this codec myself a go. Streaming Clair Obscure over my LAN via Sunshine / Moonlight is exactly my use-case and the latency could definitely be better.
- kreco 1y agoI have nothing interesting to say, but thanks for author and for the one who shared the article. It was a good read.
- kg 1y agoThe sample screenshot of Expedition 33 is really impressive quality considering it appears to be encoding at around 1 bit per pixel and (according to the post) it took a fraction of a millisecond to encode it. This is an order of magnitude faster than typical hardware encoders, AFAIK. Very cool work explained well.
- sippeangelo 1y agoI know next to nothing about video encoding, but I feel like there should be so much low hanging fruit when it comes to videogame streaming if the encoder just cooperated with the game engine even slightly. Things like motion prediction would be free since most rendering engines already have a dedicated buffer just for that for its own rendering, for example. But there's probably some nasty patent hampering innovation there, so might as well forget it!
- toast0 1y agoFor 2d sprite games, OMG yes, you could provide some very accurate motion vectors to the encoder. For 3d rendered games, I'm not so sure. The rendering engine has (or could have) motion vectors for the 3d objects, but you'd have to translate them to the 2d world the encoder works in; I don't know if it's reasonable to do that ... or if it would help the encoder enough to justify.
- sudosysgen 1y agoSchemes like DLSS already do provide 2D motion vectors, it's not necessarily a crazy ask.
- IshKebab 1y agoI don't think games do normally have a motion vector buffer. I guess they could render one relatively easily, but that's a bit of a chicken and egg problem.
- garaetjjte 1y agoThey do, one reason is postprocessing effects like motion blur, another is antialiasing like TAA or DLSS upscaling.
- IshKebab 1y agoYeah I did almost mention motion blur but do many games use that? I don't play many AAA games TBF so maybe I'm just out of date... Take something like Rocket League for example. Definitely doesn't have velocity buffers.
- superjan 1y agoI love this. The widely used standards for video compression are focused on compression efficiency, which is important if you’re netflix or youtube, but sometimes latency and low complexity is more important. Even if only to play around and learn how a video codec actually works.
- CharlesW 1y ago> The widely used standards for video compression are focused on compression efficiency, which is important if you’re netflix or youtube, but sometimes latency and low complexity is more important. That's a misconception. All modern video codecs (i.e. H.264/AVC, H.265/HEVC, AV1) have explicit, first-class tools, profiles, and reference modes aimed at both low- and high-resolution low‑latency and/or low‑complexity use. AV1: Improving RTC Video Quality at Scale: https://atscaleconference.com/av1-improving-rtc-video-quality-at-scale/ https://atscaleconference.com/av1-improving-rtc-video-qualit...
- westurner 1y agoThere are hardware AV1 encoders and decoders. Objective metrics and tools for video encoding and source signal quality: netflix/vmaf, easyVmaf, psy-ex/metrics, ffmpeg-quality-metrics, Ffmpeg settings for low-latency encoding: # h264, h265 -preset ultrafast -tune zerolatency # AV1 -c:v libsvtav1 -preset 8 -svtav1-params tune=0:latency-mode=1 -g 60 It's possible to follow along with ffmpeg encoding for visual inspection without waiting for the whole job to complete with the tee muxer and ffplay. GPU Screen Recorder and Sunlight server expose some encoder options in GUI forms, but parameter optimization is still manual; nothing does easyVmaf with thumbnails of each rendering parameter set with IDK auto-identification of encoding artifacts. Ardour has a "Loudness Analyzer & Normalizer" with profiles for specific streaming services. What are good target bitrates for low-latency livestreaming 4k with h264, h265 (HDR), and AV1?
- westurner 1y agoFFmpeg Explorer is made with ffmpeg.wasm: https://github.com/antiboredom/ffmpeg-explorer https://github.com/antiboredom/ffmpeg-explorer .. web: https://ffmpeg.lav.io/ https://ffmpeg.lav.io/
- crazygringo 1y agoFascinating! But... > The go-to solution here is GPU accelerated video compression Isn't the solution usually hardware encoding? > I think this is an order of magnitude faster than even dedicated hardware codecs on GPUs. Is there an actual benchmark though? I would have assumed that built-in hardware encoding would always be faster. Plus, I'd assume your game is already saturating your GPU, so the last thing you want to do is use it for simultaneous video encoding. But I'm not an expert in either of these, so curious to know if/how I'm wrong here? Like if hardware encoders are designed to be real-time, but intentionally trade off latency for higher compression? And is the proposed video encoding really is so lightweight it can easily share the GPU without affecting game performance?
- averne_ 1y agoHardware GPU encoders refer to dedicated ASIC engines, separate from the main shader cores. So they run in parallel and there is no performance penalty for using both simultaneously, besides increased power consumption. Generally, you're right that these hardware blocks favor latency. One example of this is motion estimation (one of the most expensive operations during encoding). The NVENC engine on NVidia GPUs will only use fairly basic detection loops, but can optionally be fed motion hints from an external source. I know that NVidia has a CUDA-based motion estimator (called CEA) for this purpose. On recent GPUs there is also the optical flow engine (another separate block) which might be able to do higher quality detection.
- miladyincontrol 1y agoIm pretty sure they arent dedicated ASIC engines anymore. Thats why hacks like nvidia-patch are a thing where you can scale up NVENC usage up to the full GPU's compute rather than the arbitrary limitation nvidia adds. The penalty for using them within those limitations tends to be negligible however. And on a similar note, NvFBC helps a ton with latency but its disabled on a driver level for consumer cards.
- theshackleford 1y ago> Im pretty sure they arent dedicated ASIC engines anymore. They are. That patch doesnt do what you think it does.
- CharlesW 1y ago> Given how niche and esoteric this codec is, it’s hard to find any actual competing codecs to compare against. It'd be interesting to see benchmarks against H.264/AVC (see example "zero‑latency" ffmpeg settings below) and JPEG XS. -c:v libx264 -preset ultrafast -tune zerolatency \ -x264-params "keyint=1:min-keyint=1:scenecut=0:rc-lookahead=0" \ -bf 0 -b:v 8M -maxrate 8M -bufsize 1M
- kookamamie 1y agoNot bad. The closest competition would be NDI from NewTek, now Vizrt. It targets similar bitrate and latency ranges.
- deleted 1y ago[deleted]
- raphman 1y agoVery cool - That's nearly exactly what I need for a research project. FWIW, there's also the non-free JPEG-XS standard [1] which also claims very low latency [2] and might be a safer choice for commercial projects, given that there is a patent pool around it. [1] https://www.jpegxs.com/ https://www.jpegxs.com/ [2] https://ds.jpeg.org/whitepapers/jpeg-xs-whitepaper.pdf https://ds.jpeg.org/whitepapers/jpeg-xs-whitepaper.pdf
- jamesfmilne 1y agoJPEG-XS is great for low latency, but it uses more bandwidth. We're using it for low-latency image streaming for film/TV post production: https://www.filmlight.ltd.uk/store/press_releases/filmlight-introduces-remote/ https://www.filmlight.ltd.uk/store/press_releases/filmlight-... We currently use the IntoPIX CUDA encoder/decoder implementation, and SRT for the low-level transport. You can definitely achieve end-to-end latencies <16ms over decent networks. We have customers deploying their machines in data centres and using them in their post-production facilities in the centre of town, usually over a 10GbE link. But I've had others using 1GbE links between countries, running at higher compression ratios.
- indolering 1y agoA patent pool doesn't make you safer: it's just a patent troll charging you to cross the bridge. They are not offering insurance against more patent trolls blackmailing you after you cross the bridge.
- raphman 1y agoWhile I am personally opposed to software patents, I'd argue that the JPEG XS patent holders [1] are not 'patent trolls' in any meaningful sense of the word. While I have no personal experience on that topic, I'd assume that a codec with a patent pool is a safer bet for a commercial project. Key aspects being protected by patents makes it less likely that some random patent troll or competitor extorts you with some nonsense patent. Also, using e.g., JPEG XS instead of e.g., pyrowave also ensures that you won't be extorted by the JPEG XS patent holders. One may call this a protection racket - but under the current system, it may make economical sense to pay for a license instead of risking expensive law suits. [1] https://www.jpegxspool.com/ https://www.jpegxspool.com/
- 10000truths 1y agoAre there any solutions to game streaming that build an RPC on top of the DirectX/Vulkan API and data structures? I feel like streaming a serialized form of the command queue over the network would be more efficient than streaming video frames.
- babypuncher 1y agoWhat's the point in streaming a video game from one computer to another if the client machine still needs the expensive and power hungry dedicated graphics hardware to display it?
- mschuster91 1y agoYou could use that to have a chungus dGPU with a massive amount of VRAM keep all the assets, position data and god knows what else to do the heavy lifting - determine what needs to be drawn and where, deal with physics, 3D audio simulation etc. - and then offload only a (comparatively) small-ish amount of work to the client GPU.
- debugnik 1y agoBut those are the cheap parts compared to the rendering itself. A small-ish amount of work is what we're already sending, the final image, because anything else takes much, much more GPU work.
- babypuncher 1y agoThe client GPU still needs all those assets in its own VRAM in order to render any scene that uses them. You would need to stream all of those assets in real time to the client, and last I checked consumer network interfaces are checks notes slower than 16 PCI Express lanes. I'm still unsure what computational resources are being saved for the client here, the actual rasterization is where the bulk of the work is done.
- wmf 1y agoI don't think this is true when you count texture uploads. Loading 8 GB of textures over the network would take a while.
- richardw 1y agoIf the system knew both sides were the same vendor or used the same algorithm, would it be better to stream the scene/instructions rather than the video? I suppose the issue would be media. Faster to load locally than push it out. Could be semi solved with typical web caching approaches.
- Firehawke 1y agoI've been mulling over the concept of a codec designed for streaming NES video. Stream the VRAM tiles and the RAM data needed to reconstruct the output locally, which could be done in a shader along with CRT simulation if desired. Very much a one-trick pony, but probably considerably less bandwidth-intensive than even the original resolution (320x224) under nearly any acceptable bitrate.
- Karliss 1y agoAre you suggesting to do the 3d rendering on client side, which would require a beefy GPU for the client? The whole point of game streaming is that the game can run on the big noisy, power hungry computer located somewhere else, but the device receiving the stream only needs minimal compute power to decode video.
- monster_truck 1y agoLooks like NDI without any of the conveniences. You're doing something wrong if nvenc is any slower, the llhp preset should be all you need.
- hashtekar 1y agoWhat a great read and such a throwback for me. I worked on video compression techniques using wavelets 30yrs ago. Computing power and networking speeds were not what they are now and I had difficulty getting the backing to carry it forward. I’m so happy that this still has such active development and boundaries are still being pushed. Bravo.
- pimlottc 1y agoWhat are existing streaming game services using? Surely there is some good previous work in this area?
- wmf 1y agoProbably GPU hardware encoding using low-latency mode. Somebody reported 1 ms latency.
- toast0 1y agoI doubt any of those are willing to drop 100mbps+ per client. If the clients can even manage it.
- theshackleford 1y ago> I doubt any of those are willing to drop 100mbps+ per client. There is zero reason to not be willing to do so on a local network. > If the clients can even manage it. They can. I use significantly more than that already.
- detaro 1y ago"Streaming game service" clearly does not refer to something in your local network
- theshackleford 1y agoPlenty of people I know refer to local game streaming setups as services, especially when they’re run as system services or hosted servers (like I do). In my world, anything that runs persistently and offers functionality, whether it’s local or the cloud, is a service. I run a local game streaming setup that fits that definition exactly and so yeah, when I hear game streaming service, that’s what I think of. That said, I get that others could associate the term more with commercial cloud solutions with some reflection. Just not where my brain goes by default, since I don’t use them and nobody I know does or would as we can host our own “services” for this purpose. My bad.
- Thaxll 1y agoThere is the creator of VLC that is working on something similar, very cutting edge. https://streaminglearningcenter.com/codecs/an-interview-with-jean-baptiste-kempf-of-kyber.html https://streaminglearningcenter.com/codecs/an-interview-with... Ultra low latency for streaming. https://www.youtube.com/watch?v=0RvosCplkCc https://www.youtube.com/watch?v=0RvosCplkCc
- torginus 1y agoHaving worked in the space, I'd have to say hardware encoders and H.264 is pretty dang good - NVENC works with very little latency (if you tell it to, and disable the features that increase it, such as multiple frame prediction, B-frames). The two things that increase latency are more advanced processing algorithms, giving the encoder more stuff to do, and schemes that require waiting multiple frames. If you go disable those, the encoder can pretty much start working on your frame the nanosecond the GPU stops rendering to it, and have it encoded in <10ms.
- dishsoap 1y ago10ms is quite long in this context.
- Wowfunhappy 1y ago> have it encoded in <10ms. For context, OP achieved 0.13 ms with his codec.
- pjc50 1y ago"0.13 ms on a RX 9070 XT on RADV." "interesting data point is that transferring a 4K RGBA8 image over the PCI-e bus is far slower than compressing it on the GPU like this, followed by copying over the compressed payload." "200mbit/s at 60 fps" It's certainly a very different set of tradeoffs, using a lot more bandwidth.
- theshackleford 1y ago> It's certainly a very different set of tradeoffs, using a lot more bandwidth. Wasnt that the point? > These use cases demand very, very low latency. Every millisecond counts here > When game streaming, the expectation is that we have a lot of bandwidth available. Streaming locally on a LAN in particular, bandwidth is basically free. Gigabit ethernet is ancient technology and hundreds of megabits over WiFi is no problem either. This shifts priorities a little bit for me at least.
- sitkack 1y agoI'd want to know how it compares to https://github.com/phoboslab/qoi https://github.com/phoboslab/qoi One thing to note when designing a new video codec is to carpet bomb around the idea with research projects to stake claim to any possible feature enhancements. Anything can have an improvement patent filed against, no matter the license.
- Almondsetat 1y agoQOI is lossless
- atiedebee 1y agoQOI is an image codex, lossless, and not parallelizable. I don't think they are comparable
- sitkack 1y agoBoth of you are falling into the trap of comparing at the lowest possible level. Here is an example of QOI being used in a video codec, https://wide-video.github.io/qov/static/demo.html https://wide-video.github.io/qov/static/demo.html https://github.com/wide-video/qov https://github.com/wide-video/qov
- theLiminator 1y agoPretty cool, though I think in practice network latency dominates so much that this kind of optimization is fairly low impact. I think the main advantage is perhaps the robustness against packet drops is better.
- babypuncher 1y agoIt could make in-home streaming actually usable for me. I've never been happy with the lag in Steam's streaming or moonlight, even when both the server and client are on the same switch. That's not a network latency problem, that's an everything else problem.
- _kb 1y agoDepends on the network. Where this style of codec is common you’re not traversing internet so transport latency, including switch forwarding, is normally in the microseconds. The killer is the display device that ends up rendering this. If you’re not careful that can add 10-100ms to glass to glass times.
- theshackleford 1y ago> though I think in practice network latency dominates so much that this kind of optimization is fairly low impact. In practice, no. Network latency is the least problematic part of the stack. I consistently get <3ms. It's largely encode/decode time which in my setup sits at around 20ms meaning any work in this area would actually have a a HUGE impact.
- tombert 1y agoOne of my bucket list things is to some day build a video codec from scratch. I have no delusions of competing with h264 or anything like that, just something that does some basic compression and can play videos in the process. Maybe I should try for that next weekend.
- nairoz 1y agoIt's really cool. I have always wondered if it would be possible to have video encoders designed for some specific games with prior knowledge about important regions to encode with more details. Example would be the center of the screen for the main character.
- DecentShoes 1y agoHell, you could do eye tracking and full on foveated rendering.
- keketi 1y agoHave an LLM transcribe what is happening in the game into a few sentences per frame, transfer the text over network and have another LLM reconstruct the frame from the text. It won't be fast, it's going to be lossy, but compression ratio is insane and it's got all the right buzzwords.
- foota 1y agoYou've got my attention
- poglet 1y agoMaybe even one day we reach point where the game can run locally on the end users' machine.
- jameshart 1y agoFrame 1: You are standing in an open field west of a white house, with a boarded front door. There is a small mailbox here.
- Y_Y 1y agokill jester
- Eduard 1y ago(user input: mouse delta: (-20, -8)) Frame 2: A few blades of grass sway gently in the breeze. The camera begins to drift slightly, as if under player control — a faint ambient sound begins: wind and birds.
- taneq 1y agoAh, this explains why there are clowns under the bed and creepy children staring at me from the forest.
- nusl 1y agoThey did this :P https://www.youtube.com/watch?v=ZpCrBBj6AWE https://www.youtube.com/watch?v=ZpCrBBj6AWE
- 1y ago
- Cadwhisker 1y agoThis CODEC uses the same base algorithm as HTJ2K (High-Throughput JPEG 2000). If the author is reading this, it would be very interesting to read about the differences between this method and HTJ2K.
- mleonhard 1y agohttps://github.com/Themaister/pyrowave https://github.com/Themaister/pyrowave
- ChadNauseam 1y agoGreat article. One thing I've always noticed is that when you get good enough at coding it just turns into math. I hope I can reach that level some day.
- _kb 1y agoThis is a really nice walkthrough of matching trade offs to acceptable distortions for a known signal type. Even if you’re selecting rather than designing a codec, it’s a great process to follow. For those interesting in the ultra low latency space (where you’re willing to trade a bit of bandwidth to gain quality and minimise latency), VSF have a pretty good wrap up of other common options and what they each optimise for: https://static.vsf.tv/download/technical_recommendations/VSF_Pro-AV-CODEC-Landscape-Report_24-10-07.pdf https://static.vsf.tv/download/technical_recommendations/VSF...
- freshtake 1y agoIf you're focused solely on local network streaming, you can throw most of the features of modern codecs out the window. The trade-off is bandwidth, but if the network can support 100 Mbps, you can get remarkably low latency with relatively little processing. For example, Microsoft's DXT codec lacks most modern features (no entropy coding, motion comp, deblocking, etc.), but delivers roughly 4x to 8x compression and is hardware decodable (saving on decoding and presentation latency). Of course, once you've tuned the end to end capture-encode-transmit-decode-display loop to sub 10 ms, you then have to contend with the 30-100 ms of video processing latency introduced by the display :-)
- Fokamul 1y agoBy any chance, does anyone know what streaming services use? Like Boosteroid or GeforceNow. Because GeforceNow has best streaming quality in the business and Boosteroid have various problems like stuttering etc.
- Almondsetat 1y agoVC-2 is an intra-only wavelet-based ultra low latency codec developed by the BBC years ago for exactly this purpose. It is royalty free and currently the only implementations are in ffmpeg and in the official BBC repository, and are CPU based. I am planning to make a CUDA accelerated version for my master thesis, since the Vulkan implementations made at GSoC last year still suck quite a bit. I would suggest people to look into this codec
- oplav 1y agoIn your experience, how does VC-2 compare to JPEG XS from a quality perspective? The JPEG XS resources I’ve seen say JPEG XS has higher visual quality, but curious what it’s like in practice.
- Almondsetat 1y agoJPEG-XS is an almost direct successor to VC-2. They use the same techniques and if you read JPEG-XS's whitepaper they explicitly cite VC-2 as an inspiration and a target to surpass. JPEG-XS is an improvement, there is not doubt about that, but unfortunately they decided to patent it for all uses. In both cases, the publicly available software implementations are very few, CPU-based, and the ones that aren't are implemented in hardware inside business AV solutions.
- averne_ 1y agoDo you mind going in some detail as to why they suck? Not a dig, just genuinely curious.
- Almondsetat 1y ago95% GPU usage but only x2 faster than the reference SIMD encoder/decoder
- actionfromafar 1y agoWhat I wonder is, how do you get the video frames to be compressed from the video card into the encoder? The only frame capture APIs I know, take the image from the GPU, to CPU RAM, then you can put it back into the GPU for encoding. Are there APIs which can sidestep the "load to CPU RAM" part? Or is it implied, that a game streaming codec has to be implemented with custom GPU drivers?
- tomaskafka 1y agoI love how the final step is “Dopamine is released in target brain” We can minimize the latency and save on rendering by doing only this step :).
- deleted 1y ago[deleted]
- maxglute 1y agoInfra really caught up to point where remote streaming my desktop to various devices mobile devices have been viable. I kind of wish windows phone OS didn't fail so hard, it would be nice to toggle mobile mode in a remote session and have years of apps built to support it.