4 ms·
Encoding speed can be very important, especially if you are seeking to decode the resulting image as quickly as possible at all times. Certain image formats are
by bob1029 4y ago
Encoding speed can be very important, especially if you are seeking to decode the resulting image as quickly as possible at all times. Certain image formats are wildly better real-time containers than others. Encoding speed is irrelevant unless it's very low or very high. Most people are thinking amortize the decode against the encode and don't even consider what would happen if you took it into the other extreme.
Did you know that, with the correct SIMD code, you can encode a 1080p bitmap into a 4:4:4 JPEG within 10 milliseconds on a typical consumer PC? Encoding times like this open up an entire dimension of new possibilities.
I do have a side project where I use libjpegturbo to synchronize a frame buffer between server & web client (i.e. 60 FPS real-time). HN may frown on using [m]jpeg for real-time applications today, but in my experience it has proven to be a very robust path. Bandwidth is really the only tradeoff (and it is a big one). This is the very last thing Netflix would want to do (short of sending RAW files to your TV), but it does provide some very compelling attributes in other areas such as streaming gaming.
Our networks are only getting faster over time. Intraframe video compression techniques are really elegant in their simplicity and resilience to motion-induced artifacts (as seen with x264, et. al.). They also can provide lower latency and easier recovery for clients (since every frame is a keyframe).
For real-time applications, I'd happily trade some extra bits per second to have a final result sooner. Especially if it looked better.
- raxxorraxor 4y agoAdditionally it should be mentioned how crazy imaging tech got lately. USB2 with 480 mbit/s is at the physical limit near the speed of light. This is why it easily fails in reality if you have the slightest disturbance. Faster busses like USB3 basically use more lines even if we still call it serial... If you would transfer raw image data (rgba) you get what for full hd images? Around 7 fps? I don't even want to see the bus for modern smartphone cameras that now seem to casually have around 100 megapixels... So very fast "on-site" encoding is pretty much mandatory today...
- mwcampbell 4y agoWhat do you think about MJPEG versus RFB (the VNC protocol)? I know that in modern VNC implementations, at least unofficial ones, JPEG is one of the available encodings. Still, IIUC, the big difference is that with MJPEG, the encoder sends a complete JPEG image for each frame, while this doesn't need to be the case with RFB.
- bob1029 4y agoI didn't know RFB was a thing. I will be looking into this. What I really want is to be able to send an initial jpeg keyframe to the client, and then use some application-level heuristic to determine if I should then send a delta-encoded frame (what to change based upon the prior) or another keyframe. For 90%+ of my frames, using a prior frame and indicating deltas would be far more efficient than sending a keyframe each time. Maintaining the previous frames on the server per-client is not a huge deal and would not impact latency if done properly. I wonder if there is some way I can use jpeg itself to half-ass this sort of approach in the browser. Maybe... encode keyframes at ~80% quality and delta frames at 100% (still not guaranteed, I know). Maybe canvas compositing or simple buffer tricks with bitmaps could get ~95% of the way there. For my application, the pixels don't have to be perfect or accurate to any particular standards.
- mwcampbell 4y agoIn case you're interested, here's a permissively licensed server-side implementation of the protocol, including all major codecs, in C: https://github.com/any1/neatvnc https://github.com/any1/neatvnc