3 ms·

I'm an insider to this space, I have streaming software customers who individually exceed the revenue of Mighty's entire lifetime. This Thume guy gets a lot wr

I'm an insider to this space, I have streaming software customers who individually exceed the revenue of Mighty's entire lifetime. This Thume guy gets a lot wrong. I'm not going to spend eons taking it apart: his ridiculous breadboard thing does not measure something precisely or accurately enough to be useful, and whatever it did measure he misinterpreted anyway. > Encoding the whole window means latency scales with window size. > something called “damage regions” that apps provide to the OS compositor > possible to diff tiles of a 4k screen image in under 2ms > One disadvantage of the hardware accelerated video encode approach is that using minimal damage regions is tricky, because all you have is a vendor-provided API to encode new full frames into a video stream, and you can’t count on the vendor to optimize their encoder for large pixel-equal screen regions getting encoded very quickly. This is just so far off base. Until he checked the "Enable H264 encoding" box, the stream was compressed on the CPU, to VP8. So it's not even hardware accelerated. It's just slow to compress on the CPU, period full stop, which of course scales with screen size much more than the dedicated compression hardware on their VMs. The low performance has nothing to do with "minimal damage" regions, that provides no meaningful benefit to a hardware encoded stream. An H264/H265 P frame is much more optimal than a "minimal damage" region, it's like comparing spaceflight to a horse drawn carriage. Large pixel equal screen regions are of course encoded extremely well in something like NvEnc and Intel QuickSync. What vendors is he even talking about? > Does it constantly send 60fps compositor updates? No. A lot of apps make this mistake but Remotey doesn’t. If you mess it up it causes WindowServer to take an extra 10-20% CPU and extra power This gets the biggest facepalm to me. This is the same crowd that thinks Unity games are slow because of "Gfx.WaitForPresentOnGfxThread." Enough about Thume. The simple reason Mighty failed is that it MADE. NO. SENSE. Their latency could have been zero, and approximately zero people would have still used it. I'm not going to toot my own horn here, there are definitely streaming business that not only make sense, but make a HUGE amount of $$$ SENSE, but remote browser streaming isn't one of them.

forgotusername6 4y ago
I'm not really sure what minimal damage regions are but I can imagine a case in something like scrolling where a context aware approach would beat a general purpose block based video codec, especially h264 at higher resolutions. With a small macroblock size a large enough vertical scroll might cause the encoder to send fresh pixel data for the blocks with a large enough vertical translation. A context aware encoder would not only not miss the translation but also know where to begin its search algorithm, improving the encode speed. In another case a window coming into view, misaligned to a macroblock boundary, would need to send more data than a context aware encoder that simply sent the window pixels and the location to draw them.
any1 4y ago
This comment seems overly contrarian to me. Perhaps you can tell us what's wrong with his "ridiculous breadboard thing" rather than just stating that it's ridiculous? About damage tracking and hardware encoding: The encoder must always diff the whole frame. There is no magic involved here. It probably does it intelligently, e.g. by hashing blocks and comparing the hashes rather than the memory regions. However, comparing the whole frame still means that you must read the entire frame from memory. He is correct in saying that an encoder that is aware of damage regions would be more efficient. This is because it would require a lot less memory bandwidth. He is also correct about 60 fps compositor updates. You probably just misinterpreted his meaning. What he's saying is that when there is nothing happening on screen, you shouldn't tell the compositor to keep updating the content because that just wastes energy on compositing something that hasn't changed. However, this is tricky when you have a constant bandwidth h264 stream because you will have incremental updates for the same frame. A trick I've used to get past this is to stop encoding when the size of the encoded packages goes below a certain threshold, at which point the image isn't going to improve much.
izacus 4y ago
What exactly is so slow about encoding on CPU? In my career working with video encoding I haven't seen any big latency differences between where the encoding was happening - if anything CPU was often better at latency just because it supported more "wierd" encoding schemes (Intra refresh, partial encode) that were better for latency than GPU encoding blocks.
ksidudwbw 4y ago
0-days in browsers? From images, videos javascript or html parsing

forgotusername6 4y ago

I'm not really sure what minimal damage regions are but I can imagine a case in something like scrolling where a context aware approach would beat a general purpose block based video codec, especially h264 at higher resolutions. With a small macroblock size a large enough vertical scroll might cause the encoder to send fresh pixel data for the blocks with a large enough vertical translation. A context aware encoder would not only not miss the translation but also know where to begin its search algorithm, improving the encode speed. In another case a window coming into view, misaligned to a macroblock boundary, would need to send more data than a context aware encoder that simply sent the window pixels and the location to draw them.

any1 4y ago

This comment seems overly contrarian to me. Perhaps you can tell us what's wrong with his "ridiculous breadboard thing" rather than just stating that it's ridiculous? About damage tracking and hardware encoding: The encoder must always diff the whole frame. There is no magic involved here. It probably does it intelligently, e.g. by hashing blocks and comparing the hashes rather than the memory regions. However, comparing the whole frame still means that you must read the entire frame from memory. He is correct in saying that an encoder that is aware of damage regions would be more efficient. This is because it would require a lot less memory bandwidth. He is also correct about 60 fps compositor updates. You probably just misinterpreted his meaning. What he's saying is that when there is nothing happening on screen, you shouldn't tell the compositor to keep updating the content because that just wastes energy on compositing something that hasn't changed. However, this is tricky when you have a constant bandwidth h264 stream because you will have incremental updates for the same frame. A trick I've used to get past this is to stop encoding when the size of the encoded packages goes below a certain threshold, at which point the image isn't going to improve much.

izacus 4y ago

What exactly is so slow about encoding on CPU? In my career working with video encoding I haven't seen any big latency differences between where the encoding was happening - if anything CPU was often better at latency just because it supported more "wierd" encoding schemes (Intra refresh, partial encode) that were better for latency than GPU encoding blocks.

ksidudwbw 4y ago

0-days in browsers? From images, videos javascript or html parsing