5 ms·

Latency is the biggest problem with video meetings and I can only imagine it's worse with this. In normal human interaction you can see people's reactions to w

Latency is the biggest problem with video meetings and I can only imagine it's worse with this. In normal human interaction you can see people's reactions to what you're saying and doing within microseconds of taking an action, you can subtly and automatically adjust your presentation, your pace and your expressions to build greater rapport. Over video calls you need to introduce artificial pauses to make sure that everyone has caught up, and it's really jarring compared to being face to face.

KaiserPro 5y ago
In a previous role we did some light research into this, you tend to notice latency in movement at around 40-80ms, but thats from your own movement vs avatar. However, you don't notice it in other people's movements until there is a mismatch between social clues, Even then you have up to 250 ms (sometimes more, depending on the pace of the conversation) In practice that means you can do an atlantic call without really noticing it. the problem with VC calls is that because video needs to be matched to audio, the audio is buffered to make sure its lined up, this means that you have a 250ms+ delay plus any time to switch presenters.
crakhamster01 5y ago
The latency on this should actually be better than video calls. Since all the character assets are downloaded locally, the only visual data that needs to be sent across network is the positioning of facial features and hand movements - which is a much smaller payload than streaming video. More akin to the latency you'd see playing Call of Duty or something. I could see this experience being higher fidelity than video calls, especially in poor network conditions.
- ghusbands 5y ago
  This is only true under the assumption that you can analyse the video and extract the movement/features as quickly as you can stream video to the network. Given how optimised streaming of live video is and how hard video analysis can be, it's not at all a given.
  - crakhamster01 5y ago
    I'm not sure I follow? When you use Workrooms you're not streaming video. The application is taking the positional data of your headset/controllers and translating that into how your avatar moves around. There isn't any video analysis going on.
    - ghusbands 5y ago
      You said "the only visual data that needs to be sent across network is the positioning of facial features and hand movements". To extract facial features and hand movements is non-trivial and requires video analysis and may hence involve lag.
filereaper 5y ago
I agree with this. There's work done by NVidia on AI Video Compression around facial ticks and reducing latency and bandwidth. Maybe these two approaches can be combined for a best-of-both worlds result. https://developer.nvidia.com/ai-video-compression https://developer.nvidia.com/ai-video-compression

KaiserPro 5y ago

In a previous role we did some light research into this, you tend to notice latency in movement at around 40-80ms, but thats from your own movement vs avatar. However, you don't notice it in other people's movements until there is a mismatch between social clues, Even then you have up to 250 ms (sometimes more, depending on the pace of the conversation) In practice that means you can do an atlantic call without really noticing it. the problem with VC calls is that because video needs to be matched to audio, the audio is buffered to make sure its lined up, this means that you have a 250ms+ delay plus any time to switch presenters.

crakhamster01 5y ago

The latency on this should actually be better than video calls. Since all the character assets are downloaded locally, the only visual data that needs to be sent across network is the positioning of facial features and hand movements - which is a much smaller payload than streaming video. More akin to the latency you'd see playing Call of Duty or something. I could see this experience being higher fidelity than video calls, especially in poor network conditions.

ghusbands 5y ago
This is only true under the assumption that you can analyse the video and extract the movement/features as quickly as you can stream video to the network. Given how optimised streaming of live video is and how hard video analysis can be, it's not at all a given.
- crakhamster01 5y ago
  I'm not sure I follow? When you use Workrooms you're not streaming video. The application is taking the positional data of your headset/controllers and translating that into how your avatar moves around. There isn't any video analysis going on.
  - ghusbands 5y ago
    You said "the only visual data that needs to be sent across network is the positioning of facial features and hand movements". To extract facial features and hand movements is non-trivial and requires video analysis and may hence involve lag.

filereaper 5y ago

I agree with this. There's work done by NVidia on AI Video Compression around facial ticks and reducing latency and bandwidth. Maybe these two approaches can be combined for a best-of-both worlds result. https://developer.nvidia.com/ai-video-compression https://developer.nvidia.com/ai-video-compression