4 ms·

I think this could be used as a great basis for next generation video conferencing. Rather than compressing raster images of people speaking, we could send a mo

I think this could be used as a great basis for next generation video conferencing. Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses. Most current networks can handle video conferencing (relatively) well. However, if 8k video or something like Light Field displays ever catch on, then we might end up needing orders of magnitude more bandwidth to drive video conferencing, so this might be an enabling technology.

ricardobeat 4y ago
Might not be that simple. In a recent demo from Unreal Engine they said that the facial animation data is the main bottleneck to shipping titles with more realistic avatars. Their minute-long demo took many TBs to store the motion data alone.
- cheschire 4y ago
  My layperson understanding makes this baffling to me. Isn't motion inherently analog, and haven't we've developed high performing analog compression already? Motion seems significantly more amenable to lossy compression as well, and I could imagine a use-case for ML-based decompression to fill in between the keyframes.
  - dmwallin 4y ago
    You also have to consider that faces are one of the things that we have the strongest perception of, with lots of our neurons dedicated to the task, so when you get things wrong it's far more noticeable than many other bodily animations would be.
    - jon_richards 4y ago
      One of my favorite “weird” conjectures is that the existence of the uncanny valley implies that at some point it was evolutionarily advantageous for us to recognize something that looked human, but wasn’t… and to be afraid of it.
      - rjeli 4y ago
        neanderthals?
        deleted 4y ago
        [deleted]
        giantrobot 4y ago
        Think also about face-like things we perceive because of pareidolia but might in fact be dangerous. One part of our brain is opportunistic about finding patterns while another is a sanity check on those patterns telling us not to just trust them outright.
      - shwoopdiwoop 4y ago
        Is there any further reading on this topic in particular that is not leading to wild conspiracy theories?
  - ricardobeat 4y ago
    I had the same reaction (note: I actually got things mixed up, the one in question is the Unity "Enemies" demo [1], not Unreal), but if you consider the amount of detail - the saliva 'sticking' to the corners of the mouth, the 40+ muscles of the face, lips, eyelids, the iris, etc - it makes sense that it would be a ton of data. IIRC they are working on some kind of ML-powered compression so that this becomes actually possible to ship (the demo is not available to download, presumably it's too big and would barely run on a 3090TI). [1] https://www.youtube.com/watch?v=eXYUNrgqWUU https://www.youtube.com/watch?v=eXYUNrgqWUU
- HWR_14 4y ago
  Facial animation is a real issue. This is facial mapping, which is an easier to solve problem, in that throwing artists at the problem is a solution.
ekianjo 4y ago
> Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses. What about latency though? Would that improve?
fortran77 4y ago
Or, at least, the faces that aren't actively speaking could be replaced by avatars. It will also help keep a clean image in case stuff is going on the the background, etc.
abeppu 4y ago
I guess, would watching an animated head avatar in 8k or a Light Field display be a better experience than watching lower resolution video? I wonder whether the experience would be too undermined by artifacts from e.g. your coworker's cat, for which the system doesn't have a model, wanders into frame partway through a call, and suddenly the system must dynamically either (a) 'downgrade' to just video (b) be able to create a new model for an arbitrary new thing on the fly (c) create some hybrid of partial video data with part viewer-rendered animation or (d) your coworker looks crazy interacting with something the system hides from you entirely.
8n4vidtmkvmk 4y ago
there's another thread about just this, but very different approach. reduces bandwidth a lot specifically for video conferencing

ricardobeat 4y ago

Might not be that simple. In a recent demo from Unreal Engine they said that the facial animation data is the main bottleneck to shipping titles with more realistic avatars. Their minute-long demo took many TBs to store the motion data alone.

cheschire 4y ago
My layperson understanding makes this baffling to me. Isn't motion inherently analog, and haven't we've developed high performing analog compression already? Motion seems significantly more amenable to lossy compression as well, and I could imagine a use-case for ML-based decompression to fill in between the keyframes.
- dmwallin 4y ago
  You also have to consider that faces are one of the things that we have the strongest perception of, with lots of our neurons dedicated to the task, so when you get things wrong it's far more noticeable than many other bodily animations would be.
  - jon_richards 4y ago
    One of my favorite “weird” conjectures is that the existence of the uncanny valley implies that at some point it was evolutionarily advantageous for us to recognize something that looked human, but wasn’t… and to be afraid of it.
    - rjeli 4y ago
      neanderthals?
      - deleted 4y ago
        [deleted]
      - giantrobot 4y ago
        Think also about face-like things we perceive because of pareidolia but might in fact be dangerous. One part of our brain is opportunistic about finding patterns while another is a sanity check on those patterns telling us not to just trust them outright.
    - shwoopdiwoop 4y ago
      Is there any further reading on this topic in particular that is not leading to wild conspiracy theories?
- ricardobeat 4y ago
  I had the same reaction (note: I actually got things mixed up, the one in question is the Unity "Enemies" demo [1], not Unreal), but if you consider the amount of detail - the saliva 'sticking' to the corners of the mouth, the 40+ muscles of the face, lips, eyelids, the iris, etc - it makes sense that it would be a ton of data. IIRC they are working on some kind of ML-powered compression so that this becomes actually possible to ship (the demo is not available to download, presumably it's too big and would barely run on a 3090TI). [1] https://www.youtube.com/watch?v=eXYUNrgqWUU https://www.youtube.com/watch?v=eXYUNrgqWUU
HWR_14 4y ago
Facial animation is a real issue. This is facial mapping, which is an easier to solve problem, in that throwing artists at the problem is a solution.

ekianjo 4y ago

> Rather than compressing raster images of people speaking, we could send a model as an initial payload and subsequent instructions on how to manipulate it as the conversation progresses. What about latency though? Would that improve?

fortran77 4y ago

Or, at least, the faces that aren't actively speaking could be replaced by avatars. It will also help keep a clean image in case stuff is going on the the background, etc.

abeppu 4y ago

I guess, would watching an animated head avatar in 8k or a Light Field display be a better experience than watching lower resolution video? I wonder whether the experience would be too undermined by artifacts from e.g. your coworker's cat, for which the system doesn't have a model, wanders into frame partway through a call, and suddenly the system must dynamically either (a) 'downgrade' to just video (b) be able to create a new model for an arbitrary new thing on the fly (c) create some hybrid of partial video data with part viewer-rendered animation or (d) your coworker looks crazy interacting with something the system hides from you entirely.

8n4vidtmkvmk 4y ago

there's another thread about just this, but very different approach. reduces bandwidth a lot specifically for video conferencing