6 ms·

The example video clips in the post look nothing like me and my team's view when using the new feature. Most of the time half of our hair gets blurred or replac

The example video clips in the post look nothing like me and my team's view when using the new feature. Most of the time half of our hair gets blurred or replaced and hand gestures will cause either our hands or head to disappear.

gundmc 6y ago
It seems to have gotten a little better recently, but my experience matches yours. It really struggles when I wear over-ear headphones - they sort of phase in and out of existence. The other thing I've noticed is the background blur absolutely annihilates my CPU. To the point where I would rather just turn off my camera if I don't want my background visible.
samtheprogram 6y ago
I can vouch for this. I haven’t really needed the background blur feature personally, but I’ve tried it and both myself, colleagues, and friends — pretty much everyone I’ve talked to that has used it — loathe Google Meet’s background blur, and prefer Zoom’s by far. In my experience, it doesn’t completely cover the background most of the time, and if you move at all, as you point out, it can’t keep up. Kind of funny to see Google engineering blogging about it when it feels extremely half baked. This makes me sad, because in all other areas, I think Meet excels well beyond the competition. EDIT: removed my general sentiment on Google
- judge2020 6y ago
  "Turns out our training set was comprised entirely of backdrops from Google HQ. Sorry everyone else!"
  - dontblink 6y ago
    Everyone is wfh @ Google
- _pastel 6y ago
  "Half baked" misrepresents the difficulty this task. Yes, Zoom does it better, but it's _still_ an excellent and interesting engineering accomplishment. I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; this not only answers that question, but provides details down to the level of model architecture and the final latency and IOU benchmarks. Of course I'd be more interested to read how Zoom manages to do even better, but I'm not holding my breath for them to publish those details.
  - enriquto 6y ago
    > I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; ML is a classical tool for image processing, what do you mean here?
madars 6y ago
At least for background blur the latency there is enough to make it almost unusable: easily over 100ms. This is with latest stable Chrome on a relatively recent Ryzen/Nvidia system. Maybe background replacement will do better once it rolls down to regular Google Meet (too lazy to log into my Google <del>Apps</del> <del>Suite</del> Workspace) :-) However, everything else about Google Meet is great and I wish I could make all my Zoom friends switch.
- riffraff 6y ago
  > However, everything else about Google Meet is great and I wish I could make all my Zoom friends switch. is it _better_ than zoom tho? I my experience, I don't see much of an improvement worth switching.
  - deelowe 6y ago
    Try using meet, teams and zoom on high latency and/or low bandwidth connections.
    - loosescrews 6y ago
      Meet seems to work better on poor connections, but it does it with a significantly more CPU intensive codec (VP9?). As a result, it only seems to work well if you have a powerful CPU. If you have a weak CPU, Zoom seems to work much better.
      - jvdvegt 6y ago
        On my 2010 Macbook Pro, Meet uses about half the CPU resources as Zoom. Also, audio-lag is much worse in Zoom. Background blur doesn't seem available in Firefox on Ubuntu unfortunately :(
        MarkyC4 6y ago
        not just Ubuntu, Firefox 82.0.2 on Mac doesn't show the option
chrischen 6y ago
I'm wondering why they didn't just use standard CV techniques like background subtraction? Does their technique work with a dynamic background as well?
- objclxt 6y ago
  I’ve done some work in this space - subtraction doesn’t perform well when other motion is present, whereas if you use pose / body detection you can ignore other bodies in view (i.e, the toddler running across the room).
cameldrv 6y ago
They have their example video clips, but they also provide data. They say that in their better model, They get an IoU of 93.8% This means 6.2% of pixels are misclassified. Either it's your hair getting cut off or the background is leaking through. 6.2% of an image is a fair bit considering your head is probably 30% of the frame.

gundmc 6y ago

It seems to have gotten a little better recently, but my experience matches yours. It really struggles when I wear over-ear headphones - they sort of phase in and out of existence. The other thing I've noticed is the background blur absolutely annihilates my CPU. To the point where I would rather just turn off my camera if I don't want my background visible.

samtheprogram 6y ago

I can vouch for this. I haven’t really needed the background blur feature personally, but I’ve tried it and both myself, colleagues, and friends — pretty much everyone I’ve talked to that has used it — loathe Google Meet’s background blur, and prefer Zoom’s by far. In my experience, it doesn’t completely cover the background most of the time, and if you move at all, as you point out, it can’t keep up. Kind of funny to see Google engineering blogging about it when it feels extremely half baked. This makes me sad, because in all other areas, I think Meet excels well beyond the competition. EDIT: removed my general sentiment on Google

judge2020 6y ago
"Turns out our training set was comprised entirely of backdrops from Google HQ. Sorry everyone else!"
- dontblink 6y ago
  Everyone is wfh @ Google
_pastel 6y ago
"Half baked" misrepresents the difficulty this task. Yes, Zoom does it better, but it's _still_ an excellent and interesting engineering accomplishment. I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; this not only answers that question, but provides details down to the level of model architecture and the final latency and IOU benchmarks. Of course I'd be more interested to read how Zoom manages to do even better, but I'm not holding my breath for them to publish those details.
- enriquto 6y ago
  > I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; ML is a classical tool for image processing, what do you mean here?

madars 6y ago

At least for background blur the latency there is enough to make it almost unusable: easily over 100ms. This is with latest stable Chrome on a relatively recent Ryzen/Nvidia system. Maybe background replacement will do better once it rolls down to regular Google Meet (too lazy to log into my Google <del>Apps</del> <del>Suite</del> Workspace) :-) However, everything else about Google Meet is great and I wish I could make all my Zoom friends switch.

riffraff 6y ago
> However, everything else about Google Meet is great and I wish I could make all my Zoom friends switch. is it _better_ than zoom tho? I my experience, I don't see much of an improvement worth switching.
- deelowe 6y ago
  Try using meet, teams and zoom on high latency and/or low bandwidth connections.
  - loosescrews 6y ago
    Meet seems to work better on poor connections, but it does it with a significantly more CPU intensive codec (VP9?). As a result, it only seems to work well if you have a powerful CPU. If you have a weak CPU, Zoom seems to work much better.
    - jvdvegt 6y ago
      On my 2010 Macbook Pro, Meet uses about half the CPU resources as Zoom. Also, audio-lag is much worse in Zoom. Background blur doesn't seem available in Firefox on Ubuntu unfortunately :(
      - MarkyC4 6y ago
        not just Ubuntu, Firefox 82.0.2 on Mac doesn't show the option

chrischen 6y ago

I'm wondering why they didn't just use standard CV techniques like background subtraction? Does their technique work with a dynamic background as well?

objclxt 6y ago
I’ve done some work in this space - subtraction doesn’t perform well when other motion is present, whereas if you use pose / body detection you can ignore other bodies in view (i.e, the toddler running across the room).

cameldrv 6y ago

They have their example video clips, but they also provide data. They say that in their better model, They get an IoU of 93.8% This means 6.2% of pixels are misclassified. Either it's your hair getting cut off or the background is leaking through. 6.2% of an image is a fair bit considering your head is probably 30% of the frame.