5 ms·
I just discovered that Zoom teleconferencing software has the ability to detect the background and replace it with an arbitrary video in real time. I have no i
by mark-r 7y ago
I just discovered that Zoom teleconferencing software has the ability to detect the background and replace it with an arbitrary video in real time. I have no idea how they do it, but it's as impressive as heck.
- gerikson 7y agoMS Teams can blur the background on video.
- GordonS 7y agoJitsi has this feature in beta too. It works pretty well, although it currently uses way too much CPU, and doesn't work quite as well as it does in MS Teams.
- ycombobreaker 7y agoHaving just used it, it feels like a few factors: - a background scene is usually static. if a pixel matches its historical average (i e. not changing) then it's probably background. - if a pixel color matches its neighbors, it's probably the same as them. i noticed my black shirt tripped it up on occasion, but was all-or-nothing. - people are blobs, not diffuse, so try to segment large regions.
- riedel 7y agoYes, it did not work for me sitting with a laptop on a swing chair. It actually works better with clear static features in the background, but it seams to be a bit more (cnn?) because it seams to isolate paintings on the wall for me as well. Wish there was more open source easy to use camera stream manipulation. Skype also has background blurring.
- MivLives 7y agoMy work is using Meets, and I second this desire to edit my camera stream. I didn't find anything even cheap to do it.
- thaumasiotes 7y agoA green screen should make it pretty straightforward?
- mark-r 7y agoYes, that would make it trivial. But Zoom gives you the option of using a green screen or not.
- codingdave 7y agoIt is a good early version of such a feature, but I've found it to fail fairly quickly in uneven light or with busy backgrounds. Still, it is a great feature, so I'm hopeful that they'll keep improving it and other services will match it.
- StavrosK 7y agoThey detect the face and then extrapolate the body. You can tell because hiding your face (or turning away) makes you background.
- Jaxan 7y agoSkype, Jitsi and MS teams also have this feature for some time now.
- sabalaba 7y agoIt’s likely some type of segmentation network like Mask RCNN or SegNet. Look up Mask RCNN for some state of the art segmentation results. This stuff has been able to run on mobile phones in real time for years now.