4 ms·

Ask HN: How does MS-Teams, Meets and Zoom virtual background works?

All right, so we need a person body segmentation, that's not a huge deal now, but then we need to serve it somewhere and we need to decide if we'll do this on client side, server side or in-between. All of these have pros and cons, and I am sure big companies already measured and optimized the hell out of the solution and this is why I am interested in the topic.

theGeatZhopa 3y ago
Computer vision, otsy algorithm, and some movement comparison like if the recorded pixel is same in value, make it's value smaller and display it. Over the time all stable pixels will loose its information, but the ones, that record movements, will be refreshed each time. With that you can easily segment background from moving bodied which you are when talking. And then one needs to think of other possibilities:)
- someinteresting 3y ago
  Thank you, but I am more interested in what can be the system architecture. It would be an obvious choice to push this to the client side, but with lower end decides it can be slow. If we push it to the server then the stream can have delay and it would cost a fortune.
  - pantulis 3y ago
    I am not sure because, eh, I'm not using lower end devices, but it seems that the most reasonable way would be the browser to check for the capabilities declared by the graphics drivers and then decide if they offer the option of background blurring. I don't think doing it server side would be cost efficient.
  - theGeatZhopa 3y ago
    Nothing has to be pushed to the server. Computer vision like edge recognition is done solely on the client. Client dont have to fast and can be very slow. What takes the most the computer power is the compression of the video stream. This is also done on the client side. for that you also do not need convolutional networks or object recognition or ML, nor pretrained models. Its pure "Do pixel x change within time t, set pixel y = 1. If no change in x, then y = y-z, with z as a factor. If y < threshold, apply blur"
scrapheap 3y ago
This is an assumption, so may not be how they do it - but I'd assume you'd push it client side and the client doesn't have the hardware or power to do an option then you don't give it to them on that device.
- someinteresting 3y ago
  This sounds feasible, this is how I would do it to save on costs. But then it can be a bad user experience. And as far as I know, you have the virtual background even when using a potato to stream. Maybe if the device is not good enough then they push it to the server side?
  - withinboredom 3y ago
    No, my linux machine doesn't have the option to blur backgrounds anywhere. I don't think they do it server-side. Also, not everyone wants it. I'm not a fan of it since it usually breaks down when multiple people are on the screen, even in the background.
    - imp0cat 3y ago
      This. At least Zoom does it client side and only if your client is fast enough. https://support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0059090 https://support.zoom.com/hc/en/article?id=zm_kb&sysparm_arti...
doix 3y ago
It definitely used to all happen client side, if your graphics stack was messed up, then those tools wouldn't apply those effects. I haven't used those tools in a couple years, but I doubt it's any different now. I know that Google Meet (the tool I use nowadays) still does it client side. Every now and then I break something when updating my Nvidia drivers on Linux and I can no longer blur my background.
samlinnfer 3y ago
It is a classic computer vision problem: image segmentation. There are open source tools like rembg (https://github.com/danielgatis/rembg https://github.com/danielgatis/rembg) which call into pre-trained models. It is also easy to train your own model, I've used code from https://github.com/SkyTNT/anime-segmentation https://github.com/SkyTNT/anime-segmentation with my own data to train an image segmentation model to separate objects with really good results.
rapsey 3y ago
I think the most popular solution for this is Google's mediapipe selfie segmentation: https://developers.google.com/mediapipe/solutions/vision/image_segmenter https://developers.google.com/mediapipe/solutions/vision/ima... They definitely do it client side. I just wish there was something more easily deployable to desktops as a C/Rust solution.
pavlov 3y ago
I'm not aware of any of the big players rendering virtual backgrounds on the server. It's all client-side. A couple years ago when I worked at Meta, I built the server bits for a demo that was a hybrid solution. The client did the segmentation as usual but instead of sending the final composite, it sent the raw alpha channel (i.e. the transparency mask) in addition to the video stream. The composite was then rendered on a server. This added an interesting level of flexibility to the composition, as you could change the background independently and move/scale the participant's masked video to adapt the layout on the fly. But I don't suppose it ever went into production.
- bobdvb 3y ago
  Teams allows you to put people in a virtual lecture theatre called "Together Mode" where the individuals are cut out. I presume it sends the alpha along with the video.
  - michaelmior 3y ago
    Zoom has a very similar feature which I would also assume is implemented like this.
  - pavlov 3y ago
    Maybe? What happens when some people on the call are not sending the alpha though? Background segmentation remains a hardware-dependent feature. Without the alpha, you can either skip the participant, render them unmasked, do the expensive segmentation on the receiver side (where the rendering happens), or do some kind of fast approximation for a mask. Personally I would probably try to implement this feature first without the alpha channels, maybe just using face detection to get a passable mask cheaply. In a layout with a grid of people and a fancy complex background design, the requirements for mask quality are not quite as strict. There are compositing tricks that could obfuscate the cheap masking.
jameshush 3y ago
What's your use case? If it's for live video calls, I'd recommend using a vendor to handle this (and every other WebRTC edge case you can think of) for you. I'm a bit biased because I work for Daily.co, who does this. Banuba is great if you're not trying to integrate any WebRTC stuff too.
- hactually 3y ago
  I spent a few minutes on the site and it all looks like it's aimed at someone who's not me. You've suggested it for enhancing/image processing on the client-side and yet, a cursory look at the landing and docs and there's nothing I could see that would help me do this. Are there any blog/howto guides that are better than your landing page?
rcarmo 3y ago
It’s done client side. Even a measly iGPU can do this today, and you don’t even get the option to turn it on if your hardware isn’t able to do it.
rspoerri 3y ago
OBS has a plugin that does background removal and incorporates different solutions to do so. https://github.com/occ-ai/obs-backgroundremoval https://github.com/occ-ai/obs-backgroundremoval

theGeatZhopa 3y ago

Computer vision, otsy algorithm, and some movement comparison like if the recorded pixel is same in value, make it's value smaller and display it. Over the time all stable pixels will loose its information, but the ones, that record movements, will be refreshed each time. With that you can easily segment background from moving bodied which you are when talking. And then one needs to think of other possibilities:)

someinteresting 3y ago
Thank you, but I am more interested in what can be the system architecture. It would be an obvious choice to push this to the client side, but with lower end decides it can be slow. If we push it to the server then the stream can have delay and it would cost a fortune.
- pantulis 3y ago
  I am not sure because, eh, I'm not using lower end devices, but it seems that the most reasonable way would be the browser to check for the capabilities declared by the graphics drivers and then decide if they offer the option of background blurring. I don't think doing it server side would be cost efficient.
- theGeatZhopa 3y ago
  Nothing has to be pushed to the server. Computer vision like edge recognition is done solely on the client. Client dont have to fast and can be very slow. What takes the most the computer power is the compression of the video stream. This is also done on the client side. for that you also do not need convolutional networks or object recognition or ML, nor pretrained models. Its pure "Do pixel x change within time t, set pixel y = 1. If no change in x, then y = y-z, with z as a factor. If y < threshold, apply blur"

scrapheap 3y ago

This is an assumption, so may not be how they do it - but I'd assume you'd push it client side and the client doesn't have the hardware or power to do an option then you don't give it to them on that device.

someinteresting 3y ago
This sounds feasible, this is how I would do it to save on costs. But then it can be a bad user experience. And as far as I know, you have the virtual background even when using a potato to stream. Maybe if the device is not good enough then they push it to the server side?
- withinboredom 3y ago
  No, my linux machine doesn't have the option to blur backgrounds anywhere. I don't think they do it server-side. Also, not everyone wants it. I'm not a fan of it since it usually breaks down when multiple people are on the screen, even in the background.
  - imp0cat 3y ago
    This. At least Zoom does it client side and only if your client is fast enough. https://support.zoom.com/hc/en/article?id=zm_kb&sysparm_article=KB0059090 https://support.zoom.com/hc/en/article?id=zm_kb&sysparm_arti...

doix 3y ago

It definitely used to all happen client side, if your graphics stack was messed up, then those tools wouldn't apply those effects. I haven't used those tools in a couple years, but I doubt it's any different now. I know that Google Meet (the tool I use nowadays) still does it client side. Every now and then I break something when updating my Nvidia drivers on Linux and I can no longer blur my background.

samlinnfer 3y ago

It is a classic computer vision problem: image segmentation. There are open source tools like rembg (https://github.com/danielgatis/rembg https://github.com/danielgatis/rembg) which call into pre-trained models. It is also easy to train your own model, I've used code from https://github.com/SkyTNT/anime-segmentation https://github.com/SkyTNT/anime-segmentation with my own data to train an image segmentation model to separate objects with really good results.

rapsey 3y ago

I think the most popular solution for this is Google's mediapipe selfie segmentation: https://developers.google.com/mediapipe/solutions/vision/image_segmenter https://developers.google.com/mediapipe/solutions/vision/ima... They definitely do it client side. I just wish there was something more easily deployable to desktops as a C/Rust solution.

pavlov 3y ago

I'm not aware of any of the big players rendering virtual backgrounds on the server. It's all client-side. A couple years ago when I worked at Meta, I built the server bits for a demo that was a hybrid solution. The client did the segmentation as usual but instead of sending the final composite, it sent the raw alpha channel (i.e. the transparency mask) in addition to the video stream. The composite was then rendered on a server. This added an interesting level of flexibility to the composition, as you could change the background independently and move/scale the participant's masked video to adapt the layout on the fly. But I don't suppose it ever went into production.

bobdvb 3y ago
Teams allows you to put people in a virtual lecture theatre called "Together Mode" where the individuals are cut out. I presume it sends the alpha along with the video.
- michaelmior 3y ago
  Zoom has a very similar feature which I would also assume is implemented like this.
- pavlov 3y ago
  Maybe? What happens when some people on the call are not sending the alpha though? Background segmentation remains a hardware-dependent feature. Without the alpha, you can either skip the participant, render them unmasked, do the expensive segmentation on the receiver side (where the rendering happens), or do some kind of fast approximation for a mask. Personally I would probably try to implement this feature first without the alpha channels, maybe just using face detection to get a passable mask cheaply. In a layout with a grid of people and a fancy complex background design, the requirements for mask quality are not quite as strict. There are compositing tricks that could obfuscate the cheap masking.

jameshush 3y ago

What's your use case? If it's for live video calls, I'd recommend using a vendor to handle this (and every other WebRTC edge case you can think of) for you. I'm a bit biased because I work for Daily.co, who does this. Banuba is great if you're not trying to integrate any WebRTC stuff too.

hactually 3y ago
I spent a few minutes on the site and it all looks like it's aimed at someone who's not me. You've suggested it for enhancing/image processing on the client-side and yet, a cursory look at the landing and docs and there's nothing I could see that would help me do this. Are there any blog/howto guides that are better than your landing page?

rcarmo 3y ago

It’s done client side. Even a measly iGPU can do this today, and you don’t even get the option to turn it on if your hardware isn’t able to do it.

rspoerri 3y ago

OBS has a plugin that does background removal and incorporates different solutions to do so. https://github.com/occ-ai/obs-backgroundremoval https://github.com/occ-ai/obs-backgroundremoval