8 ms·

OBS Plugin: Background Removal

149 points by lnyan 5y ago

nadermx 5y ago
This is awesome! I made a while ago a cmd line background remover using the same model, https://github.com/nadermx/backgroundremover https://github.com/nadermx/backgroundremover
rahimnathwani 5y ago
This is awesome. The last time I looked, the only background removal I found for OBS required a green screen. It's a shame Nvidia broadcast doesn't work on Linux. If you use this with CUDA, are the results comparable?
- Bilal_io 5y ago
  I plan to switch to Linux for my main machine and the only thing I'll miss is Nvidia Broadcast. I know the voice part of it is integrated in OBS, but not sure if that works on OBS Linux.
  - charcircuit 5y ago
    RNNoise works good enough for me
    - ninjin 5y ago
      Seconding that RNNoise works very well and ships with OBS. I do however not use it as I find it to be a bit too aggressive and close to a noise gate at times. But I fully recognise that falling back on an analogue expander is not something everyone can do.
poindi1 5y ago
Google/Mediapipe has a better performing models here https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/ https://github.com/google/mediapipe/tree/master/mediapipe/mo...
- rikroots 5y ago
  I can confirm this with a couple of Codepen demos. MediaPipe's selfie-segmentation model[1] seems to give much better results than TensorFlow's body-pix model[2] (which is the model I most often see talked about for this sort of background removal work). As much as I admire ML models (and I remain in awe of people who know how to build them), I still think there's room for more conventional chroma key filtering - for instance identifying part of a video's moving display so you can add some UI to it[3]. [1] - MediaPipe demo - https://codepen.io/kaliedarik/pen/PopBxBM https://codepen.io/kaliedarik/pen/PopBxBM [2] - TensorFlow demo - https://codepen.io/kaliedarik/pen/ZEeoZaP https://codepen.io/kaliedarik/pen/ZEeoZaP [3] - https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html
chrismorgan 5y ago
Given how severely imperfect this approach is, I’d much rather people didn’t use it: it’s disconcerting and distracting. Either do it properly (which tends to mean chroma keying, preferably with thoughtful handling of lighting too if you do anything like background replacement rather than just removal), or don’t do it at all. I have not the faintest clue whether this position of mine is popular or unpopular. Better models may improve matters (the one shown here looks to be decidedly worse than some), but even the best I’ve observed at this time is still something I’d prefer not to be used, because it’s still wrong too much of the time in annoying ways.
- alsobrsp 5y ago
  I won't use the blur or fake backgrounds normally, I find them goofy. I just let people see what is behind me. If it not appropriate for video I'll leave it off. My background has been, the living room, kitchen pantry, bedroom, forest behind my house, patio, whatever. We are real people, in real locations just show it.
- kelnos 5y ago
  Chroma keying requires a green (or other solid, "unusual" color) screen, though, right? Most people don't have those, and won't want to expend the effort to set something like that up. It's certainly imperfect, but it's useful for people who'd prefer to obscure their surroundings while on a video call without having to deal with a more "professional" setup.
  - robbrown451 5y ago
    The problem is that those people don't grasp how ridiculously awful it looks. Do you pay money for decent clothes and a decent haircut? Then spend the money on either a greenscreen, or anything else to put behind you to cover whatever you want to cover without making you look like a special effect done by a third grader.
    - trog 5y ago
      I pay about $30 every 3-4 months for a haircut and most of my clothes are 10+ years old (one of my favourite shirts, that I regularly wear in the home office, is the Australian team's football jersey from the 2006 World Cup). I cheerfully use Google Meet with it's poor blurring that constantly pops things in and out of the background without giving it a second thought. I judge people by the content of their words and the merit of their actions, not what they look like or how much effort they've gone to in order to have a perfect webcam production. This tool looks super awesome and I can absolutely see myself using it. I've tried greenscreen in OBS before and decided it wasn't worth the effort for my purposes; something that is roughly decent as this looks at a glance will be really useful.
usui 5y ago
Finally! So glad to see a plugin make an earnest attempt on this now. Been using Snap Camera green screen background combined with OBS chroma keying to make it work, but will try this out now.
KingOfCoders 5y ago
I use Nvidia Broadcast, it has the least artifacts and the roll of when blurring is also the best I think, downside it uses a lot of GPU and my RTX2080 fan spins up.
- Netcob 5y ago
  It works surprisingly well! I started using it for the noise filtering, which is even better.
- cptskippy 5y ago
  Are you using a lot of the other filters too? I use Broadcast's background removal and the tracking zoom on a Quadro 4000 mobile and it remains mostly silent.
deng 5y ago
This is a great plugin. I'm using this to have background blurring with the Linux MS Teams client, which for whatever reason does not support this natively.
rasz 5y ago
"AI". Wow this looks bad, worse than just taking a screenshot of the background and subtracting from video feed.
CharlesW 5y ago
So if a proper chromakey setup is more of a commitment/hassle than many people are willing to accept, and software background removal is imperfect to the point of distraction, why aren't depth-sensing solutions (like the $300 Intel RealSense D435) an ideal solution to this problem? What am I missing?
jedberg 5y ago
Why hasn't anyone built a plugin that says "sit still, now move left, now move right" and then just remove all the pixels that didn't move? Seems like it would be easy and work as long as the camera doesn't move. Is this a harder problem than I think it is?
- Udo 5y ago
  I think current algorithms do a version of that, but it has to be coupled together with person/outline recognition and such. Taking the background in just once doesn't work because the pixels still change over time (due to lighting and such). What's really impressive to me is that it looks like web apps like Google Meet and Discord seem to do this in JavaScript on the fly.
andrewmackrodt 5y ago
This looks cool, being both free and cross platform. A non free (4 hours free a month) solution I quite like on macOS is krisp.ai which I also use for noise removal. Background replacement was recently added to M1 chips. On the Windows side, Nvidia Broadcast has worked well for me (requires an RTX card), it also supports tracking which I really like. I've previously tried XSplit but the results were worse than whatever implementation Zoom and even Hangouts in a browser use. I rarely video call under Linux (non existent laptop drivers for my webcam) but look forward to trying this out if the need ever arises.
shirro 5y ago
My kids missed out on most of the remote schooling thing due to location but they got 8 days earlier this year and made the most of it setting up OBS under linux with this plugin and fed it into MS Teams as a virtual camera. They made up several scenes they could switch between with subtle animated backgrounds and added chroma keyed meme/animated stuff they could enable during less formal time like end of lessons. It isn't as good as the closed source offerings from big companies but those companies don't have enough users on Linux to justify developing for the platform so it is a chicken and egg thing. For what it is I was happy with it. With a bit of adjustment it worked suprisingly well in actual use. One of the kids was stuck with an old under powered laptop with integrated graphics and it had no problems at all running this. If you want to do this stuff properly you obviously need a green screen and good lighting but when you suddenly have an entire family home and online and are trying to juggle access to computers, quickly locating and finding space for multiple green screens and studio lighting isn't an option for regular people.

149 points by lnyan 5y ago

This is awesome! I made a while ago a cmd line background remover using the same model, https://github.com/nadermx/backgroundremover https://github.com/nadermx/backgroundremover

rahimnathwani 5y ago

This is awesome. The last time I looked, the only background removal I found for OBS required a green screen. It's a shame Nvidia broadcast doesn't work on Linux. If you use this with CUDA, are the results comparable?

Bilal_io 5y ago
I plan to switch to Linux for my main machine and the only thing I'll miss is Nvidia Broadcast. I know the voice part of it is integrated in OBS, but not sure if that works on OBS Linux.
- charcircuit 5y ago
  RNNoise works good enough for me
  - ninjin 5y ago
    Seconding that RNNoise works very well and ships with OBS. I do however not use it as I find it to be a bit too aggressive and close to a noise gate at times. But I fully recognise that falling back on an analogue expander is not something everyone can do.

poindi1 5y ago

Google/Mediapipe has a better performing models here https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/ https://github.com/google/mediapipe/tree/master/mediapipe/mo...

rikroots 5y ago
I can confirm this with a couple of Codepen demos. MediaPipe's selfie-segmentation model[1] seems to give much better results than TensorFlow's body-pix model[2] (which is the model I most often see talked about for this sort of background removal work). As much as I admire ML models (and I remain in awe of people who know how to build them), I still think there's room for more conventional chroma key filtering - for instance identifying part of a video's moving display so you can add some UI to it[3]. [1] - MediaPipe demo - https://codepen.io/kaliedarik/pen/PopBxBM https://codepen.io/kaliedarik/pen/PopBxBM [2] - TensorFlow demo - https://codepen.io/kaliedarik/pen/ZEeoZaP https://codepen.io/kaliedarik/pen/ZEeoZaP [3] - https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html

chrismorgan 5y ago

Given how severely imperfect this approach is, I’d much rather people didn’t use it: it’s disconcerting and distracting. Either do it properly (which tends to mean chroma keying, preferably with thoughtful handling of lighting too if you do anything like background replacement rather than just removal), or don’t do it at all. I have not the faintest clue whether this position of mine is popular or unpopular. Better models may improve matters (the one shown here looks to be decidedly worse than some), but even the best I’ve observed at this time is still something I’d prefer not to be used, because it’s still wrong too much of the time in annoying ways.

alsobrsp 5y ago
I won't use the blur or fake backgrounds normally, I find them goofy. I just let people see what is behind me. If it not appropriate for video I'll leave it off. My background has been, the living room, kitchen pantry, bedroom, forest behind my house, patio, whatever. We are real people, in real locations just show it.
kelnos 5y ago
Chroma keying requires a green (or other solid, "unusual" color) screen, though, right? Most people don't have those, and won't want to expend the effort to set something like that up. It's certainly imperfect, but it's useful for people who'd prefer to obscure their surroundings while on a video call without having to deal with a more "professional" setup.
- robbrown451 5y ago
  The problem is that those people don't grasp how ridiculously awful it looks. Do you pay money for decent clothes and a decent haircut? Then spend the money on either a greenscreen, or anything else to put behind you to cover whatever you want to cover without making you look like a special effect done by a third grader.
  - trog 5y ago
    I pay about $30 every 3-4 months for a haircut and most of my clothes are 10+ years old (one of my favourite shirts, that I regularly wear in the home office, is the Australian team's football jersey from the 2006 World Cup). I cheerfully use Google Meet with it's poor blurring that constantly pops things in and out of the background without giving it a second thought. I judge people by the content of their words and the merit of their actions, not what they look like or how much effort they've gone to in order to have a perfect webcam production. This tool looks super awesome and I can absolutely see myself using it. I've tried greenscreen in OBS before and decided it wasn't worth the effort for my purposes; something that is roughly decent as this looks at a glance will be really useful.

usui 5y ago

Finally! So glad to see a plugin make an earnest attempt on this now. Been using Snap Camera green screen background combined with OBS chroma keying to make it work, but will try this out now.

KingOfCoders 5y ago

I use Nvidia Broadcast, it has the least artifacts and the roll of when blurring is also the best I think, downside it uses a lot of GPU and my RTX2080 fan spins up.

Netcob 5y ago
It works surprisingly well! I started using it for the noise filtering, which is even better.
cptskippy 5y ago
Are you using a lot of the other filters too? I use Broadcast's background removal and the tracking zoom on a Quadro 4000 mobile and it remains mostly silent.

deng 5y ago

This is a great plugin. I'm using this to have background blurring with the Linux MS Teams client, which for whatever reason does not support this natively.

rasz 5y ago

"AI". Wow this looks bad, worse than just taking a screenshot of the background and subtracting from video feed.

CharlesW 5y ago

So if a proper chromakey setup is more of a commitment/hassle than many people are willing to accept, and software background removal is imperfect to the point of distraction, why aren't depth-sensing solutions (like the $300 Intel RealSense D435) an ideal solution to this problem? What am I missing?

jedberg 5y ago

Why hasn't anyone built a plugin that says "sit still, now move left, now move right" and then just remove all the pixels that didn't move? Seems like it would be easy and work as long as the camera doesn't move. Is this a harder problem than I think it is?

Udo 5y ago
I think current algorithms do a version of that, but it has to be coupled together with person/outline recognition and such. Taking the background in just once doesn't work because the pixels still change over time (due to lighting and such). What's really impressive to me is that it looks like web apps like Google Meet and Discord seem to do this in JavaScript on the fly.

andrewmackrodt 5y ago

This looks cool, being both free and cross platform. A non free (4 hours free a month) solution I quite like on macOS is krisp.ai which I also use for noise removal. Background replacement was recently added to M1 chips. On the Windows side, Nvidia Broadcast has worked well for me (requires an RTX card), it also supports tracking which I really like. I've previously tried XSplit but the results were worse than whatever implementation Zoom and even Hangouts in a browser use. I rarely video call under Linux (non existent laptop drivers for my webcam) but look forward to trying this out if the need ever arises.

shirro 5y ago

My kids missed out on most of the remote schooling thing due to location but they got 8 days earlier this year and made the most of it setting up OBS under linux with this plugin and fed it into MS Teams as a virtual camera. They made up several scenes they could switch between with subtle animated backgrounds and added chroma keyed meme/animated stuff they could enable during less formal time like end of lessons. It isn't as good as the closed source offerings from big companies but those companies don't have enough users on Linux to justify developing for the platform so it is a chicken and egg thing. For what it is I was happy with it. With a bit of adjustment it worked suprisingly well in actual use. One of the kids was stuck with an old under powered laptop with integrated graphics and it had no problems at all running this. If you want to do this stuff properly you obviously need a green screen and good lighting but when you suddenly have an entire family home and online and are trying to juggle access to computers, quickly locating and finding space for multiple green screens and studio lighting isn't an option for regular people.