15 ms·
Real Time Person Removal from Complex Video
- ulises314 7y agoWe have been always been at war with Eurasia.
- jackallis 7y agois it just me or arent we supposed to be terrified of this kind of development? Cringe.
- johnkpaul 7y agoYou mean specifically because of the real-time-ness of it? As far as I understand, with professional video editing tools and skills, this camera-in-fixed-location removal is trivial.
- Nasrudith 7y agoPersonally I find anything we have to be told to be terrified of instead of drawing our own conclusions eyeroll worthy manipulation.
- williesleg 7y agoI only trust video from cnn.
- twism 7y agoReminds of the movie "Rising Sun"
- xiphias2 7y agoIt would be great to see what the more advanced (but bigger and slower) ResNet model is capable of: https://github.com/tensorflow/tfjs-models/tree/master/body-pix https://github.com/tensorflow/tfjs-models/tree/master/body-p...
- lettergram 7y agoI think this can be improved if edge detection was applied. At the moment it appears to be a bounding box, this lets shoulders and stuff on the edges appear out of box
- elboru 7y agoJust yesterday I watched a video that was difficult to detect it was fake, it was too funny to be true though. Anyways I couldn't tell just by looking at the video. The video showed Mexico's president next to a phallic shaped cactus. I like to think I'm good detecting fake videos or images, but I guess now I'll have to check the sources every time. Fake (possibly NSFW): https://www.youtube.com/watch?v=4r0TCQrfMq4 https://www.youtube.com/watch?v=4r0TCQrfMq4 Real: https://youtu.be/31KzD5a3KS4?t=88 https://youtu.be/31KzD5a3KS4?t=88
- rolltiide 7y agoThat's just a composite though. Nothing currently novel there.
- fudged71 7y agoI work in a gym, where lots of people take video of themselves doing exercises (for instagram, or form checks), despite the gym's policy for privacy reasons (other members in the background). There's an immediate need for an app that will remove the background of people for privacy purposes.
- MiniGod 7y agoCombine this with facial recognition to only remove unrecognized people (or opposite)
- cartercole 7y agoi think i saw this in a black mirror episode
- uyuioi 7y agoBank heists just got that much easier
- RotaryTelephone 7y agoThat bag of cash is floating out the door all by itself!
- TuringTest 7y agoSure, you just need to hack into the camera hardware system and inject a browser-specific machine learning library. Easy-peasy
- dkdk8283 7y agoWe’re very close to not being able to trust video, I don’t think many people outside of tech realize how close we are. Imagine someone adding your face to a video of a crime.
- archivist1 7y agoYeah, it turns out "El Coco" was actually just an AI master.
- deleted 7y ago[deleted]
- btrettel 7y agoI'm not sure video ever went as far as many people think it does in terms of criminal prosecution. My experience suggests that video alone is often not enough for a conviction. I'm one of the increasing number of helmet camera cyclists. Last year, a driver nearly hit me head on and stopped nearby, so I went over to politely tell them to pay better attention. They got angry and tried to assault me with a screwdriver. Once the driver got violent, I left, shouting out his plate number for the camera, and then called the cops. The cops arrived a few hours later and loved the video. But after working with a detective, it became clear that the video probably wasn't enough to get a conviction. We had to get a second independent piece of evidence placing the driver at the scene of the crime. In this case, the driver was working for Uber, whose GPS tracking placed him at the scene at the time of the incident. (I had no idea the guy was an Uber driver, and I'm thankful that the police had the insight to suggest this possibility.) Just last week the driver plead guilty to disorderly conduct, after the charges were reduced as part of a deal made with his lawyer. You can watch the clip here if you're curious: https://www.youtube.com/watch?v=trSB3mK78bs https://www.youtube.com/watch?v=trSB3mK78bs
- syshum 7y ago>>after the charges were reduced as part of a deal made with his lawyer. That is more likely why they need the 2nd piece, They want to make a deal, they never want to go to trial, so if they have overwhelming evidence then 99% of guilty people, and 90% of innocent people will take the deal offered
- deleted 7y ago[deleted]
- mackrevinack 7y agothis would be very useful in busy tourist spots where you want a video of something, the leaning tower of piza for example, but minus all the annoying people pretending to push it over
- deleted 7y ago[deleted]
- calmworm 7y agohold it up.
- sschueller 7y agoAlso useful if you want to live stream in a country where privacy laws make it illegal unless you get everyones consent. Live face blurring would probably be enough however.
- olivierduval 7y agoIsn't it what "long exposure photography" (and photo stacking in photoshop) is used for ? ;-) I mean: if there's too much crowd, you have to wait a long time to get every little part of what's behind. So I'm not sure that ML will be useful except maybe to detect "humans" and decide what parts need to be replaced
- bangboombang 7y agoYes! Just take a lot of photos over a couple minutes (depending on how busy your scene is) from a fixed position, or just a video if you're lazy, then use imagemagick and combine them using "median" (NOT average). It's not always perfect but can deliver most of the time. That way even a command line dork like me can do it. :-)
- ansgri 7y agoA similar simple trick allows to reduce reflections from photos of flat objects under museum glasses (books, pictures, coins): take several photos from slightly different angle, co-register them using some panorama assembly techique (I've used a simple Python OpenCV script) and merge with minimum or some bottom quantile, since reflections are additive.
- alasdair_ 7y agoStalin famously had people that had “left” his administration airbrushed out of photographs. I don’t think it will be long before current-day dictators start doing the same thing with old footage containing previous allies they no longer wish to be associated with.
- deleted 7y ago[deleted]
- PavlovsCat 7y ago"tech" as seen and practiced by too many people mostly does 2 things: enable total control, and infantilize people so they can't even meaningfully discuss what they're creating. The rest is toys, breadcrumbs, distractions -- plus minor improvements that will be completely undone when the fruit of totalitarianism is ripe.
- coribuci 7y agoGerman tv WDR also removed sensitive content from a video. At least a dictator has an excuse.
- eb0la 7y agoNot just people, the famous 'Red Flag over the Reichstag' photograph was modified (added smoke and removed watches on soldier arms).
- yozel 7y agoThis project should have the name "Stalin"
- coribuci 7y agoThis will be a praise for him. Unfortunately censorship has become common practice around the world and it did not started with Stalin.
- edf13 7y agoWhat’s impressive is that this is done in JavaScript!
- dmd 7y agoWhy is that impressive? JS is one of the most powerful and performant general-purpose languages in common use.
- Traster 7y agoI was kind of disappointed with this. I may be misunderstanding this but essentially it seems to use machine learning to identify body parts and then just plasters over the background. Now what I would have expected from a 'complex video' would at the least be a moving camera - in which case you actually have to build a model of the environment ala GuaGAN and generate from it.
- dannyw 7y agoYes. It’s best to underpromise and overdeliver. If you’re not really anywhere near SOTA, then be accurate in your description, eg “Real-time person inpainting from video in JavaScript”
- greenbean4 7y agoHas "Real Time Background Removal" already implemented?
- penagwin 7y agoYeah it's been around for a long time - although many of the earlier version were rather awful. There's different methods, and not all of them use neural networks. But there's software to do it with a decent camera that works okay, you'd be better off with a green screen though in most any situation.
- Nasrudith 7y agoDepends on the background by method. At thr very least chroma key and edge detection qualify - define a target and take them to their edges - just with signal processing and no learning.
- po1nter 7y agoIn MS Teams you can blur the background while video chatting. So I'd say yes, it exists.
- lowdose 7y agoCommercially it would be more interesting to do active background removal. If every Youtuber has a professional background like a news anchor they would immediately look legit.
- sergiotapia 7y agozoom does this in real time, virtual background.
- susam 7y agoCame here to confirm this. Indeed Zoom does this in real time and replaces the real background with a virtual background or solid colour. We often use it at our workplace for fun.
- m463 7y agohttps://youtu.be/GxaAhIOgho https://youtu.be/GxaAhIOgho
- bangboombang 7y agoGreen screen? Yes, it still requires some effort (buy and set up the actual screen, proper lighting), but I doubt a proper green screen setup would be outperformed by some AI system any time soon: Got a shitty cam? AI would have to fix that too. Shitty lighting? Still a problem after any background removal. And if you're gonna invest in those two things, might as well go for your green screen.
- deadbunny 7y agoPretty sure this is already a thing: https://news.developer.nvidia.com/logitechs-new-webcam-includes-dynamic-background-removal-feature/ https://news.developer.nvidia.com/logitechs-new-webcam-inclu...
- bsenftner 7y agoBackground replacement is a commodity motion graphics tool. It has been around since the 90's, and exponentially better than this work here.
- kashishgrover 7y agoThis is what they used to remove Bojack from Horsin' Around :O
- theemathas 7y agoI have doubts this hasn't already been done given that the background is static.
- bsenftner 7y agoYes, 35+ years ago. Background replacement is a commodity motion graphics tool.
- mfer 7y agoMaking provenance of software and data easier will be of increasing importance in the future. We need to be able to prove things have not been tampered with.
- eskaytwo 7y agoThe corollary to this is that people’s faith in video will be so low, that actual video of a major event can be dismissed as potentially fake - and the more extreme the more easy to dismiss. The sword cuts both ways: false negatives being accepted as true and true positives being dismissed as fakes.
- douglaswlance 7y agoIt is easier to detect fakes than it is to create them. The datasets will always be better for fakes.
- pbhjpbhj 7y agoThat doesn't mean the fake-finders win, just that the fakers with most resources are the ones who can pass their fakes.
- mlrtime 7y agoHow is this any different than a picture? We have been making fake pictures, editing, airbrushing, photoshoping for years.
- infinity0 7y agoThose have taken much more time to produce. Now it will take just a few seconds.
- hombre_fatal 7y agoPeople on FB and Reddit are just upvoting Twitter-screenshot-style pictures and screenshots of headlines with no sources. That takes about 2 seconds to create. Just choose-your-own-caption with an accompanying image. The fear of fake videos is overblown. What amounts to a screenshot of a headline can get 30k upvotes and comments on Reddit without anyone asking for a source. That tendency is what worries me, not the medium.
- 7y ago
- bsenftner 7y agoLooks like mid 90's background replacement extremely early work. This type of thing, and exponentially better - as you know from feature film VFX - have accomplished these types of operations decades ago, and have published how it is done, and commodity versions of this logic are in open source packages like blender. Deep Fakes and Style GAN has people all excited, but this type of effect does not need ML and are fully features digital toys now. Hell, after 40 years as a developer of games, 3D graphics, interactive video, 3D games, feature film VFX, a digital double creation service, and now FR I can say with authority: this work is nothing.
- superdisk 7y agoIt's interesting though because an amateur slapped it together quickly rather than an entire industry over decades.
- bsenftner 7y agoI wrote one myself back in '93 that was used in talking head videos by a documentary company to place their talking heads into the locations of their films. The essential logic is quite simple, leveraging the fact that you've got previous frames.
- paulsutter 7y agoThe earliest cameras around 1830 already removed people, perfectly. Exposure times were as long as 8 hours, so people were invisible. only buildings, trees, etc appeared. Link below includes images of streets with people “removed”, and has the oldest known image of a person, a shoeshine that stayed in one place long enough to be seen https://www.livescience.com/60387-oldest-photographs.html https://www.livescience.com/60387-oldest-photographs.html
- dTal 7y agoIt's arguable that 8 hour exposures count as "real time", however :)