25 ms·
Self-Supervised Learning [pdf]
- planckscnst 7y agoIf we can use GANs to produce believable images of a subject and we can identify whether a video frame can exist between two others, it seems like we can produce infinitely high framerate videos that look like they were captured that way. This also means we can make slow motion video when we didn't record at a high framerate. I think with that, colorization, and possibly similar tools for audio, we might see some amazing recreations of classic performances.
- danielscrubs 7y agoScrubs in 4k!
- gwern 7y agoYep, you certainly can. 'In-betweening', like superresolution, has been a GAN thing for years now, because triplets of frames are a clean dataset but you also care more about perceptual plausibility than pixel error. People use in-betweening GANs to make things like 60 FPS anime. (Not entirely sure why, but they do.)
- usmannk 7y agoThat sounds interesting. Where can I read more about people doing this? (And other fun things with GANs and/or anime)
- zitterbewegung 7y agoGwern has a website https://www.gwern.net https://www.gwern.net
- deleted 7y ago[deleted]
- cycrutchfield 7y agoI mean, both things have been around much much longer than GANs. I think Freeman’s paper on superresolution is from the early 2000s.
- rrishi 7y agoThis is really interesting. Any place I can get to know more about this?
- ionionionioa 7y ago>People use in-betweening GANs to make things like 60 FPS anime Animation seems like an especially poor fit to me, since the actual framerate is often much lower than the video's framerate. Framerate can vary between scenes and even within different parts of one scene! Typically the background is very low framerate (sometimes as low as 4 FPS), the foreground is higher framerate (typically 8-12 FPS), while pans, zooms, and 3D elements are at a full 24 FPS. Most of the additional frames from interpolation will therefore be exact duplicates of other frames. This does little to improve the smoothness of the video. It just adds in artifacts. And, since the frames between two drawings will be interpolated while frames within one drawing will be unchanged, the framerate will be inconsistent and appear as judder. Interpolation will never work for 2D animation. No way, no how. Any worthwhile system will need to modify existing frames rather than simply adding more in between the original frames. I can understand interpolation for live action (though I still dislike it), but it is absolutely god-awful for animation.
- gwern 7y agoI think that's wrong: the whole point of GANs is that they're quite intelligent and good at faking outputs. I've seen interpolated/in-betweened videos (mostly but not entirely live-action), and it looks realistic to me. The reason I'm somewhat skeptical is that just because something looks realistic doesn't mean that it what is intended. It's a version of the 'zoom in, enhance, enhance' problem. It's like the _Hobbit_ problem: a GAN could perfectly well fake a 60FPS version of a 30FPS version of the _Hobbit_ such that you couldn't tell that it wasn't the actual 60FPS version that Peter Jackson shot... but the problem is that it's 60FPS and that just feels wrong for cinema. Animators, anime included, use the limitations of framerate and deliberate switches between animating 'on twos' etc, with reductions in framerates being deliberately done for action segments and sakuga and other reasons. An anime isn't simply a film which was unavoidably shot with a too-low framerate. (This is less true of superresolution: in most cases, if an anime studio could have afforded to animate at a higher resolution originally, they would have; and you're not compromising any 'artistic vision' if you use a GAN to do a good upscaling job instead of a lousy bilinear upscale built into your video player.)
- 7y ago
- antpls 7y agoMEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Frame Interpolation and Enhancement : https://sites.google.com/view/wenbobao/memc-net https://sites.google.com/view/wenbobao/memc-net One of the author also worked on https://news.ycombinator.com/item?id=20188316 https://news.ycombinator.com/item?id=20188316
- shostack 7y agoCouple this then be used to reduce the storage required for phones to capture slow motion video since it can simply be done with regular video in post server side?
- kachurovskiy 7y agoThis is already possible in some video editors, see e.g. DaVinci Resolve optical flow - https://youtu.be/4LsLaJq5-ow https://youtu.be/4LsLaJq5-ow
- spyder 7y agoYes: AI Learns Video Frame Interpolation | Two Minute Papers #197 https://www.youtube.com/watch?v=T_g6S3f0Z5I https://www.youtube.com/watch?v=T_g6S3f0Z5I
- l-m-z 7y agoThis is from yesterday's (June the 15th) workshop on self-supervised learning at ICML. The video of this talk can be seen here: https://www.facebook.com/icml.imls/videos/2030095370631729/ https://www.facebook.com/icml.imls/videos/2030095370631729/ (not sure if the video is also available on other platforms than facebook)
- varelse 7y agoSee also "The Revolution Will Not Be Supervised" from CS294-158... https://www.youtube.com/watch?v=PX11C5Vfo9U https://www.youtube.com/watch?v=PX11C5Vfo9U
- picozeta 7y agoThanks for that, this is an interesting talk.
- hbarka 7y agoI’m nowhere near this field, but for the experts in here I was wondering if Self-Supervised Learning changes the paradigm for the approach to self-driving cars?
- ec109685 7y agoAt what point are we able to deliver low resolution video and have the system make up on the fly believable high resolution version of it?
- felipellrocha 7y agoENHANCE!
- derefr 7y agoNah, the CSI "enhance" thing is "multi-frame super-resolution image recovery", a different (though related) ML technique. Speaking of, though: you'd think, by now, that security cameras that capture footage at very low framerates for the sake of storage space, would have ASICS in them using those models to convolve together a bunch of grainy input frames into a stream of fewer, but very good and clean frames. Any hardware on the market with this capability yet?
- ClassyJacket 7y agoIt makes sense for entertainment but not for security cameras - then you're filling it in with made up information. A security camera is supposed to be a record of truth.
- oneshot908 7y agoImagine a world where low information sorts interpret a sampling of possible hi-res reconstructions from low-res security videos as ground truth. That to me is far scarier than the OpenAI and MIRI fear-mongering about GPT-2.
- derefr 7y agoIt’s not made-up information; it’s parallax / compressed sensing, in the same way that you can see through the grate on the front of a microwave oven to what’s behind it by moving your eyes around. If it’s good enough for generating accurate fMRI images from sequentially-overlaid magnetic flux readings, it’s definitely good enough for generating visuals from slightly suckier visuals.
- p1esk 7y agoYann Lecun is also a fan of self-supervised learning: https://www.facebook.com/epflcampus/videos/yann-lecun-self-supervised-learning-could-machines-learn-likes-humans/1960325127394608/ https://www.facebook.com/epflcampus/videos/yann-lecun-self-s...
- shgidi 7y agoA post series that also summarizes this subject: https://link.medium.com/IhOvrqFEzX https://link.medium.com/IhOvrqFEzX
- suyash 7y agoMind blowing progress in the field of self supervised learning.
- amthewiz 7y agoThe DNN based techniques are new but the concept of models that can fill in the blanks is old. It used to be called content addressable memory or autoassociative memory.
- p1esk 7y agoSo, is there any difference between "self-supervised" and "unsupervised" learning?
- make3 7y agoThat guy's definition of self-supervised learning is what I believe most people in the field would call unsupervised learning.
- 6nf 7y agoIs it different from reinforcement learning?
- jokoon 7y agoThat reminds me of those neural network models that would learn to change themselves, a little like a ML algorithm that would learn about neural network that worked best. I think google did something like this some years ago?
- neruotablet 7y agoYou mean Neuroevolution? I.e using Evolutionary Algorithms https://youtu.be/L--IxUH4fac https://youtu.be/L--IxUH4fac to evolve NNs?
- nl 7y agoNo. The Google technique was reinforcement learning based and didn't use evolutionary algorithms at all. https://ai.google/research/pubs/pub45826 https://ai.google/research/pubs/pub45826
- nl 7y agoNeural Architecture Search with Reinforcement Learning[1] This isn't really a unsupervised or self-supervised technique at all. It's a combination of supervised learning with reinforcement learning (which is a whole other thing too). [1] https://ai.google/research/pubs/pub45826 https://ai.google/research/pubs/pub45826
- person_of_color 7y agoIs this the game changer we are looking for?
- p1esk 7y agoWhat are you looking for?
- Blozvez 7y agoFor me, this kind of education always seemed to be amazingly tough. I mean, I always prefered to hire tutors, I live in la https://worldpostalcode.com/united-states/california/los-angeles https://worldpostalcode.com/united-states/california/los-ang... and here it's not that hard to find a good one.