11 ms·
VideoGigaGAN: Towards detail-rich video super-resolution
- geor9e 3y agoThis is great. I look forward to when cell phones run this at 60fps. It will hallucinate wrong, but pixel perfect moons and license plate numbers.
- 1970-01-01 3y agoJust get a plate with 'AAAAA4' and blame everything on 'AAAAAA'
- MetaWhirledPeas 3y agoI look forward to VR 360 degree videos using something like this to overcome their current limitations, assuming the limit is on the capture side.
- constantcrying 3y agoThe first demo on the page alone shows that it is a huge failure. It clearly changes the expression of the person. Yes, it is impressive, but it's not what you want to actually "enhance" a movie.
- philipov 3y agoIt doesn't change the expression - the animated gifs are merely out of sync. This appears to happen because they begin animating as soon as they finish loading, which happens at different times for each side of the image.
- mlyle 3y agoReloading can get them in sync. But, it seems to stop playback of the "left" one if you drag the slider completely left, which makes it easy to get desynced again.
- turnsout 3y agoI agree that it's not perfect, though it does appear to be SoTA. Eventually something like this will just be part of every video codec. You stream a 480p version and let the TV create the 4K detail.
- ethbr1 3y agoSo, DLAA for video instead of games? https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing https://en.m.wikipedia.org/wiki/Deep_learning_anti-aliasing
- jsheard 3y agoNot really, DLAA and the current incarnation of DLSS are temporal techniques, meaning all of the detail they add is pulled from past frames. That's an approach which only really makes sense in games where you can jitter the camera to continuously generate samples at different subpixel offsets with each frame. The OP has more in common with the defunct DLSS 1.0, which tried to infer extra detail out of thin air rather than from previous frames, without much success in practice. That was like 5 years ago though so maybe the idea is worth revisiting at some point.
- constantcrying 3y agoWhy would you ever do that? If you have the high res data you can actually compress the details which are there and then recreate them. No need to have those be recreated, when you actually have them. Downscaling the images and then upscaling them is pure insanity when the high res images are available.
- turnsout 3y agoSo streaming services can save money on bandwidth
- yellow_postit 3y agoOr low connectivity scenarios that pushes more local processing. I think it a bit unimaginative to see no use cases for this.
- metalrain 3y agoVideo quality seems really good, but limitations are quite restrictive "Our model encounters challenges when processing extremely long videos (e.g. 200 frames or more)". I'd say most videos in practice are longer than 200 frames, so lot more research is still needed.
- chompychop 3y agoI guess one can break videos into 200-frame chunks and process them independent of each other.
- prmoustache 3y agoAt 30fps, which is not high, that would mean chunks of less than 7 seconds. Doable but highly impractical to say the least.
- rowanG077 3y agoYou will probably have to have some overhang of time to get the state space to match enough to minimize flicker in between fragments.
- bredren 3y agoPerhaps a second pass that focuses on smoothing out the frames where the clips are joined.
- bmicraft 3y agoYou could probably mitigate this by using overlapping clips and fading between them. Pretty crude but could be close to unnoticeable, depending on how unstable the technique actually is.
- IanCal 3y ago7s is pretty alright, I've seen HLS chunks of 6 seconds, that's pretty common I think.
- IncreasePosts 3y agoI find it interesting how it changed the bokeh from a octagon to a circular bokeh
- Jur 3y agointeresting, which scene is this?
- IncreasePosts 3y agoThe third image in the carousel, with the beer getting poured.
- rowanG077 3y agoI am personally much more interested in frame rate upscalers. A proper 60Hz just looks much better then anything. Also would really, really like to see a proper 60Hz animate upscale. Anything in that space just sucks. But when in the rare cases it works it really looks next level.
- fwip 3y agoFrame-rate upscaling is fine for video, but for animation it's awful. I think it's almost inherently so, because of the care that an artist takes in choosing keyframes, deforming the action, etc.
- rowanG077 3y agoThis just sounds like the AI is just not good enough yet. I mean it's pretty clear now that there is nothing stopping AI from producing close to or sometimes even exceeding human artists. A big problem here is good training material
- fwip 3y agoBaffling to me that you think that AI art is capable of "exceeding" the human training material.
- rowanG077 3y agoI didn't say that. I said that AI is capable of sometimes exceeding human artists. That is not the same thing as saying AI is exceeding the best human artist. If your training material is of high quality it's shouldn't be impossible to exceed human artists some or even most times, i.e. Produce better material then the average or good artists.
- whywhywhywhy 3y agoHave you tried DAIN?
- forgingahead 3y agoNo code?
- jack_riminton 3y agoAnother boon for the porn industry
- esafak 3y agoWhy, so they can restore old videos? I can't see much demand for that.
- falcor84 3y ago"I can't see much" - that's the demand
- jack_riminton 3y agoOk then?
- duskwuff 3y agoThere are a lot of old porn videos out there which have become commercially worthless because they were recorded at low resolutions (e.g. 320x240 MPEG, VHS video, 8mm film, etc). Being able to upscale them to HD resolutions, at high enough quality that consumers are willing to pay for it, would be a big deal. (It doesn't hurt that a few minor hallucinations aren't going to bother anyone.)
- falcor84 3y agoHistory demonstrates that what's good for porn is generally good for society.
- sys32768 3y agoFinally, we get to know whether the Patterson bigfoot film is authentic.
- dguest 3y agoI can't wait for the next explosion in "bigfoot" videos: wildlife on the moon, people hiding in shadows, plants, animals, and structures completely out of place. The difference will be that this time the images will be crystal clear, just hallucinated by a neural network.
- k2xl 3y agoWould be neat to see this on much older videos (maybe WW2 era) to see how it improves details.
- bberrry 3y agoYou mean _invents_ details.
- loudmax 3y agoThat is essentially what Peter Jackson did for the 2018 film They Shall Not Grow Old: https://www.imdb.com/title/tt7905466/ https://www.imdb.com/title/tt7905466/ They used digital upsampling techniques and colorization to make World War One footage into high resolution. Jackson would later do the same process for the 2021 series Get Back, upscaling 16mm footage of the Beatles taken in 1969: https://www.imdb.com/title/tt9735318/ https://www.imdb.com/title/tt9735318/ Both of these are really impressive. They look like they were shot on high resolution film recently, instead of fifty or a hundred years ago. It appears that what Peter Jackson and his team did meticulously at great effort can now be automated. Everyone should understand the limitations of this process. It can't magically extract details from images that aren't there. It is guessing and inventing details that don't really exist. As long as everyone understands this, it shouldn't be a problem. Like, we don't care that the cross-stitch on someone's shirt in the background doesn't match reality so long as it's not an important detail. But if you try to go Blade Runner/CSI and extract faces from reflections of background objects, you're asking for trouble.
- aftbit 3y agoCan this take a crappy phone video of an object and convert that into a single high resolution image?
- jampekka 3y agoThat's known as multi-frame super-resolution. https://paperswithcode.com/task/multi-frame-super-resolution https://paperswithcode.com/task/multi-frame-super-resolution
- herculity275 3y agoWonder how long until Hollywood CGI shops have these types of models running as part of their post-production pipeline. Big blockbusters often release with ridiculously broken CGI due to crunch (Black Panther's third act was notorious for looking like a retro video-game), adding some extra generative polish in those cases is a no-brainer.
- j45 3y agoA few years if not less. They will have huge budgets for compute and the makers of compute will be happy to absorb those budgets. Cloud production was already growing but this will continue to accelerate it imho
- lupusreal 3y agoWasn't Hollywood an early adopter of advanced AI video stuff, w.r.t. de-aging old famous actors?
- inhumantsar 3y agoyeah and the only reason we don't see more of it was prohibitively expensive for all but basically Disney. the compute budgets for basic run of the mill small screen 3D rendering and 2D compositing is already massive compared to most other businesses of a similar scale. the industry has been under paying their artists for decades too. I'm willing to bet that as soon as unreal or adobe or whoever comes out with a stable diffusion like model that can be consistent across a feature length movie, they'll stop bothering with artists altogether. why have an entire team of actual people in the loop when the director can just tell the model what they want to see? why shy away from revisions when the model can update colour grade or edit a character model throughout the entire film without needing to re-render?
- j45 3y agoBingo. Except it looked like magic because the tech was so expensive and only available to them. Limited access to the tech added some mystique to it too. Just like digital cameras created a lot more average photographers, it pushed photography to a higher standard than just having access to expensive equipment.
- kfarr 3y agoThis is amazing and all but at what point do we reach the point of there is no more “real” data to infer from low resolution? In other words there are all sorts of information theory research on the amount of unique entropy on a given medium and even with compression there is a limit. How does that limit relate to work like this? Is there a point at which it can say we know it’s inventing things beyond x scaling constant because of information theory research?
- incorrecthorse 3y agoI'm not sure information theory deals with this question. Since this isn't lossless decompression, the point of having no "real" data is already reached. It _is_ inventing things, and the only relevant question is how plausible are the things being invented; in other words, if the video also existed in higher resolution, how close would it actually look like the inferred version. Seems obvious that this metric increases as a function of the amount of information from the source, but I would guess the exact relationship is a very open question.
- itishappy 3y ago> This is amazing and all but at what point do we reach the point of there is no more “real” data to infer from low resolution? The start point. Upscaling is by definition creating information where there wasn't any to begin with. Nearest neighbor filtering is technically inventing information, it's just the dumbest possible approach. Bilinear filtering is slightly smarter. This approach tries to be smarter still by applying generative AI.
- thomastjeffery 3y agoThat point is the starting point. There is plenty of real information: that's what the model is trained on. That information ceases to be real the moment it is used by a model to fill in the gaps of other real information. The result of this model is a facade, not real data.
- peppertree 3y agoHave we reached peak image sensor size. Would it still make sense to shoot in fullframe when you can just upscale.
- dguest 3y agoIf you want to use your image for anything that needs to be factual (i.e. surveillance, science, automation) the up-scaling adds nothing---it's just guessing on what is probably there. If you just want the picture to be pretty, this is probably cheaper than a bigger sensor.
- itishappy 3y agoIt's impressive, but still looks kinda bad? I think the video of the camera operator on the ladder shows the artifacts the best. The main camera equipment is no longer grounded in reality, with the fiddly bits disconnected from the whole and moving around. The smaller camera is barely recognizable. The plant in the background looks blurry and weird, the mountains have extra detail. Finally, the lens flare shifts! Check out the spider too, the way the details on the leg shift is distinctly artificial. I think the 4x/8x expansion (16x/64x the pixels!) is pushing the tech too far. I bet it would look great at <2x.
- Jackson__ 3y ago>I think the 4x/8x expansion (16x/64x the pixels!) is pushing the tech too far. I bet it would look great at <2x. I believe this applies to every upscale model released in the past 8 years, yet undeterred by this scientists keep pushing on, sometimes even claiming 16x upscaling. Though this might be the first one that is pretty close to holding up at 4x in my opinion, which is not something I've seen often.
- goggy_googy 3y agoI think the hand running through the wheat (?) is pretty good, object permanence is pretty reasonable especially considering the GAN architecture. GANs are good at grounded generation--this is why the original GigaGAN paper is still in use by a number of top image labs. Inferring object permanence and object dynamics is pretty impressive for this structure. Plus, a rather small data set: REDS and Vimeo-90k aren't massive in comparison to what people speculate Sora was trained on.
- deleted 3y ago[deleted]
- skerit 3y agoNo public model available yet? Would love to test and train it on some of my datasets.
- Aissen 3y agoThis is great for entertainment (and hopefully the main application), but we need clear marking of such type of videos before hallucinated details are used as "proofs" of any kind by people not knowing how this works. Software video/photography on smartphones is already using proprietary algorithms that "infer" non-existent or fake details, and this would be at an even bigger scale.
- staminade 3y agoFunny to think of all those scenes in TV and movies when someone would magically "enhance" a low-resolution image to be crystal clear. At the time, nerds scoffed, but now we know they were simply using an AI to super-scale it. In retrospect, how many fictional villains were condemned on the basis of hallucinated evidence? :-D
- jsheard 3y agoEnemy of the State (1998) was prescient, that had a ridiculous example of "zoom and enhance" where they move the camera, but they hand-waved it as the computer "hypothesizing" what the missing information might have been. Which is more or less what gaussian splat 3D reconstructions are doing today.
- IanCal 3y agoLike Ryan Gosling appearing in a building https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-added-ryan-goslings-face-to-this-photo/ https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-a...
- RVuRnvbM2e 3y agoOr white Obama https://twitter.com/Chicken3gg/status/1274314622447820801 https://twitter.com/Chicken3gg/status/1274314622447820801 I don't see the word "bias" appear anywhere in this work :'(
- matsemann 3y agoYeah I was curious about that baby. Do they know how it looks, or just guess? What about the next video with the animals. The leaves on the bush, are they matching a tree found there, or just generic leaves perhaps from the wrong side of the world? I guess it will be like people pointing out bird sounds in movies, that those birds don't exist in that country.
- renewiltord 3y agoWow, the results are amazing. Maintaining temporal consistency was just the beginning part. Very cool.
- kouru225 3y agoI need to learn how to use these new models
- esaym 3y agoOk, how do I download it and use it though???
- scoobertdoobert 3y agoIs anyone else concerned at the societal effects of technology like this? In one of the examples they show a young girl. In the upscale example it's quite clearly hallucinating makeup and lipstick. I'm quite worried about tools like this perpetuating social norms even further.
- roughly 3y agoYes, but if you mention that here, you’ll get accused of wokeism. More seriously, though, yes, the thing you’re describing is exactly what the AI safety field is attempting to address.
- Culonavirus 3y ago> is exactly what the AI safety field is attempting to address Is it though? I think it's pretty obvious to any neutral observer that this is not the case, at least judging based on recent examples (leading with the Gemini debacle).
- fwip 3y agoYes, avoiding creating societally-harmful content is what the Gemini "debacle" was attempting to do. It clearly had unintended effects (e.g: generating a black Thomas Jefferson), but when these became apparent, they apologized and tried to put up guard rails to keep those negative effects from happening.
- Culonavirus 3y ago> societally-harmful content Who decides what is "societally-harmful content"? Isn't literally rewriting history "societally-harmful"? The black T.J. was a fun meme, but that's not what the alignment's "unintended effects" were limited to. I'd also say that if your LLM condemns right-wing mass murderers, but "it's complicated" with the left-wing mass murderers (I'm not going to list a dozen of other examples here, these things are documented and easy to find online if you care), there's something wrong with your LLM. Genocide is genocide.
- adzm 3y agoCurious how this compares to Topaz which is the current industry leader in the field.
- goofybozo111 3y ago[flagged]
- cjensen 3y agoThe video of the owl is a great example of doing a terrible job without the average Joe noticing. The real owl has fine light/dark concentric circles on its face. The app turned it into gray because it does not see any sign of the circles. The real owl has streaks of spots. The app turned them into solid streaks because it saw no sign of spots. There's more where this came from, but basically only looks good to someone who has no idea what the owl should look like.
- confused_boner 3y agoIs this considered a reincarnation of the 'rest of the owl' meme
- nomoreban 3y ago[dead]
- softfalcon 3y agoI'm curious as to how well this works when upscaling from 1080p to 4K or 4K to 8K. Their 128x128 to 1024x1024 upscales are very impressive, but I find the real artifacts and weirdness are created when AI tries to upscale an already relatively high definition image. I find it goes haywire, adding ghosting, swirling, banded shadowing, etc as it whirlwinds into hallucinations from too much source data since the model is often trained to work with really small/compressed video into an "almost HD" video.
- rjmunro 3y agoI wonder if you could specialise a model by training it on a whole movie or TV series, so that instead of hallucinating from generic images, the model generates things it has seen closer-up in other parts of the movie. You'd have to train it to go from a reduced resolution to the original resolution, then apply that to small parts of the screen at the original resolution to get an enhanced resolution, then stitch the parts together.
- hellofellows 3y agohmm is there something more specifically for lecture videos? I'm tired of watching lectures in 480p...
- therealmarv 3y agoWhen do I have that in my Nvidia Shield? I would pay $$$ to have that in real-time ;)
- bick_nyers 3y agoAre you using your Shield as an HTPC? In that case you can use the upscaler built into a TV. I prefer my LG C2 upscale (particularly the frame interpolation) compared to most Topaz AI upscales.
- sciencesama 3y agoShow me the code
- cynicalpeace 3y agoWe need to input UFO vids into this ASAP to get a better guess as what some of those could be.
- tambourine_man 3y agoVideos autoplay in full screen as I scroll in mobile. Impressive tech, but could use better mobile presentation
- can16358p 3y agoYup, same here (iPhone Safari). They go fullscreen and can't dismiss them (they expand again) unless I try it very fast a few times.
- tambourine_man 3y agoTerrible viewing experience
- 1shooner 3y agoThis seems technically very impressive, but it does occur to my more pragmatic side that I probably haven't seen videos as blurry as the inputs for ~ 10 years. I'm sure I'm unaware of important use cases, but I didn't realize video resolution was a thing we needed to solve for these days (at least inference for perceptive quality).
- sizzle 3y agoThe video comparison examples, while impressive, were basically unusable on mobile Safari because they launched in full screen view and broke the slider UI.
- can16358p 3y agoYeah, and in my case they immediately went fullscreen again the moment I dismissed them, hijacking the browser.
- imhereforwifi 3y agoThis looks great, however, things like rolling shutter or video wipes/transitions will be interesting to see how it handles that. Also, all of the sample videos the camera is locked down and not moving, or moving just ever so slightly (the ants and the car clips ). It looks like they took time to smooth out any excessive camera shake. Intergrading this with Adobe's object tracking software (in premier/after effects) may help.
- deleted 3y ago[deleted]
- smokel 3y agoIt seems likely that our brains are doing something similar. I remember being able to add a lot of detail to the monsters that I could barely make out amidst the clothes piled up on my bedroom floor.
- fladd 3y agoWhat exactly does this do? They have examples with a divider in the middle that you can move around and one side says "input" and the other "output". However, no matter where I move the slider, both sides look identical to me. What should I be focusing on exactly to see a difference?
- 7734128 3y agoIt has clearly just loaded incorrectly for you (or you need glasses desperately). The effect is significant.
- petermcneeley 3y agoI wonder how well this works with compressed video. The low res input video looks to be raw uncompressed.
- jiggawatts 3y agoSomething I've been thinking about recently is a more scalable approach to video super-resolution. The core problem is that any single AI will learn how to upscale "things in general", but won't be able to take advantage of inputs from the source video itself. E.g.: a close-up of a face in one scene can't be used elsewhere to upscale a distant shot of the same actor. Transformers solve this problem, but with quadratic scaling, which won't work any time soon for a feature-length movie. Hence the 10 second clips in most such models. Transformers provide "short term" memory, and the base model training provides "long term" memory. What's needed is medium-term memory. (This is also desirable for Chat AIs, or any long-context scenario.) LoRA is more-or-less that: Given input-output training pairs it efficiently specialises the base model for a specific scenario. This would be great for upscaling a specific video, and would definitely work well in scenarios where ground-truth information is available. For example, computer games can be rendered at 8K resolution "offline" for training, and then can upscale 2K to 4K or 8K in real time. NVIDIA uses this for DLSS in their GPUs. Similarly, TV shows that improved in quality over time as the production company got better cameras could use this. This LoRA fine-tuning technique obviously won't work for any single movie where there isn't high-resolution ground truth available. That's the whole point of upscaling: improving the quality where the high quality version doesn't exist! My thought was that instead of training the LoRA fine-tuning layers directly, we could train a second order NN that outputs the LoRA weights! This is called a HyperNet, which is the term for neural networks that output neural networks. Simply put: many differentiable functions are twice (or more) differentiable, so we can minimise a minimisation function... training the trainer, in other words. The core concept is to train a large base model on general 2K->4K videos, and then train a "specialisation" model that takes a 2K movie and outputs a LoRA for the base model. This acts as the "medium term" memory for the base model, tuning it for that specific video. The base model weights are the "long term" memory, and the activations are its "short term" memory. I suspect (but don't have access to hardware to prove) that approaches like this will be the future for many similar AI tasks. E.g.: specialising a robot base model to a specific factory floor or warehouse. Or specialising a car driving AI to local roads. Etc...
- sheepscreek 3y agoVery impressive demonstration but a terrible mobile experience. For the record, I am using iOS Safari.
- thesz 3y agoHow does it compare to fractal video compression [1]? [1] https://www.computer.org/csdl/proceedings-article/dcc/1992/00227455/12OmNBc1uvQ https://www.computer.org/csdl/proceedings-article/dcc/1992/0...
- vim-guru 3y agoHow long until we can have this run real-time in the browser? :D
- woctordho 3y agoGlad to see that Adobe is still investing on the alias-free convolutions (as in StyleGAN3), and this time they know how to fill the lost high frequency features I always thought that alias-free convolutions can produce much more natural videos
- thrdbndndn 3y agois there code/model available to try out?
- nullsmack 3y agoI was hoping this would be an open-source video upscale until I saw it was from Adobe.