4 ms·

Texture Enhancement for Video Super-Resolution

179 points by smusamashah 2y ago

pornel 2y ago
They don't explain what event-driven means, but AFAIK it's based on diffs between frames, which highlight motion and de-emphasise overall brightness/exposure: https://github.com/uzh-rpg/rpg_vid2e?tab=readme-ov-file#readme https://github.com/uzh-rpg/rpg_vid2e?tab=readme-ov-file#read...
- pavlov 2y ago
  I believe the important point about event cameras is the diffs are per-pixel and entirely asynchronous, so there is no concept of a frame. The video data is simply a stream of events which encode the time and location of a brightness change. For an immediate full-scene change (like removing the lens cap), you’d get a stream that happens to update every pixel, but there’s no particular guarantee about the ordering.
- ipsum2 2y ago
  That's incorrect. Event cameras are different hardware devices than your traditional camera.
vessenes 2y ago
So this is surprisingly bleeding edge, at least to me. I had to go learn about some hardware and physical imaging stuff I didn’t know to get my head around it. Upshot: Event Cameras are a different sort of camera in that they have an array of sensor pixels, and sensors only fire when there is a brightness change for that sensor. This has a bunch of benefits, including very high dynamic range, reduced ghosting, and high frame rates, and has some downsides, like reconstructing video, and presumably others. The paper seems to have started out with the idea that if you had event camera output, you’d be able to reconstruct more fine texture details. And, this works incredibly well, their baby model trained for 8 days significantly beats SOTA and looks a lot better in comparisons as well. They then seem to have added a step where you simulate/infer event camera data from “normal” RGB video, using a different set of networks, and use that inferred event data to do the texture recovery, and … this also works. Pretty surprising, and interesting. Their GitHub is full of people like “I want to try this” and then realizing it’s a fairly deep stack to deploy. Even as is, it seems worth someone building a GUI around this in an app, it’s quite remarkable.
- MetaWhirledPeas 2y ago
  My first thought is, is this fake? (Edit: Okay maybe not?) It looks extremely fake. (Edit: A credit to how effective it is?) Like the blurriness seems applied after the fact. It can really generate a license plate number from that shadowy rectangle? I'm expecting to see giveaways for AI but I see none, except maybe the simple oval badge on the sedan (although that might be a real vehicle make; not sure). If someone else manages to deploy and try this please share your result.
  - vessenes 2y ago
    If I understand what they did correctly it won’t have the same failure mode of hallucinations that a diffusion model has - it’s not a model that has an understanding of the world, it’s a model that’s really good at turning async per pixel light event data plus blurry rgb into sharp rgb. That said I don’t understand it very well, for instance there’s a voxel step in the pipeline and I have no idea why.
  - 4RealFreedom 2y ago
    The first thing I looked for was text and saw they used cars in an example. I thought the same thing as you 'this can actually get readable license plates?' I don't know enough about ML to understand the associated paper, though.
    - MetaWhirledPeas 2y ago
      One thing that might explain it is this logic thread: a) this is a car, b) this is where a car's license plate would normally be, and c) license plates usually have numbers like this. Upon close inspection the plate's digits look realistic, but there are some symbols that look unfamiliar to me. But I don't know what country the footage is from, so I don't know real from unreal when it comes to symbols. If the car's badge turned out to correctly match the car model, that might be a bit of a red flag. Although it's not out of the question that a model could eventually recognize car models and get badges right. It just seems unlikely that I'd see such an advancement in a video before I ever saw it in a still image.
dartharva 2y ago
So we finally have the magical "Enhance" button from sci-fi detective movies and shows, nice!
- scotty79 2y ago
  Except that it's useless for any kind of investigative work where truth matters.
  - xhkkffbf 2y ago
    It's true. And for other kinds of more common uses like watching for flaws. Imagine a system like this watching for small flaws in something with a repeated texture. The algorithm will assume the flaw is a mistake make by the camera because of poor resolution and cover up the flaw.
    - tliltocatl 2y ago
      "A camera that produces a nice picture" and "a camera that tries reproduce reality as faithfully as possible" are going to become two different products. That's also a problem because the second one is not going to benefit from the economy of scales of the first one…
  - montroser 2y ago
    No, that appears to be incorrect. It is the case that with completely generative models, you will get hallucinated details very likely to be untruthful. But with this approach, you can see blurry input images of license plates that with our naked eye we could not possibly decipher the characters, then put through this model where the output is very close to the actual ground truth. https://dachunkai.github.io/evtexture.github.io/static/images/quanlitative_reds.png https://dachunkai.github.io/evtexture.github.io/static/image... Where does this information come from? It seems they are generating synthetic "event camera-like" events just from diffs between still frames? So maybe they trained a model based on real events from a real event camera? It's hard to tell from their write-up. But these results are very impressive.
smusamashah 2y ago
Project page with a few different clips https://dachunkai.github.io/evtexture.github.io/ https://dachunkai.github.io/evtexture.github.io/
- 1970-01-01 2y ago
  Generating plate numbers is concerning.
  - MarcoZavala 2y ago
    [dead]
  - HeatrayEnjoyer 2y ago
    How so
    - 1970-01-01 2y ago
      Worst case scenario would result in a police raid on innocent people.
bilater 2y ago
It would be great if there was an upfront metric about how long the process takes (say per one minute of video) as it usually is a lot.
the8472 2y ago
All the sample clips have camera motion. Does it perform worse with a static camera or is there enough variation from frame to frame to still recover details?
- vessenes 2y ago
  I think if there’s absolutely no motion for part of a scene ever, eg an anime backdrop, this would likely not work well. But if there’s even a pixel or two of motion (so, anything actually filmed), that should be enough to infer events. Just a guess though.
mermerico 2y ago
If my interpretation of the paper is correct, they are using the high resolution event data in addition to the low resolution RGB data in order to do the reconstruction, so this technique won't enhance random videos on the internet. It's a new algorithm to take advantage of event-based cameras that usually record both high resolution event data and low resolution RGB.
ComputerGuru 2y ago
I have an assortment of low quality original encodes from the 90s (an assortment of thousands of mpeg and flv web videos, think divx and co) that I’ve refrained from reencoding in hopes that some day AI would get there and having the originals would pay off. But looking at all the “originals” in the demo, they’re all super blurry (blurry upscaling, I know, but also trademark h264 low bitrate or high deblocking). It would be ironic if I had to use h264/h265 as a deblocking upscale intermediate step before using something like this someday.
ggm 2y ago
Some historicaland b&w footage might have sold the idea. We're hoarding low resolution half scan VHS of our family super8.
K0balt 2y ago
I wonder what you get if you give it extremely low resolution pictures (say, 64x64)

179 points by smusamashah 2y ago

They don't explain what event-driven means, but AFAIK it's based on diffs between frames, which highlight motion and de-emphasise overall brightness/exposure: https://github.com/uzh-rpg/rpg_vid2e?tab=readme-ov-file#readme https://github.com/uzh-rpg/rpg_vid2e?tab=readme-ov-file#read...

pavlov 2y ago
I believe the important point about event cameras is the diffs are per-pixel and entirely asynchronous, so there is no concept of a frame. The video data is simply a stream of events which encode the time and location of a brightness change. For an immediate full-scene change (like removing the lens cap), you’d get a stream that happens to update every pixel, but there’s no particular guarantee about the ordering.
ipsum2 2y ago
That's incorrect. Event cameras are different hardware devices than your traditional camera.

vessenes 2y ago

So this is surprisingly bleeding edge, at least to me. I had to go learn about some hardware and physical imaging stuff I didn’t know to get my head around it. Upshot: Event Cameras are a different sort of camera in that they have an array of sensor pixels, and sensors only fire when there is a brightness change for that sensor. This has a bunch of benefits, including very high dynamic range, reduced ghosting, and high frame rates, and has some downsides, like reconstructing video, and presumably others. The paper seems to have started out with the idea that if you had event camera output, you’d be able to reconstruct more fine texture details. And, this works incredibly well, their baby model trained for 8 days significantly beats SOTA and looks a lot better in comparisons as well. They then seem to have added a step where you simulate/infer event camera data from “normal” RGB video, using a different set of networks, and use that inferred event data to do the texture recovery, and … this also works. Pretty surprising, and interesting. Their GitHub is full of people like “I want to try this” and then realizing it’s a fairly deep stack to deploy. Even as is, it seems worth someone building a GUI around this in an app, it’s quite remarkable.

MetaWhirledPeas 2y ago
My first thought is, is this fake? (Edit: Okay maybe not?) It looks extremely fake. (Edit: A credit to how effective it is?) Like the blurriness seems applied after the fact. It can really generate a license plate number from that shadowy rectangle? I'm expecting to see giveaways for AI but I see none, except maybe the simple oval badge on the sedan (although that might be a real vehicle make; not sure). If someone else manages to deploy and try this please share your result.
- vessenes 2y ago
  If I understand what they did correctly it won’t have the same failure mode of hallucinations that a diffusion model has - it’s not a model that has an understanding of the world, it’s a model that’s really good at turning async per pixel light event data plus blurry rgb into sharp rgb. That said I don’t understand it very well, for instance there’s a voxel step in the pipeline and I have no idea why.
- 4RealFreedom 2y ago
  The first thing I looked for was text and saw they used cars in an example. I thought the same thing as you 'this can actually get readable license plates?' I don't know enough about ML to understand the associated paper, though.
  - MetaWhirledPeas 2y ago
    One thing that might explain it is this logic thread: a) this is a car, b) this is where a car's license plate would normally be, and c) license plates usually have numbers like this. Upon close inspection the plate's digits look realistic, but there are some symbols that look unfamiliar to me. But I don't know what country the footage is from, so I don't know real from unreal when it comes to symbols. If the car's badge turned out to correctly match the car model, that might be a bit of a red flag. Although it's not out of the question that a model could eventually recognize car models and get badges right. It just seems unlikely that I'd see such an advancement in a video before I ever saw it in a still image.

dartharva 2y ago

So we finally have the magical "Enhance" button from sci-fi detective movies and shows, nice!

scotty79 2y ago
Except that it's useless for any kind of investigative work where truth matters.
- xhkkffbf 2y ago
  It's true. And for other kinds of more common uses like watching for flaws. Imagine a system like this watching for small flaws in something with a repeated texture. The algorithm will assume the flaw is a mistake make by the camera because of poor resolution and cover up the flaw.
  - tliltocatl 2y ago
    "A camera that produces a nice picture" and "a camera that tries reproduce reality as faithfully as possible" are going to become two different products. That's also a problem because the second one is not going to benefit from the economy of scales of the first one…
- montroser 2y ago
  No, that appears to be incorrect. It is the case that with completely generative models, you will get hallucinated details very likely to be untruthful. But with this approach, you can see blurry input images of license plates that with our naked eye we could not possibly decipher the characters, then put through this model where the output is very close to the actual ground truth. https://dachunkai.github.io/evtexture.github.io/static/images/quanlitative_reds.png https://dachunkai.github.io/evtexture.github.io/static/image... Where does this information come from? It seems they are generating synthetic "event camera-like" events just from diffs between still frames? So maybe they trained a model based on real events from a real event camera? It's hard to tell from their write-up. But these results are very impressive.

smusamashah 2y ago

Project page with a few different clips https://dachunkai.github.io/evtexture.github.io/ https://dachunkai.github.io/evtexture.github.io/

1970-01-01 2y ago
Generating plate numbers is concerning.
- MarcoZavala 2y ago
  [dead]
- HeatrayEnjoyer 2y ago
  How so
  - 1970-01-01 2y ago
    Worst case scenario would result in a police raid on innocent people.

bilater 2y ago

It would be great if there was an upfront metric about how long the process takes (say per one minute of video) as it usually is a lot.

the8472 2y ago

All the sample clips have camera motion. Does it perform worse with a static camera or is there enough variation from frame to frame to still recover details?

vessenes 2y ago
I think if there’s absolutely no motion for part of a scene ever, eg an anime backdrop, this would likely not work well. But if there’s even a pixel or two of motion (so, anything actually filmed), that should be enough to infer events. Just a guess though.

mermerico 2y ago

If my interpretation of the paper is correct, they are using the high resolution event data in addition to the low resolution RGB data in order to do the reconstruction, so this technique won't enhance random videos on the internet. It's a new algorithm to take advantage of event-based cameras that usually record both high resolution event data and low resolution RGB.

ComputerGuru 2y ago

I have an assortment of low quality original encodes from the 90s (an assortment of thousands of mpeg and flv web videos, think divx and co) that I’ve refrained from reencoding in hopes that some day AI would get there and having the originals would pay off. But looking at all the “originals” in the demo, they’re all super blurry (blurry upscaling, I know, but also trademark h264 low bitrate or high deblocking). It would be ironic if I had to use h264/h265 as a deblocking upscale intermediate step before using something like this someday.

ggm 2y ago

Some historicaland b&w footage might have sold the idea. We're hoarding low resolution half scan VHS of our family super8.

K0balt 2y ago

I wonder what you get if you give it extremely low resolution pictures (say, 64x64)