3 ms·
So this is surprisingly bleeding edge, at least to me. I had to go learn about some hardware and physical imaging stuff I didn’t know to get my head around it.
by vessenes 2y ago
So this is surprisingly bleeding edge, at least to me. I had to go learn about some hardware and physical imaging stuff I didn’t know to get my head around it.
Upshot: Event Cameras are a different sort of camera in that they have an array of sensor pixels, and sensors only fire when there is a brightness change for that sensor. This has a bunch of benefits, including very high dynamic range, reduced ghosting, and high frame rates, and has some downsides, like reconstructing video, and presumably others.
The paper seems to have started out with the idea that if you had event camera output, you’d be able to reconstruct more fine texture details. And, this works incredibly well, their baby model trained for 8 days significantly beats SOTA and looks a lot better in comparisons as well.
They then seem to have added a step where you simulate/infer event camera data from “normal” RGB video, using a different set of networks, and use that inferred event data to do the texture recovery, and … this also works.
Pretty surprising, and interesting. Their GitHub is full of people like “I want to try this” and then realizing it’s a fairly deep stack to deploy. Even as is, it seems worth someone building a GUI around this in an app, it’s quite remarkable.
- MetaWhirledPeas 2y agoMy first thought is, is this fake? (Edit: Okay maybe not?) It looks extremely fake. (Edit: A credit to how effective it is?) Like the blurriness seems applied after the fact. It can really generate a license plate number from that shadowy rectangle? I'm expecting to see giveaways for AI but I see none, except maybe the simple oval badge on the sedan (although that might be a real vehicle make; not sure). If someone else manages to deploy and try this please share your result.
- vessenes 2y agoIf I understand what they did correctly it won’t have the same failure mode of hallucinations that a diffusion model has - it’s not a model that has an understanding of the world, it’s a model that’s really good at turning async per pixel light event data plus blurry rgb into sharp rgb. That said I don’t understand it very well, for instance there’s a voxel step in the pipeline and I have no idea why.
- 4RealFreedom 2y agoThe first thing I looked for was text and saw they used cars in an example. I thought the same thing as you 'this can actually get readable license plates?' I don't know enough about ML to understand the associated paper, though.
- MetaWhirledPeas 2y agoOne thing that might explain it is this logic thread: a) this is a car, b) this is where a car's license plate would normally be, and c) license plates usually have numbers like this. Upon close inspection the plate's digits look realistic, but there are some symbols that look unfamiliar to me. But I don't know what country the footage is from, so I don't know real from unreal when it comes to symbols. If the car's badge turned out to correctly match the car model, that might be a bit of a red flag. Although it's not out of the question that a model could eventually recognize car models and get badges right. It just seems unlikely that I'd see such an advancement in a video before I ever saw it in a still image.
- MarcoZavala 2y ago[dead]
- boogerlad 2y agoWhat's the display equivalent for an event camera?
- nicd 2y agoInstead of writing out frames at a fixed FPS (ie, 60 fps), the driver would send out updates to individual pixels at precise moments in time, and the unaddressed pixels would remain unchanged. I'm not sure whether or where this kind of technology is used in practice.
- boogerlad 2y agoYeah I've been searching for this kind of display technology for the past couple years and concluded I'm using the wrong search terms or it just doesn't exist. Techically possible with oled / microled.