9 ms·
Interactive Dynamic Video
- pimlottc 10y agoWow, more incredible image processing stuff from MIT. What's really cool is how it shows how much hidden information is out there there in world to be captured within a normal camera. We just need to be aware of it and analyze it properly. For me, it just re-injects a bit of wonder into the everyday world we live in.
- ElijahLynn 10y ago#holyfuck #sciencebitches
- ne0phyte 10y agoHere's TED a talk about the technology http://www.ted.com/talks/abe_davis_new_video_technology_that_reveals_an_object_s_hidden_properties http://www.ted.com/talks/abe_davis_new_video_technology_that...
- Phithagoras 10y agoPapers of their findings can be found at http://www.interactivedynamicvideo.com/publications.html http://www.interactivedynamicvideo.com/publications.html
- jasonkostempski 10y agoKids would go bonkers if they could record their toys and dance with them like at the end of that video.
- arxpoetica 10y agoThe implications of this technology are staggering. He doesn't even talk about online use-cases, but imagine (just as an example) if YouTube added this algorithm to all it's video library, and started allowing you to interact with videos in very different ways. Crazy stuff.
- iKlsR 10y agoYes, so many applications, stop motion would benefit from this incredibly.
- ilovecomputers 10y agoSignal Processing was that subject I found difficult and I found no reason to be interested in audio filtering, so I didn't find an incentive to work through that difficulty. Then I see applications of signal processing, like this, that goes beyond simple audio filtering, that makes me want to learn it again. Honestly, this looks like wizardry! It is no surprise, MIT is on the bleeding edge of signal processing and it mainly is due to Dr. Oppenheim . He wrote the textbook on DSP and runs the DSP Group at MIT.
- gavinpc 10y agoWhat exactly are the implications of this for construction engineering, which must already have some way of measuring vibrations? I.e. does this measure something previously unmeasurable, or is it just that the visualization provides an extra channel for interpreting the data? Or is it that it would make this kind of tool available to anyone with a digital camera?
- tlb 10y agoThe vibrational modes of a structure are a sort of signature. When it changes, that means something in the structure has changed. Perhaps a beam has rusted or buckled, or bolts have come loose, or concrete has cracked. This is traditionally tested by placing accelerometers in the structure, but it might be much cheaper if you could do it with a camera.
- aylons 10y agoIt's kind of the last option. This kind of analysis is already done and important, but usually it takes installing a lot of sensors and giving an impulse to the structure. I have actually worked on a company that did this, using strain gauges. The fact that this is one using a commercial camera is kind of impressive. I'll take a look on the papers someday, but the obvious way to do this would be to have a framerate at least the double of the frequencies being measured. Being able to determine frequencies much higher than sample rate not only is non-trivial, but also alleviates immensely the cost of these measurements.
- abedavis 10y agoA big part of it is that measuring the vibrations of large structures, like buildings or bridges, can be inconvenient and expensive. There are limited cases where you can measure new stuff with video simply because you don't have to touch the object or shine a laser at it, but the big win is that by making vibration measurement cheap and simple/convenient, it means you can do a lot more of it. We can start taking more measurements on more structures, which will tell us where we should put resources to rebuild infrastructure. It might also make quality assurance/ product testing cheaper but that's less save-the-world :-p
- dspoka 10y agoWhen I first saw this (extracting audio from visual vibrations): http://news.mit.edu/2014/algorithm-recovers-speech-from-vibrations-0804 http://news.mit.edu/2014/algorithm-recovers-speech-from-vibr... it seemed incredible that the signal the cameras could capture was strong enough. This work is even cooler, even more non obvious and seems closer to wizardry.
- deleted 10y ago[deleted]
- radarsat1 10y agoDedicated host name for a single algorithm, TED talk, high-production value video with professional animations, narrator talking excitedly, more kitschy examples of applications than description of the algorithm itself. Yup, MIT.
- erdevs 10y agoAh, come on, it's pretty cool. :) And they do have a publications page: http://www.interactivedynamicvideo.com/publications.html http://www.interactivedynamicvideo.com/publications.html I'm all for science being promoted/marketed, even if it's in a(n endearingly) dorky way. And there are some fun use cases for this to boot (see the Pokemon GO / AR vid below).
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- WhitneyLand 10y agoJazzing up science seems to have more upside than not, as long as the quality and substance are maintained.
- mynewtb 10y agoI recently got access to some hailed MIT technology with its own shiny videos, professional photos, patent pending and slick copy. It turned out to be a barely working prototype built on standard technology. To me misleading the public is not the worst about it. Pretending scientific accomplishments can deter and demotivate other, 'competing' scientists in the field.
- icebraining 10y agoWhich tech was it?
- deleted 10y ago
- two2two 10y agoThis would be fantastic for getting a feel for the tactility of a product you've not yet seen in person.
- spyder 10y agoPokemon GO and Interactive Dynamic Video: https://www.youtube.com/watch?v=9f1fCCb3hVg https://www.youtube.com/watch?v=9f1fCCb3hVg
- palakchokshi 10y agoThat's an awesome "real world" example of where this could be used.
- erdevs 10y agoThanks for this link. Really cool use case, and maybe Niantic will check it out, if they haven't already.
- RIMR 10y agoYes, my battery doesn't drain nearly fast enough when I'm playing. This would definitely fix that.
- erdevs 10y agoFunny comment. :) But actually this is unlikely to make a big difference on battery drain. Pokemon GO appears to be built on Unity and Unity is pretty notorious for having poor battery management. Whether you're pegging CPU or not, Unity burns lots of power. So, the incremental battery life hit is likely to be low in doing image processing on top of a frame.
- vanderZwan 10y agoSeriously, fuck Unity. It's the indy game equivalent of vendor-lock in, with everything negative about it. They can't be arsed to fix a simple VSync bug on Linux, meaning even simple sprite games, or even more insulting: TIS-100, a game about optimising assembly on fake retro-hardware, deliberately made to look like a terminal, always burns 25% CPU power on my laptop. More importantly, as Paolo Pedercini[0] mentioned on Twitter recently: > I can open my Flash crap from 15 years ago no problem, but a project I made last year in Unity 5.1 crashes silently if rebuilt in Unity 5.3 I'm telling you: today's indie game production is a Unity monoculture that will disappear at the next hardware architecture cycle [0] https://twitter.com/molleindustria/status/760904169444274176 https://twitter.com/molleindustria/status/760904169444274176
- johnhenry 10y agoI would love to see what this could do when combined with light field technology https://www.youtube.com/watch?v=xNJZHFZEkYQ https://www.youtube.com/watch?v=xNJZHFZEkYQ
- a2tech 10y agoThats the next best thing to magic that I've ever seen
- jcoffland 10y agoI wonder of you could use this on car commercials and thereby deduce the actual comparative quality of construction. Assuming that better built cars would vibrate less at speed. It would be interesting to understand where the vibrations occur. A question for the authors. All the video is with vibrating but otherwise stationary objects. Must the object be stationary for these tools to work?
- infinite8s 10y agoNot the author, but I surmise it would be much more difficult to assign vibration modes to subparts of the image in the image without being able to track them across frames.
- yorwba 10y agoNot one of the authors, but I had a look at their paper http://www.interactivedynamicvideo.com/ISMB_Davis_2015.pdf http://www.interactivedynamicvideo.com/ISMB_Davis_2015.pdf They use optical flow to get displacement values for each pixel. This works well when you don't have very large movements between frames, no sudden illumination changes and no rotations. Car commercials will probably have all of those and on top of that be highly edited (e.g. you have to deal with cuts in the video) You could try to use some other approach for motion estimation, like identifying and tracking salient points, but that would require quite a lot of work on top of what this demo shows.
- initram 10y agoProbably not because in a lot of car videos there's no real car. Here's one simple example: https://www.youtube.com/watch?v=b7vTM4_rjhs https://www.youtube.com/watch?v=b7vTM4_rjhs There's a company that makes a rig that's basically just an engine and 4 wheels. You film your entire commercial using it and they replace it with computer imagery of your car in post. I can't find the video at the moment, though.
- spoiler 10y agoThis does look fascinating. The video is a bit showbusiness-like (for a lack of better expression). A few questions if the author is on HN: I wonder how long it takes to analyse a 5 second video? Also, it seems the algorithm only works with static images after the initial video analysis, or am I wrong? Also, how long does it take to render the "virtual" state of the object?
- abedavis 10y agoHey, first author here. Good questions. The simulation runs in realtime, and is interactive. Generating the simulation can take a while though. The 5 second video took about a minute to process on my laptop. But longer videos, and higher-resolution videos, can take much longer. For instance, we've used the technique on high speed video to recover audible vibration modes, and this can take hours to process because the video itself is so big. I wrote the processing code in Matlab, and the simulation in c++ and GLSL. The simulation is pretty well engineered because I wanted it to run in realtime. I didn't put a lot of time or effort into optimizing the video analysis part though, so it probably could be made faster.
- spoiler 10y agoHi! Thanks for responding to my questions! The whole project is amazing and I hope you keep working on this (and future projects) with the same enthusiasm as you do now! :D
- jefe_ 10y agoThe consumer use cases are interesting but the propagandist use cases are terrifying. Along with this (Real-time Face Capture and Reenactment): https://www.youtube.com/watch?v=ohmajJTcpNk https://www.youtube.com/watch?v=ohmajJTcpNk
- olewhalehunter 10y agoWhich country (or organization) do you think will be (or has already been) the first to implement a program of creating physical mimics of humans for military and intelligence purposes? With technologies like this and CRISPR, video, testimony, or genetic evidence could be thrown out the window in criminal or military cases.
- iamcreasy 10y agoSo if we can use this algorithm for Structural Health Monitoring (like a bridge shown in the video) - what's the major obstacle to replace all destructive testing in mechanical engineering with this kind of algorithm?
- ARothfusz 10y agoThis might make the VRML included in MPEG4 useful. Could possibly use it to pass along the physical constraints and other data needed to make it interactive. ref: https://en.wikipedia.org/wiki/MPEG-4_Part_11 https://en.wikipedia.org/wiki/MPEG-4_Part_11