4 ms·

Paper: https://arxiv.org/abs/1909.05483 https://arxiv.org/abs/1909.05483 Website: http://sniklaus.com/kenburns http://sniklaus.com/kenburns

alphakappa 7y ago
The results look impressive. Is the code or a tool available that would enable others to try it out on their images?
- myth_buster 7y ago
  I may be wrong in thinking that given two researchers are working at Adobe, the plan probably is to keep it proprietary and put it in one of their software.
  - sniklaus 7y ago
    I am planning to release the code as well as the dataset but have yet to get approval to do so. If the work is done by an intern, then Adobe is usually fairly open in terms of sharing the work.
    - Wowfunhappy 7y ago
      Thank you, I'd really love to play with this!
    - myth_buster 7y ago
      Thanks for clarifying. Looking forward to the digging into codebase.
    - puranjay 7y ago
      I'd love to play around with it for a music video (non commercial)!
    - canada_dry 7y ago
      I'm doing a video slideshow for my niece's wedding in a couple weeks and would love to add this effect to a bunch of their photos (without spending a usual hours-and-hours in photoshop). Hope you're able to release it ASAP!
kissickas 7y ago
Love your personal website.
- sniklaus 7y ago
  Thank you for you kind words! I moved away from having separate websites for each project and am currently trying to bring all my projects to my personal website. I just added the Mini Chess game last night, which used to be an Android app.
  - sorenjan 7y ago
    Really impressive results. Is your "Context-aware Synthesis for Video Frame Interpolation" available as an implementation somewhere? It looks really good. I skimmed the paper about the Ken Burns effect, and if you don't mind I have some questions. I hope I didn't miss the answers in the paper itself, I'll be sure to read it more carefully when time permits. 1. The loss function for the depth estimation is Ldepth=0.0001·Lord+Lgrad. Is Lgrad much bigger than Lord by design or will this basically make Lord tiny and almost unecessary? How did you arrive at this number and not a magnitude bigger or smaller? 2. How are you rendering the point cloud, like tiny discs in free space? When I think of point clouds I think of the typical LIDAR output renderings, but your results are continuous images, or video frames. Are the points rendered to an image plane and the result then interpolated? And a couple more general ones: 1. Are all other depth estimation techniques other than NN obsolete now? Is there no point in estimating intrinsic camera parameters and epipolar lines when the state of the art seems to be to input one or more images into a NN and let it produce the depth for you? 2. How do you decide how the NN should look like, with various downsampling layers, convolutions, etc. I've seen people start with pretrained networks and retrain them, but how do you build a novel network based on the input and desired output?
    - sniklaus 7y ago
      I unfortunately did not get the approval to release the context-aware frame interpolation, but one of my older works is open source for research purposes: https://github.com/sniklaus/pytorch-sepconv https://github.com/sniklaus/pytorch-sepconv 1. I am not sure about the scale of the individual loss functions anymore, my apologies. I determined the combination of the two losses via a simple grid search and seeing what works best (plus / minus a magnitude did not make that much of a difference). 2. The points are just splatted to an image plane, more advanced point cloud rendering techniques would be better though. There is no video frame interpolation, each individual frame in the output video is a rendering of the point cloud from a different camera perspective. 3. I am not sure about multi-view stereo, COLMAP still seems like the state of the art for that. But neural networks definitely outperform classic techniques for single image depth estimation. 4. Common architectures just did not do as well as I was hoping for so I tried about 1500 model architectures. I started with an architecture that intuitively seemed right and then gradually explored / refined alterations of it. It ultimately was a lot of trial and error.
sniklaus 7y ago
I just got the daily email from my server with some metrics. It has experienced over 500 gigabytes of traffic in the last few hours and the video is only 3 megabytes in size. I definitely did not expect such a HN-effect.
- FreeHugs 7y ago
  The video is 180 MB. So 500 GB are about 2800 full video downloads.
  - sniklaus 7y ago
    Only very few people in comparison watched the full video. Here are the logs for the content server from yesterday 5 PM PST, covering the previous 24 hours (search for /kenburns/promo.mp4): https://pastebin.com/4zTxL2aw https://pastebin.com/4zTxL2aw
    - sniklaus 7y ago
      I see your point now. This post originally pointed to the teaser video and not the project website (I do not know who changed that, must have been an admin): http://content.sniklaus.com/kenburns/promo.mp4 http://content.sniklaus.com/kenburns/promo.mp4
    - FreeHugs 7y ago
      What does this line mean: /kenburns/promo.mp4 | GET | 175574 | 33.4183748571803 That the video was watched 175574 times? That would indeed be a lot!
      - sniklaus 7y ago
        That means that there were 175574 HTTP GET requests, but my guess is that if someone fast-forwards that that might trigger another request. So the actual number of views may be a little lower.
- nacs 7y ago
  Maybe upload the video to Youtube? The video is barely loading here.

alphakappa 7y ago

The results look impressive. Is the code or a tool available that would enable others to try it out on their images?

myth_buster 7y ago
I may be wrong in thinking that given two researchers are working at Adobe, the plan probably is to keep it proprietary and put it in one of their software.
- sniklaus 7y ago
  I am planning to release the code as well as the dataset but have yet to get approval to do so. If the work is done by an intern, then Adobe is usually fairly open in terms of sharing the work.
  - Wowfunhappy 7y ago
    Thank you, I'd really love to play with this!
  - myth_buster 7y ago
    Thanks for clarifying. Looking forward to the digging into codebase.
  - puranjay 7y ago
    I'd love to play around with it for a music video (non commercial)!
  - canada_dry 7y ago
    I'm doing a video slideshow for my niece's wedding in a couple weeks and would love to add this effect to a bunch of their photos (without spending a usual hours-and-hours in photoshop). Hope you're able to release it ASAP!

kissickas 7y ago

Love your personal website.

sniklaus 7y ago
Thank you for you kind words! I moved away from having separate websites for each project and am currently trying to bring all my projects to my personal website. I just added the Mini Chess game last night, which used to be an Android app.
- sorenjan 7y ago
  Really impressive results. Is your "Context-aware Synthesis for Video Frame Interpolation" available as an implementation somewhere? It looks really good. I skimmed the paper about the Ken Burns effect, and if you don't mind I have some questions. I hope I didn't miss the answers in the paper itself, I'll be sure to read it more carefully when time permits. 1. The loss function for the depth estimation is Ldepth=0.0001·Lord+Lgrad. Is Lgrad much bigger than Lord by design or will this basically make Lord tiny and almost unecessary? How did you arrive at this number and not a magnitude bigger or smaller? 2. How are you rendering the point cloud, like tiny discs in free space? When I think of point clouds I think of the typical LIDAR output renderings, but your results are continuous images, or video frames. Are the points rendered to an image plane and the result then interpolated? And a couple more general ones: 1. Are all other depth estimation techniques other than NN obsolete now? Is there no point in estimating intrinsic camera parameters and epipolar lines when the state of the art seems to be to input one or more images into a NN and let it produce the depth for you? 2. How do you decide how the NN should look like, with various downsampling layers, convolutions, etc. I've seen people start with pretrained networks and retrain them, but how do you build a novel network based on the input and desired output?
  - sniklaus 7y ago
    I unfortunately did not get the approval to release the context-aware frame interpolation, but one of my older works is open source for research purposes: https://github.com/sniklaus/pytorch-sepconv https://github.com/sniklaus/pytorch-sepconv 1. I am not sure about the scale of the individual loss functions anymore, my apologies. I determined the combination of the two losses via a simple grid search and seeing what works best (plus / minus a magnitude did not make that much of a difference). 2. The points are just splatted to an image plane, more advanced point cloud rendering techniques would be better though. There is no video frame interpolation, each individual frame in the output video is a rendering of the point cloud from a different camera perspective. 3. I am not sure about multi-view stereo, COLMAP still seems like the state of the art for that. But neural networks definitely outperform classic techniques for single image depth estimation. 4. Common architectures just did not do as well as I was hoping for so I tried about 1500 model architectures. I started with an architecture that intuitively seemed right and then gradually explored / refined alterations of it. It ultimately was a lot of trial and error.

sniklaus 7y ago

I just got the daily email from my server with some metrics. It has experienced over 500 gigabytes of traffic in the last few hours and the video is only 3 megabytes in size. I definitely did not expect such a HN-effect.

FreeHugs 7y ago
The video is 180 MB. So 500 GB are about 2800 full video downloads.
- sniklaus 7y ago
  Only very few people in comparison watched the full video. Here are the logs for the content server from yesterday 5 PM PST, covering the previous 24 hours (search for /kenburns/promo.mp4): https://pastebin.com/4zTxL2aw https://pastebin.com/4zTxL2aw
  - sniklaus 7y ago
    I see your point now. This post originally pointed to the teaser video and not the project website (I do not know who changed that, must have been an admin): http://content.sniklaus.com/kenburns/promo.mp4 http://content.sniklaus.com/kenburns/promo.mp4
  - FreeHugs 7y ago
    What does this line mean: /kenburns/promo.mp4 | GET | 175574 | 33.4183748571803 That the video was watched 175574 times? That would indeed be a lot!
    - sniklaus 7y ago
      That means that there were 175574 HTTP GET requests, but my guess is that if someone fast-forwards that that might trigger another request. So the actual number of views may be a little lower.
nacs 7y ago
Maybe upload the video to Youtube? The video is barely loading here.