5 ms·

Self-Supervised Tracking via Video Colorization

129 points by ot 8y ago

cs702 8y ago
Very clever, and "obvious" only in hindsight: Training a deep convnet to colorize all frames in a grayscale video clip from a single color frame taken from the same clip induces the neural net to learn to track all objects in the video, with robustness to occlusions, change of viewing angles, etc. Labels are not required; only a color frame from each clip. Most impressively, the embeddings learned by the convnet (i.e., the representations learned by the next-to-last layer) are linearly separable by object. Very nice!
- Eridrus 8y ago
  Simple to explain, yes, but I feel like this isn't even really "obvious" even in hindsight. This whole thing is very clever. The only complaint I have is that it's not better than supervised object tracking, so I wonder if this idea is too late? To draw a parallel to image classification, at one point in time neural nets were trained with a bunch of unsupervised pre-training using reconstruction loss, but that technique has basically fallen by the wayside as we've gotten larger datasets and found a pile of tricks for training them from scratch.
  - cs702 8y ago
    Labeling object locations in all frames of a large number of video clips is significantly more expensive than labeling a comparably large number of images.
    - Eridrus 8y ago
      Sure, but like Imagenet, these datasets already exist. So unless these models are quite brittle to the objects being tracked, this is likely not going to be an issue.
      - vanderZwan 8y ago
        > Sure, but like Imagenet, these datasets already exist. I'm pretty sure that making it cheap to use new datasets is very valuable in the long run
- vanderZwan 8y ago
  Now I want to know if using two input videos to mimic our two eyes makes this even more robust, or even gives depths sense for free?
arnioxux 8y ago
I saw something similar before (from gifs.com's sticker editor of all things lol) where you annotate the segmentation of the first frame of the video and it will propagate that segmentation to the rest of the frames: https://medium.com/gifs-ai/interactive-segmentation-with-convolutional-neural-networks-2e171a85df82#fc74 https://medium.com/gifs-ai/interactive-segmentation-with-con...
AboutTheWhisles 8y ago
I'm very skeptical that there is any merit to the 'tracking' over other techniques, as well as the colorization being better than this 14 year old paper: http://webee.technion.ac.il/people/anat.levin/papers/colorization-siggraph04.pdf http://webee.technion.ac.il/people/anat.levin/papers/coloriz... The results in their videos look very poor.
- IanCal 8y ago
  Tracking is not the method they're using for colourizing, but the other way around. Your linked paper has no tracking.
  - AboutTheWhisles 8y ago
    I realize that. They are for some reason saying they can track things with their colorization, when their colorization is extremely unimpressive as well as the tracking that results from using it. There is no reason colorization needs to happen to do the tracking anyway. The tracking is unimpressive and now indirect. This isn't some sort of epiphany they've discovered, they are just reinventing video image segmentation very poorly. Here are half a dozen examples from a 30 second google search: https://www.youtube.com/watch?v=juDvLrFQF0U https://www.youtube.com/watch?v=juDvLrFQF0U https://www.youtube.com/watch?v=JYgyDdLf7GQ https://www.youtube.com/watch?v=JYgyDdLf7GQ https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36247.pdf https://static.googleusercontent.com/media/research.google.c... https://perso.liris.cnrs.fr/nicolas.bonneel/InteractiveMultilabelVideoSegmentation.pdf https://perso.liris.cnrs.fr/nicolas.bonneel/InteractiveMulti... http://files.is.tue.mpg.de/black/papers/TsaiCVPR2016.pdf http://files.is.tue.mpg.de/black/papers/TsaiCVPR2016.pdf https://graphics.ethz.ch/~perazzif/bvs/files/bvs.pdf https://graphics.ethz.ch/~perazzif/bvs/files/bvs.pdf The only reason this is news is because it's google and the researchers seem to think they've discovered something. Techniques like this with much better results have been shown at Siggraph for decades.
    - felippee 8y ago
      Right, I totally agree. What is more, I think this same result could actually be sold without the pompous deep learning bullshit and be received quite differently. If they did not claim to invent the wheel, but rather modestly noted their observation (which in a limited way is actually quite cool - that is from a known - at least on this forum - deep learning skeptic like me), it would make a much better impression. Same is true actually for many DL papers. They'd be actually cool, if they weren't oversold.

129 points by ot 8y ago

Very clever, and "obvious" only in hindsight: Training a deep convnet to colorize all frames in a grayscale video clip from a single color frame taken from the same clip induces the neural net to learn to track all objects in the video, with robustness to occlusions, change of viewing angles, etc. Labels are not required; only a color frame from each clip. Most impressively, the embeddings learned by the convnet (i.e., the representations learned by the next-to-last layer) are linearly separable by object. Very nice!

Eridrus 8y ago
Simple to explain, yes, but I feel like this isn't even really "obvious" even in hindsight. This whole thing is very clever. The only complaint I have is that it's not better than supervised object tracking, so I wonder if this idea is too late? To draw a parallel to image classification, at one point in time neural nets were trained with a bunch of unsupervised pre-training using reconstruction loss, but that technique has basically fallen by the wayside as we've gotten larger datasets and found a pile of tricks for training them from scratch.
- cs702 8y ago
  Labeling object locations in all frames of a large number of video clips is significantly more expensive than labeling a comparably large number of images.
  - Eridrus 8y ago
    Sure, but like Imagenet, these datasets already exist. So unless these models are quite brittle to the objects being tracked, this is likely not going to be an issue.
    - vanderZwan 8y ago
      > Sure, but like Imagenet, these datasets already exist. I'm pretty sure that making it cheap to use new datasets is very valuable in the long run
vanderZwan 8y ago
Now I want to know if using two input videos to mimic our two eyes makes this even more robust, or even gives depths sense for free?

arnioxux 8y ago

I saw something similar before (from gifs.com's sticker editor of all things lol) where you annotate the segmentation of the first frame of the video and it will propagate that segmentation to the rest of the frames: https://medium.com/gifs-ai/interactive-segmentation-with-convolutional-neural-networks-2e171a85df82#fc74 https://medium.com/gifs-ai/interactive-segmentation-with-con...

AboutTheWhisles 8y ago

I'm very skeptical that there is any merit to the 'tracking' over other techniques, as well as the colorization being better than this 14 year old paper: http://webee.technion.ac.il/people/anat.levin/papers/colorization-siggraph04.pdf http://webee.technion.ac.il/people/anat.levin/papers/coloriz... The results in their videos look very poor.

IanCal 8y ago
Tracking is not the method they're using for colourizing, but the other way around. Your linked paper has no tracking.
- AboutTheWhisles 8y ago
  I realize that. They are for some reason saying they can track things with their colorization, when their colorization is extremely unimpressive as well as the tracking that results from using it. There is no reason colorization needs to happen to do the tracking anyway. The tracking is unimpressive and now indirect. This isn't some sort of epiphany they've discovered, they are just reinventing video image segmentation very poorly. Here are half a dozen examples from a 30 second google search: https://www.youtube.com/watch?v=juDvLrFQF0U https://www.youtube.com/watch?v=juDvLrFQF0U https://www.youtube.com/watch?v=JYgyDdLf7GQ https://www.youtube.com/watch?v=JYgyDdLf7GQ https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36247.pdf https://static.googleusercontent.com/media/research.google.c... https://perso.liris.cnrs.fr/nicolas.bonneel/InteractiveMultilabelVideoSegmentation.pdf https://perso.liris.cnrs.fr/nicolas.bonneel/InteractiveMulti... http://files.is.tue.mpg.de/black/papers/TsaiCVPR2016.pdf http://files.is.tue.mpg.de/black/papers/TsaiCVPR2016.pdf https://graphics.ethz.ch/~perazzif/bvs/files/bvs.pdf https://graphics.ethz.ch/~perazzif/bvs/files/bvs.pdf The only reason this is news is because it's google and the researchers seem to think they've discovered something. Techniques like this with much better results have been shown at Siggraph for decades.
  - felippee 8y ago
    Right, I totally agree. What is more, I think this same result could actually be sold without the pompous deep learning bullshit and be received quite differently. If they did not claim to invent the wheel, but rather modestly noted their observation (which in a limited way is actually quite cool - that is from a known - at least on this forum - deep learning skeptic like me), it would make a much better impression. Same is true actually for many DL papers. They'd be actually cool, if they weren't oversold.