6 ms·

I see a potential problem with moving pictures - the neural network might decide that while red was an appropriate color for the truck in the last frame, blue i

I see a potential problem with moving pictures - the neural network might decide that while red was an appropriate color for the truck in the last frame, blue is more likely in this frame.

xiphias 11y ago
You can use the previous few frames for fixing this problem for videos
- thomasahle 11y ago
  There might be a few scenes in between the viewings of the truck.
  - pgeorgi 11y ago
    I'd imagine that many people wouldn't even notice (unless the truck's color is part of the plot)
    - thomasahle 11y ago
      Maybe it's about a truck painting shop, and every time they paint something, it either doesn't change color, or get a color different from what the customer ordered. I would watch that.
      - jcoffland 11y ago
        You mean you would watch paint dry?
        paulmd 11y ago
        Actually, someone recently filmed exactly that - a 10-hour film of paint drying, as a protest against film censorship. The British Board of Film Classification awarded it a "U" rating: universal/suitable for all. http://www.telegraph.co.uk/film/movie-news/paint-drying-bbfc-certificate/ http://www.telegraph.co.uk/film/movie-news/paint-drying-bbfc... It would have been funny to splice in a couple frames of porn throughout, Fight Club-style.
    - caltrain 11y ago
      Truck's color can not be part of the plot, its a B&W movie!
      - pgeorgi 11y ago
        Protagonist: "Do you see that red truck?" light blue truck rolls through the scene
  - xiphias 11y ago
    You're right, it would be a fun side effect to see the color change between different scenes :) I guess there's not enough in commercial interest in fixing these problems, but it can be probably done with the current algorithms. Precise 3D reconstruction is much more important than colour reconstruction.
    - pionar 11y ago
      >it would be a fun side effect to see the color change between different scenes :) Now that's a horse of a different color[0]! [0] http://oz.wikia.com/wiki/Horse_of_a_Different_Color http://oz.wikia.com/wiki/Horse_of_a_Different_Color
    - m_mueller 11y ago
      Would it be possible to have some sort of semantic naming coupled to it 'the car in this scene is red', that could be fixed by humans, maybe even by talking to the NN? That's when this stuff starts getting fun IMO - human language together with superhuman domain knowledge.
      - ThePaco 11y ago
        Or even using the previously processed frames as reference the next time the shot appears
  - nxzero 11y ago
    Are there any other issues or advantages that you're able to see that would be unique to movies? While I'm unable to think of a use, seems like there's the potiental for side channel analyis via music, dialog, background-noises, scripts, etc.
    - xiphias 11y ago
      Script analysis would take much more effort probably, but it could work if the colour is in the movie script...in that case object detection gets important as well. Just using more training data to cover all the dog breeds and car colours would help a bit though.
WalterBright 11y ago
I suspect that if an artist selected colors for a few key frames, the computer could figure out the rest of the frames.
- pbhjpbhj 11y ago
  How would they know that the vehicles, say, were supposed to be the same colour. I'm thinking The Italian Job - the same model of cars are used and are distinguished by their colours. Also what about when different vehicles are painted by the production team to look the same (for stunts say), the computer could rightly recognise them as different vehicles - how would it then know that they're supposed to be the same. For such things it seems you'd need to check every shot change.
  - omtose 11y ago
    >the same model of cars are used and are distinguished by their colours Aren't we talking about a B&W film? In that case the colour (or its lack of) wouldn't communicate any information.
  - WalterBright 11y ago
    That TIJ cars were distinguishable only by color was made possible by filming in color. A BW movie would not make a film requiring distinguishing colors that filmed as equal shades of grey. In any case, any such colorizing system would be designed to accept a bit of guidance here and there from the artist. This is much like when an OCR'd document needs a bit of touch-up. And even if it wasn't perfect, many BW movies would be made much more watchable, like the 1927 Wings, which is crying out to be colorized (and have a soundtrack added).
    - m_mueller 11y ago
      Even though already perfectly enjoyable, I'm thinking Kurosawa films like Seven Samurai and Yojimbo. Man that would be something.
      - 11y ago
aab0 11y ago
One idea here is to adopt a recent generative approach: a CNN which starts with two noise image inputs, and then repeatedly tweaks it plus a new noise image over multiple inputs until it does one last tweak to a final version. The noise serves as a RNG for making choices to the built-up image, I think. You could apply this recurrent idea to movies too: for the first BW frame, pass in a noise image and the BW frame, get out a C frame; now for the second BW frame, pass in the BW frame but also the C frame from before. The CNN may gradually learn to transfer colors from the C frame to the BW frame, thereby maintaining temporal coherency. (Or you could just try to use a RNN directly and keep hidden state from frame to frame.)

I see a potential problem with moving pictures - the neural network might decide that while red was an appropriate color for the truck in the last frame, blue is more likely in this frame.

xiphias 11y ago

You can use the previous few frames for fixing this problem for videos

thomasahle 11y ago
There might be a few scenes in between the viewings of the truck.
- pgeorgi 11y ago
  I'd imagine that many people wouldn't even notice (unless the truck's color is part of the plot)
  - thomasahle 11y ago
    Maybe it's about a truck painting shop, and every time they paint something, it either doesn't change color, or get a color different from what the customer ordered. I would watch that.
    - jcoffland 11y ago
      You mean you would watch paint dry?
      - paulmd 11y ago
        Actually, someone recently filmed exactly that - a 10-hour film of paint drying, as a protest against film censorship. The British Board of Film Classification awarded it a "U" rating: universal/suitable for all. http://www.telegraph.co.uk/film/movie-news/paint-drying-bbfc-certificate/ http://www.telegraph.co.uk/film/movie-news/paint-drying-bbfc... It would have been funny to splice in a couple frames of porn throughout, Fight Club-style.
  - caltrain 11y ago
    Truck's color can not be part of the plot, its a B&W movie!
    - pgeorgi 11y ago
      Protagonist: "Do you see that red truck?" light blue truck rolls through the scene
- xiphias 11y ago
  You're right, it would be a fun side effect to see the color change between different scenes :) I guess there's not enough in commercial interest in fixing these problems, but it can be probably done with the current algorithms. Precise 3D reconstruction is much more important than colour reconstruction.
  - pionar 11y ago
    >it would be a fun side effect to see the color change between different scenes :) Now that's a horse of a different color[0]! [0] http://oz.wikia.com/wiki/Horse_of_a_Different_Color http://oz.wikia.com/wiki/Horse_of_a_Different_Color
  - m_mueller 11y ago
    Would it be possible to have some sort of semantic naming coupled to it 'the car in this scene is red', that could be fixed by humans, maybe even by talking to the NN? That's when this stuff starts getting fun IMO - human language together with superhuman domain knowledge.
    - ThePaco 11y ago
      Or even using the previously processed frames as reference the next time the shot appears
- nxzero 11y ago
  Are there any other issues or advantages that you're able to see that would be unique to movies? While I'm unable to think of a use, seems like there's the potiental for side channel analyis via music, dialog, background-noises, scripts, etc.
  - xiphias 11y ago
    Script analysis would take much more effort probably, but it could work if the colour is in the movie script...in that case object detection gets important as well. Just using more training data to cover all the dog breeds and car colours would help a bit though.

WalterBright 11y ago

I suspect that if an artist selected colors for a few key frames, the computer could figure out the rest of the frames.

pbhjpbhj 11y ago
How would they know that the vehicles, say, were supposed to be the same colour. I'm thinking The Italian Job - the same model of cars are used and are distinguished by their colours. Also what about when different vehicles are painted by the production team to look the same (for stunts say), the computer could rightly recognise them as different vehicles - how would it then know that they're supposed to be the same. For such things it seems you'd need to check every shot change.
- omtose 11y ago
  >the same model of cars are used and are distinguished by their colours Aren't we talking about a B&W film? In that case the colour (or its lack of) wouldn't communicate any information.
- WalterBright 11y ago
  That TIJ cars were distinguishable only by color was made possible by filming in color. A BW movie would not make a film requiring distinguishing colors that filmed as equal shades of grey. In any case, any such colorizing system would be designed to accept a bit of guidance here and there from the artist. This is much like when an OCR'd document needs a bit of touch-up. And even if it wasn't perfect, many BW movies would be made much more watchable, like the 1927 Wings, which is crying out to be colorized (and have a soundtrack added).
  - m_mueller 11y ago
    Even though already perfectly enjoyable, I'm thinking Kurosawa films like Seven Samurai and Yojimbo. Man that would be something.
    - 11y ago

aab0 11y ago

One idea here is to adopt a recent generative approach: a CNN which starts with two noise image inputs, and then repeatedly tweaks it plus a new noise image over multiple inputs until it does one last tweak to a final version. The noise serves as a RNG for making choices to the built-up image, I think. You could apply this recurrent idea to movies too: for the first BW frame, pass in a noise image and the BW frame, get out a C frame; now for the second BW frame, pass in the BW frame but also the C frame from before. The CNN may gradually learn to transfer colors from the C frame to the BW frame, thereby maintaining temporal coherency. (Or you could just try to use a RNN directly and keep hidden state from frame to frame.)