4 ms·

Total Relighting: Learning to Relight Portraits for Background Replacement

67 points by Hard_Space 5y ago

dheera 5y ago
Is there code for this? There seems to be only a website in the repo: https://github.com/augmentedperception/total_relighting https://github.com/augmentedperception/total_relighting
- ipsum2 5y ago
  Don't expect code from this group, because their work usually goes into Google Pixel camera software.
  - sorenjan 5y ago
    What is the legal situation for implementing things from these kinds of papers? Am I allowed to use other's research papers to make features in my own products, or should I assume it's patented? I don't think copyright applies if I write the code myself, right? Or are neural net models copyrightable? In this case you need specialized data from a lighting rig, but what about in general when you can use your own or open data to do the training, or in the cases when no machine learning is used?
    - dheera 5y ago
      Also of note -- building that lighting rig is actually not that hard if you have a spare walk-in closet and a bunch of WS2812 lights. You can programmatically turn on and off combinations of lights and automate the photography. But you'll probably want several subjects of multiple races and genders if you want it to work reliably though. There's also this free dataset, although I haven't dug into it to see how good it is: https://drive.google.com/drive/folders/10luekF8vV5vo2GFYPRCe9Rm2Xy2DwHkT https://drive.google.com/drive/folders/10luekF8vV5vo2GFYPRCe... ( and the associated paper: https://zhhoper.github.io/dpr.html https://zhhoper.github.io/dpr.html ) Are there any decent collections of human portrait Blender scenes? If so lighting would be easy to change in a pinch, and programmatically create a simulated dataset. A collection of images of this quality would be awesome: https://blenderartists.org/t/felicia-blend/701475 https://blenderartists.org/t/felicia-blend/701475
    - ipsum2 5y ago
      It's probably patented (patent pending, because they take forever to get approved), copyright doesn't apply if you write the code and train the model yourself afaik
nakedgremlin 5y ago
Incredible work. Coming from a POV of a person who has to wrangle a lot of VFX heavy creative, it's great to see the technical walkthroughs like this, especially the workflow overviews to see how it matches with our render process.
tqi 5y ago
Interesting - as I understand it, one of the big tech innovations from Disney's The Mandalorian was LED projected virtual sets that solved the lighting issues from green screens (see ~2:10 mark of this video: https://www.youtube.com/watch?v=Ufp8weYYDE8 https://www.youtube.com/watch?v=Ufp8weYYDE8), making VFX scenes cheaper and faster to produce in exchange for higher up front costs. It had the added benefit of giving actors more to work off of. I wonder if this technique will swing the cost pendulum back in favor of traditional green screens?

67 points by Hard_Space 5y ago

Is there code for this? There seems to be only a website in the repo: https://github.com/augmentedperception/total_relighting https://github.com/augmentedperception/total_relighting

ipsum2 5y ago
Don't expect code from this group, because their work usually goes into Google Pixel camera software.
- sorenjan 5y ago
  What is the legal situation for implementing things from these kinds of papers? Am I allowed to use other's research papers to make features in my own products, or should I assume it's patented? I don't think copyright applies if I write the code myself, right? Or are neural net models copyrightable? In this case you need specialized data from a lighting rig, but what about in general when you can use your own or open data to do the training, or in the cases when no machine learning is used?
  - dheera 5y ago
    Also of note -- building that lighting rig is actually not that hard if you have a spare walk-in closet and a bunch of WS2812 lights. You can programmatically turn on and off combinations of lights and automate the photography. But you'll probably want several subjects of multiple races and genders if you want it to work reliably though. There's also this free dataset, although I haven't dug into it to see how good it is: https://drive.google.com/drive/folders/10luekF8vV5vo2GFYPRCe9Rm2Xy2DwHkT https://drive.google.com/drive/folders/10luekF8vV5vo2GFYPRCe... ( and the associated paper: https://zhhoper.github.io/dpr.html https://zhhoper.github.io/dpr.html ) Are there any decent collections of human portrait Blender scenes? If so lighting would be easy to change in a pinch, and programmatically create a simulated dataset. A collection of images of this quality would be awesome: https://blenderartists.org/t/felicia-blend/701475 https://blenderartists.org/t/felicia-blend/701475
  - ipsum2 5y ago
    It's probably patented (patent pending, because they take forever to get approved), copyright doesn't apply if you write the code and train the model yourself afaik

nakedgremlin 5y ago

Incredible work. Coming from a POV of a person who has to wrangle a lot of VFX heavy creative, it's great to see the technical walkthroughs like this, especially the workflow overviews to see how it matches with our render process.

tqi 5y ago

Interesting - as I understand it, one of the big tech innovations from Disney's The Mandalorian was LED projected virtual sets that solved the lighting issues from green screens (see ~2:10 mark of this video: https://www.youtube.com/watch?v=Ufp8weYYDE8 https://www.youtube.com/watch?v=Ufp8weYYDE8), making VFX scenes cheaper and faster to produce in exchange for higher up front costs. It had the added benefit of giving actors more to work off of. I wonder if this technique will swing the cost pendulum back in favor of traditional green screens?