4 ms·

Reminds me of when I tried to extract G-buffers from my Unity High Definition Rendering Pipeline test project: https://www.youtube.com/watch?v=Fwtc694qNUM https

Reminds me of when I tried to extract G-buffers from my Unity High Definition Rendering Pipeline test project: https://www.youtube.com/watch?v=Fwtc694qNUM https://www.youtube.com/watch?v=Fwtc694qNUM I'm not sure if this paper is really proving anything though. There's a giant-ass UNET Lora model that's being trained here, so is it really "extracting" something from an existing model, or simply creating a new model that can create channels that look like something you'd get out of a deferred rendering pipeline. After all, taking normals, albedo, and depth and combining them (deferred rendering) is just one of several techniques to create a 3d scene. Wasn't even used in videogames until the early 2000s (in a Shrek videogame for the Xbox! (https://sites.google.com/site/richgel99/the-early-history-of-deferred-shading-and-lighting https://sites.google.com/site/richgel99/the-early-history-of...) What would really be awesome is to get a LORA model that can extract the "camera" rotation and translation matrix for these image generation models. That would really demonstrate something (and be quite useful at the same time).

joefourier 3y ago
If you look at the supplementary material, they do a test where they train the Lora with a randomly initialised Unet and it is largely incapable of extracting any surface normals as opposed to using the pretrained Stable Diffusion Unet - clearly showing the features of the model are relevant to its performance.
sfink 3y ago
I don't really know what I'm talking about, but doesn't this address that? > with newly learned parameters that make up less than 0.6% of the total parameters in the generative model 0.6% sounds like a small number. Is it measuring the right thing? Certainly, I wouldn't expect the model to necessarily be encoding exactly the set of things that they're extracting, but it still seems very significant to me even if it is "just" encoding some set of things that can be cheaply (in terms of model size) and reliably mapped to normals, albedo, and depth. (I don't care what basis vectors it's using, as long as I know how to map them to mine.)
- LarsDu88 3y ago
  0.6 percent of a model with 890 million parameters is still 5.34 million parameters. That's still pretty big. Maybe big enough to fake normals, learn some albedo smoothing functions, and learn a depth estimator perhaps??

joefourier 3y ago

If you look at the supplementary material, they do a test where they train the Lora with a randomly initialised Unet and it is largely incapable of extracting any surface normals as opposed to using the pretrained Stable Diffusion Unet - clearly showing the features of the model are relevant to its performance.

sfink 3y ago

I don't really know what I'm talking about, but doesn't this address that? > with newly learned parameters that make up less than 0.6% of the total parameters in the generative model 0.6% sounds like a small number. Is it measuring the right thing? Certainly, I wouldn't expect the model to necessarily be encoding exactly the set of things that they're extracting, but it still seems very significant to me even if it is "just" encoding some set of things that can be cheaply (in terms of model size) and reliably mapped to normals, albedo, and depth. (I don't care what basis vectors it's using, as long as I know how to map them to mine.)

LarsDu88 3y ago
0.6 percent of a model with 890 million parameters is still 5.34 million parameters. That's still pretty big. Maybe big enough to fake normals, learn some albedo smoothing functions, and learn a depth estimator perhaps??