5 ms·

So I wonder if unusual angles that normally do not get photographed will be distorted? For example, underneath a table looking up.

jacobr1 4y ago
They reapply noise to the potentially distorted image and then predict the de-noised version like the originally rendered first frame. So the image is at least internally consistent for the frame (to the extend the the system generates consistency whatsoever). The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.
yarg 4y ago
It'll be delusions and guesses, rather than distortions. It'll just make up some colours and geometries that don't contradict anything it already knows from the defined perspectives. Or leave it empty.
poolio 4y ago
Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).
- salawat 4y ago
  >cat wearing sunglasses, back view") Bad prompt, missing implied antecedent/ambiguous subject... You may want: Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
  - thrown_22 4y ago
    >Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either. In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this. Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
    - salawat 4y ago
      I mean... It kinda did. Sarcasm aside. The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly... Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard. I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
  - bmpoole 4y ago
    yep our fixed strategy for view-dependent prompting is silly and there is tons of room for improvement!
LordDragonfang 4y ago
Some of the "mesh exports" used as examples on the page actually show this, to some extent. Look specifically at the plush corgi's belly and the weird geometry on the underside of the lemur's book, and to a lesser extent the underside of the bedsheet-ghost.
anentropic 4y ago
You can see that in some of these examples e.g. "plush toy of a corgi nurse"

jacobr1 4y ago

They reapply noise to the potentially distorted image and then predict the de-noised version like the originally rendered first frame. So the image is at least internally consistent for the frame (to the extend the the system generates consistency whatsoever). The example with a squirrel wearing a hoodie demonstrates an interesting edge case, the "front" of the squirrel (with hoodie over the head) show a normal hooded face as expected, but when you rotate to the "back" you get another face where the hoodie is low over the eyes. Each looks fine in isolation, but in aggregate it seems like we have a two-faced squirrel.

yarg 4y ago

It'll be delusions and guesses, rather than distortions. It'll just make up some colours and geometries that don't contradict anything it already knows from the defined perspectives. Or leave it empty.

poolio 4y ago

Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).

salawat 4y ago
>cat wearing sunglasses, back view") Bad prompt, missing implied antecedent/ambiguous subject... You may want: Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.
- thrown_22 4y ago
  >Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either. In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this. Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.
  - salawat 4y ago
    I mean... It kinda did. Sarcasm aside. The trouble is all those darn uninitiated and trying to create a generalized oracle to map their inspecific ramblings to what they mean to free them of having to actually communicate properly... Actually, funnily enough, this has cross section with philosophy in a way most programmers scoff at; but communication is frigging hard, and worse, detecting when someone is trying to get something across, but just needs a nudge in the right direction to be able to find the language to explain it is really damn hard. I run into it every time I get a haircut. I have no idea how to speak their language, so it's always "Uh... A little off the top and rounded at the back, I guess?"
- bmpoole 4y ago
  yep our fixed strategy for view-dependent prompting is silly and there is tons of room for improvement!

LordDragonfang 4y ago

Some of the "mesh exports" used as examples on the page actually show this, to some extent. Look specifically at the plush corgi's belly and the weird geometry on the underside of the lemur's book, and to a lesser extent the underside of the bedsheet-ghost.

anentropic 4y ago

You can see that in some of these examples e.g. "plush toy of a corgi nurse"