10 ms·

Zero-1-to-3: Zero-shot One Image to 3D Object

530 points by GaggiX 4y ago

HopenHeyHi 4y ago
3D reconstruction from a single image. They stress the examples are not curated, appears to... well, gosh darnit, it appears to work. If it runs fast enough I wonder whether one could just drive around with a webcam and generate these 3d models on the fly and even import them into a sort of GTA type simulation/game engine in real time. (To generate a novel view, Zero-1-to-3 takes only 2 seconds on an RTX A6000 GPU) This research is based on work partially supported by: - Toyota Research Institute - DARPA MCS program under Federal Agreement No. N660011924032 - NSF NRI Award #1925157 Oh, huh. Interesting. Future Work From objects to scenes: Generalization to scenes with complex backgrounds remains an important challenge for our method. From scenes to videos: Being able to reason about geometry of dynamic scenes from a single view would open novel research directions -- such as understanding occlusions and dynamic object manipulation. A few approaches for diffusion-based video generation have been proposed recently and extending them to 3D would be key to opening up these opportunities.
- TylerE 4y ago
  Seems like there is a bit of a gap between “runs at 0.5 fps on a $7000 workstation-grade GPU with 48GB of VRAM” and consumer applications. With the fairly shallow slope of the GPU performance curve overtime, I don’t see them just Moores Lawing out of it either. this would need two, maybe three orders of magnitude more performance.
  - jiggawatts 4y ago
    Computer power goes up exponentially thanks to Moore's law. Sprinkle some software optimisations on top, and it's conceivable for that to be running at interactive framerates on consumer GPUs within 5-10 years.
    - ffitch 4y ago
      the processing may as well shift to the clouds. With the subscription fee, of course : )
      - TylerE 4y ago
        Until we break the speed of light, I’m very bearish on cloud gaming. It just feels so bad. You’ve got like 9 layers of latency between you and the screen.
        fooker 4y ago
        You don’t have to break the speed of light, just have the ping below human perception. ~20ms is that threshold, but even 40ms latency is barely noticeable for single player games.
        TylerE 4y ago
        A wireless controller ALONE is already over 20ms, and that’s before you touch the network, actually doing with that input, wait for the display to redraw… At a 20ms total round trip, that only buys you about a 1500 mile radius, again completely ignoring all other latencies.
        enlyth 4y ago
        It's quite noticeable actually, and it adds up, it's not just an extra 20ms. For casual gamers and turn based games maybe it could work, as a niche. For FPS, multiplayer, ARPG, and so on, it's a dealbreaker, anything over 100ms feels too sluggish. We should be happy we have so much autonomy with our own hardware, I don't want some big cloud company to be able to tell me what I can play and render, unless we want the "you will own nothing and be happy" meme to become reality.
        TylerE 4y ago
        I actually, in my testing, JRPG/other turn based games were amongst the worst because there is so much “management” (inventory, loot, gear, etc) and the extra lag really throws you off
  - HopenHeyHi 4y ago
    Of course there is a gap. This is at the exploratory proof of concept stage. The fact that it works at all is what is interesting. Furthermore once you've identified the make and model of the car, its relative position in 3d, any anomalies -- that ain't just a Ford pickup, it is loaded with cargo that overhangs in a particular way -- its velocity, 'etc -- I'm quite sure that extrapolating additional information from the subsequent frames will be significantly cheaper as you don't have to generate a 3d model from scratch each time. I think this is a viable exploratory path forward. Make it work <- you are here Make it work correctly Make it work fast Edit: Scotty does know ;)
    - scotty79 4y ago
      I prefer: Make it work <- you are here Make it work correctly Make it work fast
      - deleted 4y ago
        [deleted]
  - frozenport 4y ago
    >> fairly shallow slope of the GPU performance curve overtime Not true.
King-Aaron 4y ago
That's honestly extremely impressive. I do hope that the 'in the wild' examples aren't completely curated and are actually being rendered on the fly (They appear to be, but it's hard for me to tell if that' truly the case). Pretty cool to see however.
- GaggiX 4y ago
  >and are actually being rendered on the fly They are precomputed, "Note that the demo allows a limited selection of rotation angles quantized by 30 degrees due to limited storage space of the hosting server." but I don't think they are curated, the seeds probably correspond to the seeds of the live demo you can host (they released the code and the models)
- deleted 4y ago
  [deleted]
mitthrowaway2 4y ago
> We compare our reconstruction with state-of-the-art models in single-view 3D reconstruction. Here they list "GT Mesh", "Ours", "Point-E", and "MCC". Does anyone know what technique "GT mesh" refers to? Is it simply the original mesh that generated the source image?
- haykmartiros 4y ago
  Ground truth
  - EGreg 4y ago
    Well honestly the "Ground truth" algorithm seems a lot superior to their method, it has higher fidelity in ALL the examples
    - sophiebits 4y ago
      “Ground truth” doesn’t refer to a particular algorithm; it refers to the ideal benchmark of what a perfect performance would look like, which they’re grading against.
    - simlevesque 4y ago
      Ground truth means that a human person created the model.
      - DarthNebo 4y ago
        Not necessarily, could also be synthetic. Google did the same for hand poses in BlazePalm
    - Thorrez 4y ago
      Ground truth means the original model that the image was generated from.
    - razemio 4y ago
      Haha, I am sorry. I spit my coffee reading this. It is ofc totally OK to not know what ground truth means but the irony was to funny. Yes ground truth will always be superior compared to anything else :)!
echelon 4y ago
Super cool results. This is what my startup is getting into. So I'm very interested. These aren't "game ready" - the sculpts are pretty gross. But we're clearly getting somewhere. It's only going to keep getting better. I expect we'll be building all new kinds of game engines, render pipelines, and 3D animation tools shortly.
- nico 4y ago
  And 3D printing. So quickly building physical tools too.
  - skybrian 4y ago
    For printing parts, precision matters since they likely need to fit with something else. You’ll want to be able to edit dimensions on the model to get the fit right. So maybe someday, but I think it would have to be a project that targets CAD.
- bredren 4y ago
  How do these kinds of tools complement actual 3d scanning? For example, Apple supposedly has put some time into 3d asset building (presumably in support of AR world building content). Can these inference techniques stack or otherwise help more detailed object data collection?
- redox99 4y ago
  While this is cool, this is not meant to target "game ready". For games and CGI, there's no reason to limit yourself to a single image. Photogrammetry is already extensively used, and it involves using tens or hundreds of images of the object to scan. Using many images as an input will obviously always be superior to a single one, as a single image means it has to literally make up the back side, and it has no parallax information.
  - oefrha 4y ago
    You appear to be thinking about scanning a physical object, whereas zero-shot one image to 3D object would be vastly more useful with a single (possibly AI-generated or AI-assisted) illustration. You get a 3D model in seconds at essentially zero cost, can iterate hundreds of times in a single day.
yawnxyz 4y ago
Are there any models that take an image to SVG?
nico 4y ago
This is feeling like almost thought to launch. In the last week, a lot of the ideas I’ve read about in the comments of HN, have then shown up as full blown projects in the front page. As if people are building at an insane speed from idea to launch/release.
- nmfisher 4y ago
  Just yesterday I was literally musing to myself "I wonder if NeRFs would help with 3D object synthesis", and here we are. It's definitely a fun time to be involved.
  - regegrt 4y ago
    It's not based on the NeRF concept though, is it? Its outputs can provide the inputs for NeRF training, which is why they mention NeRFs. But it's not NeRF technology.
    - deleted 4y ago
      [deleted]
  - noduerme 4y ago
    it's actually a really fun time to know how to sculpt in ZBrush and print out models.
    - nmfisher 4y ago
      If I had any artistic talent whatsoever, I'd probably agree with you!
      - noduerme 4y ago
        I won't lie... ZBrush is brutally hard. I got a subscription for work and only used it for one paid job, ever. But it's super satisfying if you just want to spend Sunday night making a clay elephant or rhinoceros, and drop $20 to have the file printed out and shipped to you by Thursday. I've fed lots of my sculpture renderings to Dali and gotten some pretty cool 2D results... but nothing nearly as cool as the little asymmetrical epoxy sculptures I can line up on the bookshelf...
gs17 4y ago
For anyone else who tried to download the weights and got Google Drive throwing a quota error at you, they're working on it: https://github.com/cvlab-columbia/zero123/issues/2 https://github.com/cvlab-columbia/zero123/issues/2
desmond373 4y ago
Would it be possible to generate cad files with this. As a base for part construction this could be game changing
- gs17 4y ago
  If you look at the example meshes, it doesn't seem very likely that it would be better than manually creating them, unless you're okay with lumpy parts that aren't exactly the right size. This is too early for it to not require a lot of cleanup to be usable.
  - flangola7 4y ago
    In other words we just need to wait 6 more months
brokensegue 4y ago
how is this different from the previous NeRF work? does it build a 3D model?
- GaggiX 4y ago
  NeRF models are trained on several views with known location and viewing direction. This model takes one image (and you don't need to train a model for each object).
  - fooker 4y ago
    Not hotdog.
  - amelius 4y ago
    But if it takes only one image, isn't it likely to hallucinate information?
    - gs17 4y ago
      Not just likely, it does. Try out the demo and see, e.g. what the backside of their Pikachu toy looks like. Or a little simpler, the paper has an example (the demo also has this) of the back of a car under different seeds.
ilaksh 4y ago
I wonder if this type of thing could be adapted to a vision system for a robot? So it would locate the camera and reconstruct an entire scene from a series of images as the robot moves around. Probably needs a ways to get there but to be able to do robust SLAM etc. With just a single camera would make things much less expensive.
- guyomes 4y ago
  You might be interested in this related recent work [1] that fits simple ellipsoids to images and then use them for the pose estimation of a camera. [1]: https://ieeexplore.ieee.org/document/9127873 https://ieeexplore.ieee.org/document/9127873
  - lefrancais 4y ago
    Same ref [1], but open : [https://hal.science/hal-02886633/document https://hal.science/hal-02886633/document]
ar9av 4y ago
It's hard to tell for certain from the paper, without going deep into the code, but it seems they created the new model the same way the depth conditioned SD models were made i.e. normal finetune. It might be possible to create a "original view + new angle" conditioned model much more easily by taking the Controlnet/T2IAdapter/GLIDE route where you freeze the original model. Text to 3d seems almost close to being solved. It also makes me think a "original character image + new pose" conditioned model would also work quite well.
bmitc 4y ago
What if you give it a picture of a cardboard cutout or billboard?
- noduerme 4y ago
  it'll build Angelyne for you, to distract your pathetic carbon-based intelligence. https://www.hollywoodreporter.com/wp-content/uploads/2017/07/zumaglobal327406_-_embed_2017.jpg?w=928 https://www.hollywoodreporter.com/wp-content/uploads/2017/07...
deleted 4y ago
[deleted]
noduerme 4y ago
Is there some kind of symmetry at work here in the deductive process?
- xotom20390 4y ago
  [dead]
jonplackett 4y ago
Would this be useful for a robot / car trying to navigate to be able to do this?
- eternalban 4y ago
  Great idea. Processing latency may be an issue. It has to be fast, small, and energy efficient.
- elif 4y ago
  unlikely. the front bumper of a car you are following has zero value for your ego's safety. most of the optimization of FSD is in removing extra data to improve latency of the mapping loop.
  - jonplackett 4y ago
    But it seems like the main problem for self drive is accurately understanding the world around the car. The actual driving is pretty easy. Being able to see a partial object and understand what the rest of it is is very useful to human drivers
throwaway4aday 4y ago
If you can produce any view angle you want of an object then can't you use photogrammetry to construct a 3D object?
- nwoli 4y ago
  See the “Single-View 3D Reconstruction” section at the bottom where they do precisely that
  - throwaway4aday 4y ago
    Cool, I missed that.
hypertexthero 4y ago
Brings to mind the Blade Runner enhance scene: https://www.youtube.com/watch?v=hHwjceFcF2Q https://www.youtube.com/watch?v=hHwjceFcF2Q
- BiteCode_dev 4y ago
  Given the data is (credible and beautiful) BS, I think it's closer to red dwarf: https://www.youtube.com/watch?v=6i3NWKbBaaU https://www.youtube.com/watch?v=6i3NWKbBaaU
- Sakos 4y ago
  Reminds me of this at the time fantastical scene in Enemy of the State https://youtu.be/3EwZQddc3kY https://youtu.be/3EwZQddc3kY
mov 4y ago
People plugging it as output of Midjourney in 3, 2, 1...
qikInNdOutReply 4y ago
What happens, if you build a circle? As in this creates a 3d object from a image and another ai creates a image from a 3d object? https://www.youtube.com/watch?v=zPqJUrfKuqs https://www.youtube.com/watch?v=zPqJUrfKuqs Does it stabilize, or refine prejudices, or go on a fractal journey of errors over the weight landscape?
hiccuphippo 4y ago
Can you obtain the 3d object from this or only an image with the new perspective? This could revolutionize indie gamedev.
- jxf 4y ago
  You can obtain a 3D object, but it's more useful for the novel views than the object, because the object isn't very good and probably needs some processing. See the bottom of the paper.
wslh 4y ago
I keep thinking in my project where we are taking multiple photos from the same angle with moving lights for rebuilding the 3D model. We are not using AI, just optic research like in [1]. We applied that on art at [2]. [1] Methods for 3D digitization of Cultural Heritage: http://www.ipet.gr/~akoutsou/docs/M3DD.pdf http://www.ipet.gr/~akoutsou/docs/M3DD.pdf [2] https://sublim.art https://sublim.art
- bogwog 4y ago
  So the business model there is: scanner + paper shredder + NFT = $$$? How many people have taken you up on that offer? Unless it's a shitty/low-effort painting, it seems insane to me that anyone would destroy their artwork in exchange for an NFT of that same artwork.
  - wslh 4y ago
    What is insane for you could be completely different for others: we have been in the last Miami Art Week and Art Basel and we don't have enough time for the number of artists that wanted to be in the process. Will expand more later (working now) but you can see AP coverage here [1]. It is also important to highlight that we are doing this project at our own risk, with our own money, have built the hardware and software, and not charging artists for the process. Just the primary market sell is split between 85% for artists and the rest for the project. Pretty generous in this risky market. [1] https://youtu.be/ajDEHSLi0iE https://youtu.be/ajDEHSLi0iE
    - tough 4y ago
      It's Banksy as a Service
    - bogwog 4y ago
      > we have been in the last Miami Art Week and Art Basel and we don't have enough time for the number of artists that wanted to be in the process. Will expand more later Please also include the number of those people who actually understand what an NFT is. As a native Miamian, I can guarantee you not a single one does. This city has always been a magnet for the get rich quick scheme types, and crypto is a good match for that because it's harder for a layman to grasp the scam part.
lofatdairy 4y ago
This is insanely impressive, looking at the 3D reconstruction results. If I'm not mistaken occlusions are where a lot of attentions being placed in pose estimation problems, and if there are enough annotated environmental spaces to create ground truths you could probably add environment reconstruction to pose reconstruction. What's nice there is that if you have multiple angles of an environment from a moving camera in a video, you can treat each previous frame as a prior which helps with prediction time and accuracy.
hombre_fatal 4y ago
Aside, I really like the UI indicators on the draggable models at the bottom that let you know you can rotate them.

530 points by GaggiX 4y ago

3D reconstruction from a single image. They stress the examples are not curated, appears to... well, gosh darnit, it appears to work. If it runs fast enough I wonder whether one could just drive around with a webcam and generate these 3d models on the fly and even import them into a sort of GTA type simulation/game engine in real time. (To generate a novel view, Zero-1-to-3 takes only 2 seconds on an RTX A6000 GPU) This research is based on work partially supported by: - Toyota Research Institute - DARPA MCS program under Federal Agreement No. N660011924032 - NSF NRI Award #1925157 Oh, huh. Interesting. Future Work From objects to scenes: Generalization to scenes with complex backgrounds remains an important challenge for our method. From scenes to videos: Being able to reason about geometry of dynamic scenes from a single view would open novel research directions -- such as understanding occlusions and dynamic object manipulation. A few approaches for diffusion-based video generation have been proposed recently and extending them to 3D would be key to opening up these opportunities.

TylerE 4y ago
Seems like there is a bit of a gap between “runs at 0.5 fps on a $7000 workstation-grade GPU with 48GB of VRAM” and consumer applications. With the fairly shallow slope of the GPU performance curve overtime, I don’t see them just Moores Lawing out of it either. this would need two, maybe three orders of magnitude more performance.
- jiggawatts 4y ago
  Computer power goes up exponentially thanks to Moore's law. Sprinkle some software optimisations on top, and it's conceivable for that to be running at interactive framerates on consumer GPUs within 5-10 years.
  - ffitch 4y ago
    the processing may as well shift to the clouds. With the subscription fee, of course : )
    - TylerE 4y ago
      Until we break the speed of light, I’m very bearish on cloud gaming. It just feels so bad. You’ve got like 9 layers of latency between you and the screen.
      - fooker 4y ago
        You don’t have to break the speed of light, just have the ping below human perception. ~20ms is that threshold, but even 40ms latency is barely noticeable for single player games.
        TylerE 4y ago
        A wireless controller ALONE is already over 20ms, and that’s before you touch the network, actually doing with that input, wait for the display to redraw… At a 20ms total round trip, that only buys you about a 1500 mile radius, again completely ignoring all other latencies.
        enlyth 4y ago
        It's quite noticeable actually, and it adds up, it's not just an extra 20ms. For casual gamers and turn based games maybe it could work, as a niche. For FPS, multiplayer, ARPG, and so on, it's a dealbreaker, anything over 100ms feels too sluggish. We should be happy we have so much autonomy with our own hardware, I don't want some big cloud company to be able to tell me what I can play and render, unless we want the "you will own nothing and be happy" meme to become reality.
        TylerE 4y ago
        I actually, in my testing, JRPG/other turn based games were amongst the worst because there is so much “management” (inventory, loot, gear, etc) and the extra lag really throws you off
- HopenHeyHi 4y ago
  Of course there is a gap. This is at the exploratory proof of concept stage. The fact that it works at all is what is interesting. Furthermore once you've identified the make and model of the car, its relative position in 3d, any anomalies -- that ain't just a Ford pickup, it is loaded with cargo that overhangs in a particular way -- its velocity, 'etc -- I'm quite sure that extrapolating additional information from the subsequent frames will be significantly cheaper as you don't have to generate a 3d model from scratch each time. I think this is a viable exploratory path forward. Make it work <- you are here Make it work correctly Make it work fast Edit: Scotty does know ;)
  - scotty79 4y ago
    I prefer: Make it work <- you are here Make it work correctly Make it work fast
    - deleted 4y ago
      [deleted]
- frozenport 4y ago
  >> fairly shallow slope of the GPU performance curve overtime Not true.

King-Aaron 4y ago

That's honestly extremely impressive. I do hope that the 'in the wild' examples aren't completely curated and are actually being rendered on the fly (They appear to be, but it's hard for me to tell if that' truly the case). Pretty cool to see however.

GaggiX 4y ago
>and are actually being rendered on the fly They are precomputed, "Note that the demo allows a limited selection of rotation angles quantized by 30 degrees due to limited storage space of the hosting server." but I don't think they are curated, the seeds probably correspond to the seeds of the live demo you can host (they released the code and the models)
deleted 4y ago
[deleted]

mitthrowaway2 4y ago

> We compare our reconstruction with state-of-the-art models in single-view 3D reconstruction. Here they list "GT Mesh", "Ours", "Point-E", and "MCC". Does anyone know what technique "GT mesh" refers to? Is it simply the original mesh that generated the source image?

haykmartiros 4y ago
Ground truth
- EGreg 4y ago
  Well honestly the "Ground truth" algorithm seems a lot superior to their method, it has higher fidelity in ALL the examples
  - sophiebits 4y ago
    “Ground truth” doesn’t refer to a particular algorithm; it refers to the ideal benchmark of what a perfect performance would look like, which they’re grading against.
  - simlevesque 4y ago
    Ground truth means that a human person created the model.
    - DarthNebo 4y ago
      Not necessarily, could also be synthetic. Google did the same for hand poses in BlazePalm
  - Thorrez 4y ago
    Ground truth means the original model that the image was generated from.
  - razemio 4y ago
    Haha, I am sorry. I spit my coffee reading this. It is ofc totally OK to not know what ground truth means but the irony was to funny. Yes ground truth will always be superior compared to anything else :)!

echelon 4y ago

Super cool results. This is what my startup is getting into. So I'm very interested. These aren't "game ready" - the sculpts are pretty gross. But we're clearly getting somewhere. It's only going to keep getting better. I expect we'll be building all new kinds of game engines, render pipelines, and 3D animation tools shortly.

nico 4y ago
And 3D printing. So quickly building physical tools too.
- skybrian 4y ago
  For printing parts, precision matters since they likely need to fit with something else. You’ll want to be able to edit dimensions on the model to get the fit right. So maybe someday, but I think it would have to be a project that targets CAD.
bredren 4y ago
How do these kinds of tools complement actual 3d scanning? For example, Apple supposedly has put some time into 3d asset building (presumably in support of AR world building content). Can these inference techniques stack or otherwise help more detailed object data collection?
redox99 4y ago
While this is cool, this is not meant to target "game ready". For games and CGI, there's no reason to limit yourself to a single image. Photogrammetry is already extensively used, and it involves using tens or hundreds of images of the object to scan. Using many images as an input will obviously always be superior to a single one, as a single image means it has to literally make up the back side, and it has no parallax information.
- oefrha 4y ago
  You appear to be thinking about scanning a physical object, whereas zero-shot one image to 3D object would be vastly more useful with a single (possibly AI-generated or AI-assisted) illustration. You get a 3D model in seconds at essentially zero cost, can iterate hundreds of times in a single day.

yawnxyz 4y ago

Are there any models that take an image to SVG?

nico 4y ago

This is feeling like almost thought to launch. In the last week, a lot of the ideas I’ve read about in the comments of HN, have then shown up as full blown projects in the front page. As if people are building at an insane speed from idea to launch/release.

nmfisher 4y ago
Just yesterday I was literally musing to myself "I wonder if NeRFs would help with 3D object synthesis", and here we are. It's definitely a fun time to be involved.
- regegrt 4y ago
  It's not based on the NeRF concept though, is it? Its outputs can provide the inputs for NeRF training, which is why they mention NeRFs. But it's not NeRF technology.
  - deleted 4y ago
    [deleted]
- noduerme 4y ago
  it's actually a really fun time to know how to sculpt in ZBrush and print out models.
  - nmfisher 4y ago
    If I had any artistic talent whatsoever, I'd probably agree with you!
    - noduerme 4y ago
      I won't lie... ZBrush is brutally hard. I got a subscription for work and only used it for one paid job, ever. But it's super satisfying if you just want to spend Sunday night making a clay elephant or rhinoceros, and drop $20 to have the file printed out and shipped to you by Thursday. I've fed lots of my sculpture renderings to Dali and gotten some pretty cool 2D results... but nothing nearly as cool as the little asymmetrical epoxy sculptures I can line up on the bookshelf...

gs17 4y ago

For anyone else who tried to download the weights and got Google Drive throwing a quota error at you, they're working on it: https://github.com/cvlab-columbia/zero123/issues/2 https://github.com/cvlab-columbia/zero123/issues/2

desmond373 4y ago

Would it be possible to generate cad files with this. As a base for part construction this could be game changing

gs17 4y ago
If you look at the example meshes, it doesn't seem very likely that it would be better than manually creating them, unless you're okay with lumpy parts that aren't exactly the right size. This is too early for it to not require a lot of cleanup to be usable.
- flangola7 4y ago
  In other words we just need to wait 6 more months

brokensegue 4y ago

how is this different from the previous NeRF work? does it build a 3D model?

GaggiX 4y ago
NeRF models are trained on several views with known location and viewing direction. This model takes one image (and you don't need to train a model for each object).
- fooker 4y ago
  Not hotdog.
- amelius 4y ago
  But if it takes only one image, isn't it likely to hallucinate information?
  - gs17 4y ago
    Not just likely, it does. Try out the demo and see, e.g. what the backside of their Pikachu toy looks like. Or a little simpler, the paper has an example (the demo also has this) of the back of a car under different seeds.

ilaksh 4y ago

I wonder if this type of thing could be adapted to a vision system for a robot? So it would locate the camera and reconstruct an entire scene from a series of images as the robot moves around. Probably needs a ways to get there but to be able to do robust SLAM etc. With just a single camera would make things much less expensive.

guyomes 4y ago
You might be interested in this related recent work [1] that fits simple ellipsoids to images and then use them for the pose estimation of a camera. [1]: https://ieeexplore.ieee.org/document/9127873 https://ieeexplore.ieee.org/document/9127873
- lefrancais 4y ago
  Same ref [1], but open : [https://hal.science/hal-02886633/document https://hal.science/hal-02886633/document]

ar9av 4y ago

It's hard to tell for certain from the paper, without going deep into the code, but it seems they created the new model the same way the depth conditioned SD models were made i.e. normal finetune. It might be possible to create a "original view + new angle" conditioned model much more easily by taking the Controlnet/T2IAdapter/GLIDE route where you freeze the original model. Text to 3d seems almost close to being solved. It also makes me think a "original character image + new pose" conditioned model would also work quite well.

bmitc 4y ago

What if you give it a picture of a cardboard cutout or billboard?

noduerme 4y ago
it'll build Angelyne for you, to distract your pathetic carbon-based intelligence. https://www.hollywoodreporter.com/wp-content/uploads/2017/07/zumaglobal327406_-_embed_2017.jpg?w=928 https://www.hollywoodreporter.com/wp-content/uploads/2017/07...

deleted 4y ago

[deleted]

noduerme 4y ago

Is there some kind of symmetry at work here in the deductive process?

xotom20390 4y ago
[dead]

jonplackett 4y ago

Would this be useful for a robot / car trying to navigate to be able to do this?

eternalban 4y ago
Great idea. Processing latency may be an issue. It has to be fast, small, and energy efficient.
elif 4y ago
unlikely. the front bumper of a car you are following has zero value for your ego's safety. most of the optimization of FSD is in removing extra data to improve latency of the mapping loop.
- jonplackett 4y ago
  But it seems like the main problem for self drive is accurately understanding the world around the car. The actual driving is pretty easy. Being able to see a partial object and understand what the rest of it is is very useful to human drivers

throwaway4aday 4y ago

If you can produce any view angle you want of an object then can't you use photogrammetry to construct a 3D object?

nwoli 4y ago
See the “Single-View 3D Reconstruction” section at the bottom where they do precisely that
- throwaway4aday 4y ago
  Cool, I missed that.

hypertexthero 4y ago

Brings to mind the Blade Runner enhance scene: https://www.youtube.com/watch?v=hHwjceFcF2Q https://www.youtube.com/watch?v=hHwjceFcF2Q

BiteCode_dev 4y ago
Given the data is (credible and beautiful) BS, I think it's closer to red dwarf: https://www.youtube.com/watch?v=6i3NWKbBaaU https://www.youtube.com/watch?v=6i3NWKbBaaU
Sakos 4y ago
Reminds me of this at the time fantastical scene in Enemy of the State https://youtu.be/3EwZQddc3kY https://youtu.be/3EwZQddc3kY

mov 4y ago

People plugging it as output of Midjourney in 3, 2, 1...

qikInNdOutReply 4y ago

What happens, if you build a circle? As in this creates a 3d object from a image and another ai creates a image from a 3d object? https://www.youtube.com/watch?v=zPqJUrfKuqs https://www.youtube.com/watch?v=zPqJUrfKuqs Does it stabilize, or refine prejudices, or go on a fractal journey of errors over the weight landscape?

hiccuphippo 4y ago

Can you obtain the 3d object from this or only an image with the new perspective? This could revolutionize indie gamedev.

jxf 4y ago
You can obtain a 3D object, but it's more useful for the novel views than the object, because the object isn't very good and probably needs some processing. See the bottom of the paper.

wslh 4y ago

I keep thinking in my project where we are taking multiple photos from the same angle with moving lights for rebuilding the 3D model. We are not using AI, just optic research like in [1]. We applied that on art at [2]. [1] Methods for 3D digitization of Cultural Heritage: http://www.ipet.gr/~akoutsou/docs/M3DD.pdf http://www.ipet.gr/~akoutsou/docs/M3DD.pdf [2] https://sublim.art https://sublim.art

bogwog 4y ago
So the business model there is: scanner + paper shredder + NFT = $$$? How many people have taken you up on that offer? Unless it's a shitty/low-effort painting, it seems insane to me that anyone would destroy their artwork in exchange for an NFT of that same artwork.
- wslh 4y ago
  What is insane for you could be completely different for others: we have been in the last Miami Art Week and Art Basel and we don't have enough time for the number of artists that wanted to be in the process. Will expand more later (working now) but you can see AP coverage here [1]. It is also important to highlight that we are doing this project at our own risk, with our own money, have built the hardware and software, and not charging artists for the process. Just the primary market sell is split between 85% for artists and the rest for the project. Pretty generous in this risky market. [1] https://youtu.be/ajDEHSLi0iE https://youtu.be/ajDEHSLi0iE
  - tough 4y ago
    It's Banksy as a Service
  - bogwog 4y ago
    > we have been in the last Miami Art Week and Art Basel and we don't have enough time for the number of artists that wanted to be in the process. Will expand more later Please also include the number of those people who actually understand what an NFT is. As a native Miamian, I can guarantee you not a single one does. This city has always been a magnet for the get rich quick scheme types, and crypto is a good match for that because it's harder for a layman to grasp the scam part.

lofatdairy 4y ago

This is insanely impressive, looking at the 3D reconstruction results. If I'm not mistaken occlusions are where a lot of attentions being placed in pose estimation problems, and if there are enough annotated environmental spaces to create ground truths you could probably add environment reconstruction to pose reconstruction. What's nice there is that if you have multiple angles of an environment from a moving camera in a video, you can treat each previous frame as a prior which helps with prediction time and accuracy.

hombre_fatal 4y ago

Aside, I really like the UI indicators on the draggable models at the bottom that let you know you can rotate them.