3 ms·

It's... neat? But I'm struggling to think of what the applications of this would actually be. 2D artwork usually doesn't have a consistent 3D space, which they

It's... neat? But I'm struggling to think of what the applications of this would actually be. 2D artwork usually doesn't have a consistent 3D space, which they acknowledge, but they don't seem to have overcome that problem in any useful sense. The scenes are barely coherent once they move from one of the originally drawn camera positions.

codetrotter 2y ago
Both Futurama and Family Guy sometimes use 3d rendering for vehicles for example, and render it in a cartoon looking style and composit it with flat 2d animations. Maybe similar kind of things could be an application of this. Another possible use-case might be a game development studio developing a license game based on a 2d cartoon, but making the game 3d. They could use this as a tool for visualization while planning and developing, to iterate quickly and to reference how the original 2d could translate into 3d.
- SiempreViernes 2y ago
  Not really? In those examples the hand crafted 3d assets already exist, this thing could at best recreate the 3d geometries the show creators made themselves. That seems useful mainly for cloning someone else's work.
  - codetrotter 2y ago
    “Similar kind of thing” meaning for another show that wants to do the same but who have not created the 3d assets yet. Team of 2d artists draw the desired vehicles for the cartoon from two or three angles. Software like this makes a usable 3d model of it.
    - jsheard 2y ago
      If you were making 2D drawings with the intent of turning them into a 3D model then you would draw them to be coherent in 3D in the first place. The whole novelty of the research in the OP is that they're trying to reconstruct drawings that were never intended to make realistic sense in 3D. Even if AI has a place in the 2D to 3D part of a pipeline, surely you'd still want the 2D artwork be unambigously representative of what the 3D asset should look like, rather than providing self-contradictory input data and praying that the AI can magically make it make sense.
      - codetrotter 2y ago
        True. For the second use-case I mentioned it still applies though. Where a studio is making a licensed 3d game based on an existing 2d cartoon.
    - superb_dev 2y ago
      A show that's being doing purely 2d art can't just integrate the 3d art in their pipeline on a whim. If they can, they probably already have the skills to make the model outright.
jameshart 2y ago
I could see some value maybe in giving an artist feedback on where the model detects inconsistencies between different viewpoints.
- jsheard 2y ago
  That assumes that consistency between viewpoints is actually desirable - part of the charm of 2D animation is that things can be stylized or exaggerated or simplified in ways that don't come naturally in a 3D workflow, where the "default" is for things to fit together realistically and any deviation from that takes additional effort. If you do want numerous 2D artworks which share a realistically defined 3D space then that can easily be done by making a very rough 3D scene and then painting over it, you don't need any AI for that.
- SiempreViernes 2y ago
  If consistency was highly desirable you'd just model the 3d space from the start...
Waterluvian 2y ago
I think this is just a device used to demonstrate and advance the technology. I doubt this has a real application in this context given how little work is needed to 3D model these kinds of environments anyways.
timdiggerm 2y ago
Maybe you could better construct a 3d model of a demolished landmark from old paintings and photos?
theultdev 2y ago
With future advancements you could pump out video games for many series. While rough, these do look better than some implementations of the artwork for cartoon games.
wongarsu 2y ago
A refined version of this could be used to make stereoscopic versions of cartoons. On the other hand you are probably better off only using the depth prediction and filling any voids in using image generation instead of this mapping process.
chungy 2y ago
SpongeBob brazenly violates 3D space rules (I mean, they also have fire underwater...). The writers and artists both draw heavy inspiration from Looney Tunes, where such rules are broken because it's funny to break them.
xsmasher 2y ago
The renders it creates are underwhelming, but it seems good at determining the location and angle of the camera. I could see it being used to create a "scratch track" that human animators animate on top of. An aid to tweening.

codetrotter 2y ago

Both Futurama and Family Guy sometimes use 3d rendering for vehicles for example, and render it in a cartoon looking style and composit it with flat 2d animations. Maybe similar kind of things could be an application of this. Another possible use-case might be a game development studio developing a license game based on a 2d cartoon, but making the game 3d. They could use this as a tool for visualization while planning and developing, to iterate quickly and to reference how the original 2d could translate into 3d.

SiempreViernes 2y ago
Not really? In those examples the hand crafted 3d assets already exist, this thing could at best recreate the 3d geometries the show creators made themselves. That seems useful mainly for cloning someone else's work.
- codetrotter 2y ago
  “Similar kind of thing” meaning for another show that wants to do the same but who have not created the 3d assets yet. Team of 2d artists draw the desired vehicles for the cartoon from two or three angles. Software like this makes a usable 3d model of it.
  - jsheard 2y ago
    If you were making 2D drawings with the intent of turning them into a 3D model then you would draw them to be coherent in 3D in the first place. The whole novelty of the research in the OP is that they're trying to reconstruct drawings that were never intended to make realistic sense in 3D. Even if AI has a place in the 2D to 3D part of a pipeline, surely you'd still want the 2D artwork be unambigously representative of what the 3D asset should look like, rather than providing self-contradictory input data and praying that the AI can magically make it make sense.
    - codetrotter 2y ago
      True. For the second use-case I mentioned it still applies though. Where a studio is making a licensed 3d game based on an existing 2d cartoon.
  - superb_dev 2y ago
    A show that's being doing purely 2d art can't just integrate the 3d art in their pipeline on a whim. If they can, they probably already have the skills to make the model outright.

jameshart 2y ago

I could see some value maybe in giving an artist feedback on where the model detects inconsistencies between different viewpoints.

jsheard 2y ago
That assumes that consistency between viewpoints is actually desirable - part of the charm of 2D animation is that things can be stylized or exaggerated or simplified in ways that don't come naturally in a 3D workflow, where the "default" is for things to fit together realistically and any deviation from that takes additional effort. If you do want numerous 2D artworks which share a realistically defined 3D space then that can easily be done by making a very rough 3D scene and then painting over it, you don't need any AI for that.
SiempreViernes 2y ago
If consistency was highly desirable you'd just model the 3d space from the start...

Waterluvian 2y ago

I think this is just a device used to demonstrate and advance the technology. I doubt this has a real application in this context given how little work is needed to 3D model these kinds of environments anyways.

timdiggerm 2y ago

Maybe you could better construct a 3d model of a demolished landmark from old paintings and photos?

theultdev 2y ago

With future advancements you could pump out video games for many series. While rough, these do look better than some implementations of the artwork for cartoon games.

wongarsu 2y ago

A refined version of this could be used to make stereoscopic versions of cartoons. On the other hand you are probably better off only using the depth prediction and filling any voids in using image generation instead of this mapping process.

chungy 2y ago

SpongeBob brazenly violates 3D space rules (I mean, they also have fire underwater...). The writers and artists both draw heavy inspiration from Looney Tunes, where such rules are broken because it's funny to break them.

xsmasher 2y ago

The renders it creates are underwhelming, but it seems good at determining the location and angle of the camera. I could see it being used to create a "scratch track" that human animators animate on top of. An aid to tweening.