4 ms·

This seems unprecedented. Imagine if you have this but you can update the scene programmatically. Ask your AI to change the location or actors. Now you have a v

This seems unprecedented. Imagine if you have this but you can update the scene programmatically. Ask your AI to change the location or actors. Now you have a very convincing artificial scene with anything you can imagine in it.

mistermann 3y ago
With all the various more specialized demos I've seen, that day doesn't seem too far away.
- MPSimmons 3y ago
  I imagine that most people will have the capacity in their home machines to render stuff like this in video games in 10ish years.
  - fragmede 3y ago
    10? I think we'll both be surprised by where we're at in 3. That's October, 2026.
    - MPSimmons 3y ago
      Re-reading the paper, I totally missed the hardware that they did this on, which was consumer-level GPUs, so I think you may be right - 3 years is probably a good time frame for seeing this kind of tech in commercial games. My reasoning for initially saying 10ish years: GPU architecture release cadence is frequent, but not THAT frequent. NVidia released the Ampere (RTX 30X0) in 2020, 3 years ago. Ada Lovelace (RTX 40X0) was released a year ago, in Oct 2022. It's _possible_ to use 4090s to do medium neural-network things right now, but to be on the leading edge, you need multiple datacenter-level cards, each of which is > $5k. Even though it's possible to do crazy things with the most recent generation of GPUs, there don't yet exist games that really take advantage of it. The closest that I'm aware of is the Cyberpunk-level games that make native use of raytracing capabilities. I think it'll probably be a year or two before we see games come out that really require the level of 40-series cards ( or 7900-series if you're of the AMD persuasion), which is a lag-time of ~3 years after the cards were released. I think the software development time and market saturation are the driving factors in the gap here. I was under the mistaken impression that the video output was produced on the datacenter cards. They got reasonable performance on a 30-series NVidia card. In 3 years, it's totally reasonable to expect that AAA game players will have that level of GPU performance in their gaming machines, so yeah, I think you're right.
      - kridsdale1 3y ago
        Developers target where the market is. Until 2027 or so, that will be the PS5, which is somewhere around RTX 3060 or 2080 I think.
ChuckMcM 3y ago
I imagine this would be helpful when making movies if you could basically play around with the scenes without having to refilm it several times to get the best one.
- Cthulhu_ 3y ago
  When it comes to perspective and the like, they already do this; multiple camera angles, CGI, and the odd reshoot. Like having Henry Cavill come back for a reshoot, then CGIing out the mustache he had for his next role.
jayd16 3y ago
I suppose you could build up what would essentially be a 4D sprite sheet or animation set of a character and use that to support natural looking arbitrary movement. I'm not sure that isn't just a mo-cap character with extra steps, though.
- kridsdale1 3y ago
  Even the most skilled animators with years of budget still can’t escape the uncanny valley which is why CG animation has converged on a style of blob-humans as the current standard. I have very little hope of AI driven animation looking ok in the next many decades. Don’t underestimate how hardwired your senses are at finding artifacts in movement. Static images are much easier to “fake”.
TeMPOraL 3y ago
Between this and LLMs, we're half-way to building a holodeck. What's missing at this point is just hard light - i.e. the being able to feel the physical substance of simulated objects, and being able to experience it all without a wearable/personal device.
- shric 3y ago
  > What's missing at this point is just hard light Yeah, "just". Even though we have no idea how to even approach that.
  - TeMPOraL 3y ago
    > Even though we have no idea how to even approach that. We'll have to cheat somehow. Sure, this is still effectively magic, but a few years ago I thought we're also anywhere near having the software layer solved - specifically, the Star Trek style computer interaction and holodeck "storyteller" - the thing that would let you create a high-fidelity interactive world with believable characters and a story generated as you play, out of a command like "computer, give me a cafe in Paris, circa 1890". Now we suddenly have all the pieces for that - we can literally just do it, as long as we constrain the medium to just text, with perhaps some generated imagery for extra mood[0]. And I'm not even talking about GPT-4 - I've had a convincing holodeck textual roleplay on AI Dungeon back ~2-3 years ago, back when GPT-3 just came out. Note: I don't want to hijack this thread and make it into yet another LLM discussion - but I want to point out we have a bunch of parts converging into an entirely new kind of experience. And while we may not crack hard light any time soon, a wearable VR/AR holodeck experience now seems in reach in under a decade. Or perhaps closer to SAO[1] than a holodeck, but still - something that felt way beyond our capabilities just a couple years ago. -- [0] - Though GPT-4V should be able to play a board game where you send an image of an updated board each turn, shouldn't it? [1] - Sword Art Online
    - bmicraft 3y ago
      > I want to point out we have a bunch of parts converging into an entirely new kind of experience We do have a bunch of parts, but I don't see any converging happening

mistermann 3y ago

With all the various more specialized demos I've seen, that day doesn't seem too far away.

MPSimmons 3y ago
I imagine that most people will have the capacity in their home machines to render stuff like this in video games in 10ish years.
- fragmede 3y ago
  10? I think we'll both be surprised by where we're at in 3. That's October, 2026.
  - MPSimmons 3y ago
    Re-reading the paper, I totally missed the hardware that they did this on, which was consumer-level GPUs, so I think you may be right - 3 years is probably a good time frame for seeing this kind of tech in commercial games. My reasoning for initially saying 10ish years: GPU architecture release cadence is frequent, but not THAT frequent. NVidia released the Ampere (RTX 30X0) in 2020, 3 years ago. Ada Lovelace (RTX 40X0) was released a year ago, in Oct 2022. It's _possible_ to use 4090s to do medium neural-network things right now, but to be on the leading edge, you need multiple datacenter-level cards, each of which is > $5k. Even though it's possible to do crazy things with the most recent generation of GPUs, there don't yet exist games that really take advantage of it. The closest that I'm aware of is the Cyberpunk-level games that make native use of raytracing capabilities. I think it'll probably be a year or two before we see games come out that really require the level of 40-series cards ( or 7900-series if you're of the AMD persuasion), which is a lag-time of ~3 years after the cards were released. I think the software development time and market saturation are the driving factors in the gap here. I was under the mistaken impression that the video output was produced on the datacenter cards. They got reasonable performance on a 30-series NVidia card. In 3 years, it's totally reasonable to expect that AAA game players will have that level of GPU performance in their gaming machines, so yeah, I think you're right.
    - kridsdale1 3y ago
      Developers target where the market is. Until 2027 or so, that will be the PS5, which is somewhere around RTX 3060 or 2080 I think.

ChuckMcM 3y ago

I imagine this would be helpful when making movies if you could basically play around with the scenes without having to refilm it several times to get the best one.

Cthulhu_ 3y ago
When it comes to perspective and the like, they already do this; multiple camera angles, CGI, and the odd reshoot. Like having Henry Cavill come back for a reshoot, then CGIing out the mustache he had for his next role.

jayd16 3y ago

I suppose you could build up what would essentially be a 4D sprite sheet or animation set of a character and use that to support natural looking arbitrary movement. I'm not sure that isn't just a mo-cap character with extra steps, though.

kridsdale1 3y ago
Even the most skilled animators with years of budget still can’t escape the uncanny valley which is why CG animation has converged on a style of blob-humans as the current standard. I have very little hope of AI driven animation looking ok in the next many decades. Don’t underestimate how hardwired your senses are at finding artifacts in movement. Static images are much easier to “fake”.

TeMPOraL 3y ago

Between this and LLMs, we're half-way to building a holodeck. What's missing at this point is just hard light - i.e. the being able to feel the physical substance of simulated objects, and being able to experience it all without a wearable/personal device.

shric 3y ago
> What's missing at this point is just hard light Yeah, "just". Even though we have no idea how to even approach that.
- TeMPOraL 3y ago
  > Even though we have no idea how to even approach that. We'll have to cheat somehow. Sure, this is still effectively magic, but a few years ago I thought we're also anywhere near having the software layer solved - specifically, the Star Trek style computer interaction and holodeck "storyteller" - the thing that would let you create a high-fidelity interactive world with believable characters and a story generated as you play, out of a command like "computer, give me a cafe in Paris, circa 1890". Now we suddenly have all the pieces for that - we can literally just do it, as long as we constrain the medium to just text, with perhaps some generated imagery for extra mood[0]. And I'm not even talking about GPT-4 - I've had a convincing holodeck textual roleplay on AI Dungeon back ~2-3 years ago, back when GPT-3 just came out. Note: I don't want to hijack this thread and make it into yet another LLM discussion - but I want to point out we have a bunch of parts converging into an entirely new kind of experience. And while we may not crack hard light any time soon, a wearable VR/AR holodeck experience now seems in reach in under a decade. Or perhaps closer to SAO[1] than a holodeck, but still - something that felt way beyond our capabilities just a couple years ago. -- [0] - Though GPT-4V should be able to play a board game where you send an image of an updated board each turn, shouldn't it? [1] - Sword Art Online
  - bmicraft 3y ago
    > I want to point out we have a bunch of parts converging into an entirely new kind of experience We do have a bunch of parts, but I don't see any converging happening