6 ms·

Small neural network enables realistic rendering of woven fabrics in real-time

181 points by wglb 2y ago

dartos 2y ago
This is what I was expecting from AI in games
- ffsm8 2y ago
  Well, the holy grail in games for AI will always be ... AI. Like, actually really pseudo intelligent companions you can converse with. Not gonna happen in the foreseeable future, and the job markets are gonna be... Well... essentially gone once that exist, so it's probably for the best that this tech isn't actually anywhere close to AGI
  - fhd2 2y ago
    Didn't downvote you (and not sure why anyone would), but I still disagree: Aren't games ultimately all about interacting with other people? It's already possible to play with actual other people, and quite popular too. Sure, those guys might not be very in-character or prioritise you having fun over them, but there's genuine intelligent interaction. The other way in which games make us interact with other humans is by experiencing things designed by them. Story, dialogues, game design, atmosphere etc. I think we might see more ad-hoc generated world/story, but I think many people would want to experience the _same_ game, in some way, being social animals. I'm not sure if and in what way generative AI or even AGI is going to disrupt the game industry. I can certainly picture the jobs of many people changing, from artists to voice actors. But I'm not sure I see a holy gray here. Lots of interesting novelties that I'm sure will be tried, but not really a fundamental change to why we play games and consequently, what games are like.
    - throwthrowuknow 2y ago
      There are plenty of games where the main feature is emergent gameplay resulting from a world simulator and the interplay of basic or complex rules. Dwarf Fortress is the OG here but also Sim City and Elite. Single player experiences are a huge segment and these will get better with LLM integration for world building and NPC agents. Fine tuned or custom models that operate on some basic state and rules could also be explored. You could train BERT family models to choose interesting state changes or actions based on the current game state for simple things like a dog NPC that would seem more intelligent by choosing the next action it should take e.g. deciding to chase a squirrel or growl at a new NPC you encountered based on its awareness of the context rather than following some rule system plus RNG. You could get rid of a lot of the stupid seeming game AI behaviour.
      - fhd2 2y ago
        True, though in my mind I put these things into my "lots of interesting novelties" category. Everything in games (including the physics engine!) is typically fine-tuned to provide fun, more than realism. The same is currently true for AI (especially enemy AI), and I believe it'll continue to be that way. Good point on the procedural aspect, Minecraft and similar games came to mind when I wrote my first post. I'd argue they are still fundamentally a designed, shareable experience, and that's my main point really. In Minecraft, you can't talk about the world much with someone else since it's generated, but the mechanics, materials, recipes etc. are the same, and provide exchange between humans playing the same game. I believe we'll even see some successful AI heavy games in the near future, but I also believe we'll cycle back to games being, first and foremost, designed upfront. I might be wrong of course and we might see some kind of holodeck 100% custom game experience become dominant. It's just my hunch that we won't, even if it's feasible. Thinking as far back as I can, people always seem to have wanted two things from games: Fun, and connection to other people.
    - deleted 2y ago
      [deleted]
  - MonortYp 2y ago
    I’d love to see a real improvement in games where the NPCs think further than a few if statements. If AI could track context of conversations and include player actions such as quests, relationships, deaths etc to influence decisions or outcomes it would add lots of enjoyment for me. For me it’s either Stardew Valley or Paradox type games for my single player itch or in the past I’ve ventured off to MMORPGs like Eve or Wurm and play in “single player” mode. MMORPGs have a greater sense of world change as players can carve mountains and build estates while slaying champions and such in Wurm online and you can stick to your plot of land and have nothing to do with it while enjoying the benefits. This doesn’t exist in single player games but it could and should. Although I don’t agree AI will be taking jobs but it will just be another tool used in the industry. We need to embrace change and use these new tools to make better content and not just assume jobs will be lost.
    - astrange 2y ago
      > If AI could track context of conversations and include player actions such as quests, relationships, deaths etc to influence decisions or outcomes it would add lots of enjoyment for me. From what people say about them Bethesda games try to do this, but it seems like the result is they're impossibly buggy because noone can test all of that. Undertale is low tech, but does track player deaths and use them in the story.
  - dartos 2y ago
    It depends on the game. That’d be great for an RPG game, but if I’m playing a action adventure game or a style fighter, I generally care about how cool it looks and how smooth it plays than I do about the intelligence of the side characters.
- Modified3019 2y ago
  We’ll have made it as a species when the harvester in C&C no longer gets stuck or wanders through the enemy base. In all seriousness though, I think RTS games (I only do single player) would be more fun for me personally if I could leave the economy/build order stuff to a reasonable AI and focus instead on military unit control. I’m just not someone that can handle all of that at once. I think the potential for an enemy tailored to challenging and occasionally surprising me without simply overwhelming me would also be great.
  - Epa095 2y ago
    What about RTS games specialising on the military part? I haven't played them myself, but I think the Company of Heroes series is like this, light on base building and focusing more on the military aspect.
    - mrguyorama 2y ago
      WARNO and it's predecessors have zero base building. You can place down a single Forward Operating Base that is just a supply truck without wheels, but everything else comes from off screen onto the game.
  - dartos 2y ago
    Totally off topic, but RTS is a weird split genre. I really like the economy / base building part, but care less for unit control. I’m more of a macro player than micro. I think a lot of players fall on one side or the other.
    - thebruce87m 2y ago
      Did you ever play settlers? I loved building the towns.
      - dartos 2y ago
        Settlers of catan? Weirdly enough, I’ve never played it
- jorvi 2y ago
  With more and more fixes for “hallucinations”, and with smaller models getting more capable (or bigger models getting more efficient, depending on how you look at it), I have a feeling that eventually it’ll be cheaper to render things neurally rather than rasterized. Imagine if you could describe what is happening on the screen, including the state of the world and everything, with a really long prompt. You dynamically alter that prompt continuously based on inputs received. There will be a point in the future where the prompt will render with 99% consistency. You’d probably still need some sort of barebones state server for things like hitboxes and to prevent cumulative divergence. Still a ways away though :)
johanvts 2y ago
> works by encoding the patterns and parameters of fabrics as a small latent vector, which can later be interpreted by a decoder to produce realistic representations of various fabrics. > Woven fabric patterns […] tend to be regular and repetitive. Chen and his colleagues set out to leverage their repetitive nature Is this “just” using the NN to approximately compress and decompress things? That doesn’t seem to explain the performance gains, unless some work can be done in the reduced space?
- Anotheroneagain 2y ago
  I believe that results like these show that our algorithms are woefully inadequate, and something much faster has to exist, if a NN can do so much better with all the overhead.
  - Arwill 2y ago
    Light/material calculation in shaders requires exponentiation, which is a costly operation. If a clever lookup table can get around that, it sure can be faster.
    - magicalhippo 2y ago
      Using a lookup table[1] or other approximations like wavelets[2] for complex BRDFs is a well-known strategy. [1]: https://github.com/HectorMF/BRDFGenerator https://github.com/HectorMF/BRDFGenerator [2]: https://hal.science/hal-01518547/document https://hal.science/hal-01518547/document
  - esperent 2y ago
    This isn't efficient at all. It requires a 3090 to get 60fps - and presumably it requires all the GPU's power to do that for the simple scene in the article: a couple of sheets and cushions on a sofa. Possibly it's RAM rather than compute limited but still. You couldn't use that in a game. Existing techniques like PBD or verlet integration would be far more efficient. Maybe on future devices with separate neural processor units this could be useful. But even then I bet those same chips could run standard real-time cloth algorithms really really fast too. Edit: I downloaded their video and there's no movement at all, only still images. Maybe I misunderstood the purpose of the technique. But I re-read the article and it certainly seems like they're claiming to do full simulation, not just static scenes. That scene with the sofa isn't even getting 60fps! It kinda feels like a disingenuous article after watching the video.
loufe 2y ago
I'm curious if the neural net ran wide and used the majority of the VRAM to achieve 60fps, or if it ran narrow and had more of a long-thread-length dependency.
sgt101 2y ago
Resnet + 16 layers isn't GPT4 but it's not small. I think that the 60fps on a consumer grade card is the win, but that says a lot about the power of consumer grade cards in 2024.
- ahoka 2y ago
  Only 60 fps on a 3090 is not exactly real time in practice.
  - johann8384 2y ago
    Why? 60fps is twice the framerate of common video streams. It seems like it's pretty realtime to me.
    - deleted 2y ago
      [deleted]
    - jsheard 2y ago
      They're spending about 14ms on inferencing and 3ms on everything else, which technically is about 60fps, but in any real world application you're probably not going to be able to blow almost your entire GPU compute budget on shading a few pieces of cloth.
      - DistractionRect 2y ago
        To their point, 30 fps is commonly acceptable, in which case you're spending 7-14ms on this, and 26-19ms on everything else. You can also use frame gen cheat 30 fps back up to 60 if one is so inclined.
        refulgentis 2y ago
        > 30 fps is commonly acceptable Interesting to hear that, consumer market has been well past 30 fps for years, to my knowledge. For example: - Phones are widely at 120 fps as of a couple years ago. - Gamers have been targeting 240 fps for years and years, since 2018? - On the low end, ex. Steam Deck requires you manually opt-in if you're happy with less than 60. I'm curious for some groups you're aware of that commonly accept 30. (note: not a trick question, ex. we know movies are 24 fps, and professional 3D renderers are happy with not-real-time, but we're thinking about 3D rendering for games)
zokier 2y ago
There was also another neural cloth rendering paper released recently, focusing on more micro-scale details (and not being real-time) https://doi.org/10.1111/cgf.15156 https://doi.org/10.1111/cgf.15156
brcmthrowaway 2y ago
How would you a neural network ever be faster than a closed form solution for fabric rendering? Neural network is the big hammer.
- eru 2y ago
  > How would you a neural network ever be faster than a closed form solution for fabric rendering? If the neural network is allowed to approximate, then why not?
- astrange 2y ago
  If the problem is difficult, you could run it 5fps or so and interpolate between its results.
speps 2y ago
Paper link: https://arxiv.org/abs/2406.17782 https://arxiv.org/abs/2406.17782
iandanforth 2y ago
For some reason I thought this was about simulating the fabric in a physical sense. Having watched the videos from the supplemental material (downloadable from the SIGGRAPH paper page) it appears to be about "just" rendering. The fabric does look nice though for some reason they chose very shiny fabrics for their video.

181 points by wglb 2y ago

dartos 2y ago

This is what I was expecting from AI in games

ffsm8 2y ago
Well, the holy grail in games for AI will always be ... AI. Like, actually really pseudo intelligent companions you can converse with. Not gonna happen in the foreseeable future, and the job markets are gonna be... Well... essentially gone once that exist, so it's probably for the best that this tech isn't actually anywhere close to AGI
- fhd2 2y ago
  Didn't downvote you (and not sure why anyone would), but I still disagree: Aren't games ultimately all about interacting with other people? It's already possible to play with actual other people, and quite popular too. Sure, those guys might not be very in-character or prioritise you having fun over them, but there's genuine intelligent interaction. The other way in which games make us interact with other humans is by experiencing things designed by them. Story, dialogues, game design, atmosphere etc. I think we might see more ad-hoc generated world/story, but I think many people would want to experience the _same_ game, in some way, being social animals. I'm not sure if and in what way generative AI or even AGI is going to disrupt the game industry. I can certainly picture the jobs of many people changing, from artists to voice actors. But I'm not sure I see a holy gray here. Lots of interesting novelties that I'm sure will be tried, but not really a fundamental change to why we play games and consequently, what games are like.
  - throwthrowuknow 2y ago
    There are plenty of games where the main feature is emergent gameplay resulting from a world simulator and the interplay of basic or complex rules. Dwarf Fortress is the OG here but also Sim City and Elite. Single player experiences are a huge segment and these will get better with LLM integration for world building and NPC agents. Fine tuned or custom models that operate on some basic state and rules could also be explored. You could train BERT family models to choose interesting state changes or actions based on the current game state for simple things like a dog NPC that would seem more intelligent by choosing the next action it should take e.g. deciding to chase a squirrel or growl at a new NPC you encountered based on its awareness of the context rather than following some rule system plus RNG. You could get rid of a lot of the stupid seeming game AI behaviour.
    - fhd2 2y ago
      True, though in my mind I put these things into my "lots of interesting novelties" category. Everything in games (including the physics engine!) is typically fine-tuned to provide fun, more than realism. The same is currently true for AI (especially enemy AI), and I believe it'll continue to be that way. Good point on the procedural aspect, Minecraft and similar games came to mind when I wrote my first post. I'd argue they are still fundamentally a designed, shareable experience, and that's my main point really. In Minecraft, you can't talk about the world much with someone else since it's generated, but the mechanics, materials, recipes etc. are the same, and provide exchange between humans playing the same game. I believe we'll even see some successful AI heavy games in the near future, but I also believe we'll cycle back to games being, first and foremost, designed upfront. I might be wrong of course and we might see some kind of holodeck 100% custom game experience become dominant. It's just my hunch that we won't, even if it's feasible. Thinking as far back as I can, people always seem to have wanted two things from games: Fun, and connection to other people.
  - deleted 2y ago
    [deleted]
- MonortYp 2y ago
  I’d love to see a real improvement in games where the NPCs think further than a few if statements. If AI could track context of conversations and include player actions such as quests, relationships, deaths etc to influence decisions or outcomes it would add lots of enjoyment for me. For me it’s either Stardew Valley or Paradox type games for my single player itch or in the past I’ve ventured off to MMORPGs like Eve or Wurm and play in “single player” mode. MMORPGs have a greater sense of world change as players can carve mountains and build estates while slaying champions and such in Wurm online and you can stick to your plot of land and have nothing to do with it while enjoying the benefits. This doesn’t exist in single player games but it could and should. Although I don’t agree AI will be taking jobs but it will just be another tool used in the industry. We need to embrace change and use these new tools to make better content and not just assume jobs will be lost.
  - astrange 2y ago
    > If AI could track context of conversations and include player actions such as quests, relationships, deaths etc to influence decisions or outcomes it would add lots of enjoyment for me. From what people say about them Bethesda games try to do this, but it seems like the result is they're impossibly buggy because noone can test all of that. Undertale is low tech, but does track player deaths and use them in the story.
- dartos 2y ago
  It depends on the game. That’d be great for an RPG game, but if I’m playing a action adventure game or a style fighter, I generally care about how cool it looks and how smooth it plays than I do about the intelligence of the side characters.
Modified3019 2y ago
We’ll have made it as a species when the harvester in C&C no longer gets stuck or wanders through the enemy base. In all seriousness though, I think RTS games (I only do single player) would be more fun for me personally if I could leave the economy/build order stuff to a reasonable AI and focus instead on military unit control. I’m just not someone that can handle all of that at once. I think the potential for an enemy tailored to challenging and occasionally surprising me without simply overwhelming me would also be great.
- Epa095 2y ago
  What about RTS games specialising on the military part? I haven't played them myself, but I think the Company of Heroes series is like this, light on base building and focusing more on the military aspect.
  - mrguyorama 2y ago
    WARNO and it's predecessors have zero base building. You can place down a single Forward Operating Base that is just a supply truck without wheels, but everything else comes from off screen onto the game.
- dartos 2y ago
  Totally off topic, but RTS is a weird split genre. I really like the economy / base building part, but care less for unit control. I’m more of a macro player than micro. I think a lot of players fall on one side or the other.
  - thebruce87m 2y ago
    Did you ever play settlers? I loved building the towns.
    - dartos 2y ago
      Settlers of catan? Weirdly enough, I’ve never played it
jorvi 2y ago
With more and more fixes for “hallucinations”, and with smaller models getting more capable (or bigger models getting more efficient, depending on how you look at it), I have a feeling that eventually it’ll be cheaper to render things neurally rather than rasterized. Imagine if you could describe what is happening on the screen, including the state of the world and everything, with a really long prompt. You dynamically alter that prompt continuously based on inputs received. There will be a point in the future where the prompt will render with 99% consistency. You’d probably still need some sort of barebones state server for things like hitboxes and to prevent cumulative divergence. Still a ways away though :)

johanvts 2y ago

> works by encoding the patterns and parameters of fabrics as a small latent vector, which can later be interpreted by a decoder to produce realistic representations of various fabrics. > Woven fabric patterns […] tend to be regular and repetitive. Chen and his colleagues set out to leverage their repetitive nature Is this “just” using the NN to approximately compress and decompress things? That doesn’t seem to explain the performance gains, unless some work can be done in the reduced space?

Anotheroneagain 2y ago
I believe that results like these show that our algorithms are woefully inadequate, and something much faster has to exist, if a NN can do so much better with all the overhead.
- Arwill 2y ago
  Light/material calculation in shaders requires exponentiation, which is a costly operation. If a clever lookup table can get around that, it sure can be faster.
  - magicalhippo 2y ago
    Using a lookup table[1] or other approximations like wavelets[2] for complex BRDFs is a well-known strategy. [1]: https://github.com/HectorMF/BRDFGenerator https://github.com/HectorMF/BRDFGenerator [2]: https://hal.science/hal-01518547/document https://hal.science/hal-01518547/document
- esperent 2y ago
  This isn't efficient at all. It requires a 3090 to get 60fps - and presumably it requires all the GPU's power to do that for the simple scene in the article: a couple of sheets and cushions on a sofa. Possibly it's RAM rather than compute limited but still. You couldn't use that in a game. Existing techniques like PBD or verlet integration would be far more efficient. Maybe on future devices with separate neural processor units this could be useful. But even then I bet those same chips could run standard real-time cloth algorithms really really fast too. Edit: I downloaded their video and there's no movement at all, only still images. Maybe I misunderstood the purpose of the technique. But I re-read the article and it certainly seems like they're claiming to do full simulation, not just static scenes. That scene with the sofa isn't even getting 60fps! It kinda feels like a disingenuous article after watching the video.

loufe 2y ago

I'm curious if the neural net ran wide and used the majority of the VRAM to achieve 60fps, or if it ran narrow and had more of a long-thread-length dependency.

sgt101 2y ago

Resnet + 16 layers isn't GPT4 but it's not small. I think that the 60fps on a consumer grade card is the win, but that says a lot about the power of consumer grade cards in 2024.

ahoka 2y ago
Only 60 fps on a 3090 is not exactly real time in practice.
- johann8384 2y ago
  Why? 60fps is twice the framerate of common video streams. It seems like it's pretty realtime to me.
  - deleted 2y ago
    [deleted]
  - jsheard 2y ago
    They're spending about 14ms on inferencing and 3ms on everything else, which technically is about 60fps, but in any real world application you're probably not going to be able to blow almost your entire GPU compute budget on shading a few pieces of cloth.
    - DistractionRect 2y ago
      To their point, 30 fps is commonly acceptable, in which case you're spending 7-14ms on this, and 26-19ms on everything else. You can also use frame gen cheat 30 fps back up to 60 if one is so inclined.
      - refulgentis 2y ago
        > 30 fps is commonly acceptable Interesting to hear that, consumer market has been well past 30 fps for years, to my knowledge. For example: - Phones are widely at 120 fps as of a couple years ago. - Gamers have been targeting 240 fps for years and years, since 2018? - On the low end, ex. Steam Deck requires you manually opt-in if you're happy with less than 60. I'm curious for some groups you're aware of that commonly accept 30. (note: not a trick question, ex. we know movies are 24 fps, and professional 3D renderers are happy with not-real-time, but we're thinking about 3D rendering for games)

zokier 2y ago

There was also another neural cloth rendering paper released recently, focusing on more micro-scale details (and not being real-time) https://doi.org/10.1111/cgf.15156 https://doi.org/10.1111/cgf.15156

brcmthrowaway 2y ago

How would you a neural network ever be faster than a closed form solution for fabric rendering? Neural network is the big hammer.

eru 2y ago
> How would you a neural network ever be faster than a closed form solution for fabric rendering? If the neural network is allowed to approximate, then why not?
astrange 2y ago
If the problem is difficult, you could run it 5fps or so and interpolate between its results.

speps 2y ago

Paper link: https://arxiv.org/abs/2406.17782 https://arxiv.org/abs/2406.17782

iandanforth 2y ago

For some reason I thought this was about simulating the fabric in a physical sense. Having watched the videos from the supplemental material (downloadable from the SIGGRAPH paper page) it appears to be about "just" rendering. The fabric does look nice though for some reason they chose very shiny fabrics for their video.