4 ms·

> works by encoding the patterns and parameters of fabrics as a small latent vector, which can later be interpreted by a decoder to produce realistic representa

> works by encoding the patterns and parameters of fabrics as a small latent vector, which can later be interpreted by a decoder to produce realistic representations of various fabrics. > Woven fabric patterns […] tend to be regular and repetitive. Chen and his colleagues set out to leverage their repetitive nature Is this “just” using the NN to approximately compress and decompress things? That doesn’t seem to explain the performance gains, unless some work can be done in the reduced space?

Anotheroneagain 2y ago
I believe that results like these show that our algorithms are woefully inadequate, and something much faster has to exist, if a NN can do so much better with all the overhead.
- Arwill 2y ago
  Light/material calculation in shaders requires exponentiation, which is a costly operation. If a clever lookup table can get around that, it sure can be faster.
  - magicalhippo 2y ago
    Using a lookup table[1] or other approximations like wavelets[2] for complex BRDFs is a well-known strategy. [1]: https://github.com/HectorMF/BRDFGenerator https://github.com/HectorMF/BRDFGenerator [2]: https://hal.science/hal-01518547/document https://hal.science/hal-01518547/document
- esperent 2y ago
  This isn't efficient at all. It requires a 3090 to get 60fps - and presumably it requires all the GPU's power to do that for the simple scene in the article: a couple of sheets and cushions on a sofa. Possibly it's RAM rather than compute limited but still. You couldn't use that in a game. Existing techniques like PBD or verlet integration would be far more efficient. Maybe on future devices with separate neural processor units this could be useful. But even then I bet those same chips could run standard real-time cloth algorithms really really fast too. Edit: I downloaded their video and there's no movement at all, only still images. Maybe I misunderstood the purpose of the technique. But I re-read the article and it certainly seems like they're claiming to do full simulation, not just static scenes. That scene with the sofa isn't even getting 60fps! It kinda feels like a disingenuous article after watching the video.
  - zokier 2y ago
    I don't see how something like PBD helps at all with rendering.. are you thinking that this work relates somehow to physics simulation? Considering that the result is pretty much just a shader, I imagine it works just as well on animated scenes as static ones.
    - esperent 2y ago
      Well, technically cloth simulation and cloth rendering are separate. But in real-time rendering the terms are often (generally, even) used interchangeably - it's rare that you'd just want static cloth without movement - so at the very least I'd expect a sentence or two clarifying that they don't do any movement simulation here, it's a pure light transport technique.
- brookst 2y ago
  Yeah it’s kind of like a FPGA implementation: it proves the concept but is by definition not optimized for efficiency. There must be a way to go from the weights to an algorithm.
CuriousCosmic 2y ago
> As for NeuMIP, it can produce results with a higher quality than ours in terms of MSE, at the cost of a higher storage and per-material training. > Furthermore, our network can be applied to three typical types of woven fabrics once trained, but NeuMIP have to be trained per material which takes a long time > Specifically, we introduce a simple encoder-decoder structure, where the encoder compresses the fabric pattern and other parameters into a material latent vector, and the decoder interprets the material latent vector with a spatial fusion component and a small angular decoder. Thanks to the lightweight encoder and decoder, our network is able to achieve real-time rendering and editing. Meanwhile, our network only occupies a small storage of 5 MB, even for scenes with multiple fabric materials. Yeah. This is really just a simple solution that can be quickly trained and that compresses the problem space better at the cost of some fidelity.
the8472 2y ago
NNs/learning/function approximation are compression.
- johanvts 2y ago
  I think more often it’s about translation. From image to classification ex. with no hope of mapping back.
  - the8472 2y ago
    The classifier still has to learn a model, which is a lossy, compressed representation of the world. Classifiers can be very lossy. The outputs from Google's Deepdream are the kind of things that a classifier would "uncompress" to.
- nightski 2y ago
  Yes, but not solely. They are also generalization.
  - plorg 2y ago
    If you let your eyes roll to the back of your head and let the spice take over generalization and compression are two sides of the same coin.
    - nightski 2y ago
      There are plenty of compression methods that have zero generalization. There are also methods of generalization that do not compress at all. It seems to me that they are orthogonal concerns, even if they may make sense to use together.
RandomThoughts3 2y ago
Everything NN-related is compress and decompress if you squint hard enough. You are trying to map inputs to outputs in an efficient way space-wise or time-wise. The NN is doing something which allows it to skip complex calculations at the expense of fidelity in a way which is not far from a more advanced look up table. The fact that what it does is so opaque remains the main flaw of it.
- taylorius 2y ago
  "The fact that what it does is so opaque remains the main flaw of it." That, and the fact that it takes rather a lot of numerical calculations to do it.

Anotheroneagain 2y ago

I believe that results like these show that our algorithms are woefully inadequate, and something much faster has to exist, if a NN can do so much better with all the overhead.

Arwill 2y ago
Light/material calculation in shaders requires exponentiation, which is a costly operation. If a clever lookup table can get around that, it sure can be faster.
- magicalhippo 2y ago
  Using a lookup table[1] or other approximations like wavelets[2] for complex BRDFs is a well-known strategy. [1]: https://github.com/HectorMF/BRDFGenerator https://github.com/HectorMF/BRDFGenerator [2]: https://hal.science/hal-01518547/document https://hal.science/hal-01518547/document
esperent 2y ago
This isn't efficient at all. It requires a 3090 to get 60fps - and presumably it requires all the GPU's power to do that for the simple scene in the article: a couple of sheets and cushions on a sofa. Possibly it's RAM rather than compute limited but still. You couldn't use that in a game. Existing techniques like PBD or verlet integration would be far more efficient. Maybe on future devices with separate neural processor units this could be useful. But even then I bet those same chips could run standard real-time cloth algorithms really really fast too. Edit: I downloaded their video and there's no movement at all, only still images. Maybe I misunderstood the purpose of the technique. But I re-read the article and it certainly seems like they're claiming to do full simulation, not just static scenes. That scene with the sofa isn't even getting 60fps! It kinda feels like a disingenuous article after watching the video.
- zokier 2y ago
  I don't see how something like PBD helps at all with rendering.. are you thinking that this work relates somehow to physics simulation? Considering that the result is pretty much just a shader, I imagine it works just as well on animated scenes as static ones.
  - esperent 2y ago
    Well, technically cloth simulation and cloth rendering are separate. But in real-time rendering the terms are often (generally, even) used interchangeably - it's rare that you'd just want static cloth without movement - so at the very least I'd expect a sentence or two clarifying that they don't do any movement simulation here, it's a pure light transport technique.
brookst 2y ago
Yeah it’s kind of like a FPGA implementation: it proves the concept but is by definition not optimized for efficiency. There must be a way to go from the weights to an algorithm.

CuriousCosmic 2y ago

> As for NeuMIP, it can produce results with a higher quality than ours in terms of MSE, at the cost of a higher storage and per-material training. > Furthermore, our network can be applied to three typical types of woven fabrics once trained, but NeuMIP have to be trained per material which takes a long time > Specifically, we introduce a simple encoder-decoder structure, where the encoder compresses the fabric pattern and other parameters into a material latent vector, and the decoder interprets the material latent vector with a spatial fusion component and a small angular decoder. Thanks to the lightweight encoder and decoder, our network is able to achieve real-time rendering and editing. Meanwhile, our network only occupies a small storage of 5 MB, even for scenes with multiple fabric materials. Yeah. This is really just a simple solution that can be quickly trained and that compresses the problem space better at the cost of some fidelity.

the8472 2y ago

NNs/learning/function approximation are compression.

johanvts 2y ago
I think more often it’s about translation. From image to classification ex. with no hope of mapping back.
- the8472 2y ago
  The classifier still has to learn a model, which is a lossy, compressed representation of the world. Classifiers can be very lossy. The outputs from Google's Deepdream are the kind of things that a classifier would "uncompress" to.
nightski 2y ago
Yes, but not solely. They are also generalization.
- plorg 2y ago
  If you let your eyes roll to the back of your head and let the spice take over generalization and compression are two sides of the same coin.
  - nightski 2y ago
    There are plenty of compression methods that have zero generalization. There are also methods of generalization that do not compress at all. It seems to me that they are orthogonal concerns, even if they may make sense to use together.

RandomThoughts3 2y ago

Everything NN-related is compress and decompress if you squint hard enough. You are trying to map inputs to outputs in an efficient way space-wise or time-wise. The NN is doing something which allows it to skip complex calculations at the expense of fidelity in a way which is not far from a more advanced look up table. The fact that what it does is so opaque remains the main flaw of it.

taylorius 2y ago
"The fact that what it does is so opaque remains the main flaw of it." That, and the fact that it takes rather a lot of numerical calculations to do it.