4 ms·

I'm assuming these 1TiB textures are procedural generated or composites? Where do this large of textures come up?

aprdm 6y ago
Can be either. You usually have digital artists creating them. https://en.wikipedia.org/wiki/Texture_artist https://en.wikipedia.org/wiki/Texture_artist
- CyberDildonics 6y ago
  Texture artists aren't painting 1 terabyte textures dude.
  - forelle2 6y ago
    The largest texture sets are heading towards 1TB in size, or at least they were when I was last involved in production support. I saw Mari projects north of 650gb and that was 5 years ago. Disclaimer : I wrote Mari, the vfx industry standard painting system. Note though these are not single 1TB textures, they’re multiple sets of textures, plus all of the layers that constitute them. Some large robots In particular had 65k 4K textures if you count the layers.
    - CyberDildonics 6y ago
      I think we both realize that it's a bit silly to have so much data in textures that you have 100x the pixel data of a 5 second shot at 4k with 32 bit float rgb. 650GB of textures would mean that even with 10gb ethernet (which I'm not sure is common yet) you would wait at least 12 minutes just for the textures to get to the computer before rendering could start and rendering 100 frames at a time would mean 100GB/s from a file server for a single shot. Even a single copy of the textures to freeze an iteration would be thousands in expensive disk space. I know it doesn't makes sense to tell your clients that what they are doing is nonsense, but if I saw something like that going on, the first thing I would do is chase down why it happened. Massive waste like that is extremely problematic while needing to make a sharper texture for some tiny piece that gets close to the camera is not a big deal.
      - forelle2 6y ago
        Texture caching in modern renders tends to be on demand and paged so it is very unlikely the full texture set is ever pulled from the filers. Over texturing like this can be a good decision depending on the production. Asset creation often starts a long time before shots or cameras are locked down. If you don’t know how an asset is to be used it makes sense to texture all of it upfront as if it will be full screen, 4K. Taking an asset off final to ‘Upres’ it for a can be a pain in the ass and more costly than just detailing it up in the first place. In isolation it’s a insane amount of detail and given perfect production planning it is normally not needed, but until directors lock down the scripts and shots it can be the simplest option.
        CyberDildonics 6y ago
        > Texture caching in modern renders tends to be on demand and paged so it is very unlikely the full texture set is ever pulled from the filers. This was easier to rely on in the days before ray tracing, when texture filtering was consistent because everything was from the camera. Ray differentials from incoherent rays aren't quite as forgiving. > If you don’t know how an asset is to be used it makes sense to texture all of it upfront as if it will be full screen, 4K. 4k textures for large parts of the asset in the UV layout can be an acceptable amount of overkill. That's not the same as putting 65,000 4k textures on something because each little part is given its own 4k texture. I know that you know this, but I'm not sure why you would conflate those two things. > Taking an asset off final to ‘Upres’ it for a can be a pain in the ass and more costly than just detailing it up in the first place It is very rare that specific textures need to be redone like that and it is not a big deal. 650GB of textures for one asset drags everything from iterations to final renders to disk usage to disk activity to network usage down for every shot in a completely unnecessary way. There isn't a fine line between these things, there is a giant gap between that much excessive texture resolution and needing to upres some piece because it gets close to the camera. > Asset creation often starts a long time before shots or cameras are locked down. This is actually fairly rare. > In isolation it’s a insane amount of detail and given perfect production planning it is normally not needed, but until directors lock down the scripts and shots it can be the simplest option. That's rarely how the time line fits together. It's irrelevant though, because there is no world where 65,000 4k textures on a single asset makes sense. It's multiple orders of magnitude out of bounds of reality. I am glad that you have that insane amount of scalability as a focus since you are making tools that people rely on heavily, and I wish way more people on the tools end thought like this. Still, it is about 1000x what would set off red flags in my mind. I apologize on behalf of whoever told you that was necessary, because they need to learn how to work within reasonable resources (which is not difficult given modern computers), no matter what project or organization they are attached to.
_3r2w 6y ago
1 terabyte sounds like an outlier, but typically texture maps are used as inputs to shading calculations. So it's not uncommon for hero assets in large-scale VFX movies to have more than 10 different sets of texture files that represent different portions of a shading model. For large assets, it may take more than fifty 4K-16K images to adequately cover the entire model such that if you were to render it from any angle, you wouldn't see the pixelation. And these textures are often stored as mipmapped 16 bit images so the renderer can choose the most optimal resolution at rendertime. So that can easily end up being several hundred gigabytes of source image data. At rendertime, only the textures that are needed to render what's visible in the camera are loaded into memory and utilized, which typically ends up being a fraction of the source data. Large scale terrains and environments typically make more use of procedural textures, and they may be cached temporarily in memory while the rendering process happens to speed up calculations
CyberDildonics 6y ago
I would take that with a huge grain of salt. Typically the only thing that would be a full terabyte is a full resolution water simulation for an entire shot. I'm unconvinced that is actually necessary, but it does happen. An entire movie at 2k, uncompressed floating point rgb would be about 4 terabytes.

Can be either. You usually have digital artists creating them. https://en.wikipedia.org/wiki/Texture_artist https://en.wikipedia.org/wiki/Texture_artist

CyberDildonics 6y ago
Texture artists aren't painting 1 terabyte textures dude.
- forelle2 6y ago
  The largest texture sets are heading towards 1TB in size, or at least they were when I was last involved in production support. I saw Mari projects north of 650gb and that was 5 years ago. Disclaimer : I wrote Mari, the vfx industry standard painting system. Note though these are not single 1TB textures, they’re multiple sets of textures, plus all of the layers that constitute them. Some large robots In particular had 65k 4K textures if you count the layers.
  - CyberDildonics 6y ago
    I think we both realize that it's a bit silly to have so much data in textures that you have 100x the pixel data of a 5 second shot at 4k with 32 bit float rgb. 650GB of textures would mean that even with 10gb ethernet (which I'm not sure is common yet) you would wait at least 12 minutes just for the textures to get to the computer before rendering could start and rendering 100 frames at a time would mean 100GB/s from a file server for a single shot. Even a single copy of the textures to freeze an iteration would be thousands in expensive disk space. I know it doesn't makes sense to tell your clients that what they are doing is nonsense, but if I saw something like that going on, the first thing I would do is chase down why it happened. Massive waste like that is extremely problematic while needing to make a sharper texture for some tiny piece that gets close to the camera is not a big deal.
    - forelle2 6y ago
      Texture caching in modern renders tends to be on demand and paged so it is very unlikely the full texture set is ever pulled from the filers. Over texturing like this can be a good decision depending on the production. Asset creation often starts a long time before shots or cameras are locked down. If you don’t know how an asset is to be used it makes sense to texture all of it upfront as if it will be full screen, 4K. Taking an asset off final to ‘Upres’ it for a can be a pain in the ass and more costly than just detailing it up in the first place. In isolation it’s a insane amount of detail and given perfect production planning it is normally not needed, but until directors lock down the scripts and shots it can be the simplest option.
      - CyberDildonics 6y ago
        > Texture caching in modern renders tends to be on demand and paged so it is very unlikely the full texture set is ever pulled from the filers. This was easier to rely on in the days before ray tracing, when texture filtering was consistent because everything was from the camera. Ray differentials from incoherent rays aren't quite as forgiving. > If you don’t know how an asset is to be used it makes sense to texture all of it upfront as if it will be full screen, 4K. 4k textures for large parts of the asset in the UV layout can be an acceptable amount of overkill. That's not the same as putting 65,000 4k textures on something because each little part is given its own 4k texture. I know that you know this, but I'm not sure why you would conflate those two things. > Taking an asset off final to ‘Upres’ it for a can be a pain in the ass and more costly than just detailing it up in the first place It is very rare that specific textures need to be redone like that and it is not a big deal. 650GB of textures for one asset drags everything from iterations to final renders to disk usage to disk activity to network usage down for every shot in a completely unnecessary way. There isn't a fine line between these things, there is a giant gap between that much excessive texture resolution and needing to upres some piece because it gets close to the camera. > Asset creation often starts a long time before shots or cameras are locked down. This is actually fairly rare. > In isolation it’s a insane amount of detail and given perfect production planning it is normally not needed, but until directors lock down the scripts and shots it can be the simplest option. That's rarely how the time line fits together. It's irrelevant though, because there is no world where 65,000 4k textures on a single asset makes sense. It's multiple orders of magnitude out of bounds of reality. I am glad that you have that insane amount of scalability as a focus since you are making tools that people rely on heavily, and I wish way more people on the tools end thought like this. Still, it is about 1000x what would set off red flags in my mind. I apologize on behalf of whoever told you that was necessary, because they need to learn how to work within reasonable resources (which is not difficult given modern computers), no matter what project or organization they are attached to.

_3r2w 6y ago

1 terabyte sounds like an outlier, but typically texture maps are used as inputs to shading calculations. So it's not uncommon for hero assets in large-scale VFX movies to have more than 10 different sets of texture files that represent different portions of a shading model. For large assets, it may take more than fifty 4K-16K images to adequately cover the entire model such that if you were to render it from any angle, you wouldn't see the pixelation. And these textures are often stored as mipmapped 16 bit images so the renderer can choose the most optimal resolution at rendertime. So that can easily end up being several hundred gigabytes of source image data. At rendertime, only the textures that are needed to render what's visible in the camera are loaded into memory and utilized, which typically ends up being a fraction of the source data. Large scale terrains and environments typically make more use of procedural textures, and they may be cached temporarily in memory while the rendering process happens to speed up calculations

CyberDildonics 6y ago

I would take that with a huge grain of salt. Typically the only thing that would be a full terabyte is a full resolution water simulation for an entire shot. I'm unconvinced that is actually necessary, but it does happen. An entire movie at 2k, uncompressed floating point rgb would be about 4 terabytes.