6 ms·

Yes they are, because existing video codecs are based on 1980s signal processing ideas, and increasingly video starts out as being 3D models that are naturally

by epeus 13y ago

Yes they are, because existing video codecs are based on 1980s signal processing ideas, and increasingly video starts out as being 3D models that are naturally better suited to a trimesh and textures approach instead. I wrote about it here: http://epeus.blogspot.com/2013/05/finally-some-progress-in-video-codecs.html http://epeus.blogspot.com/2013/05/finally-some-progress-in-v...

hnha 13y ago
What are you talking about? Film is 2D. How do you imagine it to suddenly be "encoded in 3D"?
- KaiserPro 13y ago
  Using 3d space to represent movement of the textures. Currently a frame is split up into blocks, and they are moved about in 2d (X&Y). Using a 3d model of the movement would theoretically gain you some savings, as it would require less refreshing of the moving blocks.
  - epeus 13y ago
    Quite. You can define a textures+trimeshes+motion format and have the decoder be well defined, then have encoders compete to do better, as we have seen with the 2d codecs. Clearly starting form something in GL, like a game walkthrough (How much of youtube is that?) is easier than starting with Bladerunner. A Pixar movie would be somewhere inbetween.
- cromwellian 13y ago
  Object-based encoding has long been discussed within the MPEG community. MPEG-4 included VRML as a sort of hack for scene based compositing and Structured Audio Orchestration Language et al, as a parallel. I saw early whitepapers where they had planned even to do things like recognize people's faces, and deduce geometry, thereby separately compressing a person in the foreground vs background (use case: conference calls, news anchors, etc) As for whether it is possible to deduce 3D structure 2D, there's been some progress in that regard, check this out: http://www.youtube.com/watch?v=CZiSK7OMANw http://www.youtube.com/watch?v=CZiSK7OMANw
- deleted 13y ago
  [deleted]
  - jlgreco 13y ago
    Would companies that produce computer animated movies be willing to risk exposing their 3d/texture assets like that? If even low-quality "pre-rendered"^ versions of those fell into the hands of the community then those companies would basically have to deal with a bunch of people using their stuff (probably mostly fair-use covered parody stuff on youtube, but also high quality Chinese knockoff movies and the like.) ^ You would have to pre-render that stuff in some way first, wouldn't you? The actual source assets for those movies are absolutely massive and do not render in real time.
- epeus 13y ago
  The world is 3D. I bet the last few films you saw had extensive 3D modelled scenes in them. I bet all the video games you play have 3D cutscenes.
  - KaiserPro 13y ago
    This is true, but the are pre-rendered for a reason. Anything with water you're looking at 24hour+ sim time, plus render, and 1tb of data plus. SubSurface Scattering (translucency) takes a huge amount of CPU time.
  - jfb 13y ago
    The world is 3D. Video isn't.
KaiserPro 13y ago
Gleaning useful 3d models from 2d images is rather error prone. Hence why all the cool kids are using wavelets, as they are inherently more reliable on 2d images. Movie codecs are hard. However jpeg2000 and similar wavelet codecs (like Red's(those bashful camera people) REDcode, or dirac) can yield amazing compression ratios. The main problem is processing power. Take Red cameras for example. When they first came out, it was a massive pain. Unless you spunked $5k on a super unreliable pci card you'd only get a maximum of 6 fps out of a 8-core mac (and it could only be a mac as there was no linux SDK) With the advent of 3 generations of CPU and better programming skill from people like avid, aja and the foundry we can get real time playback on a laptop.
- epeus 13y ago
  Using wavelets for encoding the textures makes sense. The challenge with JPEG200 is that you need the decoder in the read loop to know when to stop getting more detail. The degenerate case would be 2 triangles and a texture, but if you have existing world models, decimating the meshes and de-rezzing the textures appropriates would work. Read m post, it's still a Graduate Student algorithm, but cut scenes on consoles are an existence proof.
- 7952 13y ago
  I understand that wavelet is amazing, but am a little sceptical about it being used outside of a few specialist areas. The general trend seems to be towards using increased storage and power to avoid compression (raw), rather than making files smaller.
  - KaiserPro 13y ago
    The thing that replaced 35mm with a digital format is DCP. Basically its Jpeg2000 in disguise. The problem with uncompressed formats at high resolution (like cinema 4k, and its smaller "UHD" cousin.) is that they eats bandwidth. Full res 4k images with basic RLE is 50megs. Thats 1.2 gigabyte a second. You're lucky to get that out of 10gig Ethernet. Uncompressed anything is pretty much out of the reach of most consumers. iTunes, netflix, over the air, cable is all either MPEG2 or h.264

by epeus 13y ago

hnha 13y ago

What are you talking about? Film is 2D. How do you imagine it to suddenly be "encoded in 3D"?

KaiserPro 13y ago
Using 3d space to represent movement of the textures. Currently a frame is split up into blocks, and they are moved about in 2d (X&Y). Using a 3d model of the movement would theoretically gain you some savings, as it would require less refreshing of the moving blocks.
- epeus 13y ago
  Quite. You can define a textures+trimeshes+motion format and have the decoder be well defined, then have encoders compete to do better, as we have seen with the 2d codecs. Clearly starting form something in GL, like a game walkthrough (How much of youtube is that?) is easier than starting with Bladerunner. A Pixar movie would be somewhere inbetween.
cromwellian 13y ago
Object-based encoding has long been discussed within the MPEG community. MPEG-4 included VRML as a sort of hack for scene based compositing and Structured Audio Orchestration Language et al, as a parallel. I saw early whitepapers where they had planned even to do things like recognize people's faces, and deduce geometry, thereby separately compressing a person in the foreground vs background (use case: conference calls, news anchors, etc) As for whether it is possible to deduce 3D structure 2D, there's been some progress in that regard, check this out: http://www.youtube.com/watch?v=CZiSK7OMANw http://www.youtube.com/watch?v=CZiSK7OMANw
deleted 13y ago
[deleted]
- jlgreco 13y ago
  Would companies that produce computer animated movies be willing to risk exposing their 3d/texture assets like that? If even low-quality "pre-rendered"^ versions of those fell into the hands of the community then those companies would basically have to deal with a bunch of people using their stuff (probably mostly fair-use covered parody stuff on youtube, but also high quality Chinese knockoff movies and the like.) ^ You would have to pre-render that stuff in some way first, wouldn't you? The actual source assets for those movies are absolutely massive and do not render in real time.
epeus 13y ago
The world is 3D. I bet the last few films you saw had extensive 3D modelled scenes in them. I bet all the video games you play have 3D cutscenes.
- KaiserPro 13y ago
  This is true, but the are pre-rendered for a reason. Anything with water you're looking at 24hour+ sim time, plus render, and 1tb of data plus. SubSurface Scattering (translucency) takes a huge amount of CPU time.
- jfb 13y ago
  The world is 3D. Video isn't.

KaiserPro 13y ago

Gleaning useful 3d models from 2d images is rather error prone. Hence why all the cool kids are using wavelets, as they are inherently more reliable on 2d images. Movie codecs are hard. However jpeg2000 and similar wavelet codecs (like Red's(those bashful camera people) REDcode, or dirac) can yield amazing compression ratios. The main problem is processing power. Take Red cameras for example. When they first came out, it was a massive pain. Unless you spunked $5k on a super unreliable pci card you'd only get a maximum of 6 fps out of a 8-core mac (and it could only be a mac as there was no linux SDK) With the advent of 3 generations of CPU and better programming skill from people like avid, aja and the foundry we can get real time playback on a laptop.

epeus 13y ago
Using wavelets for encoding the textures makes sense. The challenge with JPEG200 is that you need the decoder in the read loop to know when to stop getting more detail. The degenerate case would be 2 triangles and a texture, but if you have existing world models, decimating the meshes and de-rezzing the textures appropriates would work. Read m post, it's still a Graduate Student algorithm, but cut scenes on consoles are an existence proof.
7952 13y ago
I understand that wavelet is amazing, but am a little sceptical about it being used outside of a few specialist areas. The general trend seems to be towards using increased storage and power to avoid compression (raw), rather than making files smaller.
- KaiserPro 13y ago
  The thing that replaced 35mm with a digital format is DCP. Basically its Jpeg2000 in disguise. The problem with uncompressed formats at high resolution (like cinema 4k, and its smaller "UHD" cousin.) is that they eats bandwidth. Full res 4k images with basic RLE is 50megs. Thats 1.2 gigabyte a second. You're lucky to get that out of 10gig Ethernet. Uncompressed anything is pretty much out of the reach of most consumers. iTunes, netflix, over the air, cable is all either MPEG2 or h.264