3 ms·

I wondered something like this—would it be better to train a CNN in say YUV color space? But if you consider that a NN approximates any function, if using YUV p

I wondered something like this—would it be better to train a CNN in say YUV color space? But if you consider that a NN approximates any function, if using YUV performed better then the network would learn to convert RGB to YUV itself. (It’s a simple linear relationship that would take a single layer.) I then found a paper that confirmed that converting input to different color spaces did not cause significant differences in performance.

jiggawatts 3y ago
Similarly I wondered if partially decompressing video and using that format as both the input and output might work. The logic is that a fully decompressed video is huge, and that extra data is by definition wasteful: it’s exactly what’s thrown away by compression! We’ve designed compression to efficiently match the human visual system and not waste bytes on irrelevant things. So I wonder if a NN trained on something like a quantised DCT as both input and output might be dramatically more efficient, roughly in line with the compression ratio of applying the same transforms to a raw video. Obviously we’d have to avoid bit-level streaming algorithms like Huffman coding. However even reusing image tiles might work via methods such as differentiable hash tables, as seen in NVIDIA’s reverse rendering neural nets! Food for thought…