3 ms·

This seems like a downgrade. Intuition suggests it should lead to better performance if you first parse/process the input to represent the actual input space be

This seems like a downgrade. Intuition suggests it should lead to better performance if you first parse/process the input to represent the actual input space better.

refulgentis 3y ago
You'd think this, but, RGB color is ~meaningless and all the image stuff works great. I long for a model that is trained in a perceptual space, and yet, I doubt it'll matter
- wizzwizz4 3y ago
  It's not faithfully representative of the post-processing performed by the average person's visual system, but that doesn't make it meaningless by any stretch of the imagination.
  - refulgentis 3y ago
    Intuitively, it’s meaningless. What’s brighter, 255 G or 255 R?
    - wizzwizz4 3y ago
      Depends on the picture. https://en.wikipedia.org/wiki/Checker_shadow_illusion https://en.wikipedia.org/wiki/Checker_shadow_illusion
      - refulgentis 3y ago
        That's called viewing conditions in color science: it's sort of like saying "in a pitch black room, the same!" Let's assume the same viewing conditions, i.e it's not a trick question: answer is green, by about 30% IIRC
    - majormajor 3y ago
      That seems much more like "incomplete" than "meaningless." There's enough meaning to answer this question, no: "what's greener, 255 G or 255 R"? Or "what's greener, #FF6600 or #FF0066?"
      - refulgentis 3y ago
        what does it mean to have "more green"? what is this "green"?
    - astrange 3y ago
      Green is brighter in all typical colorspaces. That's why it's the larget contribution to Y in the RGB->YUV conversion matrix.
rgovostes 3y ago
I wondered something like this—would it be better to train a CNN in say YUV color space? But if you consider that a NN approximates any function, if using YUV performed better then the network would learn to convert RGB to YUV itself. (It’s a simple linear relationship that would take a single layer.) I then found a paper that confirmed that converting input to different color spaces did not cause significant differences in performance.
- jiggawatts 3y ago
  Similarly I wondered if partially decompressing video and using that format as both the input and output might work. The logic is that a fully decompressed video is huge, and that extra data is by definition wasteful: it’s exactly what’s thrown away by compression! We’ve designed compression to efficiently match the human visual system and not waste bytes on irrelevant things. So I wonder if a NN trained on something like a quantised DCT as both input and output might be dramatically more efficient, roughly in line with the compression ratio of applying the same transforms to a raw video. Obviously we’d have to avoid bit-level streaming algorithms like Huffman coding. However even reusing image tiles might work via methods such as differentiable hash tables, as seen in NVIDIA’s reverse rendering neural nets! Food for thought…
ftxbro 3y ago
Yes that's the intuition, but there's an idea called 'the bitter lesson' by Richard Sutton where every 'feature engineering' is swamped and overtaken by the raw power of stacking more layers with increased parameters and exaflops and dataset sizes.
- mjburgess 3y ago
  All that's really happening there is a brute-force search for engineered features. Such brute-forcing has a very very low ROI when the necessarily huge datasets don't already exist. Indeed, creating the experimental conditions to collect that data often means engaging in some form of imagined feature engineering.

This seems like a downgrade. Intuition suggests it should lead to better performance if you first parse/process the input to represent the actual input space better.

refulgentis 3y ago

You'd think this, but, RGB color is ~meaningless and all the image stuff works great. I long for a model that is trained in a perceptual space, and yet, I doubt it'll matter

wizzwizz4 3y ago
It's not faithfully representative of the post-processing performed by the average person's visual system, but that doesn't make it meaningless by any stretch of the imagination.
- refulgentis 3y ago
  Intuitively, it’s meaningless. What’s brighter, 255 G or 255 R?
  - wizzwizz4 3y ago
    Depends on the picture. https://en.wikipedia.org/wiki/Checker_shadow_illusion https://en.wikipedia.org/wiki/Checker_shadow_illusion
    - refulgentis 3y ago
      That's called viewing conditions in color science: it's sort of like saying "in a pitch black room, the same!" Let's assume the same viewing conditions, i.e it's not a trick question: answer is green, by about 30% IIRC
  - majormajor 3y ago
    That seems much more like "incomplete" than "meaningless." There's enough meaning to answer this question, no: "what's greener, 255 G or 255 R"? Or "what's greener, #FF6600 or #FF0066?"
    - refulgentis 3y ago
      what does it mean to have "more green"? what is this "green"?
  - astrange 3y ago
    Green is brighter in all typical colorspaces. That's why it's the larget contribution to Y in the RGB->YUV conversion matrix.

rgovostes 3y ago

I wondered something like this—would it be better to train a CNN in say YUV color space? But if you consider that a NN approximates any function, if using YUV performed better then the network would learn to convert RGB to YUV itself. (It’s a simple linear relationship that would take a single layer.) I then found a paper that confirmed that converting input to different color spaces did not cause significant differences in performance.

jiggawatts 3y ago
Similarly I wondered if partially decompressing video and using that format as both the input and output might work. The logic is that a fully decompressed video is huge, and that extra data is by definition wasteful: it’s exactly what’s thrown away by compression! We’ve designed compression to efficiently match the human visual system and not waste bytes on irrelevant things. So I wonder if a NN trained on something like a quantised DCT as both input and output might be dramatically more efficient, roughly in line with the compression ratio of applying the same transforms to a raw video. Obviously we’d have to avoid bit-level streaming algorithms like Huffman coding. However even reusing image tiles might work via methods such as differentiable hash tables, as seen in NVIDIA’s reverse rendering neural nets! Food for thought…

ftxbro 3y ago

Yes that's the intuition, but there's an idea called 'the bitter lesson' by Richard Sutton where every 'feature engineering' is swamped and overtaken by the raw power of stacking more layers with increased parameters and exaflops and dataset sizes.

mjburgess 3y ago
All that's really happening there is a brute-force search for engineered features. Such brute-forcing has a very very low ROI when the necessarily huge datasets don't already exist. Indeed, creating the experimental conditions to collect that data often means engaging in some form of imagined feature engineering.