4 ms·

Time doesn't necessarily have to be another dimension. They're just what they are but organized in time.

maxbond 3y ago
You don't have to think of it as a dimension if that obfuscates the problem or isn't a useful mental model for you, but "organizing" it is what a dimension does. You can think of an image as colors or intensities "organized by" (I would call it "indexed by") width and height. If you work with videos in machine learning, you generally accept a 6 dimensional tensor (width, height, time, red, green, blue - your order may vary, time often comes first. And you may use grayscale instead of color to reduce the number of dimensions.).
- DaiPlusPlus 3y ago
  > If you work with videos in machine learning, you generally accept a 6 dimensional tensor (width, height, time, red, green, blue (Assuming you do work with ML+videos) - it's surprising to hear you say you work with RGB instead of YUV - can you briefly explain how that's the case? I'd have thought that using luma/chroma separation would be much easier to work with (not just with traditional video tooling, but ML/NNs/etc themselves would have an easier time consuming it.
  - maxbond 3y ago
    To clarify, I don't work professionally with videos, I've hacked on some projects and read some books about it. My professional experience with ML models is in writing backends to integrate with them, the models I've designed/trained were for my own education (so far, at least). The answer to your question is probably, "I'm a dilettante who doesn't know better, you may well know more than me." I take the impression that much of the time, color doesn't provide much signal and gives your model things to overfit on, so you collapse it down to grayscale. (Which is to say, most of the time you care about shape, but you don't care about color.) But I bet there are problem spaces where your intuition holds, I'm sure that there's performance the be wrung out of a model by experimenting with different color spaces who's geometry might separate samples nicely. I did something similarish a few months ago where I used LDA[1] to create a boutique grayscale model where the intensity was correlated to the classification problem at hand, rather than the luminosity of the subject. It worked better than I'd have guessed, just on it's own (though I suspect it wouldn't work very well for most problems). But the idea was to preprocess the frames of the video this way and then feed it into a CNN [2]. (Why not a transformer? Because I was still wrapping my mind around simpler architectures.) [1] https://en.wikipedia.org/wiki/Linear_discriminant_analysis https://en.wikipedia.org/wiki/Linear_discriminant_analysis [2] https://en.wikipedia.org/wiki/Convolutional_neural_network https://en.wikipedia.org/wiki/Convolutional_neural_network

You don't have to think of it as a dimension if that obfuscates the problem or isn't a useful mental model for you, but "organizing" it is what a dimension does. You can think of an image as colors or intensities "organized by" (I would call it "indexed by") width and height. If you work with videos in machine learning, you generally accept a 6 dimensional tensor (width, height, time, red, green, blue - your order may vary, time often comes first. And you may use grayscale instead of color to reduce the number of dimensions.).

DaiPlusPlus 3y ago
> If you work with videos in machine learning, you generally accept a 6 dimensional tensor (width, height, time, red, green, blue (Assuming you do work with ML+videos) - it's surprising to hear you say you work with RGB instead of YUV - can you briefly explain how that's the case? I'd have thought that using luma/chroma separation would be much easier to work with (not just with traditional video tooling, but ML/NNs/etc themselves would have an easier time consuming it.
- maxbond 3y ago
  To clarify, I don't work professionally with videos, I've hacked on some projects and read some books about it. My professional experience with ML models is in writing backends to integrate with them, the models I've designed/trained were for my own education (so far, at least). The answer to your question is probably, "I'm a dilettante who doesn't know better, you may well know more than me." I take the impression that much of the time, color doesn't provide much signal and gives your model things to overfit on, so you collapse it down to grayscale. (Which is to say, most of the time you care about shape, but you don't care about color.) But I bet there are problem spaces where your intuition holds, I'm sure that there's performance the be wrung out of a model by experimenting with different color spaces who's geometry might separate samples nicely. I did something similarish a few months ago where I used LDA[1] to create a boutique grayscale model where the intensity was correlated to the classification problem at hand, rather than the luminosity of the subject. It worked better than I'd have guessed, just on it's own (though I suspect it wouldn't work very well for most problems). But the idea was to preprocess the frames of the video this way and then feed it into a CNN [2]. (Why not a transformer? Because I was still wrapping my mind around simpler architectures.) [1] https://en.wikipedia.org/wiki/Linear_discriminant_analysis https://en.wikipedia.org/wiki/Linear_discriminant_analysis [2] https://en.wikipedia.org/wiki/Convolutional_neural_network https://en.wikipedia.org/wiki/Convolutional_neural_network