8 ms·

For DL the colorspace is irrelevant as long as it's consistent between training and inference. I.e. if you train your model in RGB, it better be receiving RGB a

For DL the colorspace is irrelevant as long as it's consistent between training and inference. I.e. if you train your model in RGB, it better be receiving RGB and not something else from the video feed or input images. Interestingly, models can also be trained in YUV (including its downsampled variants like YUYV), which can make them a tiny bit more efficient, both because there's less data, and because most camera chips can output YUV raw data directly. The net will learn whatever transform it needs to learn to make sense of the data.

taeric 8y ago
I don't think you can fully claim it is irrelevant. Especially not and point out that it can make things a bit more efficient. I think it is safe to say that the most important part is to be consistent between what you train on and what you infer against. However, it could be that one of the color spaces is more amenable to training. To my knowledge, that has not been completely ruled out. Indeed, it seems supported by your claim that they can get a tiny bit more efficient with some spaces. That said, this post is my long winded way of asking if you have links on that. :)
- h4b4n3r0 8y ago
  What makes YUYV more efficient is downsampling. You represent 2 pixels with 4 bytes instead of 6 due to the loss of spatial color information. If there's no downsampling, and there exists a transform between RGB and a given color space, your network will simply learn slightly differently. Remember that after the very first layer there's no notion of color space anymore.
  - taeric 8y ago
    Intuitively, that makes sense. However, I would still feel more comfortable to have it somewhat proven. Most intuition I have is out the window with deep learning.
dosshell 8y ago
Interessting. Why is not the "raw" sensor format, for example Bayer BG8, used more in deeplearning? Would it not contain a little bit more correct information? and if used, we could skip the Bayer-> RGB conversion completly.
- ovi256 8y ago
  Because most models are trained from files, which store in RGB. Using raw sensor format would make sense for inference, but if your model is created for RGB, why bother.
  - h4b4n3r0 8y ago
    Pretty much. And also, models almost never operate on full-size image, and if you e.g. pixel bin a bayer image you will get a de-mosaiced image, so that's not very interesting. What is interesting is, it'd be pretty cool to train on linear HDR images for instance (i.e. direct 14-stop sensor output without contrast curves and color correction), or on visible spectrum + IR + UV, but I'm not aware of any large public datasets like this. The reason why this is interesting is because a lot of vision models suffer from the lack of robustness. Their performance on the train/validation datasets might be OK, but connect them to a real live video feed and they turn to shit. To some extent this can be addressed by data augmentation during training, but you have to understand that this is an imperfect process: you're operating on the data that's already pre-cooked, so to speak, for _human_ consumption, with color correction, gamma curves, etc already applied, and there's not a whole lot you can do to make things more robust to different lighting conditions, noise in the feed when camera bumps the gain in its ADCs, color drift due to e.g. sodium lamps, over/under exposure, etc. Unsurprisingly, this is especially noticeable in efficient, quantized models, because they don't really have much leeway built in to work around the deficiencies in the input data.
    - ovi256 8y ago
      Regarding multi-spectral images, there was a recent Kaggle competition classifying ship-or-iceberg from radar multiple band data. So yeah, CNNs work well for that too. https://www.kaggle.com/c/statoil-iceberg-classifier-challenge https://www.kaggle.com/c/statoil-iceberg-classifier-challeng...
gersh 8y ago
Not quite true. Algorithms like vgg-16 typically involve subtracting out the mean at the beginning for better performance. For example, look at https://pytorch.org/docs/stable/torchvision/models.html https://pytorch.org/docs/stable/torchvision/models.html. Red has mean .486 and blue has mean .406. While this could be calculated from the dataset, it does seem like a lot of the frameworks like to hardcode it.

taeric 8y ago

I don't think you can fully claim it is irrelevant. Especially not and point out that it can make things a bit more efficient. I think it is safe to say that the most important part is to be consistent between what you train on and what you infer against. However, it could be that one of the color spaces is more amenable to training. To my knowledge, that has not been completely ruled out. Indeed, it seems supported by your claim that they can get a tiny bit more efficient with some spaces. That said, this post is my long winded way of asking if you have links on that. :)

h4b4n3r0 8y ago
What makes YUYV more efficient is downsampling. You represent 2 pixels with 4 bytes instead of 6 due to the loss of spatial color information. If there's no downsampling, and there exists a transform between RGB and a given color space, your network will simply learn slightly differently. Remember that after the very first layer there's no notion of color space anymore.
- taeric 8y ago
  Intuitively, that makes sense. However, I would still feel more comfortable to have it somewhat proven. Most intuition I have is out the window with deep learning.

dosshell 8y ago

Interessting. Why is not the "raw" sensor format, for example Bayer BG8, used more in deeplearning? Would it not contain a little bit more correct information? and if used, we could skip the Bayer-> RGB conversion completly.

ovi256 8y ago
Because most models are trained from files, which store in RGB. Using raw sensor format would make sense for inference, but if your model is created for RGB, why bother.
- h4b4n3r0 8y ago
  Pretty much. And also, models almost never operate on full-size image, and if you e.g. pixel bin a bayer image you will get a de-mosaiced image, so that's not very interesting. What is interesting is, it'd be pretty cool to train on linear HDR images for instance (i.e. direct 14-stop sensor output without contrast curves and color correction), or on visible spectrum + IR + UV, but I'm not aware of any large public datasets like this. The reason why this is interesting is because a lot of vision models suffer from the lack of robustness. Their performance on the train/validation datasets might be OK, but connect them to a real live video feed and they turn to shit. To some extent this can be addressed by data augmentation during training, but you have to understand that this is an imperfect process: you're operating on the data that's already pre-cooked, so to speak, for _human_ consumption, with color correction, gamma curves, etc already applied, and there's not a whole lot you can do to make things more robust to different lighting conditions, noise in the feed when camera bumps the gain in its ADCs, color drift due to e.g. sodium lamps, over/under exposure, etc. Unsurprisingly, this is especially noticeable in efficient, quantized models, because they don't really have much leeway built in to work around the deficiencies in the input data.
  - ovi256 8y ago
    Regarding multi-spectral images, there was a recent Kaggle competition classifying ship-or-iceberg from radar multiple band data. So yeah, CNNs work well for that too. https://www.kaggle.com/c/statoil-iceberg-classifier-challenge https://www.kaggle.com/c/statoil-iceberg-classifier-challeng...

gersh 8y ago

Not quite true. Algorithms like vgg-16 typically involve subtracting out the mean at the beginning for better performance. For example, look at https://pytorch.org/docs/stable/torchvision/models.html https://pytorch.org/docs/stable/torchvision/models.html. Red has mean .486 and blue has mean .406. While this could be calculated from the dataset, it does seem like a lot of the frameworks like to hardcode it.