If this is the case, and I believe you, then this implies: "bytes are NOT all you need". You need a good representation of your image for the network to work we

If this is the case, and I believe you, then this implies: "bytes are NOT all you need". You need a good representation of your image for the network to work well. It might so happen that one of the popular formats, maybe JPEG does a good job. But a dedicated ML person could do better most likely. This is probably what you would classically call "feature engineering"? And this is all hyperparameter search stuff in a way.