4 ms·

> For example, the camera on the iPhone is different than the camera on the iPad, so you may want to create two versions of a model and send one to iPhone users

> For example, the camera on the iPhone is different than the camera on the iPad, so you may want to create two versions of a model and send one to iPhone users of the app and the other to iPad users. Are app developers shipping models that are so brittle they cannot handle a different revision of Apple's camera? I can understand shipping more complex models for devices with better CPU/GPU or whatever Apple's AI accelerator is called, but not different cameras!

janhenr 6y ago
There might be some other reasons for shipping different models for the iPad vs the iPhone. F.e., if the iPad is more often used inside rather than outside, you could use a fine-tuned version of your big CNN to this smaller set of classes.
- Ar-Curunir 6y ago
  Another reason is that the more powerful iPad processor can handle larger networks
julvo 6y ago
Depends on the model. E.g. image enhancement or super resolution models are sensitive to the camera model and can be trained to fix artifacts introduced by specific cameras.
tomaskafka 6y ago
SE has one camera, 11 has two, iPad has two and LIDAR ...
sillysaurusx 6y ago
Image augmentations are hard to add to training. It may seem easy, but it requires a lot of thought. (To back up a bit: Image augmentations are how you solve that problem. "How do I make my model robust across different cameras?" It might be tempting to gather labeled data from a variety of cameras, but that doesn't necessarily result in a model that can handle newer, larger-res cameras. So one solution is to distort the training data with augmentations so that the model can't tell which resolution the input images are coming from.) The other way to deal with it is to just downscale the camera's image to, say, 416x416. But that introduces a question: can different cameras give images that look different when downscaled to 416x416? Sure they can! Cameras have a dizzying array of features, and they perform differently in different lighting conditions. To return to the point about image augmentations being hard to add: It's so easy to explain what your training code should do "Just distort the hue a bit" and there seem to be operations explicitly for that: https://www.tensorflow.org/api_docs/python/tf/image/adjust_hue https://www.tensorflow.org/api_docs/python/tf/image/adjust_h... but when you go to train with them, you'll discover that backpropagation isn't implemented, i.e. they break in training code. I've been trying to build an equivalent of Kornia for tensorflow https://github.com/kornia/kornia https://github.com/kornia/kornia which is a wonderful library that implements image augmentations using nothing but differentiable primitives. Work is a bit slow, but I hope to release it in Mel https://github.com/shawwn/mel https://github.com/shawwn/mel (which will hopefully look less like a TODO soon). But all of this still raises the question of which augmentations to add. Work in this area is ongoing; see Gwern's excellent writeup at https://github.com/tensorfork/tensorfork/issues/35 https://github.com/tensorfork/tensorfork/issues/35 Training a model per camera isn't necessarily a terrible idea, either. In the future I predict that we'll see more and more "on-demand" models: models that are JIT optimized for a target configuration (in this case, a specific camera). Robustness often comes at the cost of quality / accuracy (https://arxiv.org/abs/2006.14536 https://arxiv.org/abs/2006.14536 recently highlighted this). In situations where that last 2% of accuracy is crucial, there are all kinds of tricks; training separate models is but one of many.
- perturbation 6y ago
  > To return to the point about image augmentations being hard to add: It's so easy to explain what your training code should do "Just distort the hue a bit" and there seem to be operations explicitly for that: https://www.tensorflow.org/api_docs/python/tf/image/adjust_h.. https://www.tensorflow.org/api_docs/python/tf/image/adjust_h.... but when you go to train with them, you'll discover that backpropagation isn't implemented, i.e. they break in training code. Why not do the data augmentation during preprocessing (so that the transformations don't have to be done by differentiable transforms)? I.e., map over a tf.Dataset with the transformation (and append to the original dataset).
- spott 6y ago
  Why are you trying to backpropagate over data augmentations? I've never done that (or heard about it being done). Usually I just do the augmentations on the input samples and then feed the augmented samples to the network. Differentiable augmentations aren't necessary unless the augmentations are midstream (so you have to propagate parameters above the augmentations, which is weird) or have parameters (at which point you aren't learning how to work on different views of the same sample, you are learning how to modify a sample to be more learnable, which is a different problem that you are trying to solve). Don't get me wrong, augmenting samples to reduce device bias is a hard problem, but you might be making it harder than it needs to be.
  - gwern 6y ago
    The data augmentations we are interested in are in fact 'midstream', as they augment the examples before passing into the D or the classification loss but you must backprop from that back through the augmentation into the original model, because you don't want the augmentations to 'leak': the G is not supposed to generate augmented samples, the augmentation is there to regularize the D and reduce its ability to memorize real datapoints. It would probably be better to consider them as a kind of consistency or metric loss along the lines of SimCLR (which has helped inspire these very new GAN data augmentation techniques). It's a bit weird, which is perhaps why despite its simplicity (indicated by no less than 4 simultaneous inventions of it in the past few months), it hasn't been done before. You really should read the linked Github thread if you are interested.
Enginerrrd 6y ago
Fairly precise camera calibrations remain important in photogrammetry applications.
m463 6y ago
Training data pays attention to really weird things. Someone was telling me of ML cancer detection that was unexpectedly training on the ruler found in most images of cancer. I can see models based on an image sensor could inadvertently optimize for sensor size or geometry.

janhenr 6y ago

There might be some other reasons for shipping different models for the iPad vs the iPhone. F.e., if the iPad is more often used inside rather than outside, you could use a fine-tuned version of your big CNN to this smaller set of classes.

Ar-Curunir 6y ago
Another reason is that the more powerful iPad processor can handle larger networks

julvo 6y ago

Depends on the model. E.g. image enhancement or super resolution models are sensitive to the camera model and can be trained to fix artifacts introduced by specific cameras.

tomaskafka 6y ago

SE has one camera, 11 has two, iPad has two and LIDAR ...

sillysaurusx 6y ago

Image augmentations are hard to add to training. It may seem easy, but it requires a lot of thought. (To back up a bit: Image augmentations are how you solve that problem. "How do I make my model robust across different cameras?" It might be tempting to gather labeled data from a variety of cameras, but that doesn't necessarily result in a model that can handle newer, larger-res cameras. So one solution is to distort the training data with augmentations so that the model can't tell which resolution the input images are coming from.) The other way to deal with it is to just downscale the camera's image to, say, 416x416. But that introduces a question: can different cameras give images that look different when downscaled to 416x416? Sure they can! Cameras have a dizzying array of features, and they perform differently in different lighting conditions. To return to the point about image augmentations being hard to add: It's so easy to explain what your training code should do "Just distort the hue a bit" and there seem to be operations explicitly for that: https://www.tensorflow.org/api_docs/python/tf/image/adjust_hue https://www.tensorflow.org/api_docs/python/tf/image/adjust_h... but when you go to train with them, you'll discover that backpropagation isn't implemented, i.e. they break in training code. I've been trying to build an equivalent of Kornia for tensorflow https://github.com/kornia/kornia https://github.com/kornia/kornia which is a wonderful library that implements image augmentations using nothing but differentiable primitives. Work is a bit slow, but I hope to release it in Mel https://github.com/shawwn/mel https://github.com/shawwn/mel (which will hopefully look less like a TODO soon). But all of this still raises the question of which augmentations to add. Work in this area is ongoing; see Gwern's excellent writeup at https://github.com/tensorfork/tensorfork/issues/35 https://github.com/tensorfork/tensorfork/issues/35 Training a model per camera isn't necessarily a terrible idea, either. In the future I predict that we'll see more and more "on-demand" models: models that are JIT optimized for a target configuration (in this case, a specific camera). Robustness often comes at the cost of quality / accuracy (https://arxiv.org/abs/2006.14536 https://arxiv.org/abs/2006.14536 recently highlighted this). In situations where that last 2% of accuracy is crucial, there are all kinds of tricks; training separate models is but one of many.

perturbation 6y ago
> To return to the point about image augmentations being hard to add: It's so easy to explain what your training code should do "Just distort the hue a bit" and there seem to be operations explicitly for that: https://www.tensorflow.org/api_docs/python/tf/image/adjust_h.. https://www.tensorflow.org/api_docs/python/tf/image/adjust_h.... but when you go to train with them, you'll discover that backpropagation isn't implemented, i.e. they break in training code. Why not do the data augmentation during preprocessing (so that the transformations don't have to be done by differentiable transforms)? I.e., map over a tf.Dataset with the transformation (and append to the original dataset).
spott 6y ago
Why are you trying to backpropagate over data augmentations? I've never done that (or heard about it being done). Usually I just do the augmentations on the input samples and then feed the augmented samples to the network. Differentiable augmentations aren't necessary unless the augmentations are midstream (so you have to propagate parameters above the augmentations, which is weird) or have parameters (at which point you aren't learning how to work on different views of the same sample, you are learning how to modify a sample to be more learnable, which is a different problem that you are trying to solve). Don't get me wrong, augmenting samples to reduce device bias is a hard problem, but you might be making it harder than it needs to be.
- gwern 6y ago
  The data augmentations we are interested in are in fact 'midstream', as they augment the examples before passing into the D or the classification loss but you must backprop from that back through the augmentation into the original model, because you don't want the augmentations to 'leak': the G is not supposed to generate augmented samples, the augmentation is there to regularize the D and reduce its ability to memorize real datapoints. It would probably be better to consider them as a kind of consistency or metric loss along the lines of SimCLR (which has helped inspire these very new GAN data augmentation techniques). It's a bit weird, which is perhaps why despite its simplicity (indicated by no less than 4 simultaneous inventions of it in the past few months), it hasn't been done before. You really should read the linked Github thread if you are interested.

Enginerrrd 6y ago

Fairly precise camera calibrations remain important in photogrammetry applications.

m463 6y ago

Training data pays attention to really weird things. Someone was telling me of ML cancer detection that was unexpectedly training on the ruler found in most images of cancer. I can see models based on an image sensor could inadvertently optimize for sensor size or geometry.