With customized model u can do that. Models like MobileNet/SqueezeNet are designed specifically for fast inference on constrained device. Although during infere

With customized model u can do that. Models like MobileNet/SqueezeNet are designed specifically for fast inference on constrained device. Although during inference, the precision of your model can be quantized as low as single digit bits. This is an active area of research, but a lot of exciting stuff is happening and the progress made is massive.