Inference speed and net size is definitely considered important and some researchers are definitely tackling it (e.g https://arxiv.org/abs/1602.07360 https://ar

Inference speed and net size is definitely considered important and some researchers are definitely tackling it (e.g https://arxiv.org/abs/1602.07360 https://arxiv.org/abs/1602.07360). However it's more of an engineering area (meaning lots of engineers are working on it) and less of a research area (meaning fewer papers get produced per work) because there's not much in the way of performance-per-gflop benchmarks or whatever. CPU performance is good-ish when you're doing sequential stuff e.g it's not bad at old-school RNNs, but for image recognition using a cheap graphics unit is still going to be better, and if the question is "can I use cheaper hardware and get good performance" the answer is yes, you just have to do some tricks (e.g distillation). Plenty of image recognition projects on the cheap raspberry pi and its graphics unit.