I think these models will drop in requirements pretty dramatically - there is a lot of work in finding sparse subsets of NNs that actually do all the work for e

I think these models will drop in requirements pretty dramatically - there is a lot of work in finding sparse subsets of NNs that actually do all the work for example, and embedded hardware is getting a big bump in neural network inference due to massively parallel FMA blocks being added to their DSPs. Also remember that the majority of academic work is done based on benchmarks that don't include inference time or hardware requirements. As these things start becoming useful without these restrictions industry will start tweaking the cost functions that are being minimised to make it useful in the real world too.