FHE does still add a huge overhead (measured in orders of magnitude, not percentages), so it would not be applicable for large models that are comparable with s

FHE does still add a huge overhead (measured in orders of magnitude, not percentages), so it would not be applicable for large models that are comparable with state of art models. You can apply FHE to neural networks in general (e.g. https://developers.googleblog.com/2023/08/expanding-our-fully-homomorphic-encryption-offering.html https://developers.googleblog.com/2023/08/expanding-our-full... lists it as one of potential use cases), but the unsaid assumption is that they are relatively small NNs, and that's in an era where we know that simply using a larger model brings meaningful improvements to the outcome, and the main limiting factor for people training their own models is the cost of compute.