I think in general, "training" a GP, i.e. doing GP inference (or kernel regression) is not done for speed reasons, but rather because they are sample efficient.

I think in general, "training" a GP, i.e. doing GP inference (or kernel regression) is not done for speed reasons, but rather because they are sample efficient. More concretely, the practical folk wisdom regarding GPs is that when there are not much data, then GP inference with a well-chosen kernel can give you much more bang for the buck than a neural network. However, when there are a lot of data (especially in the perceptual domains like vision and language), neural networks typically train faster and generalize better. I wouldn't say I'm an expert at using GPs, so actual GP practitioners feel free to correct me if I'm wrong :)