My theory is that the latent space isn't perfectly 'disentangled' (there's not really anything in StyleGAN akin to InfoGAN which would force it to be exactly li

My theory is that the latent space isn't perfectly 'disentangled' (there's not really anything in StyleGAN akin to InfoGAN which would force it to be exactly linear/disentangled, it just nicely works out roughly disentangled), so when you train a linear model on the tags+latent and manipulate it linearly, you get slight changes in correlated features. Thus, the slight 'glitchiness'. Not something you'd notice in image samples, but more apparent when you generate interpolations of the 'same' face. If you mess with the style noise only and leave the latent vector alone, the glitchiness should be less, but you also lose a lot of possible control. Training a more powerful nonlinear classifier like a random forests should be able to compensate for the remaining nonlinearity in the latents, I think, but then it's harder to do the actual modification of the latent vector, of course, since you're no longer simply tweaking one variable but many simultaneously to try to hold the other features constant. One could try to brute force it by doing blackbox optimization of the nonlinear classifier... but I haven't seen anyone try that.