Yah, I get that and it's a big shrug from me as to how far is too far. For example, "entangled" neurons have been shown to be a necessity to some degree when a

Yah, I get that and it's a big shrug from me as to how far is too far. For example, "entangled" neurons have been shown to be a necessity to some degree when applying an informational bottleneck, and that they just tend to encode features that don't overlap very much. I'm currently of the mind that "encouraging" latent spaces to abide by certain expected distributions is often a nice way to do this (of course depending on the distribution). Historically, we've been limited to the rather vague and often inappropriate Gaussian (i.e. VAEs), but with adversarial VAEs and MMD, we're increasingly able to impose arbitrary distributions. Now of course, I'd agree that too narrow a distribution is probably not the right direction. But doing subtle things like half-continuous, half-discrete codes, or modelling rare entities with Poisson distributed codes could be interesting.