3 ms·
It's not clear from the formulas how x=[x1,...,xN] relates to y, μ, and σ since these are defined without x. Assuming y = Wx + b, and μ, σ, and π are learnable
by yobbo 1y ago
It's not clear from the formulas how x=[x1,...,xN] relates to y, μ, and σ since these are defined without x. Assuming y = Wx + b, and μ, σ, and π are learnable parameters for each output dimension. The symbol π seems to mean both weight and the constant 3.14159 in the same formula.
Overall it looks similar to radial basis activations, but the activations look to be log of weighted "stochastic" sums (weights sum to one) of a set of radial basis functions.
The biggest difference is probably log outputs.
- zakeria 1y agoThanks for the comment - uGMM neurons are not just "RBFs with log outputs". Each neuron is a mixture of Gaussians with trainable means, variances, and mixture weights, encoding uncertainty and multimodality that propagates through the network. The log-likelihood output is simply a consequence of this probabilistic formulation, not an innovation.