This might interest you: https://transformer-circuits.pub/2023/monosemantic-features https://transformer-circuits.pub/2023/monosemantic-features An embedding i

This might interest you: https://transformer-circuits.pub/2023/monosemantic-features https://transformer-circuits.pub/2023/monosemantic-features An embedding is essentially a set of neuron activations, the number of combinations of activations is much bigger than the number of neurons themselves. An embedding could store "A+B" as well as being able to store "A" and "B", but it is a little complex to understand what particular activations mean.