It just describes how two probability distributions are different. If they are the same then it’s 0. Example: an LLM gives a probability distribution of the ne

It just describes how two probability distributions are different. If they are the same then it’s 0. Example: an LLM gives a probability distribution of the next word. If it is perfectly accurate at predicting the next word then divergence is 0 (100% probability on the actual next word). If it is slightly off or unsure then the divergence goes up.