Regarding this statement about semantic space: > so long as vectors are roughly the same length, the dot product is an indication of how similar they are. Thi

Regarding this statement about semantic space: > so long as vectors are roughly the same length, the dot product is an indication of how similar they are. This potential length difference is the reason "Cosine Similarity" is used instead of dot products for concept comparisons. Cosine similarity is like a 'scale independent dot product', which represents a concept of similarity, independent of "signal strength". However, if two vectors point in the same direction, but one is 'longer' (higher magnitude) than the other, then what that indicates "semantically" is that the longer vector is indeed a "stronger signal" of the same concept. So if "happy" has a vector direction then "very happy" should be longer vector but in the same direction. Makes me wonder if there's a way to impose a "corrective" force upon model weights evolution during training so that words like "more" prefixed in front of a string can be guaranteed to encode as a vector multiple of said string? Not sure how that would work with back-propagation, but applying certain common sense knowledge about how the semantic space structures "must be" shaped could potentially be the next frontier of LLM development beyond transformers (and by transformers I really mean the attention heads specialization)