Not as much of an expert as others commenting here, but I believe the sine/cosine stuff comes in just because it’s a standard and very efficient way of comparin

Not as much of an expert as others commenting here, but I believe the sine/cosine stuff comes in just because it’s a standard and very efficient way of comparing vectors. (“Vector” is just an alternate way of talking about a coordinate point - you can say “a point is at (x,y)”, or equivalently you can also say “turn x degrees and then travel y units forward”, either method gives enough information to find the point exactly.) I don’t think sine and cosine are actually factors in human language - rather, the process of turning words into vectors captures whatever are the factors in human language, translates them into vectors, and in that translation process the factors get turned into something that sine/cosine measurements of vectors is good at picking up. A toy example would be that arithmetic doesn’t seem to be a factor in political orientation, but if you assess everyone’s political orientation with some survey questions and then put them on a line from 0 to 10, then you could do some subtraction and multiplication operations to find numbers that are close together - ie doing arithmetic to find similar politics. The reason that works is not because arithmetic has anything to do with political orientation, it’s because your survey questions captured information about political orientation and transformed it into something that arithmetic works on. I guess this explanation doesn’t do much except push the unintuitiveness into the embedding process (that’s the process of turning words into vectors in a way that captures their relationship to other words).