5 ms·

I thought they were using some kind of vector space searches like embedding.. no idea if that's the case

IshKebab 3y ago
Yeah I'm pretty sure you could do this just with the classic word embeddings (king =queen + man - woman). Maybe it doesn't work as well as with a full LLM.
- adtac 3y ago
  Addition won't work for things that depend on the order of operations. If salt + water is ocean and water + fire is steam, what's salt + water + fire? Is it salt + steam or ocean + fire? Associativity and commutivity in vector addition doesn't translate well to semantic meaning. Extrapolating your example, it'd also mean: v_king - v_queen ~= v_father - v_mother v_king - v_royalty ~= v_father - v_parent v_king - v_father ~= v_royalty - v_parent ~= v_queen - v_mother I don't see why those should all be true. Intuitively, trying to satisfy O(N^2) semantic pairings with vectors that are optimised for a very specific and different numerical operation (cosine similarity) feels like something that won't work. I'd imagine errors get amplified with 3+ operands.
  - TeMPOraL 3y ago
    Isn't the reason for lack of associativity/commutivity is that you're doing operations (addition/subtraction) that have them, and then snapping the result to the closest one of fixed number of points in your output dictionary? The addition is fine, loss of information is in the final conversion.
    - adtac 3y ago
      There's definitely some lossy compression when you snap it to the nearest known vector: enumerating every word ever written in human history wouldn't even come close to the 2^(16*D) representable points for a D-dimensional float16 embedding vector. In fact, even adding two float16 values is a form of lossy compression for most additions. But I'd be surprised if either of those were the primary reason. The words "sea" and "ocean" are different vectors but they'll be very close to each other. salt + water = sea and salt + water = ocean both sound correct to me so the problem is more about whether the v_salt + v_water can even get to the vicinity of either v_sea or v_ocean.
      - leroman 3y ago
        If we constrain our selves to a pool of words of say Wikipedia entries, minutes names and maybe some other stuff, and use a "super node" like "addition" to kind of act as a math operation.. maybe this makes more sense?

Yeah I'm pretty sure you could do this just with the classic word embeddings (king =queen + man - woman). Maybe it doesn't work as well as with a full LLM.

adtac 3y ago
Addition won't work for things that depend on the order of operations. If salt + water is ocean and water + fire is steam, what's salt + water + fire? Is it salt + steam or ocean + fire? Associativity and commutivity in vector addition doesn't translate well to semantic meaning. Extrapolating your example, it'd also mean: v_king - v_queen ~= v_father - v_mother v_king - v_royalty ~= v_father - v_parent v_king - v_father ~= v_royalty - v_parent ~= v_queen - v_mother I don't see why those should all be true. Intuitively, trying to satisfy O(N^2) semantic pairings with vectors that are optimised for a very specific and different numerical operation (cosine similarity) feels like something that won't work. I'd imagine errors get amplified with 3+ operands.
- TeMPOraL 3y ago
  Isn't the reason for lack of associativity/commutivity is that you're doing operations (addition/subtraction) that have them, and then snapping the result to the closest one of fixed number of points in your output dictionary? The addition is fine, loss of information is in the final conversion.
  - adtac 3y ago
    There's definitely some lossy compression when you snap it to the nearest known vector: enumerating every word ever written in human history wouldn't even come close to the 2^(16*D) representable points for a D-dimensional float16 embedding vector. In fact, even adding two float16 values is a form of lossy compression for most additions. But I'd be surprised if either of those were the primary reason. The words "sea" and "ocean" are different vectors but they'll be very close to each other. salt + water = sea and salt + water = ocean both sound correct to me so the problem is more about whether the v_salt + v_water can even get to the vicinity of either v_sea or v_ocean.
    - leroman 3y ago
      If we constrain our selves to a pool of words of say Wikipedia entries, minutes names and maybe some other stuff, and use a "super node" like "addition" to kind of act as a math operation.. maybe this makes more sense?