5 ms·

data + plural = number data - plural = research king - crown = (didn't work... crown gets circled in red) king - princess = emperor king - queen = kingd

data + plural = number data - plural = research king - crown = (didn't work... crown gets circled in red) king - princess = emperor king - queen = kingdom queen - king = worker king + queen = queen + king = kingdom boy + age = (didn't work... boy gets circled in red) man - age = woman woman - age = newswoman woman + age = adult female body (tied with man) girl + age = female child girl + old = female child The other suggestions are pretty similar to the results I got in most cases. But I think this helps illustrate the curse of dimensionality (i.e. distances are ill-defined in high dimensional spaces). This is still quite an unsolved problem and seems a pretty critical one to resolve that doesn't get enough attention.

pjc50 1y ago
Ah yes, 女 + 子 = girl but if combined in a kanji you get 好 = like.
Affric 2y ago
Yeah I did similar tests and got similar results. Curious tool but not what I would call accurate.
n2d4 2y ago
For fun, I pasted these into ChatGPT o4-mini-high and asked it for an opinion: data + plural = datasets data - plural = datum king - crown = ruler king - princess = man king - queen = prince queen - king = woman king + queen = royalty boy + age = man man - age = boy woman - age = girl woman + age = elderly woman girl + age = woman girl + old = grandmother The results are surprisingly good, I don't think I could've done better as a human. But keep in mind that this doesn't do embedding math like OP! Although it does show how generic LLMs can solve some tasks better than traditional NLP. The prompt I used: > Remember those "semantic calculators" with AI embeddings? Like "king - man + woman = queen"? Pretend you're a semantic calculator, and give me the results for the following:
- nbardy 2y ago
  I hate to be pedantic, but the llm is definitely doing embedding math. In fact that’s all it does.
  - n2d4 1y ago
    Sure! Although I think we both agree that the way those embeddings are transformed is significantly different ;) (what I meant to say is that it doesn't do embedding math "LIKE" the OP — not that it doesn't do embedding math at all.)
    - coolcase 1y ago
      Yeah we'd be impressed if an LLM calculated the product of a couple of 1000x1000 matrices.
- franga2000 2y ago
  This is an LLM approximating a semantic calculator, based solely on trained-in knowledge of what that is and probably a good amount of sample output, yet somehow beating the results of a "real" semantic calculator. That's crazy! The more I think about it the less surprised I am, but my initial thoughts were quite simply "now way" - surely an approximation of an NLP model made by another NLP model can't beat the original, but the LLM training process (and data volume) is just so much more powerful I guess...
  - CamperBob2 2y ago
    This is basically the whole idea behind the transformer. Attention is much more powerful than embedding alone.
    - godelski 2y ago
      The transformers are initialized by embedding models... Your embedding model is literally the translation layer converting the text to numbers. The transformers are the main processing unit of the embeddings. You can even see some self-reflection in the model as the transformer is composed of attention and a MLP sub-network. The attention mechanism generates the interrelational dependence of the data and the MLP projects up into a higher dimension before coming down so that this can untangle these relationships. But the idea is that you just repeat this process over and over. The attention mechanism has the benefit over CNN models because it has a larger receptive field, so can better process long range relationships (long range being across the input data) where CNNs bias for local relationships.
gweinberg 2y ago
I got a bunch of red stuff also. I imagine the author cached embeddings for some words but not really all that many to save on credits. I gave it mermaid - woman and got merman, but when I tried to give it boar + woman - man or ram + woman - man, it turns out it has never heard of rams or boars.
thatguysaguy 2y ago
Can you elaborate on what the unsolved problem you're referring to is?
- godelski 1y ago
  Dealing with metrics in high dimensions. As you increase dimensionality the variance decreases, leading to indistinguishablity. You can get some help in high dimensions when you're more concerned with (clearly disjoint) clusters. But this is akin to doing a dimensional reduction, treating independent clusters as individual points. (Say we have set S which has disjoint subsets {S_0,...,S_n}, your new set is now {a_0,...,a_n}, where each a_i is an element representing all elements in S_i. Think like "set of sets") But you do not get help with interrelationships (i.e. d(s_x,s_y) \in S_i \forall x≠y) and I think you can gather that when clusters are not clearly disjoint then we're in the same situation as trying to differentiate inter-cluster. Understanding this can help you understand why these models (including LLMs) are good in broader concepts like differentiating between obvious things but struggle more in nuance. A good litmus test is to ask them about any subject you have good deep knowledge in. Essentially test yourself for Murray-Gelmann Amnesia. The things are designed for human preference. When they fail they're likely to fail without warning (i.e. in ways that are not so obvious)
mathgradthrow 2y ago
Distance is extremely well defined in high dimensional spaces. That isn't the problem.
- godelski 1y ago
  Would you care to elaborate? To clarify, I mean that variance reduces as dimensionality increases
sdeframond 1y ago
Such results are inherently limited because a same word can have different meanings depending on context. The role of the Attention Layer in LLMs is to give each token a better embedding by accounting for context.
virgilp 1y ago
hacker+news-startup = golfer
charlieyu1 1y ago
I think you need to do A-B+C types? A+B or A-B wouldn’t make much sense when the magnitude changes

pjc50 1y ago

Ah yes, 女 + 子 = girl but if combined in a kanji you get 好 = like.

Affric 2y ago

Yeah I did similar tests and got similar results. Curious tool but not what I would call accurate.

n2d4 2y ago

For fun, I pasted these into ChatGPT o4-mini-high and asked it for an opinion: data + plural = datasets data - plural = datum king - crown = ruler king - princess = man king - queen = prince queen - king = woman king + queen = royalty boy + age = man man - age = boy woman - age = girl woman + age = elderly woman girl + age = woman girl + old = grandmother The results are surprisingly good, I don't think I could've done better as a human. But keep in mind that this doesn't do embedding math like OP! Although it does show how generic LLMs can solve some tasks better than traditional NLP. The prompt I used: > Remember those "semantic calculators" with AI embeddings? Like "king - man + woman = queen"? Pretend you're a semantic calculator, and give me the results for the following:

nbardy 2y ago
I hate to be pedantic, but the llm is definitely doing embedding math. In fact that’s all it does.
- n2d4 1y ago
  Sure! Although I think we both agree that the way those embeddings are transformed is significantly different ;) (what I meant to say is that it doesn't do embedding math "LIKE" the OP — not that it doesn't do embedding math at all.)
  - coolcase 1y ago
    Yeah we'd be impressed if an LLM calculated the product of a couple of 1000x1000 matrices.
franga2000 2y ago
This is an LLM approximating a semantic calculator, based solely on trained-in knowledge of what that is and probably a good amount of sample output, yet somehow beating the results of a "real" semantic calculator. That's crazy! The more I think about it the less surprised I am, but my initial thoughts were quite simply "now way" - surely an approximation of an NLP model made by another NLP model can't beat the original, but the LLM training process (and data volume) is just so much more powerful I guess...
- CamperBob2 2y ago
  This is basically the whole idea behind the transformer. Attention is much more powerful than embedding alone.
  - godelski 2y ago
    The transformers are initialized by embedding models... Your embedding model is literally the translation layer converting the text to numbers. The transformers are the main processing unit of the embeddings. You can even see some self-reflection in the model as the transformer is composed of attention and a MLP sub-network. The attention mechanism generates the interrelational dependence of the data and the MLP projects up into a higher dimension before coming down so that this can untangle these relationships. But the idea is that you just repeat this process over and over. The attention mechanism has the benefit over CNN models because it has a larger receptive field, so can better process long range relationships (long range being across the input data) where CNNs bias for local relationships.

gweinberg 2y ago

I got a bunch of red stuff also. I imagine the author cached embeddings for some words but not really all that many to save on credits. I gave it mermaid - woman and got merman, but when I tried to give it boar + woman - man or ram + woman - man, it turns out it has never heard of rams or boars.

thatguysaguy 2y ago

Can you elaborate on what the unsolved problem you're referring to is?

godelski 1y ago
Dealing with metrics in high dimensions. As you increase dimensionality the variance decreases, leading to indistinguishablity. You can get some help in high dimensions when you're more concerned with (clearly disjoint) clusters. But this is akin to doing a dimensional reduction, treating independent clusters as individual points. (Say we have set S which has disjoint subsets {S_0,...,S_n}, your new set is now {a_0,...,a_n}, where each a_i is an element representing all elements in S_i. Think like "set of sets") But you do not get help with interrelationships (i.e. d(s_x,s_y) \in S_i \forall x≠y) and I think you can gather that when clusters are not clearly disjoint then we're in the same situation as trying to differentiate inter-cluster. Understanding this can help you understand why these models (including LLMs) are good in broader concepts like differentiating between obvious things but struggle more in nuance. A good litmus test is to ask them about any subject you have good deep knowledge in. Essentially test yourself for Murray-Gelmann Amnesia. The things are designed for human preference. When they fail they're likely to fail without warning (i.e. in ways that are not so obvious)

mathgradthrow 2y ago

Distance is extremely well defined in high dimensional spaces. That isn't the problem.

godelski 1y ago
Would you care to elaborate? To clarify, I mean that variance reduces as dimensionality increases

sdeframond 1y ago

Such results are inherently limited because a same word can have different meanings depending on context. The role of the Attention Layer in LLMs is to give each token a better embedding by accounting for context.

virgilp 1y ago

hacker+news-startup = golfer

charlieyu1 1y ago

I think you need to do A-B+C types? A+B or A-B wouldn’t make much sense when the magnitude changes