9 ms·
King – man + woman is queen; but why?
- sauravjain 10y agoI really like these posts by Sanjeev Arora : 1) http://www.offconvex.org/2015/12/12/word-embeddings-1/ http://www.offconvex.org/2015/12/12/word-embeddings-1/ 2) http://www.offconvex.org/2016/02/14/word-embeddings-2/ http://www.offconvex.org/2016/02/14/word-embeddings-2/ For a more theoretical explanation : https://arxiv.org/abs/1502.03520 https://arxiv.org/abs/1502.03520
- stared 10y agoI learned about them just a few days before publishing this post (and I link to the second one, in the "technicalities" section). Though, some statements about analogies should be taken with a grain of salt, see: Tal Linzen, Issues in evaluating semantic spaces using word analogies, https://arxiv.org/abs/1606.07736 https://arxiv.org/abs/1606.07736.
- stephencanon 10y agoI've never found the "vector space" of word2vec remotely satisfying. In order to form a vector space, you need to also be able to make sense of scalar multiplication, and you need to be closed under arbitrary linear combinations. What is 2king? What is 3king - 2green + 0.5brutality? You can kind of make sense of this for adjectives, but it really breaks down with nouns.
- nightcracker 10y agoThen you really shouldn't look too closely at physics. What is a secondmeter? A voltvoltgram?
- taeric 10y agoI don't follow the point you're making. A volt plus a meter is just that, a volt plus a meter. A king minus a man plus a woman is a nonsense statement. Neat, but not as not exactly illuminating. I'm glad it can be useful. But, I agree it seems to leave a lot lacking.
- deleted 10y ago[deleted]
- jayajay 10y agoKing - man + woman actually is illuminating. Your 6 year old son might even say that's what a queen is. There is useful math here.
- taeric 10y agoI call shenanigans. Find me a place where that algebra, which lacks standard rules of common algebra, makes sense. Consider, king + woman = queen + man. Which looks neat, but is not a universal truth. It could be concubine, for example. So, is queen + man also concubine? Again. I'm glad this works for some things. But really just shows which words are often used together. It does not show any good rationale for their meanings. Unlike math, where 1 + 1 equals 2. Possibly in different encodings. But not just from convention of often being used together.
- Spooky23 10y agoThat's the deficiency of AI right now... pattern matching is only one aspect of intelligence that's fairly narrow and only moderately sophisticated. It's still useful -- you can classify millions of pictures in a meaningful way much faster than before.
- taeric 10y agoI was not claiming against its usefulness. I merely question if it truly has meaning. That is, x+y has meaning and use in most maths. Here, it seems primarily use.
- jayajay 10y agoThe answers you are looking for are Nobel-worthy. I have not formally studied linguistics, but you might wanna start there. Regarding the king equation, I think "king" is actually a vector in this context, as with the other objects. If so, then it is using the vector space formalism. That said, I definitely agree with you. An English speaker may find any of these reasonable: 1. King - man = expensive clothing 2. King - man = prince 3. King - man = queen What is "king"? What is "-"? What is "man"? If a king is a dressed up wealthy man, and you remove the man, you have wealthy clothes? Or, does removing the man mean degrading the king back into a boy? Or, does removing a man mean adding a woman? Wait -- what if a king is more than a dressed up wealthy man? Should we include his home? Do we need to subtract the home? How do you subtract a home? Is the king minus a man a prince if the king was a beggar when he was young? ... death of the universe ... Like you said, there's a combinatoric explosion here. Maybe this example is akin to trying to model each and every trajectory of all 10^23 particles in a gas. It looks like these scientists are stepping back, and looking at the big picture, instead, trying to find something more akin to PV = NkT
- im3w1l 10y agoIt seems to behave a little like positions, then. London makes sense. Paris makes sense. London - Paris makes sense. But 2*London does not.
- taejo 10y ago> London - Paris makes sense. But 2London does not. I guess you know this, but for others: London-Paris makes sense, but has a different type to London and Paris (it's a vector, not a position) and while 2London doesn't make sense, 2(London-Paris) does make sense (it's a vector with the same direction but twice the length). Such a system, with two distinct types -- positions and vectors, with vectors being the differences between positions -- is called an affine space. You can identify positions with vectors by picking a distinguished origin, but then you don't get the type-safety that forbids ridiculous expressions like 2London.
- nvahalik 10y ago> What is 2x"king"? War? Murder? :)
- Spooky23 10y agoDiety
- sp332 10y agoI thought it was putting words on the surface of a sphere and just doing math with cosine distance?
- stephencanon 10y agoThat's fine, but then you shouldn't pretend that you're in a vector space. Words have meanings, and mathematical concepts have precise definitions.
- sp332 10y agoConstraining points to a hypersphere surface only reduces dimensionality by one dimension. For example a sphere in 3-space still has a 2-dimensional surface to play with. That's still a vector space right?
- stephencanon 10y agoNo; it's not closed under linear combinations and it doesn't contain an additive identity.
- sp332 10y agoWell it does if you use the right coordinate system! As the article mentions, most people don't normalize correctly and only get passable answers by luck. You can add locations on the earth's surface if you pick a point to be the origin.
- rm999 10y agoYou may already know this part, but for anyone who doesn't know: vector space models very often use cosine distance to make comparisons, instead of Euclidean distance. In this model, you can visualize your vectors as points on a unit hypersphere, where the distance between two vectors is how far apart they are on the sphere. x+y finds the point between them, x-y finds the point between x and the antipode of y (or, perhaps more intuitively, pushes x away from y). 2*x doesn't have any additional meaning (it's the same as x). But x+2y is the equivalent of finding a point that is proportionally closer to y than x (I think this is 2 times as close to y than x, but I didn't do the math). edit: to be clear this paragraph is very hand-wavy on the math, and is just meant to create an accurate-enough visual. My intuition of why things like king + man - woman work is because the points in the vector space model happen to create a well-behaving manifold with smooth meaning changes. It's not very principled, but it does work. I wrote a series of blog posts with a coworker about doing this with music: https://tech.iheart.com/mapping-the-world-of-music-using-machine-learning-part-1-9a57fa67e366 https://tech.iheart.com/mapping-the-world-of-music-using-mac... https://tech.iheart.com/mapping-the-world-of-music-using-machine-learning-part-2-aa50b6a0304c https://tech.iheart.com/mapping-the-world-of-music-using-mac... Instead of mixing nouns and adjectives, we do things like mixing songs and artists and radio stations etc. In the second post we show how Nirvana - Kurt Cobain + Female Vocalist works remarkably well. I've studied empirically why this worked, and the best I could come up with is that the high dimensional space we created had a very dense set of points in the region of popular western music that led to a smooth manifold.
- CPLX 10y agoThose posts are really interesting.
- stared 10y agoI address this point. For scalar product, well, these word vectors are defined such that their scalar product approximate the pointwise mutual information between a pair of words (or sometimes other quantity, depending on the actual algorithm). For differences - see this section: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html#differences-and-projections http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html#.... I mention also that multiplying a word by a factor (for PMI compression) results in a word of similar meaning, just being more characteristic (bear in mind that for other models in can be related e.g. to word frequency or other properties). But you are right, that there are some problems with linear structure. Some of them were brought to me be Omer Levy (a researcher in this subject). I think an article that summarises it the best (or rather: shows empirically that it does not always work as intended) is: - Tal Linzen, Issues in evaluating semantic spaces using word analogies, https://arxiv.org/abs/1606.07736 https://arxiv.org/abs/1606.07736
- stephencanon 10y agoRight, I don't deny that you get something sort of vector-ish and some operations that make intuitive sense some of the time. That's cool enough on its own! There's no need to pretend that we've actually constructed a vector space. Just say that we've arranged words on a hypersphere with an inferred distance metric and call it a day. (Also, by "scalar product" I meant "scalar multiplication"--the product of a scalar and a vector, not the dot product. It's pretty clear how to make some sense out of the dot product, but it's pretty hard to make consistent sense out of scalar multiplication. Apologies for being unclear.)
- stared 10y agoIn any case "vector-ish" is an appropriate word here. As with logistic regression for complex problems: "Essentially, all models are wrong, but some are useful." - George Box
- moultano 10y agoThere isn't a word at every point in the space. Is that your objection to calling it a vector space? I think of every point in space as representing subtle gradiations of concepts, only some of which are instantiated by words.
- doppioandante 10y agoWell, if I had to give it any meaning I would probably choose the word with the smaller distance from the one that was computed (provided that a sensed scalar product exists here). Closure under linear combinations is not a problem if you separate the "word vector" from the actual word. With the same reasoning as above 0.999999king would still be king. The fundamental problem is that linearity is probably not that accurate in representing words relationships, but I don't think this is the goal.
- MichailP 10y agoCan someone explain a bit more on vector space model for words (and documents)? I first saw that approach in prof. Erik Demaine lecture on algorithms [1], and also here. It's fascinating how linear algebra and vector spaces pop up in unexpected places. [1] https://courses.csail.mit.edu/6.006/spring11/lectures/lec01.pdf https://courses.csail.mit.edu/6.006/spring11/lectures/lec01....
- stared 10y agoFor documents there are various approaches. I would suggest using Latent Dirichlet Allocation, see my links here: https://pinboard.in/search/u:pmigdal?query=lda https://pinboard.in/search/u:pmigdal?query=lda.
- DonaldFisk 10y agoWhy is this approach preferred to treating words as symbols connected to other symbols? E.g. (king (genl headOfState) (sex male)) (woman (genl person) (minimumAge 18) (sex female)) What are the advantages of storing words as floating point vectors? I can see how inputting huge amounts of text into a simple algorithm might be less labour intensive than manually building (or using a more complex algorithm to build) a dictionary. However, at least you can use your dictionary for other purposes, and its contents are readily verifiable by non-technical people.
- stared 10y agoIn short, manual labelling rarely works for natural language processing (and top-down mathematical approach). See: Peter Norvig, On Chomsky and the Two Cultures of Statistical Learning, http://norvig.com/chomsky.html http://norvig.com/chomsky.html Also, we want something that is automatic. It means easily adjustable to other contexts (and languages), inferring information about neologisms (e.g. semantic meaning of emoji), etc.
- DonaldFisk 10y agoWhat are non edge-case examples of the symbolic approach not working? If my recollection is correct, Peter Norvig changed his views around the time he joined Google. Google have a particular way of doing things which doesn't include symbolic processing. It should be possible to infer using a symbolic approach, or more simply just provide a new definition.
- stared 10y agoWell, AFAIK all practical systems for translation use data-centric approach, rather than any top-down one. See also: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks, http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/, and try to replicate it with any formal semantics (good luck!). Additionally, formal systems rarely incorporate for actual language, with some things being technically correct, but sounding weird, or things being incorrect, yet - prevalent (and a root for language evolution). See also: char2char translation (which accommodate for e.g. neologisms or typos).
- antiquark 10y agoFor a D&D "alignment chart", set x axis to [illegal -- legal] and y axis to [evil -- good]. Then start typing in words. Some surprises there! https://lamyiowce.github.io/word2viz/ https://lamyiowce.github.io/word2viz/
- stared 10y agoI wanted to add feature of sharing plots via URL query strings, but I need to find some time for that.
- antiquark 10y agoPreliminary findings! The most illegal thing is "heroin" The most legal thing is "CEO" The most good thing is "teacher" The most evil thing is "lucifer"
- openfuture 10y agoMurder is more legal than money Priests are about as legal and rich as criminals, same for nuns wrt. janitors 'Sad' is rich and legal, 'happy' is poor and illegal. Same delta with 'power' and 'money'.
- nitrogen 10y agoAlso interesting to use "peasant" -> "ruler" as the Y axis, while leaving the X axis as "she" -> "he". It shows a possible gender bias in language, with all of the male words being higher on the peasant-ruler scale than female words.
- deleted 10y ago[deleted]
- uncleleech 10y agoI found this interesting: https://i.imgur.com/FrqmGMz.png https://i.imgur.com/FrqmGMz.png Apparently being a secretary is more moral than being a priest.
- 10y ago
- ANaimi 10y agoHere's word2vec as REST API (sign in to interact from the browser): https://algorithmia.com/algorithms/nlp/Word2Vec https://algorithmia.com/algorithms/nlp/Word2Vec
- somer4ndmuser 10y agoAll things being equal, all men are kings and all women are queens.
- GotAnyMegadeth 10y agoInteresting how in the diagram at the top that all of the female words are either less than or equal on the queen axis to male counterparts.
- ilaksh 10y agoIts interesting but ultimately to really 'under' 'stand' language in a deep way, the systems will need representations based on lower-level (possibly virtual) sensory inputs. That is one of the main enablers for truly general intelligence because its based on this common set of inputs over time, i.e. senses. The domain is sense and motor output and this is a truly general domain. Its also a domain that is connected to the way the concepts map to the real physical world. So when the advanced agent NN systems are put through their paces in virtual 3d worlds by training on simple words, phrases, commands, etc. involving 'real-world' demonstrations of the concepts then we will see some next-level understanding.
- ilaksh 10y agoSee http://www.goertzel.org/papers/PostEmbodiedAI_June7.htm http://www.goertzel.org/papers/PostEmbodiedAI_June7.htm or start with http://courses.media.mit.edu/2004spring/mas966/Harnad%20symbol%20grounding.pdf http://courses.media.mit.edu/2004spring/mas966/Harnad%20symb...
- dottrap 10y agoI wonder, if you could train on just Disney movie scripts, would King – man + woman = Princess? Or maybe: King – man + woman = villain? (because of characters like Cruella, Maleficent, Ursula, etc.)
- sandworm101 10y agoAnd here i thought this was a statement about the strange rules of sucession in modern western monarchy. Riddle me this: why are women who marry kings made queens while men who marry queens are made only princes?
- qbrass 10y agoBecause, like a deck of cards, king trumps queen in the ranking of titles and they want to place the person who married into the family below the one who was born into it.
- sandworm101 10y agoClose, but i would say that "queen" covers two different jobs: being a female monarch and the older 'wife to the king'. Some queens are monarchs and some are only wife, with no right to rule on thier own (ie they are not daughter to any previous monarch). Kings are only ever kings by right.
- true_religion 10y agoI believe there is such a thing as the king-consort.
- candiodari 10y agoI've always found it funny what these machine learning insights mean for how humans think. You have people who focus on grammar and spelling. But word embeddings collect their insights by taking any sequence of 5 words, taking out the middle word, jumble up the result (technically they express it in a way that ignores order. They're expressed as 1 bit per word, 1 means the word is in the sentence, 0 means it's not. The sequence of the words in the input to the network is completely independent of their sequence in the sentence). And they understand that "king is to man as queen is to woman" and lots of other things. When going deeper you quickly start to realize a few things : in 90% of sentences the sequence of words does not matter. No, not even if "not" appears before or after the verb (and thus refers to the subject of object of the sentence). Word sequence. Doesn't matter. Which noun you place an adjective next to. It is semantically important. Really important. Every English (or any language I imagine) teacher will hammer the point home again and again. And yet ... it almost never matters, in the sense that getting it wrong will not cause something dumber than a human to misinterpret the resulting sentence. So why do we care ? Social reasons (ie. to fuck other humans, or more generally, to get them to do stuff for us) It's a weird thing that keeps coming back in machine learning. Humans think their reasoning high level. Yet algorithms that keep track of maybe 2 or 3 variables per individual can predict the actions of crowds with uncanny accuracy. Tens to hundreds of thousands of people, each believing they're individuals and think about what they're doing, take not just the same decision, but with an enormous probability will come to that decision within minutes of each other. I am reminded of a quote by Churchill. Humans appear smart individually, but it's a trick, an impression, it's a facade, almost an illusion. If they act in group, said intelligence is utterly gone, and they almost always act in dumb ways in large groups, even when they are acting alone. Intelligence is 95% a parlor trick used in conversation, to make friends, or to mate, like a peacock's feathers, and only 5% or less something we actually use to act. So it's purpose, from a species' perspective, is not at all to act intelligent, merely to appear intelligent to others. Second is that everyone, even if they are smart and correctly reason about the world around them, will still act stupid. Without someone to impress, you could have a triple nobel prize, you won't act it. So intelligence doesn't work in an individual, and it doesn't work in most groups. It only works in groups where the interaction of the group has people impressing each other with what they did, with some sort of reward being given for that.
- hauleth 10y agoAs a Pole you shoul know better. It isn't true in all cases, i.e. Jadwiga was king, not queen.