6 ms·

Why is this approach preferred to treating words as symbols connected to other symbols? E.g. (king (genl headOfState) (sex male)) (woman (genl

Why is this approach preferred to treating words as symbols connected to other symbols? E.g. (king (genl headOfState) (sex male)) (woman (genl person) (minimumAge 18) (sex female)) What are the advantages of storing words as floating point vectors? I can see how inputting huge amounts of text into a simple algorithm might be less labour intensive than manually building (or using a more complex algorithm to build) a dictionary. However, at least you can use your dictionary for other purposes, and its contents are readily verifiable by non-technical people.

stared 10y ago
In short, manual labelling rarely works for natural language processing (and top-down mathematical approach). See: Peter Norvig, On Chomsky and the Two Cultures of Statistical Learning, http://norvig.com/chomsky.html http://norvig.com/chomsky.html Also, we want something that is automatic. It means easily adjustable to other contexts (and languages), inferring information about neologisms (e.g. semantic meaning of emoji), etc.
- DonaldFisk 10y ago
  What are non edge-case examples of the symbolic approach not working? If my recollection is correct, Peter Norvig changed his views around the time he joined Google. Google have a particular way of doing things which doesn't include symbolic processing. It should be possible to infer using a symbolic approach, or more simply just provide a new definition.
  - stared 10y ago
    Well, AFAIK all practical systems for translation use data-centric approach, rather than any top-down one. See also: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks, http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/, and try to replicate it with any formal semantics (good luck!). Additionally, formal systems rarely incorporate for actual language, with some things being technically correct, but sounding weird, or things being incorrect, yet - prevalent (and a root for language evolution). See also: char2char translation (which accommodate for e.g. neologisms or typos).
    - DonaldFisk 10y ago
      But does the translation use word2vec or anything similar? The original Altavista Babelfish (which used SYSTRAN) used rule-based machine translation. It has been replaced by Bing Translator, but my recollection of the Babelfish was that it was accurate enough to be usable, and better than Google Translate was when it was first released. Google Translate has improved a lot recently. My only problem with it is that it doesn't understand the meaning of the words it's translating. Neural networks are perfectly suitable for perception, e.g. image recognition. No argument there.
      - matt4077 10y ago
        > My only problem with it is that it doesn't understand the meaning of the words it's translating. I think that's a fallacy that will haunt AI forever (or, more likely, will be the definitive civil rights struggle ca. March 25, 2035 6:25:45am to March 25, 2035 6:25:48am) We tend to move the goalpost whenever AI makes advances. Where many people would have considered chess a pretty good measure of at least some aspect of intelligence, it seems like mundane number crunching once you know how it works. It may be that we really mean consciousness when we say "intelligence", although if we ever find an easy formulation that creates a perfect "illusion" of consciousness, it may end up having some strong effects on people's conception of themselves that I don't necessarily want to witness.
        williamtrask 10y ago
        thanks for sharing this... stirring thoughts. Did my best to paraphrase here (https://twitter.com/iamtrask/status/818090203990659072 https://twitter.com/iamtrask/status/818090203990659072) but I really think we should spend more time on this as a society.
      - barrkel 10y ago
        I think a vector of weights in different dimensions constitutes meaning. Words don't have a single meaning; they're a shared implicit web of allusions and hints. We only understand one another to the degree that our allusions are shared - it's why nonnative speakers miss nuance, and it's why poetry is a thing. If there are symbols, they're in the dimensions of the vector; but words only probabilistically suggest meaning, they don't categorically denote it.
  - ekidd 10y ago
    > What are non edge-case examples of the symbolic approach not working? ... It should be possible to infer using a symbolic approach, or more simply just provide a new definition. I've met some really really brilliant people who've been banging their head against that particular wall since the 1980s. The MIT AI Lab crew, for example, poured untold brainpower into symbolic inference. There was the whole "expert systems" movement. This all failed miserably, in disgrace, because nobody could ever get it to work, and "AI" became a dirty word. Later, there was Cyc, which was hyped on and off throughout the 90s. http://www.cyc.com/ http://www.cyc.com/ After that, there were people who tried to reason over RDF tuples, which didn't work either: http://www.shirky.com/writings/herecomeseverybody/semantic_syllogism.html http://www.shirky.com/writings/herecomeseverybody/semantic_s... This idea pops up every 5 or 10 years and wastes a generation of brainpower. It never works. And let me be clear: There have been some terrifyingly brilliant people who were convinced that it ought to work, and who spent years of their life on it. Meanwhile, any joker who can code up Bayes theorem or single value decomposition can get some results in a couple of weeks. Probability and statistics get results (as do more advanced numeric techniques). Logic deduction fails. I'm not even sure I could explain why. But I encourage you to think long and deeply on what Norvig has written on this subject. Or just buy Norvig's two AI textbooks (written before and after he discovered the joys of probability, basically), do some exercises, and compare the results you get.
    - rvense 10y ago
      > I've met some really really brilliant people who've been banging their head against that particular wall since the 1980s. The idea of breaking meaning into an inventory of discrete components like this is at least as old as Hjelmslev, I think even Saussure touches on it. Part of it seems to me to be that you're breaking down words into words. Even if you write it in uppercase and call it a primitive, you don't have to spend too much time on cognitive linguistics to know that there's really nothing primitive about MAN or MONARCH...
      - DonaldFisk 10y ago
        At that point, you do need some sort of grounding, e.g. a human reading the text, or a neural network embedded in a robot. It isn't symbols all the way down.
- ilaksh 10y ago
  Whether its symbolic or NN based or some other representation, the main thing people are missing here is the lack of symbol grounding. That is why we the open 3d environments based on virtual senses and motor outputs are the most likely to ultimately move forward in terms of things like NLP. See http://courses.media.mit.edu/2004spring/mas966/Harnad%20symbol%20grounding.pdf http://courses.media.mit.edu/2004spring/mas966/Harnad%20symb... or http://www.goertzel.org/papers/PostEmbodiedAI_June7.htm http://www.goertzel.org/papers/PostEmbodiedAI_June7.htm
chewxy 10y ago
What you listed is called a logical form (and there can be many). A common logical form is CCG/DRS. But yet another representation structure that is a logical form that I don't see many people admit to it being a logical form is the RDF triple, used in freebase etc. You could represent the entire freebase in your notation. There is work being done to merge the formal/structural semantics and distributional semantics. I have been working on that for over a year at my startup (not necessarily having successes, mind). The benefit of using distributional semantics (word embeddings, etc) is GPU processing and libraries. My logical form parsing library uses dynamic programming and chart parsing with a LOT of tree pruning to parse a simple sentence, while distributed semantics merely require me to multiply 2 matrices/vectors together - something GPUs excel at.
- DonaldFisk 10y ago
  The frames in my previous post could be implemented in a graph database, or RDF triples. It's similar to the representation used in Cyc. I'm quite happy with a hybrid symbolic/statistical approach, which could be useful for disambiguating words and phrases in context. I think the trick is to avoid generating the parse tree in the first place. Today's hardware is pessimized for symbol and list processing. They can't be done on GPUs, and CPUs work better on contiguous data.
paulsutter 10y ago
From a survey of Deep Learning published in Nature[1]: "The issue of representation lies at the heart of the debate between the logic-inspired and the neural-network-inspired paradigms for cognition. In the logic-inspired paradigm, an instance of a symbol is something for which the only property is that it is either identical or non-identical to other symbol instances. It has no internal structure that is relevant to its use; and to reason with symbols, they must be bound to the variables in judiciously chosen rules of inference. By contrast, neural networks just use big activity vectors, big weight matrices and scalar non-linearities to perform the type of fast ‘intuitive’ inference that underpins effortless commonsense reasoning." Pithy version: "As of 2015, I pity the fool who prefers Modus Ponens over Gradient Descent." - Tomasz Malisiewicz [1] Superlong version: https://plato.stanford.edu/entries/logic-ai/ https://plato.stanford.edu/entries/logic-ai/ [1] https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pd... [2] http://www.computervisionblog.com/2015/04/deep-learning-vs-probabilistic.html http://www.computervisionblog.com/2015/04/deep-learning-vs-p...
- DonaldFisk 10y ago
  The first quote is simply not true. Symbols, even in Lisp, have properties (as well as values and possibly function values), i.e. they have deep structure. In general, symbols are explicitly linked to other symbols or algorithms. These links are the symbolic analogue of the vectors in word2vec, except that they are explicit so you can see what they mean, but you have to enter them manually or use a more complex machine learning algorithm. The author of the computer vision blog post doesn't seem to know much about symbolic AI. Some of the comments point this out.
  - paulsutter 10y ago
    The various benchmarks are a great opportunity to demonstrate the superiority of the symbolic approach over the neural network based models. I encourage you to try, there's no better way to dispel the doubters.
    - DonaldFisk 10y ago
barrkel 10y ago
This is ridiculously simplistic - take woman, for example. The distinction between woman and girl isn't nearly as simple as age. There's a sorites paradox in trying to find a dividing point of age, but actually woman connotes a bunch of other concepts probabilistically. It suggests independence and maturity vs childishness and cuteness. While girl is often used in the context not of children, but women as objects of courtship - girls on a night out, girlfriend etc. And a poem or song may play with the multiple meanings and ambiguities - where is Britney's Not a Girl in this symbolic analysis? The only way you can get symbols to work is with weights. Follow this to its logical conclusion and I think you'll end up with a system isomorphic with the vector approach, with dimensions representing something like symbols.
- DonaldFisk 10y ago
  It was a very simplified example. The Cyc database contains more realistic examples. Yes, I'd seriously consider augmenting symbols with numeric data. You could even store the vector obtained from word2vec as a property of the symbol if that's found to be useful in capturing nuance, but you can't store symbols on floating point vectors.
  - barrkel 10y ago
    The edge between a word and a symbol is where the number belongs - and arguably it should be a time series of numbers. In the vector view of the world, the dimensions of the vector are symbols - the number in a particular slot for a word vector is the number on the edge between the word and the symbol. But the "symbol" represented by a dimension may be a compound or quite abstract concept, it may not map trivially to a single word.
  - barrkel 10y ago
    Here's another example. The fat cat is sitting on a mat. Only someone who has spent too much time away from people could possibly think that the meaning of this sentence is that somewhere, there is a fat cat sitting on a mat. It's a direct allusion to books used to teach reading to young children. It's a reference to simple sentences and simple words. And even within the world of the sentence, it only has a vague meaning. What exactly makes the cat fat? Is it neutered or lazy or overfed? Why is it sitting on a mat - is the mat outside a door, is it waiting to be let in? What kind of a cat is it - could it be a big, dangerous cat? The sentence is laden with signifiers and unknowns. Western children are taught to consider sentences like these in the abstract, but it's not a natural way of thinking, because it's not practical in a life lived connected to the world. Abstract hypotheticals are the hallmark of more disconnected concerns, and we teach our children this early, in part using silly, deliberately vague sentences like these, and discouraging curious questions that might resolve the ambiguities. An AI system that's designed to handle abstract sentences like these is not one designed to understand human language, because humans don't reason like this unless they're thinking analytically - and even then, they do so blinkered with biases and errors.

stared 10y ago

In short, manual labelling rarely works for natural language processing (and top-down mathematical approach). See: Peter Norvig, On Chomsky and the Two Cultures of Statistical Learning, http://norvig.com/chomsky.html http://norvig.com/chomsky.html Also, we want something that is automatic. It means easily adjustable to other contexts (and languages), inferring information about neologisms (e.g. semantic meaning of emoji), etc.

DonaldFisk 10y ago
What are non edge-case examples of the symbolic approach not working? If my recollection is correct, Peter Norvig changed his views around the time he joined Google. Google have a particular way of doing things which doesn't include symbolic processing. It should be possible to infer using a symbolic approach, or more simply just provide a new definition.
- stared 10y ago
  Well, AFAIK all practical systems for translation use data-centric approach, rather than any top-down one. See also: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks, http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/, and try to replicate it with any formal semantics (good luck!). Additionally, formal systems rarely incorporate for actual language, with some things being technically correct, but sounding weird, or things being incorrect, yet - prevalent (and a root for language evolution). See also: char2char translation (which accommodate for e.g. neologisms or typos).
  - DonaldFisk 10y ago
    But does the translation use word2vec or anything similar? The original Altavista Babelfish (which used SYSTRAN) used rule-based machine translation. It has been replaced by Bing Translator, but my recollection of the Babelfish was that it was accurate enough to be usable, and better than Google Translate was when it was first released. Google Translate has improved a lot recently. My only problem with it is that it doesn't understand the meaning of the words it's translating. Neural networks are perfectly suitable for perception, e.g. image recognition. No argument there.
    - matt4077 10y ago
      > My only problem with it is that it doesn't understand the meaning of the words it's translating. I think that's a fallacy that will haunt AI forever (or, more likely, will be the definitive civil rights struggle ca. March 25, 2035 6:25:45am to March 25, 2035 6:25:48am) We tend to move the goalpost whenever AI makes advances. Where many people would have considered chess a pretty good measure of at least some aspect of intelligence, it seems like mundane number crunching once you know how it works. It may be that we really mean consciousness when we say "intelligence", although if we ever find an easy formulation that creates a perfect "illusion" of consciousness, it may end up having some strong effects on people's conception of themselves that I don't necessarily want to witness.
      - williamtrask 10y ago
        thanks for sharing this... stirring thoughts. Did my best to paraphrase here (https://twitter.com/iamtrask/status/818090203990659072 https://twitter.com/iamtrask/status/818090203990659072) but I really think we should spend more time on this as a society.
    - barrkel 10y ago
      I think a vector of weights in different dimensions constitutes meaning. Words don't have a single meaning; they're a shared implicit web of allusions and hints. We only understand one another to the degree that our allusions are shared - it's why nonnative speakers miss nuance, and it's why poetry is a thing. If there are symbols, they're in the dimensions of the vector; but words only probabilistically suggest meaning, they don't categorically denote it.
- ekidd 10y ago
  > What are non edge-case examples of the symbolic approach not working? ... It should be possible to infer using a symbolic approach, or more simply just provide a new definition. I've met some really really brilliant people who've been banging their head against that particular wall since the 1980s. The MIT AI Lab crew, for example, poured untold brainpower into symbolic inference. There was the whole "expert systems" movement. This all failed miserably, in disgrace, because nobody could ever get it to work, and "AI" became a dirty word. Later, there was Cyc, which was hyped on and off throughout the 90s. http://www.cyc.com/ http://www.cyc.com/ After that, there were people who tried to reason over RDF tuples, which didn't work either: http://www.shirky.com/writings/herecomeseverybody/semantic_syllogism.html http://www.shirky.com/writings/herecomeseverybody/semantic_s... This idea pops up every 5 or 10 years and wastes a generation of brainpower. It never works. And let me be clear: There have been some terrifyingly brilliant people who were convinced that it ought to work, and who spent years of their life on it. Meanwhile, any joker who can code up Bayes theorem or single value decomposition can get some results in a couple of weeks. Probability and statistics get results (as do more advanced numeric techniques). Logic deduction fails. I'm not even sure I could explain why. But I encourage you to think long and deeply on what Norvig has written on this subject. Or just buy Norvig's two AI textbooks (written before and after he discovered the joys of probability, basically), do some exercises, and compare the results you get.
  - rvense 10y ago
    > I've met some really really brilliant people who've been banging their head against that particular wall since the 1980s. The idea of breaking meaning into an inventory of discrete components like this is at least as old as Hjelmslev, I think even Saussure touches on it. Part of it seems to me to be that you're breaking down words into words. Even if you write it in uppercase and call it a primitive, you don't have to spend too much time on cognitive linguistics to know that there's really nothing primitive about MAN or MONARCH...
    - DonaldFisk 10y ago
      At that point, you do need some sort of grounding, e.g. a human reading the text, or a neural network embedded in a robot. It isn't symbols all the way down.
ilaksh 10y ago
Whether its symbolic or NN based or some other representation, the main thing people are missing here is the lack of symbol grounding. That is why we the open 3d environments based on virtual senses and motor outputs are the most likely to ultimately move forward in terms of things like NLP. See http://courses.media.mit.edu/2004spring/mas966/Harnad%20symbol%20grounding.pdf http://courses.media.mit.edu/2004spring/mas966/Harnad%20symb... or http://www.goertzel.org/papers/PostEmbodiedAI_June7.htm http://www.goertzel.org/papers/PostEmbodiedAI_June7.htm

chewxy 10y ago

What you listed is called a logical form (and there can be many). A common logical form is CCG/DRS. But yet another representation structure that is a logical form that I don't see many people admit to it being a logical form is the RDF triple, used in freebase etc. You could represent the entire freebase in your notation. There is work being done to merge the formal/structural semantics and distributional semantics. I have been working on that for over a year at my startup (not necessarily having successes, mind). The benefit of using distributional semantics (word embeddings, etc) is GPU processing and libraries. My logical form parsing library uses dynamic programming and chart parsing with a LOT of tree pruning to parse a simple sentence, while distributed semantics merely require me to multiply 2 matrices/vectors together - something GPUs excel at.

DonaldFisk 10y ago
The frames in my previous post could be implemented in a graph database, or RDF triples. It's similar to the representation used in Cyc. I'm quite happy with a hybrid symbolic/statistical approach, which could be useful for disambiguating words and phrases in context. I think the trick is to avoid generating the parse tree in the first place. Today's hardware is pessimized for symbol and list processing. They can't be done on GPUs, and CPUs work better on contiguous data.

paulsutter 10y ago

From a survey of Deep Learning published in Nature[1]: "The issue of representation lies at the heart of the debate between the logic-inspired and the neural-network-inspired paradigms for cognition. In the logic-inspired paradigm, an instance of a symbol is something for which the only property is that it is either identical or non-identical to other symbol instances. It has no internal structure that is relevant to its use; and to reason with symbols, they must be bound to the variables in judiciously chosen rules of inference. By contrast, neural networks just use big activity vectors, big weight matrices and scalar non-linearities to perform the type of fast ‘intuitive’ inference that underpins effortless commonsense reasoning." Pithy version: "As of 2015, I pity the fool who prefers Modus Ponens over Gradient Descent." - Tomasz Malisiewicz [1] Superlong version: https://plato.stanford.edu/entries/logic-ai/ https://plato.stanford.edu/entries/logic-ai/ [1] https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pd... [2] http://www.computervisionblog.com/2015/04/deep-learning-vs-probabilistic.html http://www.computervisionblog.com/2015/04/deep-learning-vs-p...

DonaldFisk 10y ago
The first quote is simply not true. Symbols, even in Lisp, have properties (as well as values and possibly function values), i.e. they have deep structure. In general, symbols are explicitly linked to other symbols or algorithms. These links are the symbolic analogue of the vectors in word2vec, except that they are explicit so you can see what they mean, but you have to enter them manually or use a more complex machine learning algorithm. The author of the computer vision blog post doesn't seem to know much about symbolic AI. Some of the comments point this out.
- paulsutter 10y ago
  The various benchmarks are a great opportunity to demonstrate the superiority of the symbolic approach over the neural network based models. I encourage you to try, there's no better way to dispel the doubters.
  - DonaldFisk 10y ago

barrkel 10y ago

This is ridiculously simplistic - take woman, for example. The distinction between woman and girl isn't nearly as simple as age. There's a sorites paradox in trying to find a dividing point of age, but actually woman connotes a bunch of other concepts probabilistically. It suggests independence and maturity vs childishness and cuteness. While girl is often used in the context not of children, but women as objects of courtship - girls on a night out, girlfriend etc. And a poem or song may play with the multiple meanings and ambiguities - where is Britney's Not a Girl in this symbolic analysis? The only way you can get symbols to work is with weights. Follow this to its logical conclusion and I think you'll end up with a system isomorphic with the vector approach, with dimensions representing something like symbols.

DonaldFisk 10y ago
It was a very simplified example. The Cyc database contains more realistic examples. Yes, I'd seriously consider augmenting symbols with numeric data. You could even store the vector obtained from word2vec as a property of the symbol if that's found to be useful in capturing nuance, but you can't store symbols on floating point vectors.
- barrkel 10y ago
  The edge between a word and a symbol is where the number belongs - and arguably it should be a time series of numbers. In the vector view of the world, the dimensions of the vector are symbols - the number in a particular slot for a word vector is the number on the edge between the word and the symbol. But the "symbol" represented by a dimension may be a compound or quite abstract concept, it may not map trivially to a single word.
- barrkel 10y ago
  Here's another example. The fat cat is sitting on a mat. Only someone who has spent too much time away from people could possibly think that the meaning of this sentence is that somewhere, there is a fat cat sitting on a mat. It's a direct allusion to books used to teach reading to young children. It's a reference to simple sentences and simple words. And even within the world of the sentence, it only has a vague meaning. What exactly makes the cat fat? Is it neutered or lazy or overfed? Why is it sitting on a mat - is the mat outside a door, is it waiting to be let in? What kind of a cat is it - could it be a big, dangerous cat? The sentence is laden with signifiers and unknowns. Western children are taught to consider sentences like these in the abstract, but it's not a natural way of thinking, because it's not practical in a life lived connected to the world. Abstract hypotheticals are the hallmark of more disconnected concerns, and we teach our children this early, in part using silly, deliberately vague sentences like these, and discouraging curious questions that might resolve the ambiguities. An AI system that's designed to handle abstract sentences like these is not one designed to understand human language, because humans don't reason like this unless they're thinking analytically - and even then, they do so blinkered with biases and errors.