5 ms·

Maybe you’re talking about a different kind of symbolic knowledge than the OP. To give an example humans can instantly tell whether an arbitrary sentence is gra

• roenxi 5y ago
  The fact that computers can't do sentence grammar and meaning right now doesn't tell us anything much about similarities or differences between humans and neural nets. It just tells us that training a neural net purely on a big corpus isn't enough to derive semantic meaning and makes it hard to work out grammatical meaning. No human has ever tried to do that either, everyone comes at text with some real-world experience. So we don't know how well they would do at it. Probably terribly. It is reasonable to believe that written language is easier to train on a neural net that is trained on both images and words so it can form visual links between words. Maybe that takes more computational grunt than we have at the moment. The failure so far proves nothing.
  • make3 5y ago
    the argument was about wether humans and neural nets learn in a similar way. I don't see how what you are saying has any impact on that
• p1esk 5y ago
  instantly tell whether an arbitrary sentence is grammatical or not You do realize we can train a neural network to perform this task? It is a binary classification problem. When I look at a grammatically incorrect sentence I don't do much symbolic reasoning - it just feels "wrong" to me. It does not match any patterns I have in my head for grammatically correct sentences. There's a lot of pattern matching in our thinking process. What's missing in the current generation of neural networks is efficient information storage and ability to recall that information (e.g. lookup) or update it (direct write).
  • xapata 5y ago
    Humans require fewer examples to learn language rules. It's not clear that humans use the same learning model a "deep net."
    • p1esk 5y ago
      Humans also require a lot of examples to learn a language - years of everyday practice for a young human. Learning algorithms are not the same, but you still need to train a large neural network - lots of neurons with lots of connections (weights) - whether it's in your head or in a datacenter.
      • alecst 5y ago
        There’s some evidence that humans have a Universal Grammar and learn through deletion. And humans can not learn any old language — only a restricted class — meanwhile there’s no reason to think that an ML model would have that problem. I’d encourage you to read a little more about the topic with an open mind. You might learn something.
  • make3 5y ago
    "You do realize we can train a neural network to perform this task" I'm doing a master's in deep learning for NLP and I'm not sure we can. Language modelling can't do this because grammatical yet semantically implausible combinations of words yield very low perplexity, like the classic being Noam Chomsky's "Colorless green ideas sleep furiously". What would be a training set for this? I assume we would first try to do parsing to extract the grammatical role of each word. Then what would be the dataset? A massive attempt at generating the set of all possible trees that are grammatical? I guess we could use massive textual datasets from reputable sources and extract their grammatical role tree, and learn from that. Generating negative examples with sufficient coverage would be very hard. Strict generative modelling without negative examples with good coverage would see the same problem as with language modelling, where acceptable but unlikely examples would have low perplexity despite being good. It would seem to me that in order to generate negative examples with good coverage, your would need to have a man made program with a definition of what grammaticality means, which would make making a neural network useless to begin with. Seems like the experts agree with my take: https://linguistics.stackexchange.com/a/1108 https://linguistics.stackexchange.com/a/1108