4 ms·

This post feels weirdly shallow? For example, a statement like "totally beyond the reach of purely symbolic computation" needs to be followed by a lot more cla

This post feels weirdly shallow? For example, a statement like "totally beyond the reach of purely symbolic computation" needs to be followed by a lot more clarification, given that all of these algorithms that have supposedly superseded symbolic computation are themselves implemented using systems that are equivalent to symbolic computation. And that means the author stops short of getting to the really interesting insights. No, really, why has symbolic AI fallen out of favor? As someone who started their career doing symbolic NLP and has since moved on to machine learning methods, I'd argue that it's a practicality problem. Symbolic methods require humans to figure out the rules and then program them into the system. And as the system gets bigger and more complex, it becomes increasingly difficult to make any change without creating a sort of butterfly effect that destabilizes some other portion of the system. I have a hunch that one of the most (popularly) underappreciated aspects of deep neural networks is that they've come up with some good mechanisms for mitigating that problem. If you squint and turn your head, that's what regularization and dropout are really about - trying to minimize the number of truly critical dependencies among different parts of the model, so that it can continue to grow without becoming unstable. By contrast, GOFAI practitioners tend to actively create such critical dependencies, because doing so limits the number of items we need to keep in our working memory when we're writing the code. Unfortunately, and rather ironically, that seems to work out about as well as greedy search ever does.

Nevermark 3y ago
The overriding bit in terms of why symbolic systems found language challenging, is language in our brains isn't a symbolic system. It has symbolic characteristics but is still an analog system inductively learned by neurons. So it is not surprising that models that use approximated neural type information processing, are better able to model something that is naturally implemented with that category of computation. And the reverse is true. Neural models, like humans, don't naturally perform well on many step exact calculations, much less complex exact reasoning. (Unless models are reduced to modules that are equivalent to primitive symbolic operations. But that is essentially symbolic design on top of a substrate whose details have been intentionally abstracted away, with the extra costs of using an arbitrary non-optimal substrate.)

Nevermark 3y ago

The overriding bit in terms of why symbolic systems found language challenging, is language in our brains isn't a symbolic system. It has symbolic characteristics but is still an analog system inductively learned by neurons. So it is not surprising that models that use approximated neural type information processing, are better able to model something that is naturally implemented with that category of computation. And the reverse is true. Neural models, like humans, don't naturally perform well on many step exact calculations, much less complex exact reasoning. (Unless models are reduced to modules that are equivalent to primitive symbolic operations. But that is essentially symbolic design on top of a substrate whose details have been intentionally abstracted away, with the extra costs of using an arbitrary non-optimal substrate.)