4 ms·

>deep learning isn't enough is to use a different approach to achieve superior results. Proponents of symbolic learning should start by just achieving results

>deep learning isn't enough is to use a different approach to achieve superior results. Proponents of symbolic learning should start by just achieving results that are even remotely close go those achieved with deep learning. Nevermind outperforming it. Because the reality is, for NLP and anything related to CV, deep learning has consistently wildly outperformed every other approach. Thousands of ai researchers have tried to make "old school" AI work for those problems with only very limited success. Now my background is in CV (not in the field anymore, but was until 2019) so I'm not sure about how well other methods stack up against DL for other use cases. But to me symbolic approaches for most of what DL excels at just seem like a completely unfeasible (and overcomplicated) pipedream. I don't disagree with the premise that current research has kind of stalled compared to the early-mid 2010s but that mostly means we need to figure out new ways to do ML and DL, not because DL has failed as a concept . & keep in mind that the fact we can even say that things have "stalled" is because we got spoiled by the huge performance leaps that DL made possible in the first place. The symbolic AI crowd have always had a "2 more weeks" narrative where they promise that they will figure out a given problem very soon if only x condition was true. The problem is that they have almost always terribly underdelivered, and that was true even when 99% of the research and funding was focused on symbolic or hard AI. Subsymbolic approaches could also be argued to have yielded somewhat underwhelming results (vs the hype) but the other methods are in a league of their own.

pishpash 5y ago
It's mainly the discomfort of the people who were left behind by DL speaking, not much else to see. We should absolutely let the data speak for itself.
- mardifoufs 5y ago
  Yeah you are right, & it's a bit just dogmatic at this point. I guess there's also a clash between purists that have an idea on how an ideal AI should be vs the others who are a lot more results focused. Nobody that I know in the DL community wouldn't be excited to switch to another method if it meant achieving actual measurable improvements. So if the data was conclusive, the alternatives would be used. No one cares about the purity of the method or whatever, because the messiness is inherent to DL. Also, the current AI community is very very focused on performance and state of the art results (that can be a problem in some cases but not w.r.t to method agnosticism) rather than on any big theories on how human intelligence ought to be translated mathematically (or not)
hajile 5y ago
It's been my observation that "good enough" ML has been one year out each year for over a decade now. Tesla has a horrible car accident where a car mistakes the side of a truck for the sky. They start over and create a new ML system only for the exact same kind of accident to happen again. His points about provable and debuggable making ML radically unsuitable for huge swaths of the tasks where AI would be most desirable are exactly on target. Meanwhile, a tiny little slug seems to understand much more about the universe than the most advanced AI we've ever designed and it does this on nanowatts of power.
- mardifoufs 5y ago
  My point isn't that ML will end up solving all our problems and lead to AGI. Not at all. What I'm saying is that while it's not perfect, it's still so far beyond anything symbolic AI approaches achieved in the real world that it's just silly to say that we need to "go back" to them. If you think the tesla accidents are bad now, wait until you get a car driven by a hard coded or logic based computer vision system. With ML there is at least a path to solve way-easier-than AGI problems like maybe (semi?) autonomous cars, given enough compute and redundancy. With anything more old-school? Maybe you'd get lane detection that works on sunny days. And yeah you will be able to "debug" and prove things more easily but what's the point if there's no real way to fix the bugs if they are inherent to the flaws of symbolic approaches? If you think Ml has a problem with edge cases (and it does), keep in mind that GOFAI methods are usually not even close to be complete enough to have "edges". "the last 1% is the hardest to deal with" becomes more of a "if we can get over 60% it's a miracle" Also, no matter how suboptimal the tesla autopilot is, I have not seen any figure that show that they do worse than humans w.r.t to fatality or accident rates. Maybe they are worse! But just the fact that they are even remotely close is not something that could've been possible even 15 years ago. But I'd be thrilled if we get to anything better than current DL methods. Hell, it might even pull me back into the field! I just really doubt it's going to be old school symbolic or causal methods that will get us to the next big steps.
  - hajile 5y ago
    I believe the answer is somewhere in the middle. Your eyes run on more-or-less fixed-function neural networks to see and recognize stuff. The part of your brain that does the actual driving is much more symbolic in how it analyzes and reacts. When you look at self-driving, there's a general pattern that it handles the 80% of easy stuff (the stuff that you could drive while texting and almost completely ignoring the road). There's a steep gradient down to "complete failure" from there. Accidents happen during a tiny, tiny subset of situations (a minuscule fraction of a percent). Vehicle deaths are in the realm of 1-2 per 100 million miles driven (even lower total accidents if you factor in multiple deaths in one vehicle). If you have a vehicle drive an average of 50mph for 8 hours every day (a very high average if not exclusively driving highways), it would take 700 years for that car to statistically have a fatal accident. For you to test an AI with a decent 95% confidence interval, you'd need hundreds of thousands of cars driving for years just to test a single model. The "enough computation" or "enough data" idea simply isn't going to work here.
The_rationalist 5y ago
> Because the reality is, for NLP and anything related to CV, deep learning has consistently wildly outperformed every other approach. Oh yes sure because you have so much erudition you can make such statements? No you do not, I though have erudition in comparison to the HN crowd and there are many important instances where symbolic outperform neural networks in NLP. For example: Sentence disambiguation AKA the task of determining the boundaries of a sentence (where it starts, where it ends). The state of the art is rule based. The task of generating a desired inflection for a given word: the state of the art is rule based. The task of quantifier scope disambiguation: algorithms are on parity with the data driven approach although in this special cases, data sets are lacking and this in fact apply to many areas or NLP, data sets are lacking and for those places, rule based parsing is the SOTA. But for the two first tasks I mentioned, datasets are not lacking, rules engines are just more correct and efficient. Because neural networks are extremely bad at achieving more than 95% accuracy, they often hit a wall because they are only an approximation engine at the end of the day,not an automated code generation of the ideal parser.
YeGoblynQueenne 5y ago
>> Proponents of symbolic learning should start by just achieving results that are even remotely close go those achieved with deep learning. The first system to win against a Grand Master in chess was Deep Blue, which, despite its name, was not a deep learning system, but a symbolic, hand-crafted, rule-based, system with alpha-beta minimax, using a hand-crafted evaluation function and an opening book of moves [1]. The first system to dominate human players in draughts (checkers) was Chinook, also a hand-crafted symbolic rule-based system [2]. The first system to outperform experts in medical diagnosis, in particular, diagnosis of infections, was MYCIN, an expert system [3]. The first system to win against a human in the question game Jeopardy was the much-maligned, but symbolic-based, and very successful in that one task, Watson [4]. SHRDLU was an NLP system used to direct a (virtual) robot hand, whose capabilities remain unsurpassed by modern systems [5]. The performance of symbolic systems remains unsurpassed in various tasks such as classical search, classical planning, SAT solving, automated theorem proving, program synthesis, etc. I assume that by "symbolic learning" you meant symbolic AI in general, btw. If you're talking about symbolic machine learning in particular, it is worth noting that decision tree learners, like CART, ID3 and C4.5, which have been wildly successful in machine learning, data mining and data science for many years, are symbolic systems that learn a propositional logic "model" (a theory). __________ [1] https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer) https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer) [2] https://en.wikipedia.org/wiki/Chinook_(computer_program) https://en.wikipedia.org/wiki/Chinook_(computer_program) [3] https://en.wikipedia.org/wiki/Mycin https://en.wikipedia.org/wiki/Mycin [4] https://en.wikipedia.org/wiki/Watson_(computer)#Comparison_with_human_players https://en.wikipedia.org/wiki/Watson_(computer)#Comparison_w... [5] https://en.wikipedia.org/wiki/SHRDLU https://en.wikipedia.org/wiki/SHRDLU
- sirwhinesalot 5y ago
  SAT solvers, by far the greatest achievement of the Machine Reasoning / Symbolic camp (IMO), have an internal learning mechanism (CDCL) that is responsible for the incredible performance of SAT solvers on real-world use-cases. They somehow learn the structure of the problem. I've seen a SAT solver used to explore complex hardware designs and not only find all good alternatives but prove there were no others. Commercial Mixed Integer Programming solvers can optimize massive problems to provably global optimality very very quickly, and they too have an internal learning mechanism (cutting-planes). They excel at reasoning but they can only learn from the outcomes of the rules they are provided and how they interact, they cannot learn from data. With the advent of soft-constraints (which have an associated weight) I wonder if such systems could be adapted to learn probabilistic rules as well. Some existing systems (like AlphaGo) already combine symbolic approaches (Monte-Carlo Tree Search) with Deep Learning, but what I'm thinking about here goes beyond that.

pishpash 5y ago

It's mainly the discomfort of the people who were left behind by DL speaking, not much else to see. We should absolutely let the data speak for itself.

mardifoufs 5y ago
Yeah you are right, & it's a bit just dogmatic at this point. I guess there's also a clash between purists that have an idea on how an ideal AI should be vs the others who are a lot more results focused. Nobody that I know in the DL community wouldn't be excited to switch to another method if it meant achieving actual measurable improvements. So if the data was conclusive, the alternatives would be used. No one cares about the purity of the method or whatever, because the messiness is inherent to DL. Also, the current AI community is very very focused on performance and state of the art results (that can be a problem in some cases but not w.r.t to method agnosticism) rather than on any big theories on how human intelligence ought to be translated mathematically (or not)

hajile 5y ago

It's been my observation that "good enough" ML has been one year out each year for over a decade now. Tesla has a horrible car accident where a car mistakes the side of a truck for the sky. They start over and create a new ML system only for the exact same kind of accident to happen again. His points about provable and debuggable making ML radically unsuitable for huge swaths of the tasks where AI would be most desirable are exactly on target. Meanwhile, a tiny little slug seems to understand much more about the universe than the most advanced AI we've ever designed and it does this on nanowatts of power.

mardifoufs 5y ago
My point isn't that ML will end up solving all our problems and lead to AGI. Not at all. What I'm saying is that while it's not perfect, it's still so far beyond anything symbolic AI approaches achieved in the real world that it's just silly to say that we need to "go back" to them. If you think the tesla accidents are bad now, wait until you get a car driven by a hard coded or logic based computer vision system. With ML there is at least a path to solve way-easier-than AGI problems like maybe (semi?) autonomous cars, given enough compute and redundancy. With anything more old-school? Maybe you'd get lane detection that works on sunny days. And yeah you will be able to "debug" and prove things more easily but what's the point if there's no real way to fix the bugs if they are inherent to the flaws of symbolic approaches? If you think Ml has a problem with edge cases (and it does), keep in mind that GOFAI methods are usually not even close to be complete enough to have "edges". "the last 1% is the hardest to deal with" becomes more of a "if we can get over 60% it's a miracle" Also, no matter how suboptimal the tesla autopilot is, I have not seen any figure that show that they do worse than humans w.r.t to fatality or accident rates. Maybe they are worse! But just the fact that they are even remotely close is not something that could've been possible even 15 years ago. But I'd be thrilled if we get to anything better than current DL methods. Hell, it might even pull me back into the field! I just really doubt it's going to be old school symbolic or causal methods that will get us to the next big steps.
- hajile 5y ago
  I believe the answer is somewhere in the middle. Your eyes run on more-or-less fixed-function neural networks to see and recognize stuff. The part of your brain that does the actual driving is much more symbolic in how it analyzes and reacts. When you look at self-driving, there's a general pattern that it handles the 80% of easy stuff (the stuff that you could drive while texting and almost completely ignoring the road). There's a steep gradient down to "complete failure" from there. Accidents happen during a tiny, tiny subset of situations (a minuscule fraction of a percent). Vehicle deaths are in the realm of 1-2 per 100 million miles driven (even lower total accidents if you factor in multiple deaths in one vehicle). If you have a vehicle drive an average of 50mph for 8 hours every day (a very high average if not exclusively driving highways), it would take 700 years for that car to statistically have a fatal accident. For you to test an AI with a decent 95% confidence interval, you'd need hundreds of thousands of cars driving for years just to test a single model. The "enough computation" or "enough data" idea simply isn't going to work here.

The_rationalist 5y ago

> Because the reality is, for NLP and anything related to CV, deep learning has consistently wildly outperformed every other approach. Oh yes sure because you have so much erudition you can make such statements? No you do not, I though have erudition in comparison to the HN crowd and there are many important instances where symbolic outperform neural networks in NLP. For example: Sentence disambiguation AKA the task of determining the boundaries of a sentence (where it starts, where it ends). The state of the art is rule based. The task of generating a desired inflection for a given word: the state of the art is rule based. The task of quantifier scope disambiguation: algorithms are on parity with the data driven approach although in this special cases, data sets are lacking and this in fact apply to many areas or NLP, data sets are lacking and for those places, rule based parsing is the SOTA. But for the two first tasks I mentioned, datasets are not lacking, rules engines are just more correct and efficient. Because neural networks are extremely bad at achieving more than 95% accuracy, they often hit a wall because they are only an approximation engine at the end of the day,not an automated code generation of the ideal parser.

YeGoblynQueenne 5y ago

>> Proponents of symbolic learning should start by just achieving results that are even remotely close go those achieved with deep learning. The first system to win against a Grand Master in chess was Deep Blue, which, despite its name, was not a deep learning system, but a symbolic, hand-crafted, rule-based, system with alpha-beta minimax, using a hand-crafted evaluation function and an opening book of moves [1]. The first system to dominate human players in draughts (checkers) was Chinook, also a hand-crafted symbolic rule-based system [2]. The first system to outperform experts in medical diagnosis, in particular, diagnosis of infections, was MYCIN, an expert system [3]. The first system to win against a human in the question game Jeopardy was the much-maligned, but symbolic-based, and very successful in that one task, Watson [4]. SHRDLU was an NLP system used to direct a (virtual) robot hand, whose capabilities remain unsurpassed by modern systems [5]. The performance of symbolic systems remains unsurpassed in various tasks such as classical search, classical planning, SAT solving, automated theorem proving, program synthesis, etc. I assume that by "symbolic learning" you meant symbolic AI in general, btw. If you're talking about symbolic machine learning in particular, it is worth noting that decision tree learners, like CART, ID3 and C4.5, which have been wildly successful in machine learning, data mining and data science for many years, are symbolic systems that learn a propositional logic "model" (a theory). __________ [1] https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer) https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer) [2] https://en.wikipedia.org/wiki/Chinook_(computer_program) https://en.wikipedia.org/wiki/Chinook_(computer_program) [3] https://en.wikipedia.org/wiki/Mycin https://en.wikipedia.org/wiki/Mycin [4] https://en.wikipedia.org/wiki/Watson_(computer)#Comparison_with_human_players https://en.wikipedia.org/wiki/Watson_(computer)#Comparison_w... [5] https://en.wikipedia.org/wiki/SHRDLU https://en.wikipedia.org/wiki/SHRDLU

sirwhinesalot 5y ago
SAT solvers, by far the greatest achievement of the Machine Reasoning / Symbolic camp (IMO), have an internal learning mechanism (CDCL) that is responsible for the incredible performance of SAT solvers on real-world use-cases. They somehow learn the structure of the problem. I've seen a SAT solver used to explore complex hardware designs and not only find all good alternatives but prove there were no others. Commercial Mixed Integer Programming solvers can optimize massive problems to provably global optimality very very quickly, and they too have an internal learning mechanism (cutting-planes). They excel at reasoning but they can only learn from the outcomes of the rules they are provided and how they interact, they cannot learn from data. With the advent of soft-constraints (which have an associated weight) I wonder if such systems could be adapted to learn probabilistic rules as well. Some existing systems (like AlphaGo) already combine symbolic approaches (Monte-Carlo Tree Search) with Deep Learning, but what I'm thinking about here goes beyond that.