3 ms·
With the exception of counterfactual regret minimisation, which has nothing to do with deep learning, all those are applications of existing approaches that hav
by YeGoblynQueenne 5y ago
With the exception of counterfactual regret minimisation, which has nothing to
do with deep learning, all those are applications of existing approaches that
have become possible because of the increased expenditure of resources-
processing power and data. There have been no theoretical advances, no
algorithmic advances to speak of, no new knowledge learned in the last 20 years
of reserach in deep learning. All that has been achieved is new SOTA on old
benchmarks [1].
To make an analogy it's as if we could sort ever larger lists because we keep
sorting them with ever bigger computers... but the only sorting algorithm we
know is bubblesort.
You bring up large language models. Language models are about as old as
computational linguistics [6]. Large language models are... large. There's
nothing that's new about them. "Attention" is an anthropomorphically misnamed
engineering tweak of the kind that was rightly lambasted by Drew McDermot fourty
years ago [7]. Self-play and reinforcement Learning are now more than 60 years
old [8].
As to "few shot" learning, that is said for a language model previously trained
on a copy of the entire web! In what sense is that "few shot"? It is truly
shocking to hear researchers make such obviously false claims with a straight
face. And that they are rewarded with publication of their work is nothing less
than a scandal.
Deep learning research is going nowhere. It is stuck in a rut. It is dead. It
has given up the spirit. It is an ex-research field. This research field is
a goner. It is dead, jim. And I'm sure it will be decades before people start to
say it out loud (the current generation of reserachers will leave the dirty job
to their students), but that doesn't change a thing.
______________
[1] Except in identifying new weaknesses of deep learning systems, as for
example Overinterpretation in the paper I cite above, adversarial examples [2],
shortcut learning [3], "the elephant in the room" [4], being "right for the
wrong reasons" [5] etc etc etc etc.
[2] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199
[3] https://www.nature.com/articles/s42256-020-00257-z https://www.nature.com/articles/s42256-020-00257-z
[4] https://arxiv.org/abs/1808.03305 https://arxiv.org/abs/1808.03305
[5] https://aclanthology.org/P19-1334.pdf https://aclanthology.org/P19-1334.pdf
Which of course are very welcome. The only way to address the limitations of
current approaches is to become aware of them and try to understand them.
Unfortunately, it is only a tiny, tiny minority in the field who does that,
while the majority is happy to spout nonsense like "[a billion plus] few shot"
learning etc.
[6] https://www.semanticscholar.org/paper/Speech-and-language-processing-an-introduction-to-Jurafsky-Martin/b54bcfca3fddc26b8889739a247a25e445818149 https://www.semanticscholar.org/paper/Speech-and-language-pr...
[7] https://www.semanticscholar.org/paper/Artificial-intelligence-meets-natural-stupidity-McDermott/d9566ac89cd9b7fccc080b764aab5107430da28c https://www.semanticscholar.org/paper/Artificial-intelligenc...
[8] Arhtur Samuel's checkers player beat a human champion in 1961. Donald
Michie build MENACE, a reinforcement learning algorithm implemented in
matchboxes in 1960:
https://rodneybrooks.com/forai-machine-learning-explained/ https://rodneybrooks.com/forai-machine-learning-explained/
Ctrl+F for "Machine learning started with games" to find the relevant section.
- Isinlor 5y agoSo much that you write is blatantly false, or stretch to the point of being as good as false. But you seem well read, so I don't think there is a point in trying to convince you. It's also interesting coming from someone researching Inductive Logic Programming that provides no results at all. From my perspective, I don't care for theoretical advances. I care about results and if scale provides results, I'm happy. For people who are just reading this exchange take a look here: https://pbs.twimg.com/media/FHHPjU0VIAUrSMA?format=jpg&name=900x900 https://pbs.twimg.com/media/FHHPjU0VIAUrSMA?format=jpg&name=... It's 16 computer generated images from the description below each image. Having capability like that is just very useful and fun. I wish you also luck in finding a "cat paying checkers in style of Salvador Dali" or a "psychedelic painting of a hamster dragon" on the internet. There is a lot of stuff on the web, but not everything.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- YeGoblynQueenne 5y ago>> From my perspective, I don't care for theoretical advances. I care about results and if scale provides results, I'm happy. Like I say you can sort ever larger lists with ever bigger computers and bubblesort. But you can go a lot further if you use the brain and come up with a better sorting algorithm. But I think you would be really surprised to find out how much humans have achieved by using their brains rather than ever bigger computers. Here's a small example that happened to be on my mind recently: https://en.wikipedia.org/wiki/Copernican_Revolution https://en.wikipedia.org/wiki/Copernican_Revolution That is an example of the difference between expensive toys like Dall-E and world-changing scientific work. Though I appreciate that the difference can sometimes appear somewhat mudddled and a machine that can generate a "hamster dragon" may seem as an important scientific breakthrough. Depends on what you're more interested in: hamster dragons, or understanding how the world works. Oh well. As to this: >> It's also interesting coming from someone researching Inductive Logic Programming that provides no results at all. That's an obvious -and odious- attempt at trolling me, but unlike yours, my HN profile is linked to my real-world identity and so I am forced to respond with politenes and to represent my field with dignity. Therefore, I will point out a famous achievement of ILP with which you are sadly unfamiliar and that I expect you to immediately discount on the grounds that you didn't know about it, so it can't have been all that important. I'm linking to an article in the popular press that will be more accessible. It doesn't go into depth over the machine learning approach used but it's ILP; I link to the scholarly article immediately after: Robot Makes Scientific Discovery All by Itself For the first time, a robotic system has made a novel scientific discovery with virtually no human intellectual input. Scientists designed “Adam” to carry out the entire scientific process on its own: formulating hypotheses, designing and running experiments, analyzing data, and deciding which experiments to run next. https://www.wired.com/2009/04/robotscientist/ https://www.wired.com/2009/04/robotscientist/ And the scholarly article: Functional genomic hypothesis generation and experimentation by a robot scientist https://www.nature.com/articles/nature02236 https://www.nature.com/articles/nature02236