3 ms·

Reading things like this on hacker news and twitter reminds me a lot like particle physicists unwilling to grapple with lack of evidence for SUSY. Every time so

Reading things like this on hacker news and twitter reminds me a lot like particle physicists unwilling to grapple with lack of evidence for SUSY. Every time someone points it out, these guys merely recite the holy texts (describe the model in gross detail with more magic words as if the criticism is because the critic don't understand it well enough), as if that addresses the critique at all. They then eventually gesture at the energy horizon, suggesting that "with more s[0] you'll eventually see my particles." It's feels a bit similar here. There are clear limitations to AI by just NNs, namely they like Stable Diffusion must be trained on so much data, far more than any single artist will ever see, and they still fall short sometimes. With NLP it seems a little more obvious because the limitations seem to lag actual speech understood by humans still, even with the massive dataset again beyond what any single human will ever digest. There are clear issues here, merely reciting to critics what your model of intelligence is doesn't address the shortcomings these systems have which any literate person can perceive. And gesturing at the data horizon (you need even more data??) is not convincing either. Apparently this cat (the interviewed author) has some baggage, but the things he's talking about (symbols) have already been studied and used by philosophers and linguists long before AI or computers were a thing. It helps to build upon what other researchers have already done, I don't know why NN researchers don't seem to map what they are doing to existing research on intelligence in other areas, namely cognitive science. Do NN researchers have any contact with linguists or cognitive scientists generally? Again, being a little ignorant of his baggage, I do think a hybrid approach sounds like it would be promising. Symbolic AI failed in the 70s, NN is getting a bit of the way there now and is pretty useful but misses a few things, why go all in on just one type of approach? [0] s is basically the center of mass energy, what particle physicists mean when they say the "energy" of an interaction (collision)

Der_Einzige 4y ago
"Every time I fire a linguist, the performance of my speech recognizer goes up"
xwolfi 4y ago
You say far more than any artist but I m not so sure. When we spend 20 years looking at stuff before doing one passably interesting work of art (bar absolute outlying geniuses), it's not a small time nor a small training cost. It may feel easy for us but our brain never stop ingesting information for decades before being original enough.
- lostmsu 4y ago
  It is impressive to me that the giant comment you replied to is factually wrong in one of its central premises about the amount of data consumed, and you are the one who gets negatives. Our vision alone uncompressed is about 13TB per hour (!) (assuming 8k per eye resolution, "full color", and about 20hz sample rate). By the time a kid knows to play with a color book, they saw an equivalent of 4 PB of imagery. For comparison Dall-E 2 only seen 175TB. And GPT-3 only a few TB.
317070 4y ago
This NN breakthrough in language is only about 2 years old now [0]. People are working on your symbolic logic hybrid approach, but not much interesting has happened yet. This is a bit expected, since the short amount of time, and the fast scaling these language models have done in those last two years. It's been hard to outscale them with any other approach, let alone by the particular one of symbolic logic. I think the main discripancy with particle physics, is that every iteration of bigger language models are also significantly better, no diminishing returns in sight. Better for concrete applications as well. There is therefore no need to claim these models are scaled for scientific reasons in the search of AI, the accountants are happy as it is. [0] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
hackinthebochs 4y ago
Yes, and the history of philosophy is replete with examples where the learned opinion on a subject stymied progress. Respect for one's intellectual forebearers is good, but being constrained by them has proven to be detrimental in areas where the learned opinion is largely speculative. The progress made by deep learning is so unlikely anything that came before, trying to shoehorn old methods and ontologies into the new paradigm would only harm progress.
- nradov 4y ago
  How do you define "progress" in the history of philosophy? By some criteria we haven't made any progress in over 2000 years. There is no clear goal or success metric.
  - hackinthebochs 4y ago
    In my view, intelligibility is the basis on which we should judge a philosophical theory. Then we can rank theories by which are more intelligible or render more phenomena more intelligible. A theory that makes some phenomena transparent, i.e. conceptually reducible to other phenomena, is more intelligible than a competing theory. Progress in philosophy is the process by which opaque phenomena are rendered transparent. Dogma hinders this kind of progress.
dwallin 4y ago
It’s worth pointing out that you are not making a fair comparison when you compare the amount of data that a NN must parse to that of an individual when learning a specific new concept. A human brain is not just trained on their own lived experiences, but also the experiences of all their ancestors via the process of evolution, in addition to years and decades of general training and fine tuning. You specifically mention Stable Diffusion, but SD has an analogue to the learning of an individual, textual inversion, which can adapt the NN to express new concepts with as few as a handful of examples.

Der_Einzige 4y ago

"Every time I fire a linguist, the performance of my speech recognizer goes up"

xwolfi 4y ago

You say far more than any artist but I m not so sure. When we spend 20 years looking at stuff before doing one passably interesting work of art (bar absolute outlying geniuses), it's not a small time nor a small training cost. It may feel easy for us but our brain never stop ingesting information for decades before being original enough.

lostmsu 4y ago
It is impressive to me that the giant comment you replied to is factually wrong in one of its central premises about the amount of data consumed, and you are the one who gets negatives. Our vision alone uncompressed is about 13TB per hour (!) (assuming 8k per eye resolution, "full color", and about 20hz sample rate). By the time a kid knows to play with a color book, they saw an equivalent of 4 PB of imagery. For comparison Dall-E 2 only seen 175TB. And GPT-3 only a few TB.

317070 4y ago

This NN breakthrough in language is only about 2 years old now [0]. People are working on your symbolic logic hybrid approach, but not much interesting has happened yet. This is a bit expected, since the short amount of time, and the fast scaling these language models have done in those last two years. It's been hard to outscale them with any other approach, let alone by the particular one of symbolic logic. I think the main discripancy with particle physics, is that every iteration of bigger language models are also significantly better, no diminishing returns in sight. Better for concrete applications as well. There is therefore no need to claim these models are scaled for scientific reasons in the search of AI, the accountants are happy as it is. [0] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165

hackinthebochs 4y ago

Yes, and the history of philosophy is replete with examples where the learned opinion on a subject stymied progress. Respect for one's intellectual forebearers is good, but being constrained by them has proven to be detrimental in areas where the learned opinion is largely speculative. The progress made by deep learning is so unlikely anything that came before, trying to shoehorn old methods and ontologies into the new paradigm would only harm progress.

nradov 4y ago
How do you define "progress" in the history of philosophy? By some criteria we haven't made any progress in over 2000 years. There is no clear goal or success metric.
- hackinthebochs 4y ago
  In my view, intelligibility is the basis on which we should judge a philosophical theory. Then we can rank theories by which are more intelligible or render more phenomena more intelligible. A theory that makes some phenomena transparent, i.e. conceptually reducible to other phenomena, is more intelligible than a competing theory. Progress in philosophy is the process by which opaque phenomena are rendered transparent. Dogma hinders this kind of progress.

dwallin 4y ago

It’s worth pointing out that you are not making a fair comparison when you compare the amount of data that a NN must parse to that of an individual when learning a specific new concept. A human brain is not just trained on their own lived experiences, but also the experiences of all their ancestors via the process of evolution, in addition to years and decades of general training and fine tuning. You specifically mention Stable Diffusion, but SD has an analogue to the learning of an individual, textual inversion, which can adapt the NN to express new concepts with as few as a handful of examples.