5 ms·

What’s really fascinating about the current state of the art in computer vision/deep learning is just how many of the top research papers basically say “We trie

What’s really fascinating about the current state of the art in computer vision/deep learning is just how many of the top research papers basically say “We tried this, it works, we’re not really sure why”

blauditore 9y ago
I've heard that claim a couple of times now, but never read or heard experts in the field say anything like that. Do you have a source? Of course, you can do fancy things with neural networks without fully understanding how they work, or why they behave like they do. But my impression is that common applications, like image classification, are quite well-understood now.
- valesco 9y ago
  I think you are right to make that distinction. Research papers are probably more about other, less known applications.
deleted 9y ago
[deleted]
hacker_9 9y ago
What? They do explain their results, and as you can see from the references section of any paper, other researchers read the published research and then build their new models using that information and continually refining the process.
- amelius 9y ago
  Yes, they explain it, but in a very hand-wavey way.
- reiinakano 9y ago
  Sure, they do their best to give an explanation after discovering it works, but nobody ever publishes the hundred other models they tried that don't work out.
  - _lpa_ 9y ago
    This is a problem in most areas of scientific research. There is no incentive to publish negative results, so no one does.
cbcoutinho 9y ago
This has been my experience as well. I'm also am afraid that these researchers will succumb to a similar replication crisis that devastated the social sciences [0]. If you can't necessarily explain why something works, it's probably difficult to explain trying it again in the future didn't work [0] https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis
- netheril96 9y ago
  The situation is slightly better than social sciences though. Social science experiments, if any, can hardly be repeated. AI experiments can be easily redone if the source code and datasets are released into the public.
- naturalgradient 9y ago
  This is not really the case. I have personally quite literally implemented papers the week after they appeared on ArXiv and they worked. Anecdotical yes, and there have been papers that nobody was able to replicate, but at large the community is quite good at recognising which work can be used, and this work is then implemented and disseminated.
- _pastel 9y ago
  It's more like the opposite problem. Psychology researchers try to justify deep, compelling theories of human behavior with small experiments that are hard to reproduce. Deep Learning experiments are comparatively easy to run and reproduce because the field has both a common set of performance benchmarks and a large community of people implementing cutting-edge ideas in common frameworks. The harder part is building useful theory on top of those experiments.
amelius 9y ago
I'm thinking that the deep learning field will become like web development: it's becoming so simple that everybody can do it. This article gives some evidence: [1], quoting: > researchers were a bit perplexed when actress and director Kristen Stewart [2] appeared as an author on a machine learning paper. [1] https://techcrunch.com/2017/01/19/kristen-stewart-co-authored-a-paper-on-style-transfer-and-the-ai-community-lost-its-mind/ https://techcrunch.com/2017/01/19/kristen-stewart-co-authore... [2] https://en.wikipedia.org/wiki/Kristen_Stewart https://en.wikipedia.org/wiki/Kristen_Stewart
- eref 9y ago
  I might be wrong, and I don't want to question the possibility that someone from a non-technical domain can contribute to technical domains (in particular because neural networks and style transfer are basically high school mathematics; and since this happened before [1]), but it seems to me that the paper in question does not contain much technical innovation, but it is rather a demonstration of how existing techniques can be used in practice [2]. [1] https://en.wikipedia.org/wiki/Hedy_Lamarr https://en.wikipedia.org/wiki/Hedy_Lamarr [2] https://arxiv.org/abs/1701.04928 https://arxiv.org/abs/1701.04928

blauditore 9y ago

I've heard that claim a couple of times now, but never read or heard experts in the field say anything like that. Do you have a source? Of course, you can do fancy things with neural networks without fully understanding how they work, or why they behave like they do. But my impression is that common applications, like image classification, are quite well-understood now.

valesco 9y ago
I think you are right to make that distinction. Research papers are probably more about other, less known applications.

deleted 9y ago

[deleted]

hacker_9 9y ago

What? They do explain their results, and as you can see from the references section of any paper, other researchers read the published research and then build their new models using that information and continually refining the process.

amelius 9y ago
Yes, they explain it, but in a very hand-wavey way.
reiinakano 9y ago
Sure, they do their best to give an explanation after discovering it works, but nobody ever publishes the hundred other models they tried that don't work out.
- _lpa_ 9y ago
  This is a problem in most areas of scientific research. There is no incentive to publish negative results, so no one does.

cbcoutinho 9y ago

This has been my experience as well. I'm also am afraid that these researchers will succumb to a similar replication crisis that devastated the social sciences [0]. If you can't necessarily explain why something works, it's probably difficult to explain trying it again in the future didn't work [0] https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis

netheril96 9y ago
The situation is slightly better than social sciences though. Social science experiments, if any, can hardly be repeated. AI experiments can be easily redone if the source code and datasets are released into the public.
naturalgradient 9y ago
This is not really the case. I have personally quite literally implemented papers the week after they appeared on ArXiv and they worked. Anecdotical yes, and there have been papers that nobody was able to replicate, but at large the community is quite good at recognising which work can be used, and this work is then implemented and disseminated.
_pastel 9y ago
It's more like the opposite problem. Psychology researchers try to justify deep, compelling theories of human behavior with small experiments that are hard to reproduce. Deep Learning experiments are comparatively easy to run and reproduce because the field has both a common set of performance benchmarks and a large community of people implementing cutting-edge ideas in common frameworks. The harder part is building useful theory on top of those experiments.

amelius 9y ago

I'm thinking that the deep learning field will become like web development: it's becoming so simple that everybody can do it. This article gives some evidence: [1], quoting: > researchers were a bit perplexed when actress and director Kristen Stewart [2] appeared as an author on a machine learning paper. [1] https://techcrunch.com/2017/01/19/kristen-stewart-co-authored-a-paper-on-style-transfer-and-the-ai-community-lost-its-mind/ https://techcrunch.com/2017/01/19/kristen-stewart-co-authore... [2] https://en.wikipedia.org/wiki/Kristen_Stewart https://en.wikipedia.org/wiki/Kristen_Stewart

eref 9y ago
I might be wrong, and I don't want to question the possibility that someone from a non-technical domain can contribute to technical domains (in particular because neural networks and style transfer are basically high school mathematics; and since this happened before [1]), but it seems to me that the paper in question does not contain much technical innovation, but it is rather a demonstration of how existing techniques can be used in practice [2]. [1] https://en.wikipedia.org/wiki/Hedy_Lamarr https://en.wikipedia.org/wiki/Hedy_Lamarr [2] https://arxiv.org/abs/1701.04928 https://arxiv.org/abs/1701.04928