3 ms·
This is an odd story that seems to gloss over the downsides of neural networks. Computational power needed to build some of the models is enough to explain how
by taeric 9y ago
This is an odd story that seems to gloss over the downsides of neural networks. Computational power needed to build some of the models is enough to explain how slow uptake was. At least in large. I would be interested to see just how many multiplications go into a typical model nowadays. In particular, the training of one.
But that still skirts the big issue, which is generalization. We are moving, it seems, to transfer learning. The danger is that we don't seem to have a good theory to why it works. At a practitioner level, I don't think this is as much of a problem. For the research, though, it is pretty shaky.
I think there is more than a strong chance this remains the future for a while. And I am layman in this field. At best. But this story presupposes that the past was wrong for not being like the present. That is a tough bar.
- seanmcdirmid 9y agoResearchers are supposed to be looking toward the future, and shouldn't be hindered by the limitations of the present. So even if your methods are not feasible today, you can still investigate them and be ready for when they will work. That many of those other scientists didn't have the foresight to see that this would work someday is definitely a big miss on their part. Let's not give them a pass.
- taeric 9y agoThat is somewhat silly. Especially when you can't investigate using the tools of the day. Be clear on that. He was not able to see the success using these techniques on older hardware. Period. People weren't dismissing him. They were doing better.
- relate 9y agoI find it strange you mention transfer learning, since one of the reasons neural networks are so popular are because they tend to excel at it. Adapting (i.e. fine-tuning) networks trained on a task with a lot of data (e.g. image classification on ImageNet) to different tasks, such as image segmentation, has proven a very successful approach.
- taeric 9y agoThat was my intent in mentioning them. My understanding, though, is that we don't have a solid foundation on why it works. Just that that is the direction we are moving, and neural networks are good at it.
- js8 9y agoI believe neural networks are not quite the right approach, and their success is misleading. Historically, almost all ML approaches were based on separation in Euclidean (vector) space, which is understandable because they were developed for much weaker computers. However, really useful ML tasks require to deal with huge nonlinearities, and the fact that you're training in linear space becomes less relevant. Neural networks have surmounted the nonlinear difficulty by increasing the number of layers. But it's a question whether a similar result couldn't be achieved with Bayesian networks on binary representations (which is approach I favor). There is some evidence that the precision of linear calculation in neuron (for example, resolution of the weights) doesn't really make much difference in the performance of neural network. Could it be that the neural learning through vector manipulation is just an artifact of the origin of the neural networks, and the really important thing is the overall organization of the network (the layers)?
- taeric 9y agoI'm assuming people are trying with Bayesian methods. Any good research it practical examples showing advantages?