4 ms·
I've been playing around with neural nets and find that swarm optimization is particularly effective; you can train high-dimensional neural net spaces with less
by dnautics 12y ago
I've been playing around with neural nets and find that swarm optimization is particularly effective; you can train high-dimensional neural net spaces with less testing per iteration. I think this works because there's a lot of arbitrary symmetrical choices embedded in the neural net. For example, hidden layer neurons are essentially arbitrarily chosen.
Also, I can't use backpropagation for what I'm trying to do.
Thanks for the suggestions. I had heard of Yann LeCun, I'm trying to do image recognition in an "different" way than CNNs, I'm going to read up on the other people in the list.
- ihodes 12y agoI hadn't heard of PSO. Looks pretty interesting. What's your use case that you cannot use stochastic gradient descent?
- dnautics 12y agoI prefer using PSO versus feed-forward gradient descent because emprically for my system, a particle size of ~100 gives comparable results to standard gradient descent, on a 400-or-so neuron neural net, which is ~4x slower. I can't use backpropagation because some of the output nodes are used to apply a transformation on the input followed by reapplication of the (now transformed) input through the neural net.
- ihodes 12y agoThat sounds similar to a recurrent neural net? Backprop definitely works on those with some modifications. I just asked some guys at CMU/Google, they suggested looking more at SGD or some other MCMC methods that may have included the word "expert" or maybe were similar to simulated annealing or Gibb's sampling… which I can't recall… I know nothing about it, and I'm totally unqualified to say anything about them, but maybe (hopefully!) some of those words will lead to something useful! The guys I chatted with don't study PSO, so maybe don't know enough to be helpful, but were a bit down on it in general.