4 ms·

Neat stuff, fun to play with. I wasn't able to get a net to classify the swiss roll. Last time I was playing around with this stuff I found the single biggest f

Neat stuff, fun to play with. I wasn't able to get a net to classify the swiss roll. Last time I was playing around with this stuff I found the single biggest factor in the success was the optimizer used. Is this just using a simple gradient descent? I would like to see a drop down for different optimizers.

deleted 11y ago
[deleted]
chestervonwinch 11y ago
What were the optimization algorithms you had most success with? Were they more successful in the sense of better out-of-sample error rate or in the sense of quicker convergence (or something else)?
8note 11y ago
http://imgur.com/ypBQEWx http://imgur.com/ypBQEWx Add some noise, and use all the inputs, and one 8 wide hidden layer edit: works better with a sigmoid activation curve, but it converges more slowly
- rmellow 11y ago
  Using syn, cos, x1, x2 with 1 six-neuron hidden layer does the trick quickly: http://imgur.com/UMv5gsH http://imgur.com/UMv5gsH No need to mess with noise or regularization :)
- andrewtbham 11y ago
  Yeh you're on the right track. Nice pattern emerges on this after 160 iterations. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=25&networkShape=8,4&seed=0.38071&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
  - deleted 11y ago
    [deleted]
- cglace 11y ago
  Using all inputs and 6 layers of varying sizes. After about 500 iterations. http://i.imgur.com/x1MOpvl.jpg http://i.imgur.com/x1MOpvl.jpg
  - visarga 11y ago
    Just 100 iterations, learning rate 0.03, activation tanh, regularization L2, rate 0.01. The network is 8,8,8 neurons per layer.
- makeset 11y ago
  > Add some noise This actually makes the dataset harder to fit to. It is not the same thing here as the "training with noise" method where random noise would be added to each batch, as an alternative means of Tikhonov regularization.
  - 8note 11y ago
    wih that particular data set, it looks like it really just adds more data, and more importantly, fills in the gaps along the spirals which is where my setup was having troubles. The noise doesn't go far enough to start confusing points between different clusters, but it adds more points. That said, my knowledge of neural nets is fairly limited.
Obi_Juan_Kenobi 11y ago
Using the defaults, I had success at about 300 iterations with all the inputs and 5 hidden layers, each with a decreasing number of neurons (i.e. 6,5,4,3,2). I don't know if that's a general feature to need fewer neurons with each layer, but that seems to work here.

deleted 11y ago

[deleted]

chestervonwinch 11y ago

What were the optimization algorithms you had most success with? Were they more successful in the sense of better out-of-sample error rate or in the sense of quicker convergence (or something else)?

8note 11y ago

http://imgur.com/ypBQEWx http://imgur.com/ypBQEWx Add some noise, and use all the inputs, and one 8 wide hidden layer edit: works better with a sigmoid activation curve, but it converges more slowly

rmellow 11y ago
Using syn, cos, x1, x2 with 1 six-neuron hidden layer does the trick quickly: http://imgur.com/UMv5gsH http://imgur.com/UMv5gsH No need to mess with noise or regularization :)
andrewtbham 11y ago
Yeh you're on the right track. Nice pattern emerges on this after 160 iterations. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=25&networkShape=8,4&seed=0.38071&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
- deleted 11y ago
  [deleted]
cglace 11y ago
Using all inputs and 6 layers of varying sizes. After about 500 iterations. http://i.imgur.com/x1MOpvl.jpg http://i.imgur.com/x1MOpvl.jpg
- visarga 11y ago
  Just 100 iterations, learning rate 0.03, activation tanh, regularization L2, rate 0.01. The network is 8,8,8 neurons per layer.
makeset 11y ago
> Add some noise This actually makes the dataset harder to fit to. It is not the same thing here as the "training with noise" method where random noise would be added to each batch, as an alternative means of Tikhonov regularization.
- 8note 11y ago
  wih that particular data set, it looks like it really just adds more data, and more importantly, fills in the gaps along the spirals which is where my setup was having troubles. The noise doesn't go far enough to start confusing points between different clusters, but it adds more points. That said, my knowledge of neural nets is fairly limited.

Obi_Juan_Kenobi 11y ago

Using the defaults, I had success at about 300 iterations with all the inputs and 5 hidden layers, each with a decreasing number of neurons (i.e. 6,5,4,3,2). I don't know if that's a general feature to need fewer neurons with each layer, but that seems to work here.