4 ms·

Has anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?

hyh1048576 11y ago
Do you consider test loss around 0.04 good?
trampi 11y ago
0.007 http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=7,3&seed=0.77306&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
deleted 11y ago
[deleted]
asab 11y ago
http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=10&networkShape=8,8&seed=0.15362&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
- asab 11y ago
  Update: after playing with this for way too long, I've found that it can converge to a spiral with 3 or 2 or even just 1 node in the 2nd hidden layer. The 1 node case is especially interesting, because when it converges the single node must learn the whole spiral pattern. Although with noise it can be less reliable with more jagged edges, as well as take longer to converge (also bumped the learning rate down), seeing the spiral encoded directly in the 2nd hidden layer is more interesting to me. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.01&regularizationRate=0&noise=10&networkShape=8,1&seed=0.58434&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
scotty79 11y ago
For me key was to use x,y and either sin x, sin y or x squared y squared as inputs and 5 or 6 neurons in hidden layer.
chronolitus 11y ago
Add both sin(X1) and sin(X2) as inputs.
moconnor 11y ago
For this simple example just choosing the largest possible fully-connected network with ReLU and L2 regularization to prevent overfit quickly converges to a nice spiral (test loss of 0.001 for me): http://playground.tensorflow.org/#activation=relu&regularization=L2&batchSize=30&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0.003&noise=0&networkShape=8,8,8,8,8,8&seed=0.38911&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=relu&regulariza...
- amelius 11y ago
  Neat. How would the number of neurons N scale with the size of the spiral? (Size=number of turns) Will N level off, meaning that it will really understand the structure of the spiral?
- findthewords 11y ago
  I wouldn't call it quick... spiral in 150 iterations, with sigmoid magic: http://playground.tensorflow.org/#activation=sigmoid&regularization=L2&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.1&regularizationRate=0.01&noise=0&networkShape=8&seed=0.63555&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=sigmoid&regular... I find the pulsating unsightly.

hyh1048576 11y ago

Do you consider test loss around 0.04 good?

trampi 11y ago

0.007 http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=7,3&seed=0.77306&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...

deleted 11y ago

[deleted]

asab 11y ago

http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=10&networkShape=8,8&seed=0.15362&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...

asab 11y ago
Update: after playing with this for way too long, I've found that it can converge to a spiral with 3 or 2 or even just 1 node in the 2nd hidden layer. The 1 node case is especially interesting, because when it converges the single node must learn the whole spiral pattern. Although with noise it can be less reliable with more jagged edges, as well as take longer to converge (also bumped the learning rate down), seeing the spiral encoded directly in the 2nd hidden layer is more interesting to me. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.01&regularizationRate=0&noise=10&networkShape=8,1&seed=0.58434&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...

scotty79 11y ago

For me key was to use x,y and either sin x, sin y or x squared y squared as inputs and 5 or 6 neurons in hidden layer.

chronolitus 11y ago

Add both sin(X1) and sin(X2) as inputs.

moconnor 11y ago

For this simple example just choosing the largest possible fully-connected network with ReLU and L2 regularization to prevent overfit quickly converges to a nice spiral (test loss of 0.001 for me): http://playground.tensorflow.org/#activation=relu&regularization=L2&batchSize=30&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0.003&noise=0&networkShape=8,8,8,8,8,8&seed=0.38911&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=relu&regulariza...

amelius 11y ago
Neat. How would the number of neurons N scale with the size of the spiral? (Size=number of turns) Will N level off, meaning that it will really understand the structure of the spiral?
findthewords 11y ago
I wouldn't call it quick... spiral in 150 iterations, with sigmoid magic: http://playground.tensorflow.org/#activation=sigmoid&regularization=L2&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.1&regularizationRate=0.01&noise=0&networkShape=8&seed=0.63555&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=sigmoid&regular... I find the pulsating unsightly.