10 ms·
Tinker with a Neural Network in Your Browser
- minimaxir 10y agoWhen it says "right here in your browser," it's not joking. On my desktop (Safari), the window becomes unresponsive after a few iterations. Does not happen in Chrome. On my phone (Safari/iOS 9.3), the default neural nework doesn't converge at all even after 300 iterations while it does on the desktop, which is legit weird: https://i.imgur.com/KNaXeHH.png https://i.imgur.com/KNaXeHH.png
- superobserver 10y agoWorking splendidly on ChromeOS, FWIW.
- shancarter 10y agoI'm sorry you're having problems with Safari. I can't reproduce on my end, but if you're still having problems you can raise an issue on github with some information about your system.
- davidgl 10y agoWorks perfectly for me on Safari 9.1 with no extensions
- deleted 10y ago[deleted]
- nefitty 10y agoYeah, it's working great in Chrome on my Galaxy Tab 3!
- koder2016 10y agoTo be honest, if it works in Chrome then it covers > 90% of people who would possibly be interested.
- CGamesPlay 10y agoNeat stuff, fun to play with. I wasn't able to get a net to classify the swiss roll. Last time I was playing around with this stuff I found the single biggest factor in the success was the optimizer used. Is this just using a simple gradient descent? I would like to see a drop down for different optimizers.
- deleted 10y ago[deleted]
- chestervonwinch 10y agoWhat were the optimization algorithms you had most success with? Were they more successful in the sense of better out-of-sample error rate or in the sense of quicker convergence (or something else)?
- 8note 10y agohttp://imgur.com/ypBQEWx http://imgur.com/ypBQEWx Add some noise, and use all the inputs, and one 8 wide hidden layer edit: works better with a sigmoid activation curve, but it converges more slowly
- rmellow 10y agoUsing syn, cos, x1, x2 with 1 six-neuron hidden layer does the trick quickly: http://imgur.com/UMv5gsH http://imgur.com/UMv5gsH No need to mess with noise or regularization :)
- andrewtbham 10y agoYeh you're on the right track. Nice pattern emerges on this after 160 iterations. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=25&networkShape=8,4&seed=0.38071&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
- danielvf 10y agoIn case you are an idiot like me, you have to train your neural network by pressing "play".
- andrewstuart2 10y ago"Okay, I don't understand. Why is my output so terrible?" I saw the play button very clearly when the page loaded, then promptly got distracted by all the dials and knobs. :-P
- shancarter 10y agoWe would've liked to have it constantly training, but didn't want to abuse your CPU :)
- dingo_bat 10y agoIt pauses when I switch tabs :(
- aab0 10y agoThis is a lot of fun. The default dataset is too easy, though, try out the Swiss Roll one!
- minimaxir 10y agoThere is a reason why sin(X) is an input property. :p
- aab0 10y agoUsing sin(x) or the other input features like x^2 goes back to making it too easy, though. So far the best I can do is 7 layers of 7 which gets a loss of 0.02. 3x7 is almost cracking the Swiss Roll but can't quite finish it off and gets stuck at 0.05: https://imgur.com/Z3f2ECc https://imgur.com/Z3f2ECc ... Surprisingly, 2x8 can do it, as long as I have noise or regularization on, but 8/7 then seriously struggles. Is 16 neurons a critical limit here?
- teraflop 10y agoI managed to get to 0.01 loss from only x1/x2, using 3 hidden layers, L1 regularization, a bit of added noise, and some patience: http://i.imgur.com/Y3zKpJF.png http://i.imgur.com/Y3zKpJF.png
- aab0 10y agoYes, noise & regularization seem to be key here. I've gotten a 2-layer with 7/8 neurons down to 0.06 and dropping but only with noise & l1: http://playground.tensorflow.org/#activation=relu®ularization=L2&batchSize=6&dataset=spiral®Dataset=reg-plane&learningRate=0.01®ularizationRate=0.03&noise=10&networkShape=8,7&seed=0.52682&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&noise_hide=false http://playground.tensorflow.org/#activation=relu®ulariza... Final loss of 0.051. Interestingly, increasing noise from 10 to 15 destroys performance, loss of 0.47.
- 10y ago
- _AllisonMobley 10y agoCan somebody explain what I'm watching when I press play?
- isuckatcoding 10y agoYeah I feel like we need some decent understanding of neural networks to have more context on this. Its kind of like being given a specialized shovel but not knowing why you need it or why you should dig holes.
- Obi_Juan_Kenobi 10y agoI think it's a playground in the best sense of the term. Take some time and actually play with it, and a lot of fun stuff happens, lightbulbs go off, etc. If you're expecting a lesson, you'll likely be disappointed, but I think there's real value in a true playground. I think the biggest improvement would be if, when hovering over a 'neuron', you get a visual representation of what feeds into it.
- ryanmonroe 10y agoFor me in Chrome on OSX you do get a visual representation of the neuron's input when hovering. It shows up behind the data points in place of the neurons' output when hovering.
- shmageggy 10y agoIt begins training the network using the backpropagation algorithm. > Next, the network is asked to solve a problem, which it attempts to do over and over, each time strengthening the connections that lead to success and diminishing those that lead to failure. On each iteration, it calculates how bad the predicted output is, then adjusts the weights between neurons to lessen that value. Google backpropagation for more info
- Mahn 10y agoOr perhaps explain how all the different inputs influence the result? I more or less get that it's just iterating over the data to approximate the given data set when you press play but I have no idea how giving it more or less neurons changes that, to name an example.
- gojomo 10y agoWhile it doesn't involve training, these 'confusion matrix' animations of NNs classifying images or digits are fun, too: http://ml4a.github.io/dev/demos/cifar_confusion.html http://ml4a.github.io/dev/demos/cifar_confusion.html http://ml4a.github.io/dev/demos/mnist_confusion.html http://ml4a.github.io/dev/demos/mnist_confusion.html Something about the high-speed updating makes me think of WOPR, in 'War Games', scoring nuclear-war scenarios.
- karpathy 10y agothis is very nice! I think that the reason swiss roll doesn't work as easily might be because of initialization. In 2 dimensions you have to be very careful with initializing the weights or biases because small networks get more easily stuck in bad local minima.
- okigan 10y agoIn this case you see that it is the swiss roll so you could say pick "proper initialization". But that technique would not work when you cannot see that it is a "swiss roll" or in multiple dimensions.
- brianchu 10y agoI'm pretty sure he wasn't talking about the swiss roll specifically. Big gains in neural net performance have been made through better initialization schemes (not dataset specific, just in general, e.g. an initialization scheme might adapt the initial weight distribution depending on the number of hidden units in the next layer), and smaller models are in general more sensitive to initialization.
- imaginenore 10y agoI wish they had more interesting data sets.
- HappyTypist 10y agoSubmit a pull request.
- nxzero 10y ago"Don’t Worry, You Can’t Break It. We Promise." (Nice, but it's completely unclear what's going on.)
- visarga 10y agoMy MacBook Pro running El Capitain froze my mouse and keyboard. I had to do a hard reset.
- eggy 10y agoI started reading about ANNs in the 1980s, and had similar confusion to those here, since it was just for fun. I suggest reading a basic book or online information that goes over the basics [1]. I struggled through $200 text books, and jumped from one to the other as an autodidact. I am now studying TWEANNs (Topology and Weight Evolving Artificial Neural Networks), which basically are what you see here with the exception that they are able to not only change their weights, but also their topology, that is how many and where the neurons and layers are. ANNs (Artificial Neural Networks - as opposed to biological ones) can be a lot of fun, and are very relevant to machine learning and big data nowadays. It was exploratory for me. I used them for generative art and music programs. Be careful: soon you'll be reading about genetic algorithms, genetic programming [2], and artificial life ;) Genetic Programming can be used to evolve neural networks as well as generate computer programs to solve a problem in a specified domain. Hint: You'll probably want to use Lisp/Scheme for genetic programming! [1] http://natureofcode.com/book/chapter-10-neural-networks/ [2] http://www.genetic-programming.com
- argonaut 10y agoAs far as the recent deep learning boom is concerned, genetic programming is really out of favor. I don't really see it in any of the deep learning (or even machine learning, for that matter) literature/successes/research groups. "Neural networks" are a really really overloaded term. A ton of stuff referred to as "neural networks" has little to do with the "neural networks" that are used in the machine learning community.
- extrapickles 10y agoIt has its niche applications. The only non machine vision application that comes to mind is one[1] that takes a pile of data, and evolves a model that fits it. Generally were its actually being used they are a bit quiet on how they go about getting the results they do. While the genetic bit is easy, the secret sauce is in guiding learning/evolution that work for the particular problem domain. [1]: http://www.nutonian.com/products/eureqa/ http://www.nutonian.com/products/eureqa/
- timroy 10y agoThis demonstration goes really well with Michael Nielsen's http://neuralnetworksanddeeplearning.com/ http://neuralnetworksanddeeplearning.com/. At the bottom of the page the author gives a shout out to Nielsen, Bengio, and others. For someone (like me) who's done a bit of reading but not much implementation, this playground is fantastic!
- seansmccullough 10y agoReally awesome article!
- pkaye 10y agoI'm not well versed in neural networks but a lot of the new neural network software stacks coming out seem to be quite plug and plug. What kind of expertise would engineers need to have a few years from now when the technology is well developed and it doesn't need to be rewritten from scratch every time?
- walrus 10y agoI'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] that are made up of simpler blocks. Learning what these do and when it's appropriate to use them requires following current literature. Operating neural networks requires some systems engineering skill. It takes a long time to train a single network and you'll find yourself trying many different architectures and hyperparameters along the way. Because of this, you'll want to distribute the training across many different systems and be able to easily monitor and deploy jobs on those systems. A solid grasp of mathematics is useful to effectively debug your networks. You'll frequently find your network doesn't converge or gives totally garbage results, so you need to know how to dig into the network internals and understand how everything works. This is especially true if you're implementing a new building block from a paper. Finally, know your machine learning and statistics fundamentals. Understand overfitting, model capacity, cross validation, probability, model ensembles, information theory, and so on. Know when a simpler model is more appropriate. [1] ftp://ftp.idsia.ch/pub/juergen/fki-207-95.ps.gz [2] http://arxiv.org/abs/1409.1259 http://arxiv.org/abs/1409.1259 [3] http://arxiv.org/abs/1505.00387 http://arxiv.org/abs/1505.00387 [4] http://arxiv.org/abs/1502.03167 http://arxiv.org/abs/1502.03167 [5] http://arxiv.org/abs/1512.03385 http://arxiv.org/abs/1512.03385
- pkaye 10y agoSo you don't think some of these details will not be automated away in the near future so that it doesn't require a specialist to do operate a neural network?
- nkozyra 10y agoIs a 50/50 training:test a normal default ratio for an ANN? I expected to see a higher amount of training data represented as the initial setting.
- nl 10y agoThis is great, but I think they should make it clear that this isn't using TensorFlow. From the title and domain I though they either had ported TF to Javascript(!) or we connecting to a server.
- sparky_ 10y agoWait - what it is using, then? I had assumed it was TF under Emscripten or similar.
- nl 10y agoIt appears to be a custom NN implementation[1] in Javascript, somewhat similar to convnet.js[2] As far as I can see the API[3] isn't much like TensorFlow. [1] https://github.com/tensorflow/playground https://github.com/tensorflow/playground [2] http://cs.stanford.edu/people/karpathy/convnetjs/ http://cs.stanford.edu/people/karpathy/convnetjs/ [3] https://github.com/tensorflow/playground/blob/master/nn.ts https://github.com/tensorflow/playground/blob/master/nn.ts
- deleted 10y ago[deleted]
- hyh1048576 10y agoOne of the finest data visualization I've seen.
- sparky_ 10y agoThis is a very cool toy. As someone with no experience in ML, this is an interesting visual approach to the absolute basics. And great for challenging your friends in an epic battle of convergence!
- visarga 10y agoIt's the classical exploration vs exploitation tradeoff. What do you do, try a radical new variation or fine tune this one?
- trgn 10y agoIf you like visual demonstrations of ML topics, you may be interested in http://ponder.hepburnave.com http://ponder.hepburnave.com. It is an interactive demonstration of a self-organizing map, generating a 2D-map from a spreadsheet with multivariate data. It's an unsupervised learning approach, good for data exploration tasks, less so for classification tasks (/shamelessPlug).
- icelancer 10y agoThis is so great. An easy way to show my friends WTF I do sometimes for math/CS work. Thank you so much.
- danblick 10y agoHas anyone been able to learn a function for the spiral (Swiss roll) data that's as good as a human-designed function would be?
- hyh1048576 10y agoDo you consider test loss around 0.04 good?
- trampi 10y ago0.007 http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=0&networkShape=7,3&seed=0.77306&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
- deleted 10y ago[deleted]
- asab 10y agohttp://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=10&networkShape=8,8&seed=0.15362&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
- asab 10y agoUpdate: after playing with this for way too long, I've found that it can converge to a spiral with 3 or 2 or even just 1 node in the 2nd hidden layer. The 1 node case is especially interesting, because when it converges the single node must learn the whole spiral pattern. Although with noise it can be less reliable with more jagged edges, as well as take longer to converge (also bumped the learning rate down), seeing the spiral encoded directly in the 2nd hidden layer is more interesting to me. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral®Dataset=reg-plane&learningRate=0.01®ularizationRate=0&noise=10&networkShape=8,1&seed=0.58434&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=true&ySquared=true&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification http://playground.tensorflow.org/#activation=tanh&batchSize=...
- plafl 10y agoBeautiful. The next time someone asks what is machine learning about I'm going to send a link to this page.
- erostrate 10y agoThe swiss roll problem also illustrates nicely the idea behind deep learning. Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d http://imgur.com/H1cvt8d Then the deep learning guys realized that you don't have to engineer all these extra features, you can just use basic features x_1, x_2 and let the network learn more complicated transformations in subsequent layers. So they would use a deep network with only x_1, x_2 as inputs: http://imgur.com/XBRjROP http://imgur.com/XBRjROP Both these approaches work here (loss < 0.01). The difference is that for the first one you have to manually choose the extra features sin(x_1), x_1^2, ... for each problem. And the more complicated the problem the harder it is to design good features. People in the computer vision community spent years and years trying to design good features for e.g. object recognition. But finally some people realized that deep networks could learn these features themselves. And that's the main idea in deep learning.
- raverbashing 10y agoThis is a very good explanation, thanks (even though I knew some of it already) I tried the swiss roll with a shallow network on the demo (and the results are not excellent, but it matches)
- beardicus 10y agoI think I learned more from your post and your two imgur links than from poking at the site for an hour. Thanks. Would it make sense for them to add a gallery of good solutions for each problem, or would they all basically be your second example network (no time to play and see for myself right now)?
- amelius 10y agoBut how will the number of neurons N grow with the number of turns in the spiral? If N levels off, then the network has grasped the concept of a spiral and can generalize to arbitrary size. If N doesn't level off, then the network isn't really learning the general case.
- halotrope 10y agoYou could totally optimise network architecture by crowdsourcing topology discovery for different problems into a multiplayer game with loss as a score.
- Your_Creator 10y agoSo glad anns are becoming mainstream Eventually it will have to be recognized as a new species of life, so I hope programmers, tinkerers and everyone else keeps that in mind because all life must be respected And this particular form will be our responsibility, we can either embrace it as we continue to merge with our technology, or we can allow ourselves to go extinct like so many other species already have For the naysayers - ever notice how attached we are to our phones? Many behave as if they are missing a limb without it - it's because they are, the brain adapts rapidly and for many, the brain has adapted to outsourcing our cognition. It used to be books, day runners, journals, diaries - now we have devices and soon they'll be implants or prosthetics The writers at marvel who came up with the idea of calling iron man's suit a prosthetic were definately onto something and suits like that are probably our best chance of successful colonization of other planets. We'll need ai to be our friend out there, working with us