4 ms·

Visualize Algorithms based on Backpropagation

hacker42 10y ago
Would be interesting to see this for various test functions with various starting parameters. Here is a large compilation of various test functions: http://infinity77.net/global_optimization/genindex.html http://infinity77.net/global_optimization/genindex.html
- p1esk 10y ago
  The functions in your link are not gradient based, not sure how they are applicable to neural networks with backprop.
  - hacker42 10y ago
    Don't know what fraction of them, but most of them look differentiable (as long it is not exactly some variant of the Weierstrass function).
    - p1esk 10y ago
      True, but I think these functions are intended for non-gradient based optimization algorithms.
choosername 10y ago
I was looking for a blogpost here that used similar, orange visualisation. The graphs seem to be an intuitively general application in geometry or analysis or whatever - I don't have the basics down; this should help.
dpratt71 10y ago
Does someone want to take on the challenge of explaining the basics of this to a lay person? For starters, I don't understand the relationship between the scatter plot and the contour plot. I would guess that the blue (?) dots relate to the darker areas of the contour and the red dots to the lighter areas.
- chestervonwinch 10y ago
  There are two classes in the scatter plot -- blue and red points. These can be separated by a line. A line has two parameters, for example, a slope and an intercept (although, other parameterizations are possible). These line parameters are the x and y axis. The lighter and darker areas of the heat map indicate better and worse lines, respectively, for separating the two classes of points.
  - dpratt71 10y ago
    That makes sense, thank you.
lormayna 10y ago
What is the advantage of using Neopy instead that others neural network Python library like Pybrain?
- itdxer 10y ago
  There are a couple of advantages: 1. It's based on Theano, so it's fast. Also you are able to run your code on CPU and GPU. 2. NeuPy supports a lot of algorithms and different layer types (http://neupy.com/docs/cheatsheet.html http://neupy.com/docs/cheatsheet.html), so you can easily construct deep neural networks. 3. In the NeuPy it's easy to read and understand network's structure. One of the newest features is a Subnetworks (http://neupy.com/docs/layers/basics.html#subnetworks http://neupy.com/docs/layers/basics.html#subnetworks). 4. Neural Network Surgery is another new feature (http://neupy.com/docs/surgery.html http://neupy.com/docs/surgery.html)

Would be interesting to see this for various test functions with various starting parameters. Here is a large compilation of various test functions: http://infinity77.net/global_optimization/genindex.html http://infinity77.net/global_optimization/genindex.html

p1esk 10y ago
The functions in your link are not gradient based, not sure how they are applicable to neural networks with backprop.
- hacker42 10y ago
  Don't know what fraction of them, but most of them look differentiable (as long it is not exactly some variant of the Weierstrass function).
  - p1esk 10y ago
    True, but I think these functions are intended for non-gradient based optimization algorithms.

choosername 10y ago

I was looking for a blogpost here that used similar, orange visualisation. The graphs seem to be an intuitively general application in geometry or analysis or whatever - I don't have the basics down; this should help.

dpratt71 10y ago

Does someone want to take on the challenge of explaining the basics of this to a lay person? For starters, I don't understand the relationship between the scatter plot and the contour plot. I would guess that the blue (?) dots relate to the darker areas of the contour and the red dots to the lighter areas.

chestervonwinch 10y ago
There are two classes in the scatter plot -- blue and red points. These can be separated by a line. A line has two parameters, for example, a slope and an intercept (although, other parameterizations are possible). These line parameters are the x and y axis. The lighter and darker areas of the heat map indicate better and worse lines, respectively, for separating the two classes of points.
- dpratt71 10y ago
  That makes sense, thank you.

lormayna 10y ago

What is the advantage of using Neopy instead that others neural network Python library like Pybrain?

itdxer 10y ago
There are a couple of advantages: 1. It's based on Theano, so it's fast. Also you are able to run your code on CPU and GPU. 2. NeuPy supports a lot of algorithms and different layer types (http://neupy.com/docs/cheatsheet.html http://neupy.com/docs/cheatsheet.html), so you can easily construct deep neural networks. 3. In the NeuPy it's easy to read and understand network's structure. One of the newest features is a Subnetworks (http://neupy.com/docs/layers/basics.html#subnetworks http://neupy.com/docs/layers/basics.html#subnetworks). 4. Neural Network Surgery is another new feature (http://neupy.com/docs/surgery.html http://neupy.com/docs/surgery.html)