5 ms·
Really interesting stuff. I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of tradi
by moron4hire 14y ago
Really interesting stuff.
I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). This RBM approach sounds better than back-propagation, but it also sounds like it would be prone to runaway feedback.
One of the performance problems with neural networks is that the number of cores on a typical machine are far less than the number of input and intermediate nodes in the network. The output nodes are less of a concern as you're trying to distill a lot of data down to a little data, but there is no reason to treat them differently. There are (very few) examples of NNs on GPUs, so that helps, but I've recently been curious to try a different, more hardware-driven approach, just because one could.
Texas Instruments has a cheap DSP chip that you'ns are probably familiar with called the MSP430. It's pretty easy to use, the tool chain is free and fairly easy to setup (especially for a bunch of professional software devs like us, right? right? Well, there's an Arduino-like tool now, too, if not), costs around 10 cents in bulk for the simplest version, requires very few external parts to run (something like a power source, 1 cap, and two resistors), and it has a couple of serial communication protocols built in. I'm quite fond of the chip; I've used it to build a number of digital synthesizers and toys.
For about $50 and quite a bit of soldering time, you could build a grid of 100 of these, each running at 16Mhz, and I bet with a clever design you could make them self programmable, i.e. propagate the program EEPROM over the grid. Load up a simple neural network program, maybe even having each chip simulating more than one node, and interface it with a PC to pump data in one end and draw it out the other. It might not be more useful than the GPGPU approach, but having physical hardware to play with and visualize node activity through other hardware would be a lot of fun.
- paxswill 14y agoIt sounds almost like you want to recreate a Connection Machine [0] using modern(ish) hardware. 0: http://en.wikipedia.org/wiki/Connection_Machine http://en.wikipedia.org/wiki/Connection_Machine
- moron4hire 14y agoYes, I think the advent of cheaply available hardware prototyping is making something like the Connection Machine and Transputers a more viable target for home-grown research. Wouldn't it be neat to be able to make a kit for building one's own Lisp machine, complete with an open, hackable, live-inspectable, Lisp OS? I.e. recreate 40 years ago. How about converting nodes between data storage and processing? There's just a lot of potential fun here.
- Retric 14y agoThe problem with stuff like this is 16Mhz * 100 is far less than 3500Mhz * 4. So doing this in software on the desktop is generally a much better idea let alone 750Mhz * 1000+ if you can get a good GPU implementation. You also hit significant speed of light and bandwidth issues if you want to network a lot of these together because neurons don't just talk to there 4 closest friends. PS: Still a fun project, just harder than you might think to scale.
- moron4hire 14y agoIt's no doubt difficult to scale, but there is no hobby-scale project that I'm aware of doing something like this. A quad-core Beast-PU from Intel or AMD is great if you're looking to get work done, but terrible if you're looking to open the hood and poke at the bits and innards. The most I've seen lately has been someone loading an existing cluster OS on 64 Raspberry Pis calling it a day. There might even be some interesting considerations for power efficiency and algorithms for disabling and reenabling nodes in the graph. There might even be some insights learned on improvements and/or exploits for other processor network systems that currently exist, like vehicle CANbuses.
- stiff 14y agoI had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). There are many ways to avoid this, for example have a look at: http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Conjugate_gradient_method http://en.wikipedia.org/wiki/Conjugate_gradient_method In traditional (I am not talking about the "deep" stuff) neural networks optimization is hardly ever the problem though, most often under- or over- fitting is the issue that produces poor performance. One of the performance problems with neural networks is that the number of cores on a typical machine are far less than the number of input and intermediate nodes in the network. This sounds odd. Certainly doing neural networks in hardware is interesting, but it sounds a bit like an imagined problem, I mean, when multiplying 20 numbers one does not complain that the number of cores is less than 20. And most often it is the training of the network that is resource-intensive not the actual running of it.
- mierle 14y agoBoth RProp and Conjugate Gradient are descent methods-- they find a path to a local minima from the starting configuration. They do not help with finding the global minimum. Using simulated annealing to guide random initializations can help find a better minima, but to get the global minima with simulated annealing takes an inordinate amount of time.
- stiff 14y ago(Poor) local minima can be sometimes reached in back propagation due to a bad setting of the learning rate and RProp does help with that. You can also repeat the learning process using one of those algorithms many times with different initial random weights and use the results of the best attempt. I think this most often suffices in practical applications of NN for getting decent results, you are right I got some of this wrong, I basically wanted to say that backprop is no longer state of the art in training and that optimization is the relatively easy part of learning.
- PaulHoule 14y ago
- jerf 14y agoYou may want this: http://www.greenarraychips.com/index.html http://www.greenarraychips.com/index.html
- moron4hire 14y agoThat's pretty neat. Thanks for the link, I know a few other people who would be interested in this, too.
- regularfry 14y agoUnless you're specifically trying to build a network with rather interesting temporal characteristics, there's no reason to try to map neurons to cores. Update rules are typically just matrix operations - no need to break out entertaining physical architectures for that.