4 ms·
story time: I tried something like this, putting a nnet output as the transfer weight of another neural net and training the first one feeding in the second net
by LoSboccacc 10y ago
story time: I tried something like this, putting a nnet output as the transfer weight of another neural net and training the first one feeding in the second net input as input and training it on the second net error, but couldn't train the first network because I didn't know how to derive the transfer function for the back propagation algorithm.
so I opted for training the first net using a randomized genetic algorithm and function descent on it, which as an afterthought is dangerously close on how biology kind of work, but it was exceptionally slow.
so I split up the training batches, went to the uni computer room and left the job running on every computer by night to collect result by morning. in the morning I'd collect the best genes from each machine, mix them all for another few round of training, select the best in the population and reseed them on all the machines by night.
after a week of painstakingly organizing, seeding and collecting results, the network never managed to converge around the problem, but boy it was fun trying! The problem was driving a car around a lap of a track using five "distance from kerb" sensor as input angled at 30deg from each other starting from center.
I remember I was inspired by an image recognition company, which was using a training network for training network for motion detection over security cameras, so this approach wasn't exactly novelty even back then (2001ish).
anyway, this got me noticed by a lab assistant and got a thesis on how to optimize neural network to run in 4.4bit fixed math for use in extra low power devices. that one worked! too bad nothing ever came out of it.
edit: some fixin
- wrong_variable 10y agoThat fact they nothing came out of it is the result of your research ! keep doing the good work :)
- LogicFailsMe 10y agoSee dp4a on GTX1080. You're a hot commodity if you can build on that. See also Google's TPU where they managed 8-bit inference but probably not 8-bit training (though that may be more of a memory or bandwidth limitation).
- blt 10y agobut reduced precision neural network evaluation is a really hot field right now!
- LoSboccacc 10y agoeh, I have the paper and the code on hand if anyone's interested, but it's printed because thesis. also matlab but that's minor.
- sillysaurus3 10y agoYes, please. I'd be happy with some photos snapped from your phone if you don't want to scan it in. The Matlab code in particular seems intriguing.
- LoSboccacc 10y agothis was ten years ago, so techniques are severely outdated etc. also it's not THAT much of code, the majority of it is standard initialization and optimization. here's the most important stuff https://imgur.com/a/w2BCy https://imgur.com/a/w2BCy
- deleted 10y ago[deleted]