4 ms·
"NNs can withstand and even benefit from noise." This is the sort of broad statement I put in the same fool file as "We used simulated annealing because it's g
by varelse 13y ago
"NNs can withstand and even benefit from noise."
This is the sort of broad statement I put in the same fool file as "We used simulated annealing because it's guaranteed to converge." Bonus points for lobbing the genetic algorithm grenade into the mix(1).
In the former case, it's the judicious use of noise rather than basing the entire application on an architecture designed to inject 1% error into every calculation (because that seems to me to be a really good definition of programmer hell) and in the latter case, simulated annealing only converges assuming you run it for INFINITE TIME. But that doesn't stop people from saying this or even Jonathan Blow, someone I otherwise respect for his indie game dev creds and success therein, making an equally uninformed statement such as equating 1% computational error to 0.00001% error.
Getting useful information out of Neural Networks requires extreme attention to detail. If it didn't, Google wouldn't be paying up to $10M a head for people who can do this - the DeepMind acquihire.
Imagine the alternative here: I train a cluster of GPUs or FPGAs and I get useful results months before the engineer working with 1% error everywhere gets a network implementation that even converges. I then design and tape out custom ASICs to do this 10x faster within a year. See bitcoin mining for a successful example of this approach.
1. My doctorate made extensive use of genetic algorithms, I'm very familiar with their strengths and weaknesses.
- Houshalter 13y agoI never claimed anything about simulated annealing, and I know the noise is less than ideal, but adjustments can be made. Really my point stands, there are plenty of ways around it, and with the supposed 10,000X speed up it would be a massive advantage.
- varelse 13y agoEven the supposed 10,000x speedup is utterly bogus... The FPU here has 5,000 transistors. If you can stomach the lack of full IEEE compliance, you can build a full 32-bit FPU with ~100,000 transistors: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.1075&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.... So really what we have here is at best a ~20x performance boost in exchange for an absolute programming nightmare. Or you can switch to DSP-like fixed point math with ~25K transistors per ALU: http://www.idosi.org/wasj/wasj2%284%29/12.pdf http://www.idosi.org/wasj/wasj2%284%29/12.pdf So now we're down to ~5x better performance in exchange for that programming nightmare. And just to be thorough, if you're hellbent on nontraditional arithmetic, here's an error-free 32-bit integer ALU made with 1696 transistors, that would be >3x more efficient than the architecture here (assuming all you care about is throughput): http://www.iosrjournals.org/iosr-jece/papers/Vol3-Issue6/B0360712.pdf http://www.iosrjournals.org/iosr-jece/papers/Vol3-Issue6/B03... So now we're talking at best 1/3 to 1/4 the performance of what they could get if they dropped logarithmic math and went to something more predictable. Someone call Vinod Khosla, we're gonna be rrrriccchhhh...