5 ms·
> Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to
by orlp 5y ago
> Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to the spec sheet, the appliance only uses 6-bit inputs and 8-bit outputs, so the relative errors are probably on the percent level. This makes it hard to believe that any signal will propagate through something like a DNN without completely drowning in noise.
Maybe you aren't aware, but half-precision (16 bit float) is already well-established in the AI community: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format. In context single-precision isn't all that puny!
And there have already been successful experiments with stronger quantization, like 8-bit neural nets, or even 1-bit (!) neural nets. There is a lot of evidence that neural networks can be very resilient to quantization noise.
- ISL 5y agoI'd be real surprised if the neurons in our brains have ADC-equivalents better than ~4 bits.
- tgv 5y agoTrue, but ANNs are nowhere near as good as our brains, nor do they operate in the same way.
- xwolfi 5y agoI think you missed the fact we were talking about neural network, not an animal brain self replicating and branching and competing with itself for billions of years until it becomes aware of itself. Give us the the same time.
- Retric 5y agoThe earliest and simplest brains where still useful. Even insects can fly around in 3D space, I doubt you need something as complicated as a mouse brain to run a self driving car let alone a drone.
- ben_w 5y agoI’m not sure how much it matters given this thread looks like it’s going off on several successive tangents, but the important (and hard) thing with a self-driving car is making sure it doesn’t hit stuff, not the actual driving part. And drones, trivially agree: Megaphragma mymaripenne has 7400 neurones, compared to the 71/14 million in a house mouse nervous system/brain.
- robwwilliams 5y agoAnd to stay on this odd tangent: Best estimate I have for total cell numbers in mouse brain—about 75 million neurons and 35 million other cell types. This estimate is from a 476 mg brain of a C57BL/6J case—the standard mouse used by many researchers. Based on much other work with discrete neuron populations the range among different genometypes of mice probable +/- 40%. for details see: www.nervenet.org/papers/brainrev99.html Expect many more (and I hope better) estimates soon from Clarity/SHIELD whole brain lighsheet counting with Al Johnson and colleagues at Duke and team at Life Canvas Tech.
- gnatman 5y agoInsects also crash into stuff constantly!
- GravitasFailure 5y agoBats do, too, despite being much smarter than insects. It seems nature prefers the "fail often and recover quickly" system.
- lostmsu 5y agoThey are simply too light to hurt themselves.
- deleted 5y ago[deleted]
- bodhiandphysics 5y agoIt’s so annoying!!!! I work in computer graphics and scientific computation and it’s getting hard to get gpus with adequate double performance.
- chillee 5y agoYou might be interested in the upcoming AMD Mi200 GPUs, which have 96 teraflops of fp64 performance.
- bodhiandphysics 5y agoUnfortunately, I need cuda (for now)
- sudosysgen 5y agoWith some tweaking, I've had sucess running CUDA code with AMD's HIP framework. You'll need to do some changes probably.
- bee_rider 5y agoI think we'll just have to embrace it. Anyway, dealing with reduced precision is more fun that throwing bits at the problem.
- simplestats 5y agoI think it's hilarious. when I was in school, microprocessors were 8-bit. Then as the world got digitized it was 16-bit microprocessors. A least in my narrow world. Then 32 bit and floats came along and finally double floats. Each step was a big product effort and launch we sell as better technology. And just as we are finally got past the need to always make both 32 and 64 bit versions of everything, we turn around and head back down. Though I saw someone (Qualcomm?) with a press release about their next gen microprocessor supporting 32 bit floats! Not sure if it counts as another U-turn or they're just a straggler.
- wtallis 5y ago
- thesz 5y agoThese 1-bit-per-coefficient neural nets need pretty good floating point implementation to train them. From what I remember, they are trained with rounding - basically, a floating point weight gets rounded to -1, 0 or 1 and computed floating point gradient is added to a floating point weight. Inference is cheap, training is not.
- stjohnswarts 5y agobut 16 bits >> 8 bits? Can the AI community get by with 8 bits?