7 ms·

From the website of the manufacturer [1] it appears that the co-processor is essentially an analog computer for matrix-vector multiplications. I am quite scepti

by fsh 5y ago

From the website of the manufacturer [1] it appears that the co-processor is essentially an analog computer for matrix-vector multiplications. I am quite sceptical about the accuracy and value range of the computations. Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to the spec sheet, the appliance only uses 6-bit inputs and 8-bit outputs, so the relative errors are probably on the percent level. This makes it hard to believe that any signal will propagate through something like a DNN without completely drowning in noise. [1] https://lighton.ai/lighton-appliance/ https://lighton.ai/lighton-appliance/

orlp 5y ago
> Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to the spec sheet, the appliance only uses 6-bit inputs and 8-bit outputs, so the relative errors are probably on the percent level. This makes it hard to believe that any signal will propagate through something like a DNN without completely drowning in noise. Maybe you aren't aware, but half-precision (16 bit float) is already well-established in the AI community: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format. In context single-precision isn't all that puny! And there have already been successful experiments with stronger quantization, like 8-bit neural nets, or even 1-bit (!) neural nets. There is a lot of evidence that neural networks can be very resilient to quantization noise.
- ISL 5y ago
  I'd be real surprised if the neurons in our brains have ADC-equivalents better than ~4 bits.
  - tgv 5y ago
    True, but ANNs are nowhere near as good as our brains, nor do they operate in the same way.
  - xwolfi 5y ago
    I think you missed the fact we were talking about neural network, not an animal brain self replicating and branching and competing with itself for billions of years until it becomes aware of itself. Give us the the same time.
    - Retric 5y ago
      The earliest and simplest brains where still useful. Even insects can fly around in 3D space, I doubt you need something as complicated as a mouse brain to run a self driving car let alone a drone.
      - ben_w 5y ago
        I’m not sure how much it matters given this thread looks like it’s going off on several successive tangents, but the important (and hard) thing with a self-driving car is making sure it doesn’t hit stuff, not the actual driving part. And drones, trivially agree: Megaphragma mymaripenne has 7400 neurones, compared to the 71/14 million in a house mouse nervous system/brain.
        robwwilliams 5y ago
        And to stay on this odd tangent: Best estimate I have for total cell numbers in mouse brain—about 75 million neurons and 35 million other cell types. This estimate is from a 476 mg brain of a C57BL/6J case—the standard mouse used by many researchers. Based on much other work with discrete neuron populations the range among different genometypes of mice probable +/- 40%. for details see: www.nervenet.org/papers/brainrev99.html Expect many more (and I hope better) estimates soon from Clarity/SHIELD whole brain lighsheet counting with Al Johnson and colleagues at Duke and team at Life Canvas Tech.
      - gnatman 5y ago
        Insects also crash into stuff constantly!
        GravitasFailure 5y ago
- bodhiandphysics 5y ago
  It’s so annoying!!!! I work in computer graphics and scientific computation and it’s getting hard to get gpus with adequate double performance.
  - chillee 5y ago
    You might be interested in the upcoming AMD Mi200 GPUs, which have 96 teraflops of fp64 performance.
    - bodhiandphysics 5y ago
      Unfortunately, I need cuda (for now)
      - sudosysgen 5y ago
        With some tweaking, I've had sucess running CUDA code with AMD's HIP framework. You'll need to do some changes probably.
  - bee_rider 5y ago
    I think we'll just have to embrace it. Anyway, dealing with reduced precision is more fun that throwing bits at the problem.
  - simplestats 5y ago
    I think it's hilarious. when I was in school, microprocessors were 8-bit. Then as the world got digitized it was 16-bit microprocessors. A least in my narrow world. Then 32 bit and floats came along and finally double floats. Each step was a big product effort and launch we sell as better technology. And just as we are finally got past the need to always make both 32 and 64 bit versions of everything, we turn around and head back down. Though I saw someone (Qualcomm?) with a press release about their next gen microprocessor supporting 32 bit floats! Not sure if it counts as another U-turn or they're just a straggler.
    - wtallis 5y ago
- thesz 5y ago
  These 1-bit-per-coefficient neural nets need pretty good floating point implementation to train them. From what I remember, they are trained with rounding - basically, a floating point weight gets rounded to -1, 0 or 1 and computed floating point gradient is added to a floating point weight. Inference is cheap, training is not.
- stjohnswarts 5y ago
  but 16 bits >> 8 bits? Can the AI community get by with 8 bits?
kortex 5y ago
It's not a transmission line though, SNR does not apply in the same way. It's more like CMOS where the signal is refreshed at each gate. Each stage of an ANN applies some weight and activation. You can think of each input vector as a vector with a true value plus some noise. As long as that feature stays within some bounds, it is going to represent the same "thought vector". It may require some architecture changes to make training feasible, but it's far from a nonstarter. And that is only considering backprop learning. The brain does not use backprop, and has way higher noise levels.
- dahart 5y ago
  I think the parent was referring to the same noise that you are, compute precision, not transmission, and was suggesting that perhaps it won’t easily stay within bounds due to the fact that some kinds of repeated calculations lose more precision at every step. Maybe it’s application dependent, maybe NNs or other matrix-heavy domains can tolerate low precision much more easily than scientific simulations. It certainly wouldn’t surprise me if these “LightOPS” processors work well in a narrow range of applications, and won’t improve or speed up just anything that needs a matrix multiply.
tdrdt 5y ago
But arent there other applications where this is ok? For example path tracing (ray tracing) doesn't need to be very accurate because multiple samples per pixel are used. A gaming card that uses less power is very welcome in laptops for example.
- moonchrome 5y ago
  What do you mean? What kind of worlds can you represent with 8 bits units ? Some small blocky voxel box ?
  - tdrdt 5y ago
    I mean inacurate vector math
    - moonchrome 5y ago
      But that is what I'm saying if your vectors are reduced to 8 bit scalar components you can represent a 256x256x256 worth of detail in the world (doesn't need to be linear but still really limited details) ?
      - montjoy 5y ago
        I’m totally out of my expertise here but I have a question - from my understanding ray tracing is primarily used for lighting/shadows/reflections- wouldn’t it be OK for something like shadows to be inaccurate- maybe some sort of amalgamation over frame refreshes? Real world light is messy anyway. I’m talking about a game type scenario not something scientific. Maybe another way to ask is- we’re trying to simulate a real world “analog” scene - maybe using an analog processing technique could actually be quite faithful for generating it? Or not. Like I said I don’t understand too much of this.
        moonchrome 5y ago
        Think about it like this : you have a spaceship model fits in 256x256x256m - to get the maximum resolution while still fitting in 8 bits you would make each axis in 1m increments and you have 256 values. So you can't have sub 1m details in geometry. Floating point is different because you have an exponent so it's not linear, you can technically have larger scale, but you sacrifice even more precision in mantissa. Now I'm not sure what this 6/8 bit precision or analog precision means so I can't say with confidence, but if your scalars are that low precision you can't really do much. You could technically encode it with some fancy tricks like instead of storing coordinates for each vertex you store the delta from previous one etc. but I think this wouldn't work if the device was just some dumb analog matrix multiplier with baked logic. Also having low detail shadows creates visual artifacts, see this for example [1] [1] https://digitalrune.github.io/DigitalRune-Documentation/media/Shadow-Resolution.jpg https://digitalrune.github.io/DigitalRune-Documentation/medi...
bjornsing 5y ago
If there’s too much noise just lower the dropout probability from 50 to 30%. ;) Joking aside, it is interesting how much noise and quantization these neural networks can work with. I think there’s a lot of room for low precision noisy computation here.

by fsh 5y ago

orlp 5y ago

> Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to the spec sheet, the appliance only uses 6-bit inputs and 8-bit outputs, so the relative errors are probably on the percent level. This makes it hard to believe that any signal will propagate through something like a DNN without completely drowning in noise. Maybe you aren't aware, but half-precision (16 bit float) is already well-established in the AI community: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format. In context single-precision isn't all that puny! And there have already been successful experiments with stronger quantization, like 8-bit neural nets, or even 1-bit (!) neural nets. There is a lot of evidence that neural networks can be very resilient to quantization noise.

ISL 5y ago
I'd be real surprised if the neurons in our brains have ADC-equivalents better than ~4 bits.
- tgv 5y ago
  True, but ANNs are nowhere near as good as our brains, nor do they operate in the same way.
- xwolfi 5y ago
  I think you missed the fact we were talking about neural network, not an animal brain self replicating and branching and competing with itself for billions of years until it becomes aware of itself. Give us the the same time.
  - Retric 5y ago
    The earliest and simplest brains where still useful. Even insects can fly around in 3D space, I doubt you need something as complicated as a mouse brain to run a self driving car let alone a drone.
    - ben_w 5y ago
      I’m not sure how much it matters given this thread looks like it’s going off on several successive tangents, but the important (and hard) thing with a self-driving car is making sure it doesn’t hit stuff, not the actual driving part. And drones, trivially agree: Megaphragma mymaripenne has 7400 neurones, compared to the 71/14 million in a house mouse nervous system/brain.
      - robwwilliams 5y ago
        And to stay on this odd tangent: Best estimate I have for total cell numbers in mouse brain—about 75 million neurons and 35 million other cell types. This estimate is from a 476 mg brain of a C57BL/6J case—the standard mouse used by many researchers. Based on much other work with discrete neuron populations the range among different genometypes of mice probable +/- 40%. for details see: www.nervenet.org/papers/brainrev99.html Expect many more (and I hope better) estimates soon from Clarity/SHIELD whole brain lighsheet counting with Al Johnson and colleagues at Duke and team at Life Canvas Tech.
    - gnatman 5y ago
      Insects also crash into stuff constantly!
      - GravitasFailure 5y ago
bodhiandphysics 5y ago
It’s so annoying!!!! I work in computer graphics and scientific computation and it’s getting hard to get gpus with adequate double performance.
- chillee 5y ago
  You might be interested in the upcoming AMD Mi200 GPUs, which have 96 teraflops of fp64 performance.
  - bodhiandphysics 5y ago
    Unfortunately, I need cuda (for now)
    - sudosysgen 5y ago
      With some tweaking, I've had sucess running CUDA code with AMD's HIP framework. You'll need to do some changes probably.
- bee_rider 5y ago
  I think we'll just have to embrace it. Anyway, dealing with reduced precision is more fun that throwing bits at the problem.
- simplestats 5y ago
  I think it's hilarious. when I was in school, microprocessors were 8-bit. Then as the world got digitized it was 16-bit microprocessors. A least in my narrow world. Then 32 bit and floats came along and finally double floats. Each step was a big product effort and launch we sell as better technology. And just as we are finally got past the need to always make both 32 and 64 bit versions of everything, we turn around and head back down. Though I saw someone (Qualcomm?) with a press release about their next gen microprocessor supporting 32 bit floats! Not sure if it counts as another U-turn or they're just a straggler.
  - wtallis 5y ago
thesz 5y ago
These 1-bit-per-coefficient neural nets need pretty good floating point implementation to train them. From what I remember, they are trained with rounding - basically, a floating point weight gets rounded to -1, 0 or 1 and computed floating point gradient is added to a floating point weight. Inference is cheap, training is not.
stjohnswarts 5y ago
but 16 bits >> 8 bits? Can the AI community get by with 8 bits?

kortex 5y ago

It's not a transmission line though, SNR does not apply in the same way. It's more like CMOS where the signal is refreshed at each gate. Each stage of an ANN applies some weight and activation. You can think of each input vector as a vector with a true value plus some noise. As long as that feature stays within some bounds, it is going to represent the same "thought vector". It may require some architecture changes to make training feasible, but it's far from a nonstarter. And that is only considering backprop learning. The brain does not use backprop, and has way higher noise levels.

dahart 5y ago
I think the parent was referring to the same noise that you are, compute precision, not transmission, and was suggesting that perhaps it won’t easily stay within bounds due to the fact that some kinds of repeated calculations lose more precision at every step. Maybe it’s application dependent, maybe NNs or other matrix-heavy domains can tolerate low precision much more easily than scientific simulations. It certainly wouldn’t surprise me if these “LightOPS” processors work well in a narrow range of applications, and won’t improve or speed up just anything that needs a matrix multiply.

tdrdt 5y ago

But arent there other applications where this is ok? For example path tracing (ray tracing) doesn't need to be very accurate because multiple samples per pixel are used. A gaming card that uses less power is very welcome in laptops for example.

moonchrome 5y ago
What do you mean? What kind of worlds can you represent with 8 bits units ? Some small blocky voxel box ?
- tdrdt 5y ago
  I mean inacurate vector math
  - moonchrome 5y ago
    But that is what I'm saying if your vectors are reduced to 8 bit scalar components you can represent a 256x256x256 worth of detail in the world (doesn't need to be linear but still really limited details) ?
    - montjoy 5y ago
      I’m totally out of my expertise here but I have a question - from my understanding ray tracing is primarily used for lighting/shadows/reflections- wouldn’t it be OK for something like shadows to be inaccurate- maybe some sort of amalgamation over frame refreshes? Real world light is messy anyway. I’m talking about a game type scenario not something scientific. Maybe another way to ask is- we’re trying to simulate a real world “analog” scene - maybe using an analog processing technique could actually be quite faithful for generating it? Or not. Like I said I don’t understand too much of this.
      - moonchrome 5y ago
        Think about it like this : you have a spaceship model fits in 256x256x256m - to get the maximum resolution while still fitting in 8 bits you would make each axis in 1m increments and you have 256 values. So you can't have sub 1m details in geometry. Floating point is different because you have an exponent so it's not linear, you can technically have larger scale, but you sacrifice even more precision in mantissa. Now I'm not sure what this 6/8 bit precision or analog precision means so I can't say with confidence, but if your scalars are that low precision you can't really do much. You could technically encode it with some fancy tricks like instead of storing coordinates for each vertex you store the delta from previous one etc. but I think this wouldn't work if the device was just some dumb analog matrix multiplier with baked logic. Also having low detail shadows creates visual artifacts, see this for example [1] [1] https://digitalrune.github.io/DigitalRune-Documentation/media/Shadow-Resolution.jpg https://digitalrune.github.io/DigitalRune-Documentation/medi...

bjornsing 5y ago

If there’s too much noise just lower the dropout probability from 50 to 30%. ;) Joking aside, it is interesting how much noise and quantization these neural networks can work with. I think there’s a lot of room for low precision noisy computation here.