3 ms·

When I read "inference" here I thought about probabilistic programming and Bayesian inference, but the article says: "Each processing element contains eight par

When I read "inference" here I thought about probabilistic programming and Bayesian inference, but the article says: "Each processing element contains eight parallel lanes of 8-way multiply-accumulate (MAC) units, each with a precision of 8-bit (as is getting common in inference)." 8-bit precision is good for neural network weights, but not for Bayesian inference, where from my understanding 32-bit floats might be the minimum. I wonder if anyone is working on hardware for accelerating probabilistic programming. Some approaches are harder to accelerate, such as monte carlo sampling (e.g. no u-turn sampler (NUTS)), while variational inference may be better suited to running on this type of custom hardware.

xiphias2 7y ago
Do you know any production Bayesian inference that really needs specialized chips? The problem with NN inference / training is that they are eating up the datacenters. At the same time you can't achieve 20x speedup compared to GPUs if you need 32 bit floats, because in that case your relative energy utilization is not that bad.
nabla9 7y ago
The world is used in the DL context where learning and inference can be often separated. Normal GPU's can be used for both learning and inference tasks. Dedicated inference processors have limited numerical accuracy and are more suitable only for inference, but have lower cost and are more power efficient.
bitL 7y ago
FYI, training - make model out of training dataset and a predefined architecture. Inference - use model for predictions/classifications. Usually training runs in FP32 or mixed-mode FP16/FP32, inference can be often used in FP8/FP16 after model pruning and normalization. Nothing to do with probabilistic inference, PGM etc. All it tells you there are some tensor cores running fast with low precision and some are now super popular like Tesla T4. For Bayesian inference you might be forced to use super expensive FP64 cards instead, as logarithm sums and small multiplications can get ugly pretty fast in FP32.

xiphias2 7y ago

Do you know any production Bayesian inference that really needs specialized chips? The problem with NN inference / training is that they are eating up the datacenters. At the same time you can't achieve 20x speedup compared to GPUs if you need 32 bit floats, because in that case your relative energy utilization is not that bad.

nabla9 7y ago

The world is used in the DL context where learning and inference can be often separated. Normal GPU's can be used for both learning and inference tasks. Dedicated inference processors have limited numerical accuracy and are more suitable only for inference, but have lower cost and are more power efficient.

bitL 7y ago

FYI, training - make model out of training dataset and a predefined architecture. Inference - use model for predictions/classifications. Usually training runs in FP32 or mixed-mode FP16/FP32, inference can be often used in FP8/FP16 after model pruning and normalization. Nothing to do with probabilistic inference, PGM etc. All it tells you there are some tensor cores running fast with low precision and some are now super popular like Tesla T4. For Bayesian inference you might be forced to use super expensive FP64 cards instead, as logarithm sums and small multiplications can get ugly pretty fast in FP32.