3 ms·

What does it mean by "inference" in this context?

• syntaxing 3y ago
  You’re trying to get an output from giving an input to a bunch of matrices (tensors if you will). You do not touch anything in the matrices themselves which happens during training. Essentially, they’re great from deploying models but not making one.
• hangsi 3y ago
  Neural networks have two different compute costs: training and inference. These are roughly analogous to compile time vs runtime for compiled programming languages. Training is in general a more intensive task. However, in an ideal scenario training is run once and inference is run millions of times, so the lifetime cost of inference is bigger - this is why it might make sense to optimize for intense.
  • imtringued 3y ago
    Inference consists of a single forward pass, which is easy to compute. Meanwhile training has both a forward pass and backward pass (back propagation). The minimum required precision for training is higher since you want to be able to have both fast and slow rates of change. Bfloat16 is preferred for training, while int8 tends to be good enough for inference.
• deleted 3y ago
  [deleted]