The 168MB is for storing all of the weights (parameters of the model) and the activations from passing one sample through the network. When you are training, yo

The 168MB is for storing all of the weights (parameters of the model) and the activations from passing one sample through the network. When you are training, you want to pass a whole minibatch (32 samples in this article) through the network, so the activations (not the weights) are repeated 32 times - that's where the 2GB figure comes from. You need to keep these activations around for backprop to work, so you either store them all, or as the article later suggests, you keep the ones that are expensive to compute and re-calculate the cheap ones. Also note that when training you need to store the gradient of every weight too, so this adds 104mb in this example - almost negligible next to 2GB of activations. For running a trained model, it depends how it's being used. If you're computing one sample at a time, then the 168MB figure is an upper bound on that (one copy of weights + one set of activations). You could optimize this by only keeping the activations for the current layer because you don't need to save the intermediate activations. If however you're batch processing samples at inference time, that's where you want to make most efficient use of the compute resources, so chances are minibatches are going to be the best route; again the same kind of optimization with not keeping intermediate activations applies here. As for whether GPUs are useful at inference time, that heavily depends on your model and available deployment hardware. TensorFlow has some support for 8-bit quantized models for targeting mobile/embedded/CPU, so there are ways to optimize for this use case. Even at inference time though, most of your compute work is going to be in matrix multiplies or convolutions so there's plenty of cases where GPUs can help.