I feel like Tensorflow tends to allocate scratch space in GPU RAM for the result of every math operation. So if you have a big tensor and want to do math on it

I feel like Tensorflow tends to allocate scratch space in GPU RAM for the result of every math operation. So if you have a big tensor and want to do math on it, you need space to effectively store multiple copies of the tensor. I think TF does this for speed reasons.