Batch size is an orthogonal concept. The reason your memory increases so much is because with larger batches, you need to store more intermediate results before

Batch size is an orthogonal concept. The reason your memory increases so much is because with larger batches, you need to store more intermediate results before running your optimizer. This would be true with a pure CPU implementation of an SGD (stochastic gradient descent) optimizer as well.