Increasing batch size does not increase the cost of your training. The opposite actually: With bigger batch size (to an extent), models tend to converge slightl

Increasing batch size does not increase the cost of your training. The opposite actually: With bigger batch size (to an extent), models tend to converge slightly faster so you need less GPU hours. As for the rest, training for 1h with batch size of 2048 on 10 TPUs, or training for 8h with batch size of 256 on 1 TPU has the exact same cost, the cost is just spread over a longer time.