Appreciate the work to make GPT training accessible! Do you leave hyperparams (like learning rate, batch size) the same when switching from 8xA100 to fewer GPU

Appreciate the work to make GPT training accessible! Do you leave hyperparams (like learning rate, batch size) the same when switching from 8xA100 to fewer GPUs, or do these need to be adjusted? Separately, when going from 8xA100 GPU to a single A100 GPU, in the worst case we can expect the same model performance after training 8x as long correct? (And likely a bit better because we get more gradient updates in with smaller batch size)