Nothing prevents interleaving inference and gradient descent / RMAD / backpropagation from being interleaved. Is a man a father or a son? It's a false dilemma,

Nothing prevents interleaving inference and gradient descent / RMAD / backpropagation from being interleaved. Is a man a father or a son? It's a false dilemma, it can be both. Is a training step pretraining or finetuning? It's the same mathematical operation, with the only difference the intention or reason of applying the training step.