Technically yes, only if you do a gradient step with data sampled from the exact same weights is it an online step. With our training recipe this can be easily

Technically yes, only if you do a gradient step with data sampled from the exact same weights is it an online step. With our training recipe this can be easily done by accumulating the gradients across the entire batch and only doing one step with optimizer before sampling more responses. In our experiments, however, we found the advantages of doing multiple gradient steps outweighed any potential drift in policy. Ultimately the online-ness of data is on a spectrum and while more online data is better, other factors may be more important.