The results are showing essentially 12 unique responses from 30 tries… not what you would expect from mixing tokens. I think it groups the batch up differently

The results are showing essentially 12 unique responses from 30 tries… not what you would expect from mixing tokens. I think it groups the batch up differently, so if I have a batch of 10, and it groups it up into 2 groups of 5, if my prompt makes it to the second group or 1st group I get a different answer. But if I’m in the same location in the batch, then I get the same answer. The whole batch is deterministic given the same batch (sequences and ordering), but if you shuffle the batch then you lose that determinism.